NT8 Strategy Analyzer Problems - futures io
futures io



NT8 Strategy Analyzer Problems


Discussion in NinjaTrader

Updated
      Top Posters
    1. looks_one johns52 with 6 posts (1 thanks)
    2. looks_two koganam with 1 posts (0 thanks)
    3. looks_3 Fat Tails with 1 posts (2 thanks)
    4. looks_4 arni with 1 posts (0 thanks)
    1. trending_up 885 views
    2. thumb_up 3 thanks given
    3. group 5 followers
    1. forum 8 posts
    2. attach_file 2 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 150,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

NT8 Strategy Analyzer Problems

(login for full post details)
  #1 (permalink)
 johns52 
Harbert MI
 
Experience: Intermediate
Platform: NinjaTrader
Trading: futures
 
Posts: 13 since Jul 2014
Thanks: 9 given, 4 received

NT8's Strategy Analyzer - I use commission, slippage, Tick Replay, COET - sometimes I get 12.5 profit factors and $30K per month in Total Net Profit. I only optimize for bar type and bar size. Then I run the strategy in Sim Trade on a chart - usually 15 minute - and steadily lose. What to do? The SA seems a quick way to evaluate a strategy when the market is running and I can't do Market Replay but what's the use if the results are so invalid? Is there a better way?

Started this thread Reply With Quote

Can you help answer these questions
from other members on futures io?
SafeDayTradingorg
Trading Reviews and Vendors
convert sierra chart graph to Ninja traders 8
NinjaTrader
Paid coding for NT7 to NT8 Custom Plot() conversion
NinjaTrader
Jumpstart Trading
Trading Reviews and Vendors
TradingView like VOLUME indi for Ninja8 ?!?!
NinjaTrader
 
 
(login for full post details)
  #2 (permalink)
 Fat Tails 
Market Wizard
Berlin, Europe
 
Experience: Advanced
Platform: NinjaTrader, MultiCharts
Broker: Interactive Brokers
Trading: Keyboard
 
Fat Tails's Avatar
 
Posts: 9,878 since Mar 2010
Thanks: 4,238 given, 26,929 received


johns52 View Post
NT8's Strategy Analyzer - I use commission, slippage, Tick Replay, COET - sometimes I get 12.5 profit factors and $30K per month in Total Net Profit. I only optimize for bar type and bar size. Then I run the strategy in Sim Trade on a chart - usually 15 minute - and steadily lose. What to do? The SA seems a quick way to evaluate a strategy when the market is running and I can't do Market Replay but what's the use if the results are so invalid? Is there a better way?

When there is a difference between the strategy analyzer and market replay, you need to analyze the differences trade by trade.
There are many potential sources for errors and discrepancies. You will only find out what is happening when you analyze it.

Typical problems:

- Use of exotic bar types which are not backtestable: What bar type do you use?
- Use of fast bars (high frequency trading) such that many bars have the same time stamp and orders can not be properly executed.
- Use of other than market orders without selecting the correct order fill resolution or adding a secondary bar series to the strategy.

As there are many potential sources of errors, you need to give more information on your strategy, such as

- bar types used
- bar period used
- order types used

In most cases, when a strategy backtest goes wrong, it is the result of someone having committed a few mathematical crimes. :-)

In case you do not want to share further information, just run a backtest over a short period of time and examine the trades one by one, comparing the backtest and the market replay.
Also you may want to strip down the strategy to a simpler version and then go step by step adding features and look whether it has an impact on the discrepancy between backtest and market replay.

Reply With Quote
The following 2 users say Thank You to Fat Tails for this post:
 
(login for full post details)
  #3 (permalink)
koganam
Garner, NC/USA
 
 
Posts: 38 since Dec 2009
Thanks: 16 given, 42 received



johns52 View Post
NT8's Strategy Analyzer - I use commission, slippage, Tick Replay, COET - sometimes I get 12.5 profit factors and $30K per month in Total Net Profit. I only optimize for bar type and bar size. Then I run the strategy in Sim Trade on a chart - usually 15 minute - and steadily lose. What to do? The SA seems a quick way to evaluate a strategy when the market is running and I can't do Market Replay but what's the use if the results are so invalid? Is there a better way?

Is your strategy running on Renko bars, or any other kind of synthetic bars, by any chance?

Reply With Quote
 
(login for full post details)
  #4 (permalink)
 johns52 
Harbert MI
 
Experience: Intermediate
Platform: NinjaTrader
Trading: futures
 
Posts: 13 since Jul 2014
Thanks: 9 given, 4 received

Hello, thanks for your responses. I am generally using minute bars. I always use market orders as I want to be sure and get a fill. I should have had all my information lined up in advance so to catch up I ran a study today. It's a simple Donchian Strategy with no stops or profit targets: I think it was the stops & profit targets hurting the prior results. I use Low3Ago (Low[3]) to exit the longs and High3Ago (High[3] to exit the shorts. I usually run the backtest on 15 min, then optimize from there. Attached are today's results. The time periods are a little weird since I started with 5 minutes and advanced by 10s. I was surprised to see that longer time frames gave better results. I have no way to determine the drawdowns other than go through the trades with the chart's Strategy Performance/Historical. I assume longer time frames have larger drawdowns.

The Strategy Analyzer for the 2nd best result (115 min) give $66K for the 6/1-8/22 period with tick replay, commission and slippage of 1. Market Replay gave +$12.7. (Great though).
The SA for for the 5th best result (65 min) gives $71.3K - MR was +$4.1K.
The SA for the 10th best result (35 min) gives $91K - MR was +$13.9K.
The SA shows the best results for a 95 min period with +$75K. Unfortunately, the Market Replay stalled out after 6 days. I will have to run it again.
Well, this looks promising and gives me some hope. A little more tweaking and a lot more testing and I may be ready to try it.

Is there any reason why I can't run 4 ES charts at once with the same strategy? Why would one stall out on me?

Could the Strategy Analyzer software be skewed to be wildly optimistic?

Does running Market Relay at 1000x hurt the results?

Strategy Analyzer Donch3 8-23-2022


Market Replay ES 6_1 - 8_22 4 times 1000x

Started this thread Reply With Quote
The following user says Thank You to johns52 for this post:
 
(login for full post details)
  #5 (permalink)
 johns52 
Harbert MI
 
Experience: Intermediate
Platform: NinjaTrader
Trading: futures
 
Posts: 13 since Jul 2014
Thanks: 9 given, 4 received

from the NinjaScript guide: https://ninjatrader.com/support/helpGuides/nt8/NT%20HelpGuide%20English.html?discrepancies_real-time_vs_bac.htm


Quoting 
You should expect that a strategy running real-time (live brokerage account, live market simulation, Playback connection etc...) will produce different results than the performance results generated during a backtest.

I guess that says it all: Strategy Analyzer will not give the same results as real-time or playback. That's a big disappointment.

Started this thread Reply With Quote
 
(login for full post details)
  #6 (permalink)
 johns52 
Harbert MI
 
Experience: Intermediate
Platform: NinjaTrader
Trading: futures
 
Posts: 13 since Jul 2014
Thanks: 9 given, 4 received

from the NinjaTrader Forum: https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/100192-comparing-real-time-historical-and-replay-performance

Quoting 
Comparing real-time, historical, and replay performance
07-23-2017, 05:12 PM
Citizens of the NinjaTrader Community,

A common question we hear from clients is 'why are results from backtest different from real-time or from market replay?'.

The main reason is due to a lack of intra-bar granularity.

See more about intra-bar granularity.
https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/94098-isfirsttickofbar-vs-onbarclose-for-backtest-live?p=773377

By default when backtesting no intra-bar granularity is added. An added 1-tick series is necessary to increase fill accuracy to within 1 tick.
Further, TickReplay is necessary for intra-bar indicator values to update and recalculate for on each tick or on price change Calculate setting.

As a heads up, TickReplay cannot be used for accurate order fill prices and High Order Fill Resolution cannot be used in conjunction with TickReplay.

Apparently my question is very common. In my case, I am getting fantastic results with COET and Tick Replay and an added 1-tick data series. It's when I go to Market Replay/Playback that the 12.5 profit factor drops to 1.1 and the drawdowns go from $650 to $6500.

Wish I knew how to rectify that. Is it slippage? Do I need to use a Stop-Market order? Why are entries & exits happening a bar late?

Started this thread Reply With Quote
 
(login for full post details)
  #7 (permalink)
 johns52 
Harbert MI
 
Experience: Intermediate
Platform: NinjaTrader
Trading: futures
 
Posts: 13 since Jul 2014
Thanks: 9 given, 4 received

Starting to figure this out.....

Here's a very helpful video from Chelsea B:


Here's the Forum page: https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/100192-comparing-real-time-historical-and-replay-performance

Zip file for the RealtimeReplayHistoricalComparisonsExample_NT8.zip : https://ninjatrader.com/support/forum/filedata/fetch?id=908317

Started this thread Reply With Quote
 
(login for full post details)
  #8 (permalink)
 johns52 
Harbert MI
 
Experience: Intermediate
Platform: NinjaTrader
Trading: futures
 
Posts: 13 since Jul 2014
Thanks: 9 given, 4 received

My tests suggest that Tick Replay must also be used with Market Replay to get closer to IRL results. Try it - you'll see huge differences between with and without Tick Replay. My tests also suggest you will need to have a 12.5 profit factor or better in the Strategy Analyzer to get a possible 1.5 profit factor in Market Replay/Tick Replay.

Here's a tip: for a Market Replay from 8/1 - 8/22, set Days to Load in the Data Series to 22. They you can access Strategy Performance >> RealTime and get the Trades, Max Drawdown, Profit Factor, etc.

I am learning all this by drips and drops. Does someone teach this? Kevin Davey works with Tradestation which is a much simpler system; Easy Language not C#.

Started this thread Reply With Quote
 
(login for full post details)
  #9 (permalink)
arni
Hamburg, Germany
 
 
Posts: 2 since Dec 2021
Thanks: 0 given, 0 received

Hi,

I have also run into problems with NT8 backtesting vs real-life trading. If you run your backtest with Order fill Resolution = Standard (fastest), there is a strange calculation being used in the background which leads to the results being just wrong. Instead, set the Order fill Resolution = High, then 1 tick for highest accuracy. Definitely takes much longer to run the backtest, but your results will be as close to real trading as possible. Thats actually what they replied to me in an email when I asked them about it.

Reply With Quote


futures io Trading Community Platforms and Indicators NinjaTrader > NT8 Strategy Analyzer Problems


Last Updated on September 27, 2022


Upcoming Webinars and Events
 

NinjaTrader Indicator Challenge!

Ongoing
     



Copyright © 2022 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada), info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts