Backtest not matching Optimization - futures io
futures io futures trading



Backtest not matching Optimization


Discussion in NinjaTrader

Updated
      Top Posters
    1. looks_one d1g1talfr3ak with 3 posts (0 thanks)
    2. looks_two Big Mike with 2 posts (0 thanks)
    3. looks_3 NetTecture with 1 posts (0 thanks)
    4. looks_4 grimReaper with 1 posts (0 thanks)
    1. trending_up 1,765 views
    2. thumb_up 0 thanks given
    3. group 7 followers
    1. forum 9 posts
    2. attach_file 0 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 125,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

Backtest not matching Optimization

(login for full post details)
  #1 (permalink)
Huntsville AL/USA
 
Experience: Beginner
Platform: NinjaTrader
Trading: Stocks, Futures
 
d1g1talfr3ak's Avatar
 
Posts: 24 since Nov 2011
Thanks: 31 given, 46 received

in the past when I develop a strategy I first run the optimizer on a range of historical data to find best settings for parameters. Then I enter the settings the optimizer found as best and run backtest on same time period with same settings and ensure that I do get the value returned by the optimizer (sort of a sanity check on making sure I have settings entered correctly) then I run backtests on out of sample time periods. Only after that do I move on to sim trading.

I have a strategy now though, that when I optimize and get some results and settings, and enter those settings to do backtest... it does not match the results the optimizer claimed to have gotten with the same settings. I have meticulously checked every setting and they are set the same on backtest as on best run from optimizer. The results aren't even close.

Any ideas?

Follow me on Twitter Started this thread Reply With Quote

Can you help answer these questions
from other members on futures io?
Automatic pattern research with NT
Elite Algorithmic NinjaTrader Trading
How to implement this?
MultiCharts
COTbase for quant trading?
Elite Algorithmic NinjaTrader Trading
Regarding Multicharts strategy trigger
MultiCharts
If ES tracks SPX, why do support/resistance, Level 2, vo …
Emini and Emicro Index
 
 
(login for full post details)
  #3 (permalink)
Site Administrator
Swing Trader
Data Scientist & DevOps
Manta, Ecuador
 
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
 
Big Mike's Avatar
 
Posts: 49,785 since Jun 2009
Thanks: 32,314 given, 97,581 received


Are you using exotic bar types?

Are you using genetic optimizer or exhaustive? If genetic, using the NT one or third party?

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #4 (permalink)
Huntsville AL/USA
 
Experience: Beginner
Platform: NinjaTrader
Trading: Stocks, Futures
 
d1g1talfr3ak's Avatar
 
Posts: 24 since Nov 2011
Thanks: 31 given, 46 received

I am using the Ninjatrader built-in genetic optimizer. I would have thought that each step of the genetic algorithm it would internally run a backtest with selected values... then score value and run genetic routine to choose next generation of parameters. So I am stumped as to why running a backtest with the same settings the optimizer found doesn't match the results it showed. I have verified and re-verified that all settings are the same... including slippage & commisions, etc.

I can optimize other strats and then backtest on same time period with same values and get the same results. At first I thought that it was because this strat used an indicator that repainted past values. I tried telling the backtester and optimizer to not run multithreaded and a few other ideas... no go. I then wrote my own version of the indicator that did not repaint past values and still the same thing.

Thanks for any ideas and help

Follow me on Twitter Started this thread Reply With Quote
 
(login for full post details)
  #5 (permalink)
Huntsville AL/USA
 
Experience: Beginner
Platform: NinjaTrader
Trading: Stocks, Futures
 
d1g1talfr3ak's Avatar
 
Posts: 24 since Nov 2011
Thanks: 31 given, 46 received

Also... I am using Better Renko. I am using it on several other strategies and not having the same issue.

Follow me on Twitter Started this thread Reply With Quote
 
(login for full post details)
  #6 (permalink)
McKinney, TX
 
Experience: Intermediate
Platform: ThinkOrSwim, NinjaTrader
Broker: TOS & Rithmic
Trading: The indexes, ES, YM, NQ & RTY
 
Posts: 171 since Aug 2012
Thanks: 216 given, 157 received

I had problems using Ranko bars. I had a tested strategy that was giving me very good results, then I ran that strategy against live data on the sim account and got a totally different return.

I don't know for sure, but what I think happens is that the Renko bar historical data is truncated so that the bar high/low values are used for the optimization. When running live, it appears that there are ticks outside of the Renko bars, those wide range ticks were hitting my stop loss and changing the results. What I would see is the execution of orders outside of the Renko bar.

Not sure if that is your problem, but when I tried the same strategy with Range bars, I couldn't duplicate the great results in the Analyzer but they matched the results better when I had a live data feed. Because of this, I won't use Renko bars on a strategy.

Reply With Quote
 
(login for full post details)
  #7 (permalink)
Szczecin
 
Experience: Intermediate
Platform: Ninja, writing own now
 
Posts: 212 since Mar 2010

Yes. Ninja does not use any data outside the bars, so you need bars that show true high low or better anther bar in the strat (tick 1) and then issue trades against this one.

Reply With Quote
 
(login for full post details)
  #8 (permalink)
Site Administrator
Swing Trader
Data Scientist & DevOps
Manta, Ecuador
 
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
 
Big Mike's Avatar
 
Posts: 49,785 since Jun 2009
Thanks: 32,314 given, 97,581 received

Renko bars and other exotic bar types have false opens, which makes backtesting them totally false.

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #9 (permalink)
Miami FL, USA
 
Experience: Beginner
Platform: NinjaTrader
Trading: Forex
 
Posts: 1 since Apr 2012
Thanks: 5 given, 0 received

I have also had similar experiences with the NinjaTrader built in Genetic Optimizer. In my case, I used a plain old 5 minute chart, ran the optimizer, then backtested the winner and received wildly different results.

Upon inspecting the actual trades in the NinjaTrader Strategy Analyzer, it looks as if the optimizer wasn't using all of the price data available. What I mean is that if the strategy had a rule to close a position on a MA crossover, the optimized strategy did not close the position whereas the backtested strategy (with identical settings) did close the position on the crossover.

Reply With Quote
 
(login for full post details)
  #10 (permalink)
Los Angeles, CA
 
 
Posts: 50 since Nov 2011



dynoweb View Post
I had problems using Ranko bars. I had a tested strategy that was giving me very good results, then I ran that strategy against live data on the sim account and got a totally different return.

Perhaps overfitting?

Reply With Quote


futures io Trading Community Platforms and Indicators NinjaTrader > Backtest not matching Optimization


Last Updated on November 11, 2012


Upcoming Webinars and Events
 

NinjaTrader Indicator Challenge!

Ongoing
 

Battlestations! Show us your trading desk - $1,500 in prizes!

March
 

Importance of Finding Your Own Way w/Adam Grimes

Elite only
 

Journal Challenge w/Jigsaw

April
     



Copyright © 2021 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts