in the past when I develop a strategy I first run the optimizer on a range of historical data to find best settings for parameters. Then I enter the settings the optimizer found as best and run backtest on same time period with same settings and ensure that I do get the value returned by the optimizer (sort of a sanity check on making sure I have settings entered correctly) then I run backtests on out of sample time periods. Only after that do I move on to sim trading.
I have a strategy now though, that when I optimize and get some results and settings, and enter those settings to do backtest... it does not match the results the optimizer claimed to have gotten with the same settings. I have meticulously checked every setting and they are set the same on backtest as on best run from optimizer. The results aren't even close.
Any ideas?
Can you help answer these questions from other members on futures io?
I am using the Ninjatrader built-in genetic optimizer. I would have thought that each step of the genetic algorithm it would internally run a backtest with selected values... then score value and run genetic routine to choose next generation of parameters. So I am stumped as to why running a backtest with the same settings the optimizer found doesn't match the results it showed. I have verified and re-verified that all settings are the same... including slippage & commisions, etc.
I can optimize other strats and then backtest on same time period with same values and get the same results. At first I thought that it was because this strat used an indicator that repainted past values. I tried telling the backtester and optimizer to not run multithreaded and a few other ideas... no go. I then wrote my own version of the indicator that did not repaint past values and still the same thing.
I had problems using Ranko bars. I had a tested strategy that was giving me very good results, then I ran that strategy against live data on the sim account and got a totally different return.
I don't know for sure, but what I think happens is that the Renko bar historical data is truncated so that the bar high/low values are used for the optimization. When running live, it appears that there are ticks outside of the Renko bars, those wide range ticks were hitting my stop loss and changing the results. What I would see is the execution of orders outside of the Renko bar.
Not sure if that is your problem, but when I tried the same strategy with Range bars, I couldn't duplicate the great results in the Analyzer but they matched the results better when I had a live data feed. Because of this, I won't use Renko bars on a strategy.
Yes. Ninja does not use any data outside the bars, so you need bars that show true high low or better anther bar in the strat (tick 1) and then issue trades against this one.
I have also had similar experiences with the NinjaTrader built in Genetic Optimizer. In my case, I used a plain old 5 minute chart, ran the optimizer, then backtested the winner and received wildly different results.
Upon inspecting the actual trades in the NinjaTrader Strategy Analyzer, it looks as if the optimizer wasn't using all of the price data available. What I mean is that if the strategy had a rule to close a position on a MA crossover, the optimized strategy did not close the position whereas the backtested strategy (with identical settings) did close the position on the crossover.
