What is everyones experience with tradestation walk forward optimizer. I am curious if it is the same as market replay optimization that I have read about on other trading platforms. I am working on a strategy and receiving pass results on walk forward optimizer and great ratios and returns on the performance reports. But I have not been doing this for long so I find it hard to believe something good be this true.
Also, what differences can I expect between simulated auto trading an real cash money auto trading
I thought I was going to be a billionaire at least 20 times when I used Tradestations backtesting and walk forward testing features.
I found out that many strategies written on lower time frames especially tick charts will give you awsome results backtesting and walkforward , but when run live totally flop mostly due to the way historically the bars are reported to be filled verses how they really would of been filled is usually quite different.
Now on the other hand if your using a higher timeframe in your strategy you will reduce that problem significantly.You will want to probly focus on getting some intrabar granularity coded in to your strategy using Multy Time Frames to get more accurate backtest results also.
Its been a few years since I used them and switched to Ninjatrader and It isnt perfect either but maybe some current users will post some better insight for you.
Trust your instincts. "Too good to be true" probably is. By questioning the results, instead of just accepting them at face value, you are thinking above most of the crowd, and that is great.
It could be a timeframe issue, as @quantismo pointed out. Or, it could be a hundred different things, either with the type of bars you are using, the type of orders, the backtesting method, the walk forward optimizer, assumptions about slippage and commission, etc. The list of potential issues is long.
The best thing to do is to assume that the great results are a fraud, because they most likely are. Then, dig into it and figure out why they are fraudulent. That will pay huge dividends in the future, as you develop new strategies - you'll avoid at least this source of "too good to be true."
If you have any questions please send me a Private Message or use the futures.io "Ask Me Anything" thread
The following 2 users say Thank You to kevinkdog for this post:
What kevindog said is VERY IMPORTANT. It's how I found my first profitable strategy. Figure out why the results are skewed.
What I found was that they were skewed because if improper slippage assumptions - lack of tick data older than 6 months- and a few other things - when I finally fixed all the "broken" pieces I had a winning formula. It turns out the results were skewed to look way too good - but after all of the fixing it really was an edge - Just not quite as robust as the original results showed.
The following user says Thank You to Yoyo4830 for this post:
i have found the walk-forward optimizer to be irrelevant. i now optimize leaving a sizable chunk of data out of sample by setting the last date some weeks past, and then check how the strategy does when extended to the out of data sample to evaluate if has a chance of working live or not. this is better and faster than using the w-f o.
higher time frames or bigger bar sizes are better. as for simulated versus live trading, strategies that trade with limit orders on the simulator generate results that are not credible, because getting fills with limit orders is really hard trading live. market orders on the simulator are closer to the real thing, provided you consider big enough slippage values as fills tend to be very bad.