Walk Forward Testing & Optimization Experiences and Best Practices

September 30th, 2012, 08:15 PM

Just curious what are others experiences and best practices for using NinjaTraders Walk Forward Optimization?

I've read that the optimal optimization period (in sample) to testing period (out of sample) ratio is around 3:1 with the number of trades being the most important (~100 minimum). I've also learned that its important to optimize against data which includes an up trend period, a down trend period, and a consolidation period to lesson against a biased result.

On to hardware... Obviously the faster the better, and I'm stuck with using a 1.8Ghz AMD Sempron w/ 2GB at the moment, but is an overclocked system the best way to go or has anyone found a satisfactory baseline? This will probably be a non-issue if they ever move Strategy Analyzer computation to the GPU. EDIT: Looks like for now the fastest quietest CPU you can get the better.

Any thoughts are appreciated!

October 8th, 2012, 11:54 PM

So far the two minor annoyances I'm running into are:

Finding the optimal in-sample and out-of-sample periods.
Having enough computing resources.

Regarding #1, I found on the NinjaTrader Support Forum this thread on How to optimize the Optimization and Test Periods?.

December 29th, 2013, 01:31 PM

I've used WFO for several months across over 100 strategies (across portfolio of futures, stocks, ETFs) and it's been a frustrating experience. Even strategies with few parameters that perform well (Profit Factor>1.6, APR>20%, MAR>0.5) break down in WFO.

Wanted to share my learnings and generate a discussion around best practices..

1) Objective function really matters - choosing "Highest/Lowest" metrics (e.g. Highest Profit Factor, Highest Select Net Profit, Highest MAR, etc) set you up for failure - by definition, they pick the outliers in-sample period which invariably fail in out-of-sample periods. Ideally an objective function should do well across 3 independent metrics/dimensions through some sort of a ranking function (e.g. 80 percentile ranks in each of these dimensions). The three dimensions are
- measurement of trade performance - e.g. Profit Factor, Trade Efficiency
- measurement of capital utilization - Net Profit, Exposure, Select Net Profit
- measurement of portfolio drawdown risk - e.g. Max Drawdown, Lake ratio, Sortino Ratio, etc.

I haven't come across any tool that offers this kind of objective function. If you have any suggestions please advise.

2) In WFO of portfolios/baskets of symbols, per-symbol strategy parameter values perform much better than average parameters across basket/portfolio. I haven't seen tools that support per-symbol optimized values (not sure of NinjaTrader, but most others don't).

3) WFO time period - while the 3:1 (3 in-sample, 1 out-of-sample period) ratio is well-establish, there are no guidelines of the absolute period of WFO - for a daily strategy, is it every 3 months, 1 year etc. Ideally, this should mirror the regime change period of the symbol. In fact, in absence of a tool that does interval optimization across different time periods to determine best regime period for each symbol and runs WFO over that regime, most WFOs will fail.

I've reached a conclusion that unless you do WFO correctly that accomodate these 3 factors, you will tend to discard robust strategies with good potential and get disillusioned.

-> Wanted to get your sense of success rates with WFO - of all the strategies you've backtested/optimized and felt excited/comfortable, how many have you successfully walked forward? And how? Any pointers from experts would be helpful.

thanks
Kiran

Walk Forward Testing & Optimization Experiences and Best Practices

Discussion in NinjaTrader

Walk Forward Testing & Optimization Experiences and Best Practices