Backtesting & Optimization -- worthless? How to get value from them?
I've seen a lot of replies buried in threads that subtlety suggest that backtesting & optimization do not have as much value as we might be led to believe. People have posted "successful" strategies, and advanced users quickly reply to say the results are unrealistic and can not be replied upon (and often fail, miserably, in live trading).
Simple questions -- Why? And how should we be utilizing these functions to realize the most value from them?
I'm sitting here trying to develop a strategies (as a new NT user) using the Backtest feature of Ninjatrader. I have been giving weight and considerations to the minute differences in results from the Backtesting engine. However, after reading some of the posts here at Futures.io, I wonder if I am throwing out profitable strategies, going in the wrong directions towards improving a strategy due to unreliable(?) data coming from the Backtesting .
I've seen some people suggest to use optimizer on half of your data series, then backtest using the optimized parameters for the other half. If that's something that is sound advice, that is the kind of information & tips that I am seeking with this thread.
Hopefully the experts and advanced users can shed some light here.
Last edited by Jayswiss; July 17th, 2016 at 12:43 PM.
The following user says Thank You to Jayswiss for this post:
It boils down to 'know' what you are doing and what to expect from back-testing and optimization.
It is possible to take a period of data and find 'magic' settings that create the best possible (optimized)
result over that period. Now adding data from another period, or just from tomorrow, will change the
results and they may be horrible.
Is all optimization bad ? no, you need to understand what you are doing
If a series of values for a parameter 16,17,18,19,20 all create positive profitable results and you pick
the best value, the chances are that your system is less vulnerable. If you have 18 being profitable
and all the rest loosers, guaranteed your strategy will fail in forward.
Probably a good read would be the book of Kevin. it walks you step by step through an approach.
Is it the only possible approach ? no.. for sure not
Is it a good approach ? if you apply and the strategy passes the tests, then you are on good track
Can you link to "The book of Kevin"? I tried some searches and I guess that's not the title?
You bring up a good point with regards to general optimizations. For example, many people favor SMAs (or EMAs) of 50/200 or 9/50. I assume these numbers are the favorites because they were tested (long before NT existed) and seemed to be the most consistently proven winners across the board? Or were they just rabbitoutofhat starting points that stuck after a couple people wrote books about them? Even today on many wise TA educational sites, you will find "standard" sets of numbers to use for SMA crossovers (and RSI, and MACD, ADX, etc. etc.). But of course I agree... 50/200 may be hat magic today, but terrible tomorrow.
For optimization specifically though, I do like the idea of using the same specific periods of difficult historical trading periods + a few "regular" trading periods to test against as a sort of benchmark. A backtesting profile? Something like that. Then when you find something that worked through the bulk of those periods within your standard profile of periods to test, retry against a current or more recent period of time and see how it works out.
With regard to back testing I'm interested in larger historical time periods and thus have a need for a good data source. I am specifically interested in YM, ES, NQ, ZN,ZB,E6 DAX etc.... I'm not interested in free stuff etc... Just a quality source where I can purchase the needed data and upload it into NT7. Anyone have any experience with this, hopefully going back at least 3 years.