I remember from my statistics class that 30 data points is the minimum sample size needed to properly test the significance of a set of data. What I'm wondering is what units to use to see if my strategy can be profitable long-term. The choices would be trades, days, weeks, or months. Any ideas from the veterans?
One more question if anyone cares to answer. Is using Market Replay on NT a reliable way to backtest where the results would correspond very closely to live trading? Would 100 MR trades provide enough evidence to attempt going live if the trades were taken the same way as you'd take them live?
Back testing is fraught with danger if you do not take proper precautions to prevent what is known as "curve cutting." That is, tweaking with the parameters to get the best results from your data set. Your first line of defense is to have two data sets. Train on one and then pass your algorithm over the out of sample set. Only if the results are similar can you have confidence in SIM trading it. If SIM trading shows promise, then on to small money trading.
Curve fitting is only the largest rock in the stream when it comes to back testing. Other considerations include how close you get to tick data and the kind of slippage your analysis incorporates. I could go on, but ...
If it is to be, it is up to me.
All I Know About Trading Options I Learned in Flight School
The following user says Thank You to HowardCohodas for this post:
On Ninja, Market Replay is a lot more reliable than their backtesting function. I'm not sure it's equivalent to live trading, but it is closer to it than Ninja's backtesting is. As for going live based on this... thats a different question and there are many different books, threads, and opinions on what kind of testing is needed before going live. Also depends on your time frame and your strategy. The market is changing all the time so it's not for sure that it will still behave similarly to how it did when you tested the 100 trades. It's very common for people to create systems that perform well for a while, but eventually crash and burn. Generally you can expect a system to perform worse live than it does in testing, so you want it to perform very well in testing before you think about going live. Good luck!
The following user says Thank You to GoldStandard for this post: