Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
This is for anyone who automates their strategies. I was wondering at what point you consider a strategy good enough to go into sim/live trading. What kind of sample size do you look for, do you have a profit VS drawdown you look for. Do you have a PF requirement? Risk of ruin etc? Any insight would be greatly appreciated as I feel that I don't really know at what point I can consider a strategy (that has not been over-optimized) ready to go. Thanks in advance.
That's a very, very open-ended and highly subjective question
I'm currently reading the 500+ page book "Evidence Based Technical Analysis", and even this book, as excellent as it is, cannot answer your post in a definitive fashion so I don't think you can really ever expect an answer.
But, in loose generalized outline form, I look for:
- At least 1 year of tick level data
- At least 1,000 trades in sample
- Always do a out of sample/walk forward analysis, make sure the results are at least comparable to the in sample results
- Always include commission costs
- Always include slippage costs
- Always sim test for 1-2 weeks before going live, and check every order carefully
- Don't use Renko, Point and Figure, Kagi or Line Break charts
- Incorporate daily stop and targets that you can live with* (more on this below)
* This is a big one. If your automated strategy says it will make a million bucks, but you lose faith in it when on cash simply because it is in a drawdown period, it does no good for anyone. Put another way, if you can't stomach a $ 1,000 daily draw down for two weeks straight, then make sure your strategy can't either. If you can't stomach having a $ 1,000 profitable trade turn into $ 0, or even a loss, then make sure your strategy can't either. In other words, make sure that you will not be tempted to override the strategy or stop trading the strategy, etc etc, or otherwise you end up negating all the backtesting.
- Don't have separate parameters for Longs vs. Shorts, to do so introduces a hindsight bias and curve fitting bias, not unless you can predict the future
"""" - Don't use Renko, Point and Figure, Kagi or Line Break charts """"
Is this a general rule on all markets?
Are there specific reasons for each of these chart types, or is this a general subjective &/or objective determination?
Yes, it's a general rule, all markets, related to backtesting problems only.
Basically, if you don't know about the problems generated by backtesting these chart types, then do not use them. If you do know, then you know how to overcome them and all the extra code it will take, and you can disregard.
Thanks for the advise. I have been in the game enough to understand what it means to accept risk and stick with a plan. I just have a hard time determining at what point these automated plans are good enough to go live with. If I can ask you this, once you have your sample set and you have your historical draw down defined, how do you determine your point of system failure? Do you just say that if it draws down X*previousDrawdown that you will pull the plug?
Well, we know that a backtest is data mining and curve fitting. The likelihood that a future result will outperform a backtest, given sufficient sample size, is not very high.
So, if you accept that the backtest results were "optimal", you can accept that all of your performance measuring metrics were also optimal, and that going forward you should allow for wiggle room. For instance, if over 1,000 trades your win percentage was 50%, and your win/loss dollar ratio was 1.5, and your maximum close-to-close drawdown was $5,000, then you could reasonably define 'failure' by using these figures as guidelines and adding the 'wiggle room', such as a 45% win rate, a 1.2 wld ratio, or a drawdown exceeding $ 7,500.
There are more scientific ways of doing this such as using a standard deviation measurement. MultiCharts actually has a lot of these types of metrics built-in, whereas NinjaTrader does not.
If you still have doubts then I suggest you post the performance data for one of your strategies in question. Let's look at the sample size and the IS and OOS results and then take a look at a two-week period of live (sim) trading and see what we have.
Also you really can't place enough emphasis on using out of sample data. The real problem is that once out of sample data is used just once, it really is no longer virgin out of sample data.
For instance, if you have Strategy A and you run a backtest on 360 days of in-sample data, and 180 days of out-of-sample data. Let's say you didn't much like the results of the OOS data.
You can't just make changes to Strategy A and then re-run your backtest on the same IS and OOS data. You've effectively turned the OOS data into IS data, at least in your own mind, because you have the hindsight of know if your changes to the strategy have "improved" the result. Naturally, it's not OOS data if you can make a change to it and measure if the change is better or worse.
it's my opinion (for what it's worth) that it's as hard to develop a good metric with which to evaluate an automated strategy as it is to develop the strategy itself. it's also much more important to have a good metric than to have a good strategy - without a good metric you cannot know whether or not the strategy you have is any good.
the problem is that you have to find some function that takes into account all the various characteristics of your strategy's performance:
- expectancy
- draw-down
- cumulative profit
- cumulative/contiguous risk
- resilience to changes in market conditions
- etc..
... and it has to represent all of that all in a single number. but how do you weigh the various values with respect to each other?
i'd love to see some more discussion on what people consider to be a good measure of an automated strategy. and, no, for me the sharpe ratio doesn't cut it...
