I have developed 3 indicators that each test profitably. I've determined the optimal parameters by optimization (periods, thresholds, etc.). I do not expect to get the same results in the future, but I prefer to use the optimized values rather than some arbitrary values. For example everyone may use 12,26,9 for their MACD but if 10,22,8 give better results in backtesting them it'd be better to use that right?
My question is this: I'm now working on combining these 3 into one signal (short, flat, long). I've tried two different approaches to do this:
1 - I use the optimal parameters that I determined on each indicator individually
2 - I re-optimized all parameters together.
#1 seems to be more realistic, with the acknowledgment that the performance will not be the same as the backtests, due to the performance of each system not being the same. This I know. So the final results will probably not be as good. But this seems robust.
#2 - Seems to be more optimal, with an even stronger acknowledgment that the results will not be as good as the backtest. However there is a greater risk of curve fitting due to the increased rules and degrees of freedom. In defense of the optimization I will say that lots of attempts produced unacceptable results so I believe that if optimization finds something good say PF > 3.0 then it's very likely to be positive in forward testing even though the PF will most likely be less.
I'm curious what people think about these two approaches. I am currently forward testing both #1 & #2 but since they trade on daily charts and not very often, it will take a while to have something meaningful.
I've developed systems that have held up and systems that have fallen apart. I understand the limitations of backtesting and automation. So I prefer not to debate that but focus on which approach would be best.
The following user says Thank You to cunparis for this post:
I've read about this but I've never done it, which is probably why most of my strategies stopped working.
one of my indicators has 500 trades since 2000. I'm going to optimize it for the first 100 trades and then walk it forward the next 100. Then optimize the first 200 and walk it forward the next 100. So I'll have:
RJay - I think we're talking about different things. I made an indicator, but the indicator uses lookback periods, moving averages, thresholds, etc. It's those that I want to optimize to find the optimal values for my indicator. Just like someone came up with 14,3,3 for stochastic.
Technically both of the approaches you describe above are curve fitting =)
Global parameter search (method #2) is a much better approach because a disjoint search does not take into consideration the relationships between the different indicators.
You want to find a 'stable region' of parameters, ie a range of numbers near each other which all give similar performance, that is good curve fitting.
Bad curve fitting is when you find the best set of parameters which may be an outlier in an unstable region, ie it may have high profit now but the likelihood of it continuing to work in the future is lower.
See the attached matlab plots for an illustration of this. The colors represent performance of a strategy based on normalized Sharpe ratio, red means highest sharpe ratio blue is lowest.
You might be tempted to pick a value along the sharp red ridge that runs along Param1=20, but actually a better choice for the long run is one of the flatter red bumps at the back (near Param1=50, Param2 = 225)
The sharpness of the first ridge makes it inherently unstable, think of it just like in real life if you were standing on a sharp peak and it moved a little bit, you would fall off (which in this case results in drawdown) whereas if you are standing on a flatter hill you will not be affected as much if it shifts slightly.
(BTW.. i generated these images but they are of a strategy from a mathworks webinar, if any of you are matlab users you can download the code and webinar from their website)
Last edited by sefstrat; August 30th, 2009 at 07:41 AM.
The following 6 users say Thank You to sefstrat for this post:
I'm sad to report the walk forward tests weren't that good. I started with 2000-2003 to give me 100 trades. Then I walked forward year by year. The first couple years it did well on the walk forward test. But then around 2007-present it didn't. I can explain that because that's when we went from bull to bear market. So if we figure the bear market started roughly end of 2007, the problem is I don't have enough trades during the bear market. And even then things changed in March when we started the bull market (or bear market rally). And the market direction isn't the only thing that changed, volatility has completely changed over the past few years.
For me this is the hardest part of creating a mechanical system on a daily timeframe. I've spent a lot of hours on this but I'm afraid it's back to the drawing board.
Lesson learned: With two moving averages you can come up with a profitable system using just about any data.
I still think there's some predictive power here but I have to find a way to prove it..
Sorry it didn't work, but now you know what to do and hopefully next time you will succeed. Don't blame it on bull or bear market. As I understand you have a trend following system, so it should work on both sides.
In this case I'm using minute data from 2000 - present, using Tradestation. This is a big problem with zen-fire not having continuous contracts and long history. I have iqfeed for this purpose for NT.
I have read pardo's book and need to reread the parts about walk forward testing. I worked a lot on my system and when it didn't hold up I was very disappointed. so now I'm focusing on non-mechanical trading. One thing I want to do is try my algorithm with other data sources and that should be interesting. if they can be curve fitted than anything can. if they cannot then maybe my original idea was on to something.