Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
I have developed 3 indicators that each test profitably. I've determined the optimal parameters by optimization (periods, thresholds, etc.). I do not expect to get the same results in the future, but I prefer to use the optimized values rather than some arbitrary values. For example everyone may use 12,26,9 for their MACD but if 10,22,8 give better results in backtesting them it'd be better to use that right?
My question is this: I'm now working on combining these 3 into one signal (short, flat, long). I've tried two different approaches to do this:
1 - I use the optimal parameters that I determined on each indicator individually
2 - I re-optimized all parameters together.
#1 seems to be more realistic, with the acknowledgment that the performance will not be the same as the backtests, due to the performance of each system not being the same. This I know. So the final results will probably not be as good. But this seems robust.
#2 - Seems to be more optimal, with an even stronger acknowledgment that the results will not be as good as the backtest. However there is a greater risk of curve fitting due to the increased rules and degrees of freedom. In defense of the optimization I will say that lots of attempts produced unacceptable results so I believe that if optimization finds something good say PF > 3.0 then it's very likely to be positive in forward testing even though the PF will most likely be less.
I'm curious what people think about these two approaches. I am currently forward testing both #1 & #2 but since they trade on daily charts and not very often, it will take a while to have something meaningful.
I've developed systems that have held up and systems that have fallen apart. I understand the limitations of backtesting and automation. So I prefer not to debate that but focus on which approach would be best.
I've read about this but I've never done it, which is probably why most of my strategies stopped working.
one of my indicators has 500 trades since 2000. I'm going to optimize it for the first 100 trades and then walk it forward the next 100. Then optimize the first 200 and walk it forward the next 100. So I'll have:
this will make 4 walk forward tests. I'll compare the results of each walk forward with the same period optimized. I will also compare the optimal values for each test.
Does this sound right?
I'll post the results as I go. Here's the equity curve for the results of the optimization of the entire 9 year period (500 trades).
RJay - I think we're talking about different things. I made an indicator, but the indicator uses lookback periods, moving averages, thresholds, etc. It's those that I want to optimize to find the optimal values for my indicator. Just like someone came up with 14,3,3 for stochastic.
Technically both of the approaches you describe above are curve fitting =)
Global parameter search (method #2) is a much better approach because a disjoint search does not take into consideration the relationships between the different indicators.
You want to find a 'stable region' of parameters, ie a range of numbers near each other which all give similar performance, that is good curve fitting.
Bad curve fitting is when you find the best set of parameters which may be an outlier in an unstable region, ie it may have high profit now but the likelihood of it continuing to work in the future is lower.
See the attached matlab plots for an illustration of this. The colors represent performance of a strategy based on normalized Sharpe ratio, red means highest sharpe ratio blue is lowest.
You might be tempted to pick a value along the sharp red ridge that runs along Param1=20, but actually a better choice for the long run is one of the flatter red bumps at the back (near Param1=50, Param2 = 225)
The sharpness of the first ridge makes it inherently unstable, think of it just like in real life if you were standing on a sharp peak and it moved a little bit, you would fall off (which in this case results in drawdown) whereas if you are standing on a flatter hill you will not be affected as much if it shifts slightly.
(BTW.. i generated these images but they are of a strategy from a mathworks webinar, if any of you are matlab users you can download the code and webinar from their website)
I'm sad to report the walk forward tests weren't that good. I started with 2000-2003 to give me 100 trades. Then I walked forward year by year. The first couple years it did well on the walk forward test. But then around 2007-present it didn't. I can explain that because that's when we went from bull to bear market. So if we figure the bear market started roughly end of 2007, the problem is I don't have enough trades during the bear market. And even then things changed in March when we started the bull market (or bear market rally). And the market direction isn't the only thing that changed, volatility has completely changed over the past few years.
For me this is the hardest part of creating a mechanical system on a daily timeframe. I've spent a lot of hours on this but I'm afraid it's back to the drawing board.
Lesson learned: With two moving averages you can come up with a profitable system using just about any data.
I still think there's some predictive power here but I have to find a way to prove it..
Hi,
Sorry it didn't work, but now you know what to do and hopefully next time you will succeed. Don't blame it on bull or bear market. As I understand you have a trend following system, so it should work on both sides.
cunparis have you looked at "The Evolution & Optimization of Trading Strategies" by Pardo? Interesting chapters on curve fitting vs. optimization.
Out of curiosity what are you using for historical data and in what timeframe? I'm working on range and tick based charts and it would great if I could test more than the last two-three contracts.
In this case I'm using minute data from 2000 - present, using Tradestation. This is a big problem with zen-fire not having continuous contracts and long history. I have iqfeed for this purpose for NT.
I have read pardo's book and need to reread the parts about walk forward testing. I worked a lot on my system and when it didn't hold up I was very disappointed. so now I'm focusing on non-mechanical trading. One thing I want to do is try my algorithm with other data sources and that should be interesting. if they can be curve fitted than anything can. if they cannot then maybe my original idea was on to something.
