Let's say I back tested and tuned a model on 10 years of data 2004-2013. Not that it matters but lets say my instrument isn't something simple like ES, but say an energy spread, that is generally mean reverting.
I then forwarded tested my data on 2014 and it performed well so I went live with my model in 2015 and it has performed well also.
As I go into 2016 should I still be using the model tuned on the original 10 years of data 2004-2013 data? Should I update that to 11 years 2004-2014 or keep it 10 years but move it forward 2005-2014, then re-tune, and walk forward on 2015 data just to make sure its still good? Or should I even consider re-tuning on the 12 years 2004-2015 data or even the most recent 10 years 2006-2015?
The following 5 users say Thank You to SMCJB for this post:
That is a good question. I'm not sure there is a correct answer, but there are some alternatives...
1. What you describe is what many people call a standard "out of sample" test. The typical way these are run do not have any mechanism for re-optimizing. So, whatever model you initially chose, which also worked on out of sample data, would be used indefinitely, and you'd never re-optimize. You could trade this model for 10-20 years and the model would never change. It may or may not hold up.
2. But, you could have done this as a walkforward test with a moving window. Then every year, you'd re-tune your model with the most recent 10 years of data. So, your model could change every year. This is walkforward with unanchored testing (the in sample optimization window moves).
3. Finally, you could re-optimize your model every year, but use all the data available. So, the first optimization is 10 years of data. Next one will use 11 years or data, and so on. The model still gets re-optimized every year. This is a walkforward optimization that is anchored (start point never moves).
Which is best? I guess it depends who you ask. I personally use option 2. I don't like #1 because it never changes, and I don't like #3 because old data keeps impacting the optimization well into the future.
The one thing about your case is that it is not really walkforward optimization yet, since you only had one out of sample period. That one period of out of sample might not be significant - that's why true walkforward testing has 10-20+ out of sample periods.
Question for you: how much does the model change between these 3 alternatives:
1. keep original model
2. re-optimize model with 10 years of data
3. re-optimize model with 11 years of data
Is there a huge difference in the models?
If you have any questions please send me a Private Message or use the futures.io "Ask Me Anything" thread
The following 6 users say Thank You to kevinkdog for this post:
It's interesting what you said in 2) as that relates to the next part of my question.
When I originally started on this model, I arbitrarily picked the last 11 years (04-14), 10 years to look at (04-13), and 1 year (14) as an out of sample test. Once I completed the first model (which is the model I'm actually still trading today) I started delving deeper into the data. I looked at using the last 9, 8, 7, 6 & 5 years rather than 10. The results showed that the model with the best fit used the last 7 years (07-13) rather than 10 years (04-14) of data. I could tell from looking at the models that the higher energy prices of the more recent years (2008, 10-13) where heavily influencing the models. Since the 7 year models results weren't significantly better than the 10 year, I decided to just stick with the original 10 year.
I actually then built a new model, that included a 'year variable'. This model was significantly better than any of the other models. The problem is the addition of the year variable gave values for 2015 significantly higher than anything previously seen, and of course 2016 would probably be even higher. Despite this model having better results I did not use it, and kept with the original 10 year model.
As we all know, energy prices have plummeted in the last year. The model I am using (built based upon 10 years (04-14)) has performed admirably. The model based upon 7 years of data would have been profitable, but not nearly as good. The model that included the 'year variable' would have got destroyed.
I'm comfortable with my decision to use the 10 year vs the 7 year, but by choosing the 10 year over the 'year variable' I feel like I allowed my personal bias to overrule what should be a more systematic and statistical approach. The fact that my personal bias was correct, obviously makes me feel good, but I'm still concerned I've contaminated the process.
So getting back to your point 2
you could have done this as a walkforward test with a moving window. Then every year, you'd re-tune your model with the most recent 10 years of data. So, your model could change every year. This is walkforward with unanchored testing (the in sample optimization window moves).
This is actually what I was planning to do. The problem is, I already know what happened in 2015. If I roll my window forward to 04-14 with an out of sample 2015, I know what the results will be. I know that any model more heavily based upon the recent high energy prices will perform far worse than a model with a longer tenor of data. I strongly suspect that a new model with a "polynomial year variable" (something like -(year-2011.5)^2) would probably perform quite well, but I know that because I know energy prices have had a concave trajectory over the last 15 years. I feel like whatever I do, my personal bias's/knowledge are corrupting the process.
I know there's no actual question in there, but I was curious if you have any additional thoughts on the matter?
The following 3 users say Thank You to SMCJB for this post: