Over Optimization/Curve Fitting? - futures io
futures io



Over Optimization/Curve Fitting?


Discussion in Traders Hideout

Updated
      Top Posters
    1. looks_one kevinkdog with 5 posts (7 thanks)
    2. looks_two WoodyFox with 4 posts (2 thanks)
    3. looks_3 artemiso with 3 posts (6 thanks)
    4. looks_4 fxFlux with 1 posts (1 thanks)
      Best Posters
    1. looks_one artemiso with 2 thanks per post
    2. looks_two kevinkdog with 1.4 thanks per post
    3. looks_3 fxFlux with 1 thanks per post
    4. looks_4 WoodyFox with 0.5 thanks per post
    1. trending_up 1,964 views
    2. thumb_up 16 thanks given
    3. group 7 followers
    1. forum 15 posts
    2. attach_file 1 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 125,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

Over Optimization/Curve Fitting?

(login for full post details)
  #1 (permalink)
WhatIfGod
Birmingham, England
 
 
Posts: 21 since Nov 2018
Thanks: 18 given, 2 received

Currently I create the code, backtest on 6 months worth of data, then optimize that 6 month period and see whether those optimized settings work on previous historical data for the same instrument in a backtest. I know some of you guys will say 'read the fucking literature' and I hear you loud and clear, I will, but I'm keen to hear what people think about that simple method. Am I curve fitting the data for the 6month initial test period and as a result destined for it not to succeed on other historical data. Just want to make sure I'm not barking up the wrong tree. Peace.

Reply With Quote

Journal Challenge April 2021 results (now extended!):
Competing for $1800 in prizes from Jigsaw
looks_oneMaking a Living with the Microsby sstheo
(114 thanks from 18 posts)
looks_twoSalao's Journalby Salao
(36 thanks from 8 posts)
looks_3Deetee’s DAX Trading Journal (time based)by Deetee
(33 thanks from 14 posts)
looks_4Learning to Profit - A journey in algorithms and optionsby Syntax
(14 thanks from 9 posts)
looks_5Maybe a little bit different journalby Malykubo
(13 thanks from 12 posts)
 
Best Threads (Most Thanked)
in the last 7 days on futures io
Would You Sell Your System?
75 thanks
Big Mike in Ecuador
57 thanks
The Crude Dude Oil Trading System
52 thanks
The New Micro Contract - MICRO BITCOIN coming May 2021
25 thanks
futures io site changelog and issues/problem reporting
24 thanks
 
(login for full post details)
  #3 (permalink)
 artemiso 
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
 
Posts: 1,129 since Jul 2012
Thanks: 765 given, 2,540 received



WhatIfGod View Post
Currently I create the code, backtest on 6 months worth of data, then optimize that 6 month period and see whether those optimized settings work on previous historical data for the same instrument in a backtest. I know some of you guys will say 'read the fucking literature' and I hear you loud and clear, I will, but I'm keen to hear what people think about that simple method. Am I curve fitting the data for the 6month initial test period and as a result destined for it not to succeed on other historical data. Just want to make sure I'm not barking up the wrong tree. Peace.

Some people argue there's no regime shift between the 2 periods so it doesn't matter if you optimize on recent (e.g. 2018) and test on earlier data (e.g. 2017). In my experience though, working strategies tend to have a decaying convexity i.e. make less money over time so I prefer to go the other way, among other reasons.

Reply With Quote
The following 2 users say Thank You to artemiso for this post:
 
(login for full post details)
  #4 (permalink)
 kevinkdog   is a Vendor
 
 
Posts: 2,987 since Jul 2012
Thanks: 1,589 given, 5,933 received


artemiso View Post
Some people argue there's no regime shift between the 2 periods so it doesn't matter if you optimize on recent (e.g. 2018) and test on earlier data (e.g. 2017). In my experience though, working strategies tend to have a decaying convexity i.e. make less money over time so I prefer to go the other way, among other reasons.

I see that too. Why do you think that is? (I have my own suspicions, but am very curious what you think).

Follow me on Twitter Reply With Quote
The following user says Thank You to kevinkdog for this post:
 
(login for full post details)
  #5 (permalink)
 kanepa 
philadelphia pa
 
Experience: Intermediate
Platform: ninja
Broker: NinjaTrader Brokerage
Trading: es
 
Posts: 179 since Jul 2017
Thanks: 330 given, 364 received


WhatIfGod View Post
Currently I create the code, backtest on 6 months worth of data, then optimize that 6 month period and see whether those optimized settings work on previous historical data for the same instrument in a backtest. I know some of you guys will say 'read the fucking literature' and I hear you loud and clear, I will, but I'm keen to hear what people think about that simple method. Am I curve fitting the data for the 6month initial test period and as a result destined for it not to succeed on other historical data. Just want to make sure I'm not barking up the wrong tree. Peace.

Just my 2cents.

It depends on which 6 month has market condition been. If that 6 month has been volatile or bear market, and system made a good amount of profit, I would start to worry about when market condition becomes to norm.

Just my 2 cents.

Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #6 (permalink)
 artemiso 
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
 
Posts: 1,129 since Jul 2012
Thanks: 765 given, 2,540 received


kevinkdog View Post
I see that too. Why do you think that is? (I have my own suspicions, but am very curious what you think).

Philosophically, it's just a byproduct of no-free-lunch. If it makes money, chances are that more people will find it and get better at competing away your alpha over time conditioned on a fixed strategy.

But quantitatively, I've seen this manifest in many different ways. In some cases the intuition is pretty direct: the number of mispriced orders goes down so you explicitly see your volume decay over time. I actually see this so often that I have plots like this all over my desktop, where a strategy starts out maybe trading 80-150k contracts per day and drops down to 40k.


Reply With Quote
The following 3 users say Thank You to artemiso for this post:
 
(login for full post details)
  #7 (permalink)
 kevinkdog   is a Vendor
 
 
Posts: 2,987 since Jul 2012
Thanks: 1,589 given, 5,933 received


artemiso View Post
Philosophically, it's just a byproduct of no-free-lunch. If it makes money, chances are that more people will find it and get better at competing away your alpha over time conditioned on a fixed strategy.

But quantitatively, I've seen this manifest in many different ways. In some cases the intuition is pretty direct: the number of mispriced orders goes down so you explicitly see your volume decay over time. I actually see this so often that I have plots like this all over my desktop, where a strategy starts out maybe trading 80-150k contracts per day and drops down to 40k.


Thanks. I agree with the competition aspect. Another thing I see is that many people expect a backtest to continue at same pace. Since there is some "good luck" built into every backtest, if that random good luck becomes bad luck, the strategy will level off. Same end result occurs due to the biases we bake into every backtest (and try like heck to avoid adding in!). Profit due to those biases tends to go away with future unseen data...

Follow me on Twitter Reply With Quote
The following 3 users say Thank You to kevinkdog for this post:
 
(login for full post details)
  #8 (permalink)
 artemiso 
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
 
Posts: 1,129 since Jul 2012
Thanks: 765 given, 2,540 received


kevinkdog View Post
Since there is some "good luck" built into every backtest, if that random good luck becomes bad luck, the strategy will level off. Same end result occurs due to the biases we bake into every backtest (and try like heck to avoid adding in!). Profit due to those biases tends to go away with future unseen data...

Hm to avoid confusion, all of my above statements are only within the out of sample set on first try. So there's no additional overfitting baked into it over time. But certainly, depending on the information leakage, that can also cause error in generalizing into the future.

Reply With Quote
The following user says Thank You to artemiso for this post:
 
(login for full post details)
  #9 (permalink)
 WoodyFox 
Orlando, Florida
 
Experience: Intermediate
Platform: NinjaTrader
Trading: Futures
 
WoodyFox's Avatar
 
Posts: 300 since May 2016
Thanks: 117 given, 603 received


kevinkdog View Post
I see that too. Why do you think that is? (I have my own suspicions, but am very curious what you think).

To fix this I use non-correlated conditions with a linear operator. This will keep any one condition from bending out of usefulness and also usually keeps the strategy from decaying as fast over time. I think of it as wrapping the market around my strategy, not wrapping the strategy around the market. Also it takes a little more advanced math and most traders will not take their strategies that far which results in more time for decay.

Reply With Quote
The following 2 users say Thank You to WoodyFox for this post:
 
(login for full post details)
  #10 (permalink)
 kevinkdog   is a Vendor
 
 
Posts: 2,987 since Jul 2012
Thanks: 1,589 given, 5,933 received



WoodyFox View Post
To fix this I use non-correlated conditions with a linear operator. This will keep any one condition from bending out of usefulness and also usually keeps the strategy from decaying as fast over time. I think of it as wrapping the market around my strategy, not wrapping the strategy around the market. Also it takes a little more advanced math and most traders will not take their strategies that far which results in more time for decay.

I'll admit, your response has me baffled, definitely over my head.

I'm guessing though that your approach works for you, and that is the awesome part!

Follow me on Twitter Reply With Quote
 
(login for full post details)
  #11 (permalink)
 WoodyFox 
Orlando, Florida
 
Experience: Intermediate
Platform: NinjaTrader
Trading: Futures
 
WoodyFox's Avatar
 
Posts: 300 since May 2016
Thanks: 117 given, 603 received


kevinkdog View Post
I'll admit, your response has me baffled, definitely over my head.

I'm guessing though that your approach works for you, and that is the awesome part!


Maybe not the best at explaining LOL...but yes it works well, and really not all that difficult?

Reply With Quote
 
(login for full post details)
  #12 (permalink)
 kevinkdog   is a Vendor
 
 
Posts: 2,987 since Jul 2012
Thanks: 1,589 given, 5,933 received


WoodyFox View Post
Maybe not the best at explaining LOL...but yes it works well, and really not all that difficult?

"non-correlated conditions with a linear operator"
"bending out of usefulness"
"keeps the strategy from decaying as fast over time"
"wrapping the market around my strategy, not wrapping the strategy around the market"
"most traders will not take their strategies that far which results in more time for decay."

Any of these you care to expand on would be appreciated. But no worries if not, you understand the terms and that is what is important.

Maybe it is a water thing - your drinking water is probably from the Olentangy River, and mine is from Lake Erie! LOL.

Follow me on Twitter Reply With Quote
The following 2 users say Thank You to kevinkdog for this post:
 
(login for full post details)
  #13 (permalink)
 WoodyFox 
Orlando, Florida
 
Experience: Intermediate
Platform: NinjaTrader
Trading: Futures
 
WoodyFox's Avatar
 
Posts: 300 since May 2016
Thanks: 117 given, 603 received


kevinkdog View Post
"non-correlated conditions with a linear operator"
"bending out of usefulness"
"keeps the strategy from decaying as fast over time"
"wrapping the market around my strategy, not wrapping the strategy around the market"
"most traders will not take their strategies that far which results in more time for decay."

Any of these you care to expand on would be appreciated. But no worries if not, you understand the terms and that is what is important.

Maybe it is a water thing - your drinking water is probably from the Olentangy River, and mine is from Lake Erie! LOL.

Yes that is what's important, so Thank you.
Knowing you just dissected my 4 sentence paragraph into 5 sentences. LOL. I don't want my next answer to get logarithmic? So I will pass.

Reply With Quote
 
(login for full post details)
  #14 (permalink)
 kevinkdog   is a Vendor
 
 
Posts: 2,987 since Jul 2012
Thanks: 1,589 given, 5,933 received


WoodyFox View Post
Yes that is what's important, so Thank you.
Knowing you just dissected my 4 sentence paragraph into 5 sentences. LOL. I don't want my next answer to get logarithmic? So I will pass.


I am sorry I tried to understand your confusing reply in a thread that has been dead for 18 months. I will not make that mistake again.

Follow me on Twitter Reply With Quote
The following user says Thank You to kevinkdog for this post:
 
(login for full post details)
  #15 (permalink)
 WoodyFox 
Orlando, Florida
 
Experience: Intermediate
Platform: NinjaTrader
Trading: Futures
 
WoodyFox's Avatar
 
Posts: 300 since May 2016
Thanks: 117 given, 603 received


kevinkdog View Post
I am sorry I tried to understand your confusing reply in a thread that has been dead for 18 months. I will not make that mistake again.

No hard feelings. Yes old post but still interesting. I too know better now.
Thanks,

Reply With Quote
 
(login for full post details)
  #16 (permalink)
fxFlux
London United kingdom
 
 
Posts: 11 since May 2018
Thanks: 0 given, 1 received

Why would you optimise over 2018 and test over 2017? shouldn't it be optimise over 2018 test over 2019.

I have been a victim of over optimization in the past. What I do now is that I check that the strategy is profitable over many many sets of the input parameters to confirm that the strategy is not only profitable at such a finely tuned set of criteria but is generally profitable over all, next I select the set of input parameters which produces the minimum amount of drawdown and a healthy profit factor. I forward test over an equal duration to the optimisation period. I repeat this as a sliding 1 year or 2 year window to check it is continuously profitable.

Reply With Quote
The following user says Thank You to fxFlux for this post:


futures io Trading Community Traders Hideout > Over Optimization/Curve Fitting?


Last Updated on July 16, 2020


Upcoming Webinars and Events
 

NinjaTrader Indicator Challenge!

Ongoing
 

Journal Challenge w/$1,800 in prizes!

April

Seven Trading Mistakes Solved With Smart Trading Tools w/Brannigan Barrett

Elite only
     



Copyright © 2021 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts