NexusFi: Find Your Edge


Home Menu

 





Over Optimization/Curve Fitting?


Discussion in Traders Hideout

Updated
      Top Posters
    1. looks_one kevinkdog with 5 posts (7 thanks)
    2. looks_two WoodyFox with 4 posts (2 thanks)
    3. looks_3 artemiso with 3 posts (6 thanks)
    4. looks_4 Quick Summary with 1 posts (0 thanks)
      Best Posters
    1. looks_one artemiso with 2 thanks per post
    2. looks_two kevinkdog with 1.4 thanks per post
    3. looks_3 fxFlux with 1 thanks per post
    4. looks_4 WoodyFox with 0.5 thanks per post
    1. trending_up 2,969 views
    2. thumb_up 16 thanks given
    3. group 6 followers
    1. forum 15 posts
    2. attach_file 1 attachments




 
Search this Thread

Over Optimization/Curve Fitting?

  #1 (permalink)
WhatIfGod
Birmingham, England
 
Posts: 21 since Nov 2018
Thanks Given: 18
Thanks Received: 2

Currently I create the code, backtest on 6 months worth of data, then optimize that 6 month period and see whether those optimized settings work on previous historical data for the same instrument in a backtest. I know some of you guys will say 'read the fucking literature' and I hear you loud and clear, I will, but I'm keen to hear what people think about that simple method. Am I curve fitting the data for the 6month initial test period and as a result destined for it not to succeed on other historical data. Just want to make sure I'm not barking up the wrong tree. Peace.

Reply With Quote

Can you help answer these questions
from other members on NexusFi?
NexusFi Journal Challenge - April 2024
Feedback and Announcements
Better Renko Gaps
The Elite Circle
Are there any eval firms that allow you to sink to your …
Traders Hideout
Futures True Range Report
The Elite Circle
Deepmoney LLM
Elite Quantitative GenAI/LLM
 
Best Threads (Most Thanked)
in the last 7 days on NexusFi
Get funded firms 2023/2024 - Any recommendations or word …
61 thanks
Funded Trader platforms
39 thanks
NexusFi site changelog and issues/problem reporting
26 thanks
The Program
18 thanks
GFIs1 1 DAX trade per day journal
18 thanks
  #3 (permalink)
 artemiso 
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
Posts: 1,152 since Jul 2012
Thanks Given: 784
Thanks Received: 2,685



WhatIfGod View Post
Currently I create the code, backtest on 6 months worth of data, then optimize that 6 month period and see whether those optimized settings work on previous historical data for the same instrument in a backtest. I know some of you guys will say 'read the fucking literature' and I hear you loud and clear, I will, but I'm keen to hear what people think about that simple method. Am I curve fitting the data for the 6month initial test period and as a result destined for it not to succeed on other historical data. Just want to make sure I'm not barking up the wrong tree. Peace.

Some people argue there's no regime shift between the 2 periods so it doesn't matter if you optimize on recent (e.g. 2018) and test on earlier data (e.g. 2017). In my experience though, working strategies tend to have a decaying convexity i.e. make less money over time so I prefer to go the other way, among other reasons.

Reply With Quote
Thanked by:
  #4 (permalink)
 kevinkdog   is a Vendor
 
Posts: 3,645 since Jul 2012
Thanks Given: 1,890
Thanks Received: 7,338


artemiso View Post
Some people argue there's no regime shift between the 2 periods so it doesn't matter if you optimize on recent (e.g. 2018) and test on earlier data (e.g. 2017). In my experience though, working strategies tend to have a decaying convexity i.e. make less money over time so I prefer to go the other way, among other reasons.

I see that too. Why do you think that is? (I have my own suspicions, but am very curious what you think).

Follow me on Twitter Reply With Quote
Thanked by:
  #5 (permalink)
 kanepa 
philadelphia pa
 
Experience: Intermediate
Platform: ninja
Broker: NinjaTrader Brokerage
Trading: es
Posts: 202 since Jul 2017
Thanks Given: 348
Thanks Received: 409


WhatIfGod View Post
Currently I create the code, backtest on 6 months worth of data, then optimize that 6 month period and see whether those optimized settings work on previous historical data for the same instrument in a backtest. I know some of you guys will say 'read the fucking literature' and I hear you loud and clear, I will, but I'm keen to hear what people think about that simple method. Am I curve fitting the data for the 6month initial test period and as a result destined for it not to succeed on other historical data. Just want to make sure I'm not barking up the wrong tree. Peace.

Just my 2cents.

It depends on which 6 month has market condition been. If that 6 month has been volatile or bear market, and system made a good amount of profit, I would start to worry about when market condition becomes to norm.

Just my 2 cents.

Visit my NexusFi Trade Journal Reply With Quote
  #6 (permalink)
 artemiso 
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
Posts: 1,152 since Jul 2012
Thanks Given: 784
Thanks Received: 2,685


kevinkdog View Post
I see that too. Why do you think that is? (I have my own suspicions, but am very curious what you think).

Philosophically, it's just a byproduct of no-free-lunch. If it makes money, chances are that more people will find it and get better at competing away your alpha over time conditioned on a fixed strategy.

But quantitatively, I've seen this manifest in many different ways. In some cases the intuition is pretty direct: the number of mispriced orders goes down so you explicitly see your volume decay over time. I actually see this so often that I have plots like this all over my desktop, where a strategy starts out maybe trading 80-150k contracts per day and drops down to 40k.


Reply With Quote
Thanked by:
  #7 (permalink)
 kevinkdog   is a Vendor
 
Posts: 3,645 since Jul 2012
Thanks Given: 1,890
Thanks Received: 7,338


artemiso View Post
Philosophically, it's just a byproduct of no-free-lunch. If it makes money, chances are that more people will find it and get better at competing away your alpha over time conditioned on a fixed strategy.

But quantitatively, I've seen this manifest in many different ways. In some cases the intuition is pretty direct: the number of mispriced orders goes down so you explicitly see your volume decay over time. I actually see this so often that I have plots like this all over my desktop, where a strategy starts out maybe trading 80-150k contracts per day and drops down to 40k.


Thanks. I agree with the competition aspect. Another thing I see is that many people expect a backtest to continue at same pace. Since there is some "good luck" built into every backtest, if that random good luck becomes bad luck, the strategy will level off. Same end result occurs due to the biases we bake into every backtest (and try like heck to avoid adding in!). Profit due to those biases tends to go away with future unseen data...

Follow me on Twitter Reply With Quote
Thanked by:
  #8 (permalink)
 artemiso 
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
Posts: 1,152 since Jul 2012
Thanks Given: 784
Thanks Received: 2,685


kevinkdog View Post
Since there is some "good luck" built into every backtest, if that random good luck becomes bad luck, the strategy will level off. Same end result occurs due to the biases we bake into every backtest (and try like heck to avoid adding in!). Profit due to those biases tends to go away with future unseen data...

Hm to avoid confusion, all of my above statements are only within the out of sample set on first try. So there's no additional overfitting baked into it over time. But certainly, depending on the information leakage, that can also cause error in generalizing into the future.

Reply With Quote
Thanked by:
  #9 (permalink)
 
WoodyFox's Avatar
 WoodyFox 
Columbus, Ohio
 
Experience: Intermediate
Platform: NinjaTrader
Trading: Futures
Posts: 409 since May 2016
Thanks Given: 196
Thanks Received: 876


kevinkdog View Post
I see that too. Why do you think that is? (I have my own suspicions, but am very curious what you think).

To fix this I use non-correlated conditions with a linear operator. This will keep any one condition from bending out of usefulness and also usually keeps the strategy from decaying as fast over time. I think of it as wrapping the market around my strategy, not wrapping the strategy around the market. Also it takes a little more advanced math and most traders will not take their strategies that far which results in more time for decay.

Reply With Quote
Thanked by:
  #10 (permalink)
 kevinkdog   is a Vendor
 
Posts: 3,645 since Jul 2012
Thanks Given: 1,890
Thanks Received: 7,338



WoodyFox View Post
To fix this I use non-correlated conditions with a linear operator. This will keep any one condition from bending out of usefulness and also usually keeps the strategy from decaying as fast over time. I think of it as wrapping the market around my strategy, not wrapping the strategy around the market. Also it takes a little more advanced math and most traders will not take their strategies that far which results in more time for decay.

I'll admit, your response has me baffled, definitely over my head.

I'm guessing though that your approach works for you, and that is the awesome part!

Follow me on Twitter Reply With Quote




Last Updated on July 16, 2020


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts