Over Optimization/Curve Fitting? - futures io
futures io futures trading



Over Optimization/Curve Fitting?


Discussion in Traders Hideout

Updated by WoodyFox
      Top Posters
    1. looks_one kevinkdog with 5 posts (6 thanks)
    2. looks_two WoodyFox with 4 posts (2 thanks)
    3. looks_3 artemiso with 3 posts (6 thanks)
    4. looks_4 Quick Summary with 1 posts (0 thanks)
    1. trending_up 533 views
    2. thumb_up 14 thanks given
    3. group 7 followers
    1. forum 14 replies
    2. attach_file 1 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 100,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

Over Optimization/Curve Fitting?

(login for full post details)
  #1 (permalink)
Birmingham, England
 
 
Posts: 21 since Nov 2018
Thanks: 18 given, 2 received

Currently I create the code, backtest on 6 months worth of data, then optimize that 6 month period and see whether those optimized settings work on previous historical data for the same instrument in a backtest. I know some of you guys will say 'read the fucking literature' and I hear you loud and clear, I will, but I'm keen to hear what people think about that simple method. Am I curve fitting the data for the 6month initial test period and as a result destined for it not to succeed on other historical data. Just want to make sure I'm not barking up the wrong tree. Peace.

Reply With Quote

Can you help answer these questions
from other members on futures io?
Two Data Series in Ninjatrader 7 for Order confirmation
NinjaTrader
Execute NT8-Optimizer by script
NinjaTrader
Short OTM options with a stop order?
Options
Volume Chart Trend Lines
TradeStation
Options Backtesting Software with Market Replay tool/feature?
Options
 
Best Threads (Most Thanked)
in the last 7 days on futures io
Are sharks watching on the other side?
54 thanks
VWAP for stock index futures trading?
35 thanks
Sierra Chart ACSIL for Beginners.
16 thanks
Converting NT7 indicators to NT8 for free
14 thanks
Spoo-nalysis ES e-mini futures S&P 500
14 thanks
 
(login for full post details)
  #3 (permalink)
Market Wizard
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
 
Posts: 1,060 since Jul 2012
Thanks: 733 given, 2,211 received



WhatIfGod View Post
Currently I create the code, backtest on 6 months worth of data, then optimize that 6 month period and see whether those optimized settings work on previous historical data for the same instrument in a backtest. I know some of you guys will say 'read the fucking literature' and I hear you loud and clear, I will, but I'm keen to hear what people think about that simple method. Am I curve fitting the data for the 6month initial test period and as a result destined for it not to succeed on other historical data. Just want to make sure I'm not barking up the wrong tree. Peace.

Some people argue there's no regime shift between the 2 periods so it doesn't matter if you optimize on recent (e.g. 2018) and test on earlier data (e.g. 2017). In my experience though, working strategies tend to have a decaying convexity i.e. make less money over time so I prefer to go the other way, among other reasons.

Reply With Quote
The following 2 users say Thank You to artemiso for this post:
 
(login for full post details)
  #4 (permalink)
Market Wizard
Cleveland Ohio/United States
 
Experience: Advanced
Platform: Tradestation
Broker: Tradestation, DeCarley, others
Trading: futures
 
Posts: 2,773 since Jul 2012
Thanks: 1,424 given, 5,336 received


artemiso View Post
Some people argue there's no regime shift between the 2 periods so it doesn't matter if you optimize on recent (e.g. 2018) and test on earlier data (e.g. 2017). In my experience though, working strategies tend to have a decaying convexity i.e. make less money over time so I prefer to go the other way, among other reasons.

I see that too. Why do you think that is? (I have my own suspicions, but am very curious what you think).

Follow me on Twitter Reply With Quote
The following user says Thank You to kevinkdog for this post:
 
(login for full post details)
  #5 (permalink)
philadelphia pa
 
Experience: Intermediate
Platform: ninja
Broker: NinjaTrader Brokerage
Trading: es
 
Posts: 137 since Jul 2017
Thanks: 298 given, 279 received


WhatIfGod View Post
Currently I create the code, backtest on 6 months worth of data, then optimize that 6 month period and see whether those optimized settings work on previous historical data for the same instrument in a backtest. I know some of you guys will say 'read the fucking literature' and I hear you loud and clear, I will, but I'm keen to hear what people think about that simple method. Am I curve fitting the data for the 6month initial test period and as a result destined for it not to succeed on other historical data. Just want to make sure I'm not barking up the wrong tree. Peace.

Just my 2cents.

It depends on which 6 month has market condition been. If that 6 month has been volatile or bear market, and system made a good amount of profit, I would start to worry about when market condition becomes to norm.

Just my 2 cents.

Reply With Quote
 
(login for full post details)
  #6 (permalink)
Market Wizard
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
 
Posts: 1,060 since Jul 2012
Thanks: 733 given, 2,211 received


kevinkdog View Post
I see that too. Why do you think that is? (I have my own suspicions, but am very curious what you think).

Philosophically, it's just a byproduct of no-free-lunch. If it makes money, chances are that more people will find it and get better at competing away your alpha over time conditioned on a fixed strategy.

But quantitatively, I've seen this manifest in many different ways. In some cases the intuition is pretty direct: the number of mispriced orders goes down so you explicitly see your volume decay over time. I actually see this so often that I have plots like this all over my desktop, where a strategy starts out maybe trading 80-150k contracts per day and drops down to 40k.


Reply With Quote
The following 3 users say Thank You to artemiso for this post:
 
(login for full post details)
  #7 (permalink)
Market Wizard
Cleveland Ohio/United States
 
Experience: Advanced
Platform: Tradestation
Broker: Tradestation, DeCarley, others
Trading: futures
 
Posts: 2,773 since Jul 2012
Thanks: 1,424 given, 5,336 received


artemiso View Post
Philosophically, it's just a byproduct of no-free-lunch. If it makes money, chances are that more people will find it and get better at competing away your alpha over time conditioned on a fixed strategy.

But quantitatively, I've seen this manifest in many different ways. In some cases the intuition is pretty direct: the number of mispriced orders goes down so you explicitly see your volume decay over time. I actually see this so often that I have plots like this all over my desktop, where a strategy starts out maybe trading 80-150k contracts per day and drops down to 40k.


Thanks. I agree with the competition aspect. Another thing I see is that many people expect a backtest to continue at same pace. Since there is some "good luck" built into every backtest, if that random good luck becomes bad luck, the strategy will level off. Same end result occurs due to the biases we bake into every backtest (and try like heck to avoid adding in!). Profit due to those biases tends to go away with future unseen data...

Follow me on Twitter Reply With Quote
The following 3 users say Thank You to kevinkdog for this post:
 
(login for full post details)
  #8 (permalink)
Market Wizard
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
 
Posts: 1,060 since Jul 2012
Thanks: 733 given, 2,211 received


kevinkdog View Post
Since there is some "good luck" built into every backtest, if that random good luck becomes bad luck, the strategy will level off. Same end result occurs due to the biases we bake into every backtest (and try like heck to avoid adding in!). Profit due to those biases tends to go away with future unseen data...

Hm to avoid confusion, all of my above statements are only within the out of sample set on first try. So there's no additional overfitting baked into it over time. But certainly, depending on the information leakage, that can also cause error in generalizing into the future.

Reply With Quote
The following user says Thank You to artemiso for this post:
 
(login for full post details)
  #9 (permalink)
Columbus, Ohio
 
Experience: Intermediate
Platform: NinjaTrader
Broker: Continuum
Trading: Futures
 
WoodyFox's Avatar
 
Posts: 198 since May 2016
Thanks: 67 given, 337 received


kevinkdog View Post
I see that too. Why do you think that is? (I have my own suspicions, but am very curious what you think).

To fix this I use non-correlated conditions with a linear operator. This will keep any one condition from bending out of usefulness and also usually keeps the strategy from decaying as fast over time. I think of it as wrapping the market around my strategy, not wrapping the strategy around the market. Also it takes a little more advanced math and most traders will not take their strategies that far which results in more time for decay.

Reply With Quote
The following 2 users say Thank You to WoodyFox for this post:
 
(login for full post details)
  #10 (permalink)
Market Wizard
Cleveland Ohio/United States
 
Experience: Advanced
Platform: Tradestation
Broker: Tradestation, DeCarley, others
Trading: futures
 
Posts: 2,773 since Jul 2012
Thanks: 1,424 given, 5,336 received



WoodyFox View Post
To fix this I use non-correlated conditions with a linear operator. This will keep any one condition from bending out of usefulness and also usually keeps the strategy from decaying as fast over time. I think of it as wrapping the market around my strategy, not wrapping the strategy around the market. Also it takes a little more advanced math and most traders will not take their strategies that far which results in more time for decay.

I'll admit, your response has me baffled, definitely over my head.

I'm guessing though that your approach works for you, and that is the awesome part!

Follow me on Twitter Reply With Quote


futures io Trading Community Traders Hideout > Over Optimization/Curve Fitting?


June 28, 2020


Upcoming Webinars and Events
 

Free BloodHound Licenses for everyone!

June
 

Every journal equals ten meals for the hungry

June
     



Copyright © 2020 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts