Why using the term "curve-fitting" is wrong - Traders Hideout | futures.io
futures.io futures trading

Go Back   futures.io

> Futures Trading, News, Charts and Platforms > Traders Hideout

Why using the term "curve-fitting" is wrong
Started:January 21st, 2013 (04:47 PM) by Outlier Views / Replies:687 / 1
Last Reply:January 21st, 2013 (04:47 PM) Attachments:0

Welcome to futures.io.

Welcome, Guest!

This forum was established to help traders (especially futures traders) by openly sharing indicators, strategies, methods, trading journals and discussing the psychology of trading.

We are fundamentally different than most other trading forums:
  • We work extremely hard to keep things positive on our forums.
  • We do not tolerate rude behavior, trolling, or vendor advertising in posts.
  • We firmly believe in openness and encourage sharing. The holy grail is within you, it is not something tangible you can download.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community. It's free and simple, and we will never resell your private information.

-- Big Mike

Thread Tools Search this Thread

Why using the term "curve-fitting" is wrong

Old January 21st, 2013, 04:47 PM   #1 (permalink)
Elite Member
Futures Experience: Advanced
Platform: TradeStation
Favorite Futures: Futures
Posts: 88 since May 2012
Thanks: 53 given, 90 received

Why using the term "curve-fitting" is wrong

After having seen the term being misused and abused hundreds of times, and having been guilty of it myself just until a few years ago, I had to write this.

Curve-fitting is a term from non-linear regression analysis and means constructing a curve, such as a high order polynomial, that best fits a series of data points. It is commonly used as an aid for visualization. Think of a curve that cuts right through your data. Curve-fitting alone is neither good nor bad in the sense that it makes no claims at all about any extrapolation or generalization performance.

Over-fitting, the correct term from statistics and machine learning, means that a model generalizes poorly. On the training set (in-sample), the model has good performance. But on the validation set and test set (out-of-sample), the model has bad performance. Over-fitting, also called high variance, occurs when a model has too many degrees of freedom, or capacity, that during training is fitted to random noise (or sampling error), rather than the underlying structure.

The opposite is under-fitting, also called high bias. An under-fitted model has very similar performance on the training and validation set. But each time, the performance is poor because the capacity of the model is too small to capture enough of the underlying structure.

Desirable is a good fit, a solution to the bias-variance tradeoff.

Over-fitting is countered by the following:
- increasing the number of trades, by raising their frequency and/or using more data, which naturally reduces the sampling error (best approach)
- reduction of the model capacity
- advanced techniques like regularization, early stopping, pruning

Under-fitting is countered by increasing the model capacity.

More at https://en.wikipedia.org/wiki/Curve_fitting

Successful systematic traders know about this.

"I can talk a little more about over-fitting, if not my personal proprietary techniques. First of all I like the [term] over-fitting rather than curve-fitting because curve-fitting is a term from non-linear regression analysis. It is where you have a lot of data and you are fitting the data points to some curve. Well, you are not doing that with futures. Technically there is no curve-fitting here; the term does not apply. But what you can do is you can over-fit. The reason I like the term over-fit rather than curve-fit is that over-fit shows that you also can under-fit. The people who do not optimize are under-fitting." -- William Eckhardt

William Eckhardt: The man who launched 1,000 systems

Related to these issues is data-mining bias. When a large number of systems is evaluated during training or even validation, the best systems may meet your criteria just by chance. The more systems are tested, the higher the random variation in results.

It can be countered by:
- evaluating a selected group of systems on a 2nd validation set, taking into account the whole distribution of performance
- evaluating the final choice of system(s) on the test set that is only used once, to get an unbiased estimate of performance

None of this, however, can protect against regime changes. The markets could change enough to invalidate any statistical and structural edge. The whole training-validation-test approach of data splitting works under the assumption that all segments are drawn from the same distribution. Unfortunately, that distribution may change substantially in the future. For example, the advent of HFT was a serious regime change for discretionary stock scalpers. Fortunately, regime changes happen gradually over time, giving the trader time to adapt. It may also help to be slightly on the high-bias side.

Reply With Quote
The following 7 users say Thank You to Outlier for this post:

Old January 21st, 2013, 04:47 PM   #2 (permalink)
Quick Summary
Quick Summary Post

Quick Summary is created and edited by users like you... Add FAQ's, Links and other Relevant Information by clicking the edit button in the lower right hand corner of this message.



futures.io > Futures Trading, News, Charts and Platforms > Traders Hideout > Why using the term "curve-fitting" is wrong

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Upcoming Webinars and Events (4:30PM ET unless noted)

An Afternoon with FIO trader bobwest

Elite only

NinjaTrader 8: Programming Profitable Trading Edges w/Scott Hodson

Elite only

Anthony Drager: Executing on Intermarket Correlations & Order Flow, Part 2

Elite only

Adam Grimes: Five critically important keys to professional trading

Elite only

Machine Learning Concepts w/FIO member NJAMC

Elite only

MarketDelta Cloud Platform: Announcing new mobile features

Dec 1

NinjaTrader 8: Features and Enhancements

Dec 6

Similar Threads
Thread Thread Starter Forum Replies Last Post
Fukushima: "China Syndrome Is Inevitable" ... "Huge Steam Explosions", or "Nuclear Bo Quick Summary News and Current Events 0 November 22nd, 2011 01:50 AM
How to change "Stop Loss" and "Take Profit" in "shElderImpulse" Strategy javed759 NinjaTrader 1 November 1st, 2011 10:44 PM
"mid", "buy", "sell" volumes lokgotkent Traders Hideout 6 September 30th, 2011 01:24 PM
Warren Buffet is "wrong" on taxes RM99 News and Current Events 4 July 29th, 2011 10:10 AM
very short-term "Swing" trading heywally The Elite Circle 2 May 13th, 2011 01:30 PM

All times are GMT -4. The time now is 04:31 PM.

Copyright © 2016 by futures.io. All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts

Page generated 2016-10-21 in 0.08 seconds with 19 queries on phoenix via your IP