optimization/overfitting and mess with degrees of freedom

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors – all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you don’t need to worry about fake reviews.

We are fundamentally different than most other trading sites:

We are here to help. Just let us know what you need.

We work extremely hard to keep things positive in our community.

We do not tolerate rude behavior, trolling, or vendors advertising in posts.

We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.

We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community. It's free and simple.

optimization/overfitting and mess with degrees of freedom

The purpose of this topic is to establish some facts about degrees of freedom and its impact on overfitting.

The 1st definition which i found is in Pardo book 'The Evaluation and Optimization of Trading Strategies'

Quite different definitions are in MSA and Adaptrade Builder help, another tools which are dealing with optimization and overfitting

So the main questions is rising: what is a correct approach ????

In case of Adaptrade Builder/MSA as the approach is based on number of trades and rules immediate questions are coming to my mind:

1) what is the exact proportion which should guarantee no overfitting
2) its easy to manipulate with number of trades to inflate the results e.g for crossover strategy when the trades open to generate the
trades on every bar till opposite crossover takes place
3) definition also uses number of inputs. Is input with e.g. 10 values (range 1-10, step 1) equal with input with 100000 values ??
4) what is the origin of this type of approach ??

Hopefully somebody can answer this. Similar thread i'm opening on the adaptrade builder google group hoping to get some answers from Adaptrade/MSA author (M. Bryant)

Regards, Krzysztof

expert from Pardo book

Consider degrees of freedom as the simulation sample size adjusted
for the number of conditions and rules placed upon it. The simulation test
space is reduced in proportion to the number of degrees of freedom that
are consumed by the rules and variables of the trading strategy.

Consider the following simple example as an illustration. Assume a
trading strategy uses a single moving average with a length of 30 days and
a test window size of 100 days. It is said that one degree of freedom is consumed
by each data point that is used in its calculation. Our 30-day moving
average then uses 30 degrees of freedom. To understand the relevance of
this in a more practical way, we determine the remaining degrees of freedom
according to the following formula.
DF = Degrees of Freedom
Udf = Used Degrees of Freedom
Odf = Original Degrees of Freedom
Rdf% = Remaining Percentage of Degrees of Freedom
Rdf = 100 × [1 − (Udf/Odf)]
Rdf = 100 × [1 − (30/100)]
Rdf = 70%
The degrees of freedom left in our example are 70 percent, which
is not that attractive. As a rule of thumb, we would prefer that the
remaining degrees of freedom exceed 90 percent. Therefore, there is no
point in performing this simulation as stipulated.
On the other hand, consider a trading system that again uses a single
moving average with a period of 10 days and a test window size of
1,000 days.
Rdf % = 100 × [1 - (10/1,000)]
Rdf = 99%
This simulation can proceed with 99 percent degrees of freedom. This
will produce a more statistically reliable historical simulation.
Consider two other applications of this principle. In the first, a trading
strategy has 100 rules and a simulation with one hundred 100 days of data
is considered. Applying our formula, we see that this leaves us with no
degrees of freedom. It is easy to see that this test is absurd. In contrast,
consider a simulation of a trading strategy with 1 rule and 100 days of data.
Even though it is a small data sample, the 99 percent degrees of freedom is
acceptable.

THE CAUSES OF OVERFITTING
Overfitting is a direct result of the violation of some or all of the rules
of evaluation and optimization. These violations generally fall into five
categories:
1. Insufficient degrees of freedom
2. Inadequate data and trade sample
3. Incorrect optimization methods
4. A big win in a small trade sample
5. Absence of a Walk-Forward Analysis
Degrees of Freedom
It is a cardinal rule of statistical analysis that too many constraints—or
too few degrees of freedom—on a data sample will lead to untrustworthy
results. In other words, if the calculation of the formulae of a trading strategy
consumes too large a proportion of the data sample, the results of the
optimization will lack sufficient statistical validity and hence become unreliable.
Degrees of freedom and sample size are inextricably intertwined.
Insufficient degrees of freedom are still a major cause of overfitting.
To a large extent, degrees of freedom are simply a way to determine
whether there are enough data remaining to produce a valid trade sample
after all deductions have been made for the price data that are used to
calculate the trading rules, indicators, and so forth.
Measuring Degrees of Freedom
It is simple to measure degrees of freedom. To begin, it can be thought that
each data point in the sample represents “one degree of freedom.” If the
sample size is one thousand data points, then it begins with one thousand
degrees of freedom, that is, all of the data are unconstrained.
A degree of freedom then is said to be consumed or used by each trading
rule and by every data point necessary to calculate indicators.
To illustrate, consider two examples. Both use the same data sample,
which is a four data-point, two year, price history composed of opens,
highs, lows, and closes, or a total of 2,080 data points.
Example one is a trading strategy that uses a 10-day average of highs
and a 50-day average of lows. Average one uses 11 degrees of freedom:
10 highs plus 1 more as a rule. Average two uses 51 degrees of freedom:
50 lows plus 1 as a rule. The total is 62 degrees of freedom used. To convert
that to a percentage, divide degrees of freedom used by total available
degrees of freedom. The result is 3 percent. This is perfectly acceptable.
Example two is a trading strategy that uses a 50-day average of closes
and a 150-day average of closes. Average one uses 51 degrees of freedom:
50 closes plus 1 as a rule. Average two uses only 102 degrees of freedom:
100 additional closes plus 1 as a rule. The total degrees of freedom used
are 152. Converting to a percentage, we get 7.3 percent.
While this is still acceptable, from these examples, it is easy to see
how adding more indicators and rules or decreasing sample size can easily
lead to decreased confidence in the results. This will be made clear in the
examples in the next section.

Than from help from the Adaptrade Builder

Poor out-of-sample performance is usually caused by one of several factors.
One important factor is the so-called number of degrees-of-freedom in the in-sample segment.
The number of degrees-of-freedom, which is equal to the number of trades minus the number of rules and conditions
of the strategy, determines how tightly the strategy fits the data. Provided inputs are added for each parameter
in the strategy, the number of strategy inputs can be used as a proxy for the number of rules and conditions.
For example, if a strategy has 100 trades and 10 inputs, it has 90 degrees-of-freedom.
The more degrees-of-freedom, the less likely it is that the strategy will be over-fit to the market and the more likely
it is that it will have good out-of-sample performance.

The number of degrees-of-freedom can be increased during the build process by adjusting the weights for the number
of trades and/or the strategy complexity. All other things being equal, increasing the performance weighting for the number of trades will result in strategies with more trades and therefore more degrees-of-freedom.
Likewise, increasing the performance weighting for the complexity metric will result in strategies with fewer inputs, which will also increase the number of degrees-of-freedom.

Builder also incorporates the degrees-of-freedom into the build process via the significance performance metric.
In Builder, “significance” is based on the Student’s t test applied to the average trade.
It measures the statistical significance of the average trade; that is, the probability that the average trade will be greater than zero. The t test is based on the number of degrees-of-freedom but is a more complete measure of whether a strategy is over-fit than the number of degrees-of-freedom alone. One way, then, to improve out-of-sample performance is to use the significance metric to generate strategies that have a high statistical significance.

The following user says Thank You to krzysiaczek99 for this post:

Quick Summary is created and edited by users like you... Add FAQ's, Links and other Relevant Information by clicking the edit button in the lower right hand corner of this message.

Strategy is optimized on 75000 1 min bars, OOS period 1440 bars. Anchored WF test clearly shows that strategy is overfitted, all In sample net profit are positive, but OOS profits mostly negative.

than is left free like 99% of DFs. So it looks it don't work.

See attached screenschots.

I tried to clone this thread on Adaptrade google group but it didn't shown up on the list, most likely R. Bryant
(Adaptrade Builder/MSA seller) who is an administrator of this group just blocked it. LOVE IT !!!!!!!!!!!!!

My another post that one of the most important features of this tool - usage of custom indicator, is useless, was blocked also.....

Any comments ???

Krzysztof

The following user says Thank You to krzysiaczek99 for this post:

1. Varför addera de värde indicatorerna använder. Borde väl räcka med att ta hänsyn till indikatorn med störst värde?

2. Om man då bara tradar chartpatterns. Borde hänsyn tas storleken på dessa dvs antalet perioder de sträcker sig över. Dessa kan ju oxå variera precis som med antal perioder för de som använder indikatorer. Vad gäller då här??