optimization/overfitting and mess with degrees of freedom 
December 29th, 2012, 12:44 PM  #1 (permalink) 
Trading Apprentice
stockholm sweden
Futures Experience: Master
Platform: metatrader
Favorite Futures: forex
Posts: 6 since Sep 2012
Thanks: 1 given,
2
received

optimization/overfitting and mess with degrees of freedom
The purpose of this topic is to establish some facts about degrees of freedom and its impact on overfitting.
The 1st definition which i found is in Pardo book 'The Evaluation and Optimization of Trading Strategies' Quite different definitions are in MSA and Adaptrade Builder help, another tools which are dealing with optimization and overfitting So the main questions is rising: what is a correct approach ???? In case of Adaptrade Builder/MSA as the approach is based on number of trades and rules immediate questions are coming to my mind: 1) what is the exact proportion which should guarantee no overfitting 2) its easy to manipulate with number of trades to inflate the results e.g for crossover strategy when the trades open to generate the trades on every bar till opposite crossover takes place 3) definition also uses number of inputs. Is input with e.g. 10 values (range 110, step 1) equal with input with 100000 values ?? 4) what is the origin of this type of approach ?? Hopefully somebody can answer this. Similar thread i'm opening on the adaptrade builder google group hoping to get some answers from Adaptrade/MSA author (M. Bryant) Regards, Krzysztof expert from Pardo book Consider degrees of freedom as the simulation sample size adjusted for the number of conditions and rules placed upon it. The simulation test space is reduced in proportion to the number of degrees of freedom that are consumed by the rules and variables of the trading strategy. Consider the following simple example as an illustration. Assume a trading strategy uses a single moving average with a length of 30 days and a test window size of 100 days. It is said that one degree of freedom is consumed by each data point that is used in its calculation. Our 30day moving average then uses 30 degrees of freedom. To understand the relevance of this in a more practical way, we determine the remaining degrees of freedom according to the following formula. DF = Degrees of Freedom Udf = Used Degrees of Freedom Odf = Original Degrees of Freedom Rdf% = Remaining Percentage of Degrees of Freedom Rdf = 100 × [1 − (Udf/Odf)] Rdf = 100 × [1 − (30/100)] Rdf = 70% The degrees of freedom left in our example are 70 percent, which is not that attractive. As a rule of thumb, we would prefer that the remaining degrees of freedom exceed 90 percent. Therefore, there is no point in performing this simulation as stipulated. On the other hand, consider a trading system that again uses a single moving average with a period of 10 days and a test window size of 1,000 days. Rdf % = 100 × [1  (10/1,000)] Rdf = 99% This simulation can proceed with 99 percent degrees of freedom. This will produce a more statistically reliable historical simulation. Consider two other applications of this principle. In the first, a trading strategy has 100 rules and a simulation with one hundred 100 days of data is considered. Applying our formula, we see that this leaves us with no degrees of freedom. It is easy to see that this test is absurd. In contrast, consider a simulation of a trading strategy with 1 rule and 100 days of data. Even though it is a small data sample, the 99 percent degrees of freedom is acceptable. THE CAUSES OF OVERFITTING Overfitting is a direct result of the violation of some or all of the rules of evaluation and optimization. These violations generally fall into five categories: 1. Insufficient degrees of freedom 2. Inadequate data and trade sample 3. Incorrect optimization methods 4. A big win in a small trade sample 5. Absence of a WalkForward Analysis Degrees of Freedom It is a cardinal rule of statistical analysis that too many constraints—or too few degrees of freedom—on a data sample will lead to untrustworthy results. In other words, if the calculation of the formulae of a trading strategy consumes too large a proportion of the data sample, the results of the optimization will lack sufficient statistical validity and hence become unreliable. Degrees of freedom and sample size are inextricably intertwined. Insufficient degrees of freedom are still a major cause of overfitting. To a large extent, degrees of freedom are simply a way to determine whether there are enough data remaining to produce a valid trade sample after all deductions have been made for the price data that are used to calculate the trading rules, indicators, and so forth. Measuring Degrees of Freedom It is simple to measure degrees of freedom. To begin, it can be thought that each data point in the sample represents “one degree of freedom.” If the sample size is one thousand data points, then it begins with one thousand degrees of freedom, that is, all of the data are unconstrained. A degree of freedom then is said to be consumed or used by each trading rule and by every data point necessary to calculate indicators. To illustrate, consider two examples. Both use the same data sample, which is a four datapoint, two year, price history composed of opens, highs, lows, and closes, or a total of 2,080 data points. Example one is a trading strategy that uses a 10day average of highs and a 50day average of lows. Average one uses 11 degrees of freedom: 10 highs plus 1 more as a rule. Average two uses 51 degrees of freedom: 50 lows plus 1 as a rule. The total is 62 degrees of freedom used. To convert that to a percentage, divide degrees of freedom used by total available degrees of freedom. The result is 3 percent. This is perfectly acceptable. Example two is a trading strategy that uses a 50day average of closes and a 150day average of closes. Average one uses 51 degrees of freedom: 50 closes plus 1 as a rule. Average two uses only 102 degrees of freedom: 100 additional closes plus 1 as a rule. The total degrees of freedom used are 152. Converting to a percentage, we get 7.3 percent. While this is still acceptable, from these examples, it is easy to see how adding more indicators and rules or decreasing sample size can easily lead to decreased confidence in the results. This will be made clear in the examples in the next section. Than from help from the Adaptrade Builder Poor outofsample performance is usually caused by one of several factors. One important factor is the socalled number of degreesoffreedom in the insample segment. The number of degreesoffreedom, which is equal to the number of trades minus the number of rules and conditions of the strategy, determines how tightly the strategy fits the data. Provided inputs are added for each parameter in the strategy, the number of strategy inputs can be used as a proxy for the number of rules and conditions. For example, if a strategy has 100 trades and 10 inputs, it has 90 degreesoffreedom. The more degreesoffreedom, the less likely it is that the strategy will be overfit to the market and the more likely it is that it will have good outofsample performance. The number of degreesoffreedom can be increased during the build process by adjusting the weights for the number of trades and/or the strategy complexity. All other things being equal, increasing the performance weighting for the number of trades will result in strategies with more trades and therefore more degreesoffreedom. Likewise, increasing the performance weighting for the complexity metric will result in strategies with fewer inputs, which will also increase the number of degreesoffreedom. Builder also incorporates the degreesoffreedom into the build process via the significance performance metric. In Builder, “significance” is based on the Student’s t test applied to the average trade. It measures the statistical significance of the average trade; that is, the probability that the average trade will be greater than zero. The t test is based on the number of degreesoffreedom but is a more complete measure of whether a strategy is overfit than the number of degreesoffreedom alone. One way, then, to improve outofsample performance is to use the significance metric to generate strategies that have a high statistical significance. 
The following user says Thank You to krzysiaczek99 for this post:  