Thanks for this thread. Can you say more about the phenomena you describe in the first paragraph please. That is why you think your backtest results did not translate well into real time trading. The subsequent paragraphs dont seem to clearly point to an explaination of why you had such a hard time with backtesting - or maybe I am just not understanding...
Back testing often results in curve-fitted systems. That is your system is perfectly adapted to the past, but not the future. Ideally you would want to test your system on tomorrow's data but I have yet to find a data vendor who sells these data..
That is hard to understand but markets are not static, they change. And this does not mean mere price changes but the way markets behave changes. In statistical language: the underlying distribution changes.
Let's make an overly simplified example: you test a system for a market which moves up from the start of your data till present with some corrections in between. Your system tells you to go long after a retraction (small down move).
You go live with your system. At this point of time the market starts a serious down trend. You will buy in a down move, maybe see a little profit when the market retraces a bit up but then lose because it turns down again viciously.
There are all kinds of things to try to prevent curve-fitting e.g. walk forward testing, out of sample testing. They may help but all of them cannot avoid curve-fitting.
Change of trend is just a very simple example. There are changes of volatility, of liquidity, how markets react to external events etc. Testing very long time-frames does not help either, because markets change over time.
Optimizing parameters, signals, whatever, makes it worse only. There is a proverb: if you torture the data long enough they'll confess to anything.
Is back testing totally useless? No, You need a really good and preferably simple idea. And then look when it worked well and when not and try to figure out why.
The following 6 users say Thank You to FGBL07 for this post:
Agreed . What I and many others experience is finding something , a pattern or indicator , that when tested against a market produces profits but when traded live fails to produce . What I then found is that perhaps I traded that something in conditions that werent complimentary I.E. against a dicernable trend or during low volume .
So , what I and others missed is that trying to maybe force my will on a market doesnt work . But it may work a few times or enough times during a sampling to lead us to believe in it . What I found to be way more effective is to study an instrument or market and then use a pattern or indicator to detect the signs that its inherent moves were about to happen . This way I have an idea about the markets behaviour pattern since the same players are likely to be involved and play it the same way next time I.E. big banks or funds .
The following 8 users say Thank You to Eric j for this post:
Something that is hard to accept when trading is missing a huge move after you exit with a profit or missing the whole thing altogether . Accepting whatever happens is what makes great traders .
Getting out when your target is hit is an achievement and stopping out when your stop is hit is equally an achievement . Both events say something about you , you carry out your plan . Regret has no place in your trading regime and if you let it in it'll erode your confidence like rust on steel .
Stopping out at your predetermined level says that you refuse to be stubborn and insist on protecting your capital . Exiting at your predetermined level says you trust you abilities and your analysis and insist on building your capital base .
Watching price fly way beyond your exit level and regretting not staying in is a fools game . Praising yourself for sticking to your guns and letting the odds guide you is the route you take when you understand that trading is a game that depends on probabilities . Finding for yourself market(s) , setups , average move size , average # of waves etc. and knowing the statistics to guide you in and out of that market is all you need to know about trading .
Then when you stop out a couple times you have your stats to lean on and keep you from losing faith in your approach and finally yourself . You also have those same stats to keep you from getting high on a long winning streak . What I do to keep me centered while playing the market is when I stop out I recall that a winner is likely on the next trade and when I get my profit targets I recall that a loser is likely on the next trade . I dont expect a winner or loser I just remind myself constantly that the results of the last trade have zero bearing on the next trade - zero .
The following 22 users say Thank You to Eric j for this post:
From Merriam webster online dictionary - Common sense , noun : sound and prudent judgment based on a simple perception of the situation or facts . Since its a noun its a "thing" and not an action and it requires we consciously apply effort to put it into action . That action is where I had the most trouble using common sense .
Common sense only exists after the fact . "You used common sense" or "thats just common sense" describe actions that produced any result of an action of common sense . No where is it implied that if you practice common sense will you always get better results in all aspects of your life . Nothing happens "always". I happen to know people that have no common sense whatsoever that do great in life .
BUT , when you are a trader and you dont apply common sense you will only help to feed the families of the traders that do apply common sense - consistently .
So , what are some examples of common sense trading decisions ? Heres some of mine ......
* follow the leaders
* get out when Im wrong and before Im very wrong
* take whats mine when its mine to take then get out
* listen to me not you
* resolve to pull the trigger (opposite of hesitate)
* let the market come to me when its ready
There you have it . Unfortunately , common sense is up to you . Either you want it and you can achieve it through practice or you can talk about it and never use it or something in between but the fact is that without it you will fail at trading , sorry . Its better to come to grips with the facts about yourself to protect yourself , your capital and your loved ones . I have seen and do know some that are gambling addicts and for all intents and purposes ( if risking money is the common denominator ) gambling = trading . They know that they cant gamble , or make trades , because they are addicted to the rush and thrill of it all . I am very proud of these people that take steps to address their affliction - Gamblers Anonymous .
Its true that most of us arent addicted to trading or gambling in the literal sense and we seek to enhance our income , derive an income or just plain kill the markets . For us its a function of doing what needs to be done and nothing more . What needs to be done is right in front of you written on your markets price chart and its up to you to learn its language , see its story and understand whats the best course of action to flow with your chosen market .
I hope this has helped someone out there and I felt it was my duty to share all this . Ive been around this joint since the start and have followed Mikes blog before that and his spirit of sharing appeals to me as it does all of us , thanks .
The following 11 users say Thank You to Eric j for this post:
I just hope people have the common sense to listen to you advice.
I've often said, " Most aspiring traders want a strict rule based system, that has a high degree of profitability, where there is little to no risk incurred, and minimal capitalization or knowledge of the markets required. They want rules they can count upon to make money, now and forever. While it is alluring to think that there is a single, unchanging order to the marketplace and that some particular contrivance can discern that order, past experience and current practice, consistently contradicts this fantasy."
They are always looking for the next "shortcut" and are unwilling to take the time to learn about the markets and what drives price - too much emphasis on the "how" and not the "why", and too much emphasis on the "empirical" and not on the "human." Trades, don't make themselves, people make trades, and if you understand traders' actions, you will understand the market's actions.
Focusing on the big picture "why" allows the trader to better understand complex situations, develop a strong conviction about their trade decisions, and avoid being too surprised by the vagaries of the market- in other words, allows one to be better prepared for what the market throws at you.
And, complements the "how" perfectly.
Perhaps, this is something Keynesian central bankers should keep in mind. It's not "statistics" that drives the global economy, but people. Adopting a praxeological approach is simply common sense.
Last edited by tigertrader; October 29th, 2011 at 11:47 AM.
The following 8 users say Thank You to tigertrader for this post:
Conviction - thats it right there , conviction . How many times has this scenario taken place .......
1. A trader makes a string of winning trades and continues trading their plan .
2. That trader gets a stopout and keeps trading their plan .
3. That trader gets another stopout and hesitates to take the next trade according to their plan.
4. That trader starts looking for a different "way" to trade .
A lot , tons . Sticking to your method , vision , approach , plan is what trading is all about . Be diciplined enough to trust your decisions and you're miles ahead of the pack .
The big ( and BIG ) picture is the road map your market has made . The little curves in that road occur over and over into eternity . Standing back looking at your market will tell you what you should be looking to exploit . Trying to guess what will work in your market or appying an approach you would like to have work ( pretty indis ) in your market is just not good enough .
The following 5 users say Thank You to Eric j for this post:
An excellent article, by Brent Steenbarger, in reference to the impact of non-stationarity on indicators, that is a " must read".
Every trader is familiar with what Victor Niederhoffer calls the “ever-changing cycles” within the market. Just as a pattern makes itself evident in the market, the pattern shifts to a new configuration. For example, the market may trade for a while within a range, offering nice buy and sell signals with a 14 period RSI. Then, abruptly, the market will break out and an overbought RSI will stay overbought or oversold for a prolonged period as the market makes a trending move.
It is because of these ever-changing cycles that traditional tools of technical analysis cannot be successfully applied in a purely mechanical fashion. The website Barchart.com has a nice feature where they track trading signals from such standard tools as moving averages. Over time, it is clear from their tracking that the signals do not perform better than random chance. For a while the signals will prove profitable, only to degrade once the cycles change.
It is ironic that traders spend considerable time researching better indicators and models while giving little thought to the time frame over which these trading tools might be valid. If, indeed, the market consists of ever-changing cycles, then any system or indicator is apt to degrade in its performance over time. In fact, if one waits for an indicator or system to develop a fine historical track record, the odds are good that their useful life are limited.
What can a trader do in the face of such uncertainty?
The statistician’s term for ever-changing cycles is stationarity. A number series is stationary if the process that generated the series has been constant. Clifford Sherry, in his excellent text The Mathematics of Technical Analysis explains, “A stationary time series is one in which the underlying rules that generate the time series do not change over time.” (p. 9).
My favorite example is the Las Vegas casino. Let’s say that you are playing blackjack and think that you have a superior card-counting strategy that will help you make money. By counting the number of picture cards vs. other cards that have been dealt, you can assess the probability of drawing a picture card on subsequent hands, tilting odds in your favor.
Such a strategy will work as long as the number of decks employed by the dealer is constant. If, however, the dealer intermittently and secretly changes the number of decks in the shoe, the card counting strategy would be imperiled. If the gambler assumed that twelve cards worth 10 or higher were left in the deck because eight had been dealt, the assumption would be faulty if two decks instead of one were being used. By changing the rules for dealing cards, the dealer creates a distribution that is nonstationary. Clifford Sherry notes the importance of nonstationarity for traders: “If you use these methods and techniques and find that your time series is nonstationary, it is probably best to stop and think carefully about your investment strategy. Nonstationarity implies that the underlying rules that ‘generated’ your time series change from time to time without warning. Therefore, you are dealing with maximum uncertainty about the potential outcome of your investment.” (p. 6). Sherry’s phrase “from time to time” is important. If, say, the card dealer changed the number of decks in the shoe after each and every hand, no card counting strategy would be possible. What makes counting viable is that the cycles are changing, but not constantly changing. A regime—a period in which the market follows a stable set of rules—can last for a while, allowing an alert trader to
profit while it is in force.
What should be clear is that a skill essential to trading success is early identification of regime change: those occasions when the cycles are shifting and the distributions of price changes are significantly varying from their recent norms. If a card counter can quickly identify when the number of decks in the shoe have changed, he can avoid betting his old system and take the time to develop a new one. Similarly, once a trader notes that market behavior has shifted, he or she can stand back and identify the new rules that the market is following and position themselves for the new regime.
Amazingly, very few traders bother to look for stationarity and even fewer shift their trading strategies according to the characteristics of recent price change series. This includes technical analysts who employ the same indicators and indicator values across all markets and quantitative traders who fail to properly adjust their lookback periods when testing a relationship between predictors and price change. Would we expect the time series from 2000 to 2002 to provide an accurate database for gauging relationships in the 2003 market? Did the market from 1998 to 2000 provide useful guides over the subsequent two years?
Assuming stationarity when it is not there is one of the cardinal errors of trading.If you are trading a pattern that has been valid in the past and you don’t know if thecurrent distribution of price changes match those from the past, you are flying blind. Successful trading requires that you identify the rules the market is following and base your strategy on those.
Fundamental Uncertainty in Trading
Let’s go back to that last sentence. The trader knows that there are ever-changing cycles, but makes a fundamental assumption. That assumption is that the regime that is in place will not change over the next trading interval. The trader assumes that the market’s rules will continue to be in force at least one more time. Without that assumption, the trader is either assuming randomness or is assuming regime change in the absence of concrete evidence of such. The only way we know if a regime has changed is by seeing an actual shift in the distribution of price changes. That means that there is a fundamental uncertainty in trading. The next trade may be the one in which the cycles shift. We cannot know for sure. Any trading strategy needs sound money management for this reason. Betting the house on a single trade—or during a single time frame—is courting ruin.
This has some interesting implications. For example, a well-researched trade that loses money may be an important source of trading information. If I have tested a historical period and found stationarity and then test a relationship between predictors and prospective price change over that period, my trade should have a high probability of success—if the market is remaining stationary. A losing streak with well-researched trades is often a sign that the markets are changing. Standing aside, waiting for evidence of the new regime, and remodeling the market over the more recent time frame corresponding to the new regime may allow the trader to learn from losses—and recoup them as well!
The fundamental uncertainty of trading is highest in daytrading the stock market —particularly index futures such as the SP/ES and ND/NQ. This is because markets are nonstationary on an intraday basis—almost without fail. Markets are most volatile early in the day’s trading, retreat to lowest volatility in early afternoon, and then pick up volatility toward the close (only to plunge in volatility during Globex trading). It is rare indeed that the distribution of price changes from 09:30 – 11:30 AM ET will match those of 11:30 AM – 13:30 PM ET. Using the same indicators and indicator values in morning trading as in early afternoon and Globex sessions is a sure road to the poorhouse. Conversely, identifying regime change and valid relationships with each intraday regime shift requires a nimbleness—and an ability to control losses—that most traders lack.
Interestingly, markets exhibit greater stationarity from day to day and week to week than from hour to hour. That is one of the factors that has sped my transition from intraday trading to swing trading. But if stationarity is as important to trading as Sherry and I believe it to be, then it makes little sense to pigeonhole oneself as a short-term trader, a long-term trader, a daytrader, etc. One should trade the time frames that offer the greatest stationarity. If the market is stationary over a period of weeks and if you can clearly identify the rules the market is following over that period, it makes sense to trade those rules. Later, the market may exhibit stationarity over a shorter time frame, covering a series of days. The rules that capture that regime will provide the basis for trading.
Many times, we hear of the distinction between mechanical and discretionary trading. This is a false dichotomy, because both mechanical and discretionary trading often fail to take ever-changing cycles into account. The real alternative to mechanical trading is flexible trading that searches for regimes and the rules guiding regimes, exploiting these in a rule-governed manner.
From Theory to Practice
Analyzing the market for trades should begin with tests for stationarity. In my new swing trading system, I begin my analysis by identifying the longest swing period in which the markets are exhibiting a stationary series of price changes. (There may be more than one such stationary swing period, permitting diversification of trading by time frame, and—of course—there may be stationarity for certain instruments and not others, permitting diversification by trading vehicles.) My procedure for assessing stationarity is to divide the time series into halves and statistically test to see if the means and standard deviations for the halves are equivalent. For readers interested in the math involved, Sherry’s book outlines a practical procedure for testing stationarity. The math is simple; I employ a quick-and-dirty t-test to the data and conduct the test entirely within Excel. What takes time is the repetitive testing of various lookback periods to find the proper window of stationarity.
Once I have that window, I then analyze the market qualitatively. I look at my indicators and observe how they have behaved during the stationary lookback period. The indicators that have consistently traced swing highs and lows over that period are the ones I will use to plan my next trade. I test signals yielded by the indicators (individually and in concert) over the lookback period to examine their entries, exits, and drawdowns. When I have a cadre of indicators that have performed well over the lookback period, I
rely on them for my next trade.
But that’s curve-fitting, you might protest. Isn’t it dangerous to overfit the data with an optimized model?
My response is that optimization is only a problem when you fail to take stationarity into account. If you know you are trading within a stable regime, it makes sense to do your best to capture the rules the market is following over that period. My swing trading methodology might best be described as serial optimization: continually hunting for periods of stationary market behavior and trading optimized models derived from those periods.
Now here’s the rub. When markets shift regimes, the window for the new regime is small. In testing the indicators that best follow the new, emerging rules, there aren’t enough instances to properly conduct statistical tests. That is where historical tests become important. By identifying past periods of market history where the markets were following the same rules as today, we can see if the indicators and signals that work in the recent lookback period also worked back then. The crucial assumption is that markets that exhibit stationarity and equivalent means and standard deviations in price changes are following the same rules—regardless of whether those markets were taking place in 2003, 1993, or 1983. If the strategy that we’ve optimized in the recent, stationary market window also produces profitable trading signals during past, similar regimes, we increase our confidence in the strategy and, indeed, can even test its signals statistically to ensure their departure from randomness.
Perhaps this is why we see so few traders incorporating stationarity into their analyses: It is time-consuming to assess market windows, operative trading rules, and test strategies for exploiting those rules. It is easier—and far more beguiling—to assume that a single system or indicator will produce consistent profits. More than one person has encouraged me to make my writing, research, and trading strategies less complex so that they can be more readily understood and accepted by the bulk of traders who attend seminars, buy trading books, and hire gurus for advice. One seminar organizer even fretted that I might be a threat to the self-esteem of traders, because the majority of traders lack the data and/or statistical background to conduct my kind of trading. I took that, of course, as quite a compliment.
If you read the Trading Psychology Weblog with any frequency, you’ll notice that many of the charts that I post have a common—and seemingly random—starting date. For instance, as I write this (12/28/03), many of my charts begin with 8/1/03. This is not an accident. The period from August through December represents one of those stationary windows from which we can extract useful swing trading strategies. By posting charts over stationary time frames in the market, the Weblog can assist you in identifying tradable market patterns.
Incorporating stationarity into your market thinking and trading opens the door to innovative trading approaches. For instance, within a longer stationary window (several months), you might identify a smaller window (the past several days) for a short-term trade. By nesting and aligning the short-term trade within a longer-term pattern, you can formulate some high probability trades. Beginning January, 2004, I will be posting real time swing trades to the Weblog that take advantage of rules derived over one or more stationary windows.
Yet another avenue for research is the use of very short-term nonstationarities to identify points of larger regime change. A while ago, when I was exclusively trading the SP on an intraday basis, I noticed how short-term shifts in the NYSE Composite TICK tended to occur at points of trend change in the market. The short-term nonstationarity was a marker for longer-term trend change. I believe the same occurs at all time frames. By monitoring shifts in short-term patterns and indicators, we may be able to hop aboard early phases of regime change.
Last edited by tigertrader; October 29th, 2011 at 01:11 PM.
The following 10 users say Thank You to tigertrader for this post:
Great post TT, I don't think I've seen direct mention of this before. Honestly it's a little beyond me (never liked homework) but I guess this is why ultimately one has to rely on PASR. For guys like me anyway it still works no matter how the market changes....a bit oversimplified but I think you would agree.
I would also like to give a gold star to Eric...great thread, I agree with everything you have said and said in a manner that everyone can understand.
The following 2 users say Thank You to kbit for this post: