Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
Walk Forward Analysis - the only logical successor to backtesting [DISCUSS]
Walk Forward Analysis - the only logical successor to backtesting [DISCUSS]
Hello,
I'm Darwin and this is my second article in which I will try to explain how a walk-forward-analysis works and what benefits it brings you as an EA trader.
But I do not just want to explain how a walk-forward-analysis works (as others already did this), no, I want to explain why it is the only logical way to analyse EAs.
And as people keep asking why I release free stuff, let me explain my motivations: I am targeting a job in the trading-economy, and this is not easy to get. Beeing known in the community would help me a lot to accomplish this goal, so I decided to release some of my private tools for free and/or open source to make myself a name over the next months. I am not here to sell you stuff, keep that in mind, please!
I know that the title is a bit provocative, but it's often easier to start a discussion with a controversy
Also, I know that the best EA-traders can use normal backtesting and still be profitable - but most can not. And even if you can, a WFA is still better :P
I WILL RELEASE A WALK-FORWARD-ANALYZER-TOOL FOR FREE WITHIN THE NEXT FEW DAYS THAT DOES ALL OF THIS 100% AUTOMATED; UNTIL THEN, USE THE TIME TO MAKE YOURSELF FAMILIAR WITH THE CONCEPTS.
Nevertheless, none of this article is needed to use the walk-forward-analyzer tool, it will be as easy to use as the metatrader4-backtester.
After reading, you have 2 choices: Agree with my arguments and once and for all get rid of the flawed backtesting-approach and use WFA in the future. Disagree with my arguments, but then please try to argue with me.. Do yourself a favour, don't just stick to backtests because you "know" them or something like that.
Btw, a backtest has many more disadvantages compared to a Walk Forward Analysis than the ones I describe here. If you want to learn more on this topic, read my first article:
"Why backtests are worthless, fixed-logic-EAs are flawed and your parameters are bad [DISCUSS!]" (look it up in google or through forum-search)
Initial Situation
First, let's sum up what the essential parts of every trading system are!
1. The system's logic
The most obvious part! And for a lot beginners it's the only part they know, which is dangerous.
This might be a manual trading system or an expert advisor or any other form of fixed trading-logic / trading-system / trading-strategy (btw: all 3 terms name the same thing in this article)
So far, so good. But you all know that every strategy has some kind of variables/parameters (like the periods of moving averages or stop-loss levels etc), that are NOT FIXED(!) but can vary, which brings us to the second part.
(If you just set them to a fixed value because "this should work".. well, it wont, at least not in the long term)
2. The system's parameter-ranges
The ranges of the parameters are an ESSENTIAL part of every trading-system, as they determine the exact behaviour of it (tough the trading-logic always stays the same).
So, for example, a moving-average-period might range from 5-15 to capture short-term price movements.
It is not 6 and not 11 and not 14, it is 5-15, as the markets change, we can not choose a concrete value, ALWAYS a range!
3. The market, the amount of data, the desired characteristics
Every strategy trades on a market, so we want to determine on which. (eg EURUSD / H4)
But thats not enough, we also have to determine how many past-price-data we want to use to evaluate our possible parameter-choices.
Because, as I said, a system always has parameter-ranges, but for live trading we have to choose concrete values!
And we do this by evaluating all parameter-possibilities on the last X years of price-data, and after evaluation,
we end up with a huge list of possible and "independent" trading-systems (each of them with different parameters, but the same main-logic).
And each has it's own characteristics like "profit" or "profit factor" or "relative drawdown".
So, we also have to determine how to pick the "best" parameters.
But it's not as simple as saying "I want much profit", because the characteristics often don't hold in the future!
Instead we want to choose in a way that gives us a high probability of picking parameters that will succeed during live trading.
Simple, isn't it?
An illustrating example
The system's logic:
Let's suppose a very basic trading-system: "If the price moved more than X pips in the last Y days, a course correction will happen"
(this is not a valid strategy, its just thin air for the sake of simplicity).
The parameters would be X and Y in this case.
The system's parameter-ranges:
To make it all simple, I chose X to be 100-200 pips in this example, and Y to be 2-3 days. (also, just thin air!)
Amount of data & prefered characteristics:
Here we choose the last 10 years of data to evaluate the possible parameters on, and "profit" as prefered characteristic.
(tough, as I said, in reality profit is not a very good indicator for parameters that will have a good future performance)
The process:
Ok, now before we can trade that system, we make an optimisation on the last 10 years.
That means we backtest every possible parameter-combination for our system and choose the best in terms of "profit".
For sake of simplicity, here is a cropped example:
"If the price moved more than 100 pips in the last 2 days, a course correction will happen"
=> 1000$ in the last 10 years
"If the price moved more than 150 pips in the last 2 days, a course correction will happen"
=> 1200$ in the last 10 years
"If the price moved more than 200 pips in the last 2 days, a course correction will happen"
=> 1500$ in the last 10 years
"If the price moved more than 100 pips in the last 3 days, a course correction will happen"
=> 900$ in the last 10 years
"If the price moved more than 150 pips in the last 3 days, a course correction will happen"
=> 950$ in the last 10 years
"If the price moved more than 200 pips in the last 3 days, a course correction will happen"
=> 950$ in the last 10 years
Soo, According to our prefered-characteristic (profit), we would choose X = 200pips; Y = 2 days, and then just trade the strategy.
Well, that's the "normal" process of EA-trading, and I claim it does not work this way.
EA-Analysis: How to ask the right questions
Ok, now that I have described the current process and how it is all done, here comes the "new" part.
The goal itself always stays the same, we want to pick the best parameters (based on some kind of evaluation on the past), and then we want to trade live!
Remember: The only thing a backtest can tell is "How good did my system+parameters perform in the past".
But that is NOT what we want to know! Be sure that you really understand this.
Initial Question; What we actually want to prove with analysis. "Does the way we choose parameters for live trading ('pick the one with best profit over the last 10 years' in the above example) give us a high probability to pick parameters that are profitable during live trading?"
So we are actually interested in the relationship of past-performance&future-performance, not backtest-results!!
If the answer is No, one or more of the 3 things described in "Initial Situation" are wrong. Might be the logic itself, the parameterranges etc..
If the answer is Yes, the performance in the past and the performance in the future are somehow correlated for our EA, and we can trade the system!
The logical evolution; From Backtests to Walk-Forward-Analysis
Step One: Backtesting - in it's worst form
Pro:
We will get parameters that performed well on a wide range of data
Contra:
Every single EA trader that used this method and then tried to trade an EA live, based on good backtests, can tell you: It just does not work this way.
Overfitting / Curvefitting!! We first optimise the parameters, and then test them, all on the same data.
That means we have no clue if we have valid parameters or overfitted ones.
Overfitting means, we optimised towards a random behaviour within our data, that just exists in this particular dataset, and will not hold in the future.
That means, we captured a relationship that existed but was not a sound one. Like this:
Don't fool yourself in thinking "ah, this wont happen"... Almost all "relationships" within the markets are like this, as most price-movements are random!!
If you do not understand overfitting, google for more information, as it is our archenemy in mechanical trading.
No significance for future performance!! Remember that the initial question is not how good our parameters performed on the past, but how high the probability of succes AFTER the optimisation-timespan (so, "in the future", during live trading) will be.
As we did not do any tests with our parameters that take into account the relative future, we did not even try to answer the initial question.
We just answered the question "How good did our parameters perform in the past", not taking into account anything about the "future" => very bad.
Even if you could somehow magically invalidate my above points, because the parameters worked well on a huge amount of data, they are not really the best for the current market - just average good on all market-conditions.
Step Two: Backtesting using unseen/out-of-sample data
Notice: The first dataset, we use to optimise our parameters on, is called "in-sample" (is). The second, unseen, dataset is called "out-of-sample" (oos).
Pro:
We now have a lower chance to get overfitted parameters, as we use an independent dataset to validate our parameter-choices.
Contra:
Due to the infinite amount of senseless/unsound relationships within the markets, we still have a (too high) risk for overfitting, as chances are too high that we just got parameters that are valid (curvefitted) on both datasets, but not valid in the future.
If the system did not work in out-of-sample and you then begin to tune your parameters until you get good oos-results, your oos-results are not longer "unseen" and becoming "in-sample", which makes the whole approach using 2 datasets useless!
We still use a very larg part of our data (in-sample) to find the best parameters, which also means we use a lot "old" data. That is not a good decision as the behaviour of the markets in the past is not equal to the behaviour of today.
Not just our in-sample dataset is too huge, also our out-of-sample dataset is too huge and therefore un-realistic. In the example above it would be a few years, but would you really like to trade a system for years before choosing new, re-adjusted, parameters? I would not!
Step Three: Backtesting using a more realistic data-amount
Pro:
We now only use the recent market-behaviour to optimise our parameters, so we capture the market "at the moment", and not "10 years ago".
We not test our parameters on a timespan that is more realistic (as it is not years but months!)
Contra:
We only used a small part of the available price-data for our tests. This is not very efficient!
Ok, remember the initial situation, where we have settled on parameter ranges, amount of data to optimise on, and the "desired characteristic". Our analysis has the purpose to verify these choices, wether they are valid or not.
But in this case, we only made one test with them, so we optimised on one part of the data, then we chose 1 parameter-combination and tested it on 1 "unseen" dataset.
Facing the million/billion possible parameter-combinations an EA can have, and the infinite ways the markets can change to generate new and "unseen" behaviour, do you really think that 1 test, 1 datapoint, 1 past->future relationship, is enough to judge from? Of course not! So why are you still using normal backtests?
Step Four: Walk Forward Analysis
So, as you might see, a Walk Forward Analysis is the same thing like doing a normal back- & out of sample-test, but we do it over and over again, so we end up not just with 1 test-case but with many (100-150 in most cases, up to 1000 if we choose very small test-period).
That way we can verify our system + our optimisation-methodology on many, many independent test-cases, which is THE reason why we want to use WFA instead of every other analysis-method described in here.
Pro:
For our final analysis-report, we only take into account the green test-results, as they are the "unseen future" relative to the red optimisation-windows.
That way, we simulate the same process we would face during live trading: Optimisation on the past, trading on the (relative) future!
That allows us to draw meaningfull answers to the initial question, as we only analyse performance in "the future".
We use all data available for our testing
We have 100-150 independent "PAST=>FUTURE"-relationship-tests, which gives us a clue about the future performance, not the past performance!
We avoid overfitting, as we use different datasets to optimise and verify our parameters
If we want to trade live, we simply make "one more step" of the WFA, optimise on the last available data (the "red" dataset would then end at the end of the chart), and then trade "in the future" (the "green" dataset would be our live trading). So we trade the system using the EXACT same methodology we have tested 100-150 times already.
Due to the frequent re-optimisation of parameters, the EA is also continuously re-adapted to the markets, which will most likely increase the overall profit.
A traditional backtest answers the question "How good was my EA in the past", whereas a Walk Forward Analysis answers the question "How good will my EA be in the future, during live trading".
It does not only evaluate an EA, it also evaluates the corresponding trading plan that determines how to pick the best parameters for live trading.
""Contra"":
Most EAs will not pass this test. But this is not bad, because lets be honest, almost all EAs in existance are bullshit. So if almost all EAs tested with this approach would give bad results, that would be great.
Even if a lot people do not like to be disillusioned about their "holy grail money printing machines", it's better to face the truth during EA-development and not during live trading.
Contra:
There are some limitations regarding this process which will be discussed in a later article, stay tuned! Also, I am currently working on more sophisticated anlysis algorithms, but it will take a few months until I can show you something.
The main advantage is that we get 100-150 independent test-cases, whereas a Backtest+Out-of-sample-test gives us only 1 test-case (or 1 datapoint).
But the best thing would be to discuss it public here on the forum, so please do so if you have anything to say on the topic, thanks!
-Darwin
EDIT: Sorry, link was to the reddit #forex irc channel, if somebody wants to discuss it live
Can you help answer these questions from other members on NexusFi?
Think whatever after reading the arguments, and doubt your intentions. In your first post on futures.io (formerly BMT), you already promoted your 'private walk-forward solution', which would solve all problems and you'd happily share. For someone keen on sharing, you've only posted about your own backtesting ideas and have not thanked a single futures.io (formerly BMT) post by another user. Even though you're an advanced trader, there's more than enough high-value content on futures.io (formerly BMT) for you here.
You might have the best intentions, but forgive my scepticism if a new futures.io (formerly BMT) user has a strong need to tell other people how to backtest and perform walk-forward analyses, while flaunting with some solution that would make everything great.
I really doubt people keep asking you that, but which free things have you released? Or are you still flaunting with your 'private walk-forward analyzer'?
Furthermore, while futures.io (formerly BMT) is a great forum, I really doubt an investment bank or trading company is impressed by a few posts made on an online forum with a pseudonym.
Besides that, your posts also use terms that are not used in the industry, which will probably make professionals sceptical (e.g., "past-price-date" when meaning historical data, "expert advisor" while a professional would say quantitative/algorithmic trading system, not differentiating between overfitting and curve fitting).
Thanks for the critical arguments, far too few of them came up in my threads!
It's not a "private walk-forward solution", I just did not have the time to release it yet. But I will start tomorrow on some forums and on the others at the beginning of next week
Also, I plan to release the source, so it's as far from private as possible
Well yea, I did not read a single post by another user to be honest. Not because I think they are useless or something, I just don't have the time for this at the moment And I am 100% sure there is a lot high quality content here, and I'm looking forward reading through it when I have the time.
Well, I don't tell ppl how to backtest, I just wanted to start a discussion because, let's be honest, a lot people working with EAs have a somehow wrong view at the process of backtesting.
So if I sound somehow provocative, excuse me, but it's the best way to start discussions imo.
Also, I tried hard to put parts of my knowledge into logical argumentation/in this threads, because I want ppl to understand how the tool they will be using soon (hopefully some will :P) works, thats why I always refer to my "solution that would make everything great."
still flaunting
Well, mike asked me here on the forums, and on forex-factory there was a recent post about "do you want to sell us stuff", also on other threads and in IRC. So doubt it or not, people are very sceptic about releasing free stuff, they all asume some kind of hidden scam or I don't know. So why not posting my intentions right away?
Don't get me wrong, I know that there will not be someone saying "Oh yes, we need this guy posting 2 threads on the forum", I am not that naive.
But if traders read my threads, and then use my program, and spread it, and more and more people do this, eventually somewhen at a job interview I can say "Oh and btw, I also wrote a tool that xyz traders are using every day, and here, look at all the threads where you have a bit of an insight in my knowledge" You know?
This would at least give me an edge.
Well, I never claimed to be a professional, but using the "official" terms or not, my argumentation and the content stays the same.
Also, these threads are meant for non-professionals regarding algo-trading, so why should I talk about "quantitative/algorithmic trading system" if "expert advisor" is something all traders understand.
Sounds like this @Darwin guy is going to just end up directing you to his website where he will collect your info in exchange for "giving" you his idea, or he will just flat out sell it and will have used this entire thread as a ruse to market his product.
In 5 years of running the forum, I've never seen anything other than this happen from a thread that started like this, where a user who registers for first time posts such a thing, and spams the same identical messages on multiple message boards simultaneously.
So I am done with this thread. Everything he posted about has already been previously discussed on futures.io (formerly BMT). I am closing this thread.
