Walk Forward Analysis - the only logical successor to backtesting [DISCUSS] - Currency Futures | futures io social day trading
futures io futures trading


Walk Forward Analysis - the only logical successor to backtesting [DISCUSS]
Updated: Views / Replies:2,837 / 4
Created: by Darwin Attachments:0

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Closed Thread
 
Thread Tools Search this Thread
 

Walk Forward Analysis - the only logical successor to backtesting [DISCUSS]

  #1 (permalink)
Trading Apprentice
Regensburg, Germany
 
Futures Experience: Advanced
Platform: MetaTrader
Favorite Futures: Forex
 
Darwin's Avatar
 
Posts: 9 since Oct 2013
Thanks: 0 given, 3 received

Walk Forward Analysis - the only logical successor to backtesting [DISCUSS]

Walk Forward Analysis - the only logical successor to backtesting [DISCUSS]

Hello,

I'm Darwin and this is my second article in which I will try to explain how a walk-forward-analysis works and what benefits it brings you as an EA trader.

But I do not just want to explain how a walk-forward-analysis works (as others already did this), no, I want to explain why it is the only logical way to analyse EAs.

And as people keep asking why I release free stuff, let me explain my motivations: I am targeting a job in the trading-economy, and this is not easy to get. Beeing known in the community would help me a lot to accomplish this goal, so I decided to release some of my private tools for free and/or open source to make myself a name over the next months. I am not here to sell you stuff, keep that in mind, please!


I know that the title is a bit provocative, but it's often easier to start a discussion with a controversy
Also, I know that the best EA-traders can use normal backtesting and still be profitable - but most can not. And even if you can, a WFA is still better :P

I WILL RELEASE A WALK-FORWARD-ANALYZER-TOOL FOR FREE WITHIN THE NEXT FEW DAYS THAT DOES ALL OF THIS 100% AUTOMATED; UNTIL THEN, USE THE TIME TO MAKE YOURSELF FAMILIAR WITH THE CONCEPTS.

Nevertheless, none of this article is needed to use the walk-forward-analyzer tool, it will be as easy to use as the metatrader4-backtester.



After reading, you have 2 choices:
Agree with my arguments and once and for all get rid of the flawed backtesting-approach and use WFA in the future.
Disagree with my arguments, but then please try to argue with me.. Do yourself a favour, don't just stick to backtests because you "know" them or something like that.


Btw, a backtest has many more disadvantages compared to a Walk Forward Analysis than the ones I describe here. If you want to learn more on this topic, read my first article:
"Why backtests are worthless, fixed-logic-EAs are flawed and your parameters are bad [DISCUSS!]" (look it up in google or through forum-search)





Initial Situation

First, let's sum up what the essential parts of every trading system are!

1. The system's logic
The most obvious part! And for a lot beginners it's the only part they know, which is dangerous.
This might be a manual trading system or an expert advisor or any other form of fixed trading-logic / trading-system / trading-strategy (btw: all 3 terms name the same thing in this article)

So far, so good. But you all know that every strategy has some kind of variables/parameters (like the periods of moving averages or stop-loss levels etc), that are NOT FIXED(!) but can vary, which brings us to the second part.
(If you just set them to a fixed value because "this should work".. well, it wont, at least not in the long term)


2. The system's parameter-ranges
The ranges of the parameters are an ESSENTIAL part of every trading-system, as they determine the exact behaviour of it (tough the trading-logic always stays the same).

So, for example, a moving-average-period might range from 5-15 to capture short-term price movements.
It is not 6 and not 11 and not 14, it is 5-15, as the markets change, we can not choose a concrete value, ALWAYS a range!


3. The market, the amount of data, the desired characteristics
Every strategy trades on a market, so we want to determine on which. (eg EURUSD / H4)

But thats not enough, we also have to determine how many past-price-data we want to use to evaluate our possible parameter-choices.
Because, as I said, a system always has parameter-ranges, but for live trading we have to choose concrete values!

And we do this by evaluating all parameter-possibilities on the last X years of price-data, and after evaluation,
we end up with a huge list of possible and "independent" trading-systems (each of them with different parameters, but the same main-logic).
And each has it's own characteristics like "profit" or "profit factor" or "relative drawdown".

So, we also have to determine how to pick the "best" parameters.

But it's not as simple as saying "I want much profit", because the characteristics often don't hold in the future!
Instead we want to choose in a way that gives us a high probability of picking parameters that will succeed during live trading.


Simple, isn't it?





An illustrating example

The system's logic:
Let's suppose a very basic trading-system: "If the price moved more than X pips in the last Y days, a course correction will happen"
(this is not a valid strategy, its just thin air for the sake of simplicity).
The parameters would be X and Y in this case.

The system's parameter-ranges:
To make it all simple, I chose X to be 100-200 pips in this example, and Y to be 2-3 days. (also, just thin air!)

Amount of data & prefered characteristics:
Here we choose the last 10 years of data to evaluate the possible parameters on, and "profit" as prefered characteristic.
(tough, as I said, in reality profit is not a very good indicator for parameters that will have a good future performance)

The process:
Ok, now before we can trade that system, we make an optimisation on the last 10 years.
That means we backtest every possible parameter-combination for our system and choose the best in terms of "profit".

For sake of simplicity, here is a cropped example:

"If the price moved more than 100 pips in the last 2 days, a course correction will happen"
=> 1000$ in the last 10 years
"If the price moved more than 150 pips in the last 2 days, a course correction will happen"
=> 1200$ in the last 10 years
"If the price moved more than 200 pips in the last 2 days, a course correction will happen"
=> 1500$ in the last 10 years
"If the price moved more than 100 pips in the last 3 days, a course correction will happen"
=> 900$ in the last 10 years
"If the price moved more than 150 pips in the last 3 days, a course correction will happen"
=> 950$ in the last 10 years
"If the price moved more than 200 pips in the last 3 days, a course correction will happen"
=> 950$ in the last 10 years

Soo, According to our prefered-characteristic (profit), we would choose X = 200pips; Y = 2 days, and then just trade the strategy.
Well, that's the "normal" process of EA-trading, and I claim it does not work this way.





EA-Analysis: How to ask the right questions

Ok, now that I have described the current process and how it is all done, here comes the "new" part.
The goal itself always stays the same, we want to pick the best parameters (based on some kind of evaluation on the past), and then we want to trade live!

Remember: The only thing a backtest can tell is "How good did my system+parameters perform in the past".
But that is NOT what we want to know! Be sure that you really understand this.

Initial Question; What we actually want to prove with analysis.
"Does the way we choose parameters for live trading ('pick the one with best profit over the last 10 years' in the above example) give us a high probability to pick parameters that are profitable during live trading?"

So we are actually interested in the relationship of past-performance&future-performance, not backtest-results!!

If the answer is No, one or more of the 3 things described in "Initial Situation" are wrong. Might be the logic itself, the parameterranges etc..
If the answer is Yes, the performance in the past and the performance in the future are somehow correlated for our EA, and we can trade the system!






The logical evolution; From Backtests to Walk-Forward-Analysis

Step One: Backtesting - in it's worst form

Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).

Pro:
  • We will get parameters that performed well on a wide range of data

Contra:
  • Every single EA trader that used this method and then tried to trade an EA live, based on good backtests, can tell you: It just does not work this way.


  • Overfitting / Curvefitting!! We first optimise the parameters, and then test them, all on the same data.
    That means we have no clue if we have valid parameters or overfitted ones.

    Overfitting means, we optimised towards a random behaviour within our data, that just exists in this particular dataset, and will not hold in the future.

    That means, we captured a relationship that existed but was not a sound one. Like this:
    Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


    Don't fool yourself in thinking "ah, this wont happen"... Almost all "relationships" within the markets are like this, as most price-movements are random!!

    If you do not understand overfitting, google for more information, as it is our archenemy in mechanical trading.


  • No significance for future performance!! Remember that the initial question is not how good our parameters performed on the past, but how high the probability of succes AFTER the optimisation-timespan (so, "in the future", during live trading) will be.

    As we did not do any tests with our parameters that take into account the relative future, we did not even try to answer the initial question.
    We just answered the question "How good did our parameters perform in the past", not taking into account anything about the "future" => very bad.


  • Even if you could somehow magically invalidate my above points, because the parameters worked well on a huge amount of data, they are not really the best for the current market - just average good on all market-conditions.










Step Two: Backtesting using unseen/out-of-sample data

Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).

Notice: The first dataset, we use to optimise our parameters on, is called "in-sample" (is). The second, unseen, dataset is called "out-of-sample" (oos).


Pro:
  • We now have a lower chance to get overfitted parameters, as we use an independent dataset to validate our parameter-choices.


Contra:
  • Due to the infinite amount of senseless/unsound relationships within the markets, we still have a (too high) risk for overfitting, as chances are too high that we just got parameters that are valid (curvefitted) on both datasets, but not valid in the future.

  • If the system did not work in out-of-sample and you then begin to tune your parameters until you get good oos-results, your oos-results are not longer "unseen" and becoming "in-sample", which makes the whole approach using 2 datasets useless!

  • We still use a very larg part of our data (in-sample) to find the best parameters, which also means we use a lot "old" data. That is not a good decision as the behaviour of the markets in the past is not equal to the behaviour of today.

  • Not just our in-sample dataset is too huge, also our out-of-sample dataset is too huge and therefore un-realistic. In the example above it would be a few years, but would you really like to trade a system for years before choosing new, re-adjusted, parameters? I would not!










Step Three: Backtesting using a more realistic data-amount

Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).

Pro:
  • We now only use the recent market-behaviour to optimise our parameters, so we capture the market "at the moment", and not "10 years ago".

  • We not test our parameters on a timespan that is more realistic (as it is not years but months!)


Contra:
  • We only used a small part of the available price-data for our tests. This is not very efficient!

  • Ok, remember the initial situation, where we have settled on parameter ranges, amount of data to optimise on, and the "desired characteristic". Our analysis has the purpose to verify these choices, wether they are valid or not.

    But in this case, we only made one test with them, so we optimised on one part of the data, then we chose 1 parameter-combination and tested it on 1 "unseen" dataset.

    Facing the million/billion possible parameter-combinations an EA can have, and the infinite ways the markets can change to generate new and "unseen" behaviour, do you really think that 1 test, 1 datapoint, 1 past->future relationship, is enough to judge from? Of course not! So why are you still using normal backtests?










Step Four: Walk Forward Analysis

Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).

So, as you might see, a Walk Forward Analysis is the same thing like doing a normal back- & out of sample-test, but we do it over and over again, so we end up not just with 1 test-case but with many (100-150 in most cases, up to 1000 if we choose very small test-period).

That way we can verify our system + our optimisation-methodology on many, many independent test-cases, which is THE reason why we want to use WFA instead of every other analysis-method described in here.

Pro:
  • For our final analysis-report, we only take into account the green test-results, as they are the "unseen future" relative to the red optimisation-windows.
    That way, we simulate the same process we would face during live trading: Optimisation on the past, trading on the (relative) future!

    That allows us to draw meaningfull answers to the initial question, as we only analyse performance in "the future".

  • We use all data available for our testing

  • We have 100-150 independent "PAST=>FUTURE"-relationship-tests, which gives us a clue about the future performance, not the past performance!

  • We avoid overfitting, as we use different datasets to optimise and verify our parameters

  • If we want to trade live, we simply make "one more step" of the WFA, optimise on the last available data (the "red" dataset would then end at the end of the chart), and then trade "in the future" (the "green" dataset would be our live trading). So we trade the system using the EXACT same methodology we have tested 100-150 times already.

  • Due to the frequent re-optimisation of parameters, the EA is also continuously re-adapted to the markets, which will most likely increase the overall profit.

  • A traditional backtest answers the question "How good was my EA in the past", whereas a Walk Forward Analysis answers the question "How good will my EA be in the future, during live trading".

  • It does not only evaluate an EA, it also evaluates the corresponding trading plan that determines how to pick the best parameters for live trading.


""Contra"":
  • Most EAs will not pass this test. But this is not bad, because lets be honest, almost all EAs in existance are bullshit. So if almost all EAs tested with this approach would give bad results, that would be great.

    Even if a lot people do not like to be disillusioned about their "holy grail money printing machines", it's better to face the truth during EA-development and not during live trading.


Contra:
  • There are some limitations regarding this process which will be discussed in a later article, stay tuned! Also, I am currently working on more sophisticated anlysis algorithms, but it will take a few months until I can show you something.




The main advantage is that we get 100-150 independent test-cases, whereas a Backtest+Out-of-sample-test gives us only 1 test-case (or 1 datapoint).




But the best thing would be to discuss it public here on the forum, so please do so if you have anything to say on the topic, thanks!


-Darwin

EDIT: Sorry, link was to the reddit #forex irc channel, if somebody wants to discuss it live


Last edited by Darwin; November 26th, 2013 at 05:00 PM. Reason: removed redirect link to unknown third party site
The following 2 users say Thank You to Darwin for this post:
 
  #2 (permalink)
Quick Summary
Quick Summary Post

Quick Summary is created and edited by users like you... Add FAQ's, Links and other Relevant Information by clicking the edit button in the lower right hand corner of this message.

 
  #3 (permalink)
 Vendor: tradingcode.net 
The Netherlands
 
Futures Experience: None
Platform: MultiCharts, TradingView
Favorite Futures: ...
 
Jura's Avatar
 
Posts: 774 since Apr 2010
Thanks: 2,347 given, 661 received


Okay, I'll bite:


Darwin View Post
(...)
After reading, you have 2 choices:
Agree with my arguments and once and for all get rid of the flawed backtesting-approach and use WFA in the future.
Disagree with my arguments, but then please try to argue with me.. Do yourself a favour, don't just stick to backtests because you "know" them or something like that.
(...)

We also have a third choice:

Think whatever after reading the arguments, and doubt your intentions. In your first post on futures.io (formerly BMT), you already promoted your 'private walk-forward solution', which would solve all problems and you'd happily share. For someone keen on sharing, you've only posted about your own backtesting ideas and have not thanked a single futures.io (formerly BMT) post by another user. Even though you're an advanced trader, there's more than enough high-value content on futures.io (formerly BMT) for you here.

You might have the best intentions, but forgive my scepticism if a new futures.io (formerly BMT) user has a strong need to tell other people how to backtest and perform walk-forward analyses, while flaunting with some solution that would make everything great.


Darwin View Post
(...)
And as people keep asking why I release free stuff, let me explain my motivations: I am targeting a job in the trading-economy, and this is not easy to get. Beeing known in the community would help me a lot to accomplish this goal, so I decided to release some of my private tools for free and/or open source to make myself a name over the next months. I am not here to sell you stuff, keep that in mind, please!
(...)

I really doubt people keep asking you that, but which free things have you released? Or are you still flaunting with your 'private walk-forward analyzer'?

Furthermore, while futures.io (formerly BMT) is a great forum, I really doubt an investment bank or trading company is impressed by a few posts made on an online forum with a pseudonym.

Besides that, your posts also use terms that are not used in the industry, which will probably make professionals sceptical (e.g., "past-price-date" when meaning historical data, "expert advisor" while a professional would say quantitative/algorithmic trading system, not differentiating between overfitting and curve fitting).

 
  #4 (permalink)
Trading Apprentice
Regensburg, Germany
 
Futures Experience: Advanced
Platform: MetaTrader
Favorite Futures: Forex
 
Darwin's Avatar
 
Posts: 9 since Oct 2013
Thanks: 0 given, 3 received

Hey Jura

Thanks for the critical arguments, far too few of them came up in my threads!



It's not a "private walk-forward solution", I just did not have the time to release it yet. But I will start tomorrow on some forums and on the others at the beginning of next week

Also, I plan to release the source, so it's as far from private as possible


Quoting 
have not thanked a single futures.io (formerly BMT) post by another user

Well yea, I did not read a single post by another user to be honest. Not because I think they are useless or something, I just don't have the time for this at the moment And I am 100% sure there is a lot high quality content here, and I'm looking forward reading through it when I have the time.


Quoting 
has a strong need to tell other people how to backtest and perform walk-forward analyses

Well, I don't tell ppl how to backtest, I just wanted to start a discussion because, let's be honest, a lot people working with EAs have a somehow wrong view at the process of backtesting.

So if I sound somehow provocative, excuse me, but it's the best way to start discussions imo.

Also, I tried hard to put parts of my knowledge into logical argumentation/in this threads, because I want ppl to understand how the tool they will be using soon (hopefully some will :P) works, thats why I always refer to my "solution that would make everything great."


Quoting 
really doubt people keep asking you that, but which free things have you released? Or are you still flaunting with your 'private walk-forward analyzer'?

still flaunting
Well, mike asked me here on the forums, and on forex-factory there was a recent post about "do you want to sell us stuff", also on other threads and in IRC. So doubt it or not, people are very sceptic about releasing free stuff, they all asume some kind of hidden scam or I don't know. So why not posting my intentions right away?



Quoting 
Furthermore, while futures.io (formerly BMT) is a great forum, I really doubt an investment bank or trading company is impressed by a few posts made on an online forum with a pseudonym.

Don't get me wrong, I know that there will not be someone saying "Oh yes, we need this guy posting 2 threads on the forum", I am not that naive.

But if traders read my threads, and then use my program, and spread it, and more and more people do this, eventually somewhen at a job interview I can say "Oh and btw, I also wrote a tool that xyz traders are using every day, and here, look at all the threads where you have a bit of an insight in my knowledge" You know?
This would at least give me an edge.


Quoting 
Besides that, your posts also use terms that are not used in the industry, which will probably make professionals sceptical (e.g., "past-price-date" when meaning historical data, "expert advisor" while a professional would say quantitative/algorithmic trading system, not differentiating between overfitting and curve fitting).

Well, I never claimed to be a professional, but using the "official" terms or not, my argumentation and the content stays the same.
Also, these threads are meant for non-professionals regarding algo-trading, so why should I talk about "quantitative/algorithmic trading system" if "expert advisor" is something all traders understand.

 
  #5 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,220 received

Sounds like this @Darwin guy is going to just end up directing you to his website where he will collect your info in exchange for "giving" you his idea, or he will just flat out sell it and will have used this entire thread as a ruse to market his product.

In 5 years of running the forum, I've never seen anything other than this happen from a thread that started like this, where a user who registers for first time posts such a thing, and spams the same identical messages on multiple message boards simultaneously.

So I am done with this thread. Everything he posted about has already been previously discussed on futures.io (formerly BMT). I am closing this thread.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.


Closed Thread



futures io > > > > Walk Forward Analysis - the only logical successor to backtesting [DISCUSS]

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Linda Bradford Raschke: Reading The Tape

Elite only

Adam Grimes: TBA

Elite only

NinjaTrader: TBA

January

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Is Walk Forward Optimization always a must? Albnd Traders Hideout 15 October 28th, 2016 10:00 AM
Walk Forward Optimizer haonkered TradeStation 4 April 16th, 2014 04:03 AM
Walk Forward Experiment Silver Dragon Elite Automated Trading 327 September 9th, 2012 12:56 AM
Include ATM parameters in optimization / walk forward analysis? Krislassen NinjaTrader Programming 3 April 14th, 2012 01:12 PM
Walk Forward caprica NinjaTrader 4 August 31st, 2009 06:51 PM


All times are GMT -4. The time now is 09:57 PM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-12 in 0.20 seconds with 19 queries on phoenix via your IP 54.221.93.187