Why backtests are useless, EAs are flawed and their parameters are bad [DISCUSS] - Currency Futures | futures io social day trading
futures io futures trading


Why backtests are useless, EAs are flawed and their parameters are bad [DISCUSS]
Updated: Views / Replies:1,924 / 14
Created: by Darwin Attachments:0

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Closed Thread
 
Thread Tools Search this Thread
 

Why backtests are useless, EAs are flawed and their parameters are bad [DISCUSS]

  #1 (permalink)
Trading Apprentice
Regensburg, Germany
 
Futures Experience: Advanced
Platform: MetaTrader
Favorite Futures: Forex
 
Darwin's Avatar
 
Posts: 9 since Oct 2013
Thanks: 0 given, 3 received

Why backtests are useless, EAs are flawed and their parameters are bad [DISCUSS]

Hi, I am Darwin and I am new here.

tl;dr In the next weeks, I am planing to releasy my private Walk Forward Analyzer, but before I do this, first I want to make sure that also the beginners in mechanical trading understand why common backtests are quite useless and why walk forward analysis is better in so many ways, so I decided to start this discussion.

I WILL RELEASE THIS TOOL FOR FREE AND OPEN SOURCE!

And as people keep asking why I release free stuff, let me explain my motivations: I am targeting a job in the trading-economy, and this is not easy to get. Beeing known in the community would help me a lot to accomplish this goal, so I decided to release some of my private tools for free and/or open source to make myself a name over the next months. I am not here to sell you stuff, keep that in mind, please!


For the record: I do not claim that WFA is the holy grail in system testing, but I argue that it is way better than normal backtests.



THE GENERAL APPROACH TO SYSTEMATIC TRADING

Take a trading logic => Generate an EA that trades exactly like it => Optimise Parameters => Do a backtest => Trade it live => $$$

BUT: Everybody that has ever had an EA that was a money-printing machine in the backtests (and this is very, very easy to be done), knows that a good backtest does not imply profitable live trading.
And that is the problem. We need a test-method on whichs evaluation we can rely!




STATEMENT 1: Your parameter choices are not good

Well, every EA consists of 2 things:

1. the logic/script itself
2. the parameters (like periods for moving averages, stop loss values etc. Just everything that can be adjusted!)

The first thing is static and "given". And a lot traders only focus on this part.
But, an EA can behave in very different ways, depending on the parameters. And there can be billions of possible parameter-combinations (=parameterset). There are 3 approaches to determine the parameterset for live trading.

1. the intuitive approach (non adapting):
The trader just chooses the parameters based on his expert knowledge.
But, there are just too many possible parametersets. You can't just "guess" them without evaluation and testing.
It does not matter how good you are, you are never good enough to "think through" this huge amount of possibilities in a reliable fashion!

2. the "optimise on all data" approach (non adapting):
The trader chooses the parameters based on all past data.
So, the configuration/parameters that worked best for the last 12 years, for example, are chosen to be traded live.
But, to stay up-to-date it is not a very good method to optimise the parameters on so much and therefore OLD data.
Because the market today is not the same as it was 12 years ago.

3. the "optimise on the last few years" approach:
The trader chooses the parameters based on the last few years of data.
But you can not just take this approach without testing how this "optimisation method" would have worked in the past.
And this is exactly what a walk forward analysis will do: "Optimise on the last few years", but it tests this approach on all data in the past!




STATEMENT 2: A non-adapting EA can never make longterm profits

I mean, there is an infinite amount of possible trading-systems AND an infinite amount of ways the market can change AND an "infinite" amount of possible parameters for your trading-system.

Considering this, would you really want to bet money on the fact that you have a trading system that will always work, in the same way, without adaption, on all future market conditions? I would not.

"But why shouldn't I be able to put a traders knowledge into a script and trade it?"
Its simple: A trader always learns, he takes input from many sources, he has knowledge about the markets and therefore he adapts his trading strategy so he can always be as close to the markets as possible.
A simple EA script can't do this, a Walk Forward Analysis can (to some extent).

Reason #1 why there are many "profitable" EAs out there
The EA really works, it has a sound strategy and was developed properly. But the markets always change, and they can change in infinite ways.
So, at some point, the traded market-inefficiency WILL(!) change, and the EA can not adapt to this change and therefore will lose its profitability.

Reason #2 why there are many "profitable" EAs out there
is that some of them, sound or unsound systems, are just lucky.
If you send 10.000 people to a casino and let them play for a while, just due to chance, some of them will make profit over some time.
The same is for EAs, if there are enough of them, some will really make profit (even if they are in fact useless).
But as they keep trading, they will lose, as the probabilities are against them (same as in a casino).

Reason #3 why there are many "profitable" EAs out there
is the huge risk some EAs take (grid trading, martingale systems etc).
These EAs take very huge risks to make small profits. But somewhen, the risk strikes, and at that point, the EA will to lose (all) money.

Reason #4 why there are many "profitable" EAs out there
is the small timespan on which they are profitable. Its no magic to make profit for months, but its hard to make it for many, many years.




STATEMENT 3: A backtest does not tell you anything about the future performance

Well, don't get me wrong, I strongly hold the opinion that simulations on the past are the only way to really test a trading system.
But what does a backtest tell you? Just that your system performed well on the past.

But is trading really about having a system that represents the past?
No, trading is about designing a system on the past and then trade it in the future.

THIS IS A FUNDAMENTAL DIFFERENT QUESTION THAN WHAT A BACKTEST CAN TELL YOU!
You want to answer the question "how good is my live trading performance" with the answer to the question "how good did my system perform in the past". That logic is flawed, of course.

"But if I use out-of-sample data to verify my backtest..."
Out of sample testing is a good idea. But you only have one optimisation-dataset and one test-dataset, which is not very reliable. Walk forward analysis, somehow, is out-of-sample testing on steroids. It uses the same method, but generates 10-1000 opti/test dataset-pairs.

"So, are all backtests useless?"
No, it can help you to get sound and good trading systems, if done right.
But my point is that a good performance in a backtest does not make sure the system is also good in live trading.

Trading is a game of probabilities, and the probability that an EA is profitable, only based on a good backtest, is very, very low.

"But why should a system that performed good over the past simply stop working"
The main reason is, and that might be the most important one, OVERFITTING.
That means, the EA tackles an "random" inefficiency of the market that was just there in the past, but that was no real thing.

I found a very good picture that explains overfitting:

<IE MARKETSHARE VS MURDER RATE> (can not post images yet, here it is: i.imgur.com/CIqiSuF.jpg)



You see? There are infinite of such useless "relationships" in the price data, and most EAs tackle one of them which gives them very good backtest results, but that was just due to luck, and these "relationships" will not hold in the past.
And as there are a lot more of these overfitted solutions than sound ones, the chances are very high that you optimised towards such an overfitted system.

The other reason is that the markets always change. So even if you have a non-overfitted, good performing backtest, you will run into serveral problems that you can read in Statement 2



THE SOLUTION: WALK FORWARD ANALYSIS

A Walk Forward Analyser is an external program that can take EVERY EXPERT ADVISOR and, using Metatrader4, do the analysis.
SO YES; YOU CAN CONTINUE TO USE YOUR ALREADY WRITTEN EAs!

Well, some of you will already know how a walk forward analysis works. For the others here is a short overview, as I will explain the process in more details in a future thread.

A WFA takes, for example, the data from 2000 to optimise your system, then tests it on 2001 (which is, from the point of optimisation, the "future" or "live trading").
Then it walks forward and optimises the system on 2001, test it on 2002. Then optimise on 2002, test on 2003 etc.
Do this until you walked through your whole data and only consider the "live trading" in your evaluation.

You see? With this simple tactic you can tackle the 3 problems I have described above, the lack of adaptability, the need for a evaluated parameter selection process and the uselessness of a "past-performance"-backtest.


And then, when you want to trade the system live, you do the exact same thing that you have tested:
1) Optimise on the last available data, with the same optimisation settings you tested in WFA
2) Trade it in the future, with the same procedure you tested in WFA

And that is the main difference to normal backtests: YOU TRADE YOUR EA IN THE EXACT SAME WAY AS YOU TESTED IT!!!



So, this is what I wanted to say, now feel free to tell me what you think on this topic, please
- Darwin

 
  #2 (permalink)
Quick Summary
Quick Summary Post

Quick Summary is created and edited by users like you... Add FAQ's, Links and other Relevant Information by clicking the edit button in the lower right hand corner of this message.

 
  #3 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,240 since Jun 2009
Thanks: 29,356 given, 83,237 received


@Darwin,

Please confirm you do not have a website associated with trading where any product or service is for sale, nor plan to have a website like this in the future. Please also confirm you are not associated with any other vendor or provider of trading related products or services in any way.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

 
  #4 (permalink)
Elite Member
Alesia E.U.
 
Futures Experience: Intermediate
Platform: Sierra chart
Favorite Futures: Futures
 
Posts: 166 since Feb 2011
Thanks: 139 given, 109 received

Hi

Can your system do the walk?

The following 2 users say Thank You to crazybears for this post:
 
  #5 (permalink)
Trading Apprentice
Regensburg, Germany
 
Futures Experience: Advanced
Platform: MetaTrader
Favorite Futures: Forex
 
Darwin's Avatar
 
Posts: 9 since Oct 2013
Thanks: 0 given, 3 received


Big Mike View Post
@Darwin,
Please confirm you do not have a website associated with trading where any product or service is for sale, nor plan to have a website like this in the future. Please also confirm you are not associated with any other vendor or provider of trading related products or services in any way.

Hello Mike. Well yes, I can confirm I have no website (and no plans for a website) where I sell anything. [tough, perhaps I make one where I can upload my FREE stuff. So website - perhaps. Sell stuff - not planned!]
Also, I am not associated with ANY vendor or ANY forex related website/product/service/company.



@crazybears:

A few words about why you postet that link would be nice

This article is about WFA, yes, but it is not a very good one and not one that convinces people that they want WFA instead of backtests. (which is the purpose of my post).

I will try to explain the WFA procedure a bit more intuitive and easier in a future thread. But until then, I just want to make clear why WFA is much better than a backtest.


- Darwin

 
  #6 (permalink)
Elite Member
Alesia E.U.
 
Futures Experience: Intermediate
Platform: Sierra chart
Favorite Futures: Futures
 
Posts: 166 since Feb 2011
Thanks: 139 given, 109 received

hi

only to add some very basic info about the topic .

I'd like to see more info about WFA , i made some TS and made WFA with AB and MC but wasn't able to made them work

 
  #7 (permalink)
Elite Member
Alesia E.U.
 
Futures Experience: Intermediate
Platform: Sierra chart
Favorite Futures: Futures
 
Posts: 166 since Feb 2011
Thanks: 139 given, 109 received

read also this book :
Amazon.com: Design, Testing, and Optimization of Trading Systems (Wiley Trader&#39;s Exchange) eBook: Robert Pardo: Kindle Store
i hope you can put new insights about WFA , and SC team add it

 
  #8 (permalink)
Trading Apprentice
Regensburg, Germany
 
Futures Experience: Advanced
Platform: MetaTrader
Favorite Futures: Forex
 
Darwin's Avatar
 
Posts: 9 since Oct 2013
Thanks: 0 given, 3 received


crazybears View Post
hi

only to add some very basic info about the topic .

I'd like to see more info about WFA , i made some TS and made WFA with AB and MC but wasn't able to made them work

Hey. I will make a new thread, before releasing my tool, that will explain how a WFA is done. Tough, its not the topic of this one

TS=Trading System; MC=Monte Carlo; ?
What is AB?

"Wasn't able to make them work"

Well, as long as you have a standardt EA, it will be just a few clicks to make a walk forward analysis with my tool.
So this sould not be a problem soon

To your Book suggestion:
I do not think there is any technique in any book that covers a good WFA, as a good WFA can only be done using program code, as there are up to 1000 optimisations and 100.000.000 backtests needed to get the whole picture. (I said UP TO, 100 optimisations and 100.000 backtests are more realistic).
Afterwards, this huge amount of data needs to be processed to find the ideal parameters...
So no human is able to do this in a reasonable time.

And I think a book that is only usable by coders would not be a bestseller xD

Btw: I am currently working on such an extended wfa algorithm, I will release it here in some months (free, yes. open source, i dont know) Stay tuned!


Last edited by Darwin; November 7th, 2013 at 12:33 PM.
The following user says Thank You to Darwin for this post:
 
  #9 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,240 since Jun 2009
Thanks: 29,356 given, 83,237 received

@Darwin,

Why have you chosen now to create accounts on trading forums and post this identical message to them?

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

 
  #10 (permalink)
Trading Apprentice
Regensburg, Germany
 
Futures Experience: Advanced
Platform: MetaTrader
Favorite Futures: Forex
 
Darwin's Avatar
 
Posts: 9 since Oct 2013
Thanks: 0 given, 3 received


Hi Mike

As I said, my target is to get known in the community (see the first few sentences of my first post).
I want them all to use my program. I want to go to the job interview, say "Hi, i am darwin" and they should be like "ah cool, we know your tools" - thats what it is all about

-Darwin


Closed Thread



futures io > > > > Why backtests are useless, EAs are flawed and their parameters are bad [DISCUSS]

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Jigsaw Trading: TBA

Elite only

FuturesTrader71: TBA

Elite only

NinjaTrader: TBA

Jan 18

RandBots: TBA

Jan 23

GFF Brokers & CME Group: Futures & Bitcoin

Elite only

Adam Grimes: TBA

Elite only

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Is the DOM useless? PeterOhlson Traders Hideout 21 November 19th, 2013 10:46 AM
Bid / Ask spread in backtests Adfra Elite Automated NinjaTrader Trading 4 August 1st, 2013 10:19 AM
The World's Flawed Business Model .. kbit News and Current Events 0 November 17th, 2011 10:05 PM
Why Getting Rid of Useless Regulations May Not Be Easy Quick Summary News and Current Events 0 January 22nd, 2011 10:10 PM
Backtests and live results way off lookOutBelow NinjaTrader 23 October 22nd, 2010 06:13 PM


All times are GMT -4. The time now is 08:52 AM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-18 in 0.19 seconds with 19 queries on phoenix via your IP 107.20.115.174