NexusFi: Find Your Edge


Home Menu

 





How can a strategy be profitable when backtested but not when actually used?


Discussion in NinjaTrader

Updated
      Top Posters
    1. looks_one Big Mike with 5 posts (3 thanks)
    2. looks_two shanemcdonald28 with 5 posts (0 thanks)
    3. looks_3 kevinkdog with 4 posts (9 thanks)
    4. looks_4 sra18376 with 2 posts (0 thanks)
      Best Posters
    1. looks_one artemiso with 4 thanks per post
    2. looks_two kevinkdog with 2.3 thanks per post
    3. looks_3 wcandrews with 2 thanks per post
    4. looks_4 Big Mike with 0.6 thanks per post
    1. trending_up 7,618 views
    2. thumb_up 22 thanks given
    3. group 10 followers
    1. forum 27 posts
    2. attach_file 0 attachments




 
Search this Thread

How can a strategy be profitable when backtested but not when actually used?

  #11 (permalink)
 
Big Mike's Avatar
 Big Mike 
Manta, Ecuador
Site Administrator
Developer
Swing Trader
 
Experience: Advanced
Platform: Custom solution
Broker: IBKR
Trading: Stocks & Futures
Frequency: Every few days
Duration: Weeks
Posts: 50,399 since Jun 2009
Thanks Given: 33,175
Thanks Received: 101,541


shanemcdonald28 View Post
quote "huh"


Well if the data being used for backtesting only has the open, close, low and high data, then the buy and sell signals will be totally different than live tick by tick data .

In another scenario, a strategy that uses volume rate of change, would not trigger buy and sell signals in backtesting in the same way as live data.

Also, volatility expansion type signals will fire on live data much differently than backtesting.

A macd crossover on a daily chart would not matter much I suppose.

This is false.

All of the above work fine with backtesting. Simply use tick data.

Mike

We're here to help: just ask the community or contact our Help Desk

Quick Links: Change your Username or Register as a Vendor
Searching for trading reviews? Review this list
Lifetime Elite Membership: Sign-up for only $149 USD
Exclusive money saving offers from our Site Sponsors: Browse Offers
Report problems with the site: Using the NexusFi changelog thread
Follow me on Twitter Visit my NexusFi Trade Journal Reply With Quote

Can you help answer these questions
from other members on NexusFi?
The space time continuum and the dynamics of a financial …
Emini and Emicro Index
Futures True Range Report
The Elite Circle
Ninja Mobile Trader VPS (ninjamobiletrader.com)
Trading Reviews and Vendors
Build trailing stop for micro index(s)
Psychology and Money Management
My NT8 Volume Profile Split by Asian/Euro/Open
NinjaTrader
 
Best Threads (Most Thanked)
in the last 7 days on NexusFi
Get funded firms 2023/2024 - Any recommendations or word …
60 thanks
Funded Trader platforms
37 thanks
NexusFi site changelog and issues/problem reporting
24 thanks
GFIs1 1 DAX trade per day journal
22 thanks
The Program
19 thanks
  #12 (permalink)
shanemcdonald28
new york
 
Posts: 355 since Mar 2012
Thanks Given: 665
Thanks Received: 582

Are you saying that historical tick data and live tick data behave the same ?

Reply With Quote
  #13 (permalink)
 
Big Mike's Avatar
 Big Mike 
Manta, Ecuador
Site Administrator
Developer
Swing Trader
 
Experience: Advanced
Platform: Custom solution
Broker: IBKR
Trading: Stocks & Futures
Frequency: Every few days
Duration: Weeks
Posts: 50,399 since Jun 2009
Thanks Given: 33,175
Thanks Received: 101,541



shanemcdonald28 View Post
Are you saying that historical tick data and live tick data behave the same ?

I am saying ...


shanemcdonald28 View Post
Volatility of price action is not recorded in the data used for backtesting.

This is false.


shanemcdonald28 View Post
Wild swings up or down can trigger many signals in live trading, but these same signals may not fire at all in backtesting.

This is false.


shanemcdonald28 View Post
Well if the data being used for backtesting only has the open, close, low and high data, then the buy and sell signals will be totally different than live tick by tick data .

So use tick data instead of minute or daily data. It's simple, if you want more accuracy then increase the resolution of your data.


shanemcdonald28 View Post
In another scenario, a strategy that uses volume rate of change, would not trigger buy and sell signals in backtesting in the same way as live data.

This is false.


shanemcdonald28 View Post
Also, volatility expansion type signals will fire on live data much differently than backtesting.

False.

My point being, I don't agree with anything you've said. Now anyone that reads my posts know that I routinely talk about all the pitfalls of backtesting, so don't misconstrue my meaning. But what you've described here is not accurate, not unless you are purposely making it inaccurate by using minute or daily data.

Mike

We're here to help: just ask the community or contact our Help Desk

Quick Links: Change your Username or Register as a Vendor
Searching for trading reviews? Review this list
Lifetime Elite Membership: Sign-up for only $149 USD
Exclusive money saving offers from our Site Sponsors: Browse Offers
Report problems with the site: Using the NexusFi changelog thread
Follow me on Twitter Visit my NexusFi Trade Journal Reply With Quote
Thanked by:
  #14 (permalink)
shanemcdonald28
new york
 
Posts: 355 since Mar 2012
Thanks Given: 665
Thanks Received: 582

Ok, sounds good.

Now that you have negated everything I have said about backtesting, can you offer an answer for the original question that was posted ?

The original poster wanted to know why a strategy that backtested good should not be used in live trading ?
He was referring to a video that you made about automated strategies.

Reply With Quote
  #15 (permalink)
 
DarkPoolTrading's Avatar
 DarkPoolTrading   is a Vendor
 
Posts: 1,036 since May 2012
Thanks Given: 1,244
Thanks Received: 1,326


shanemcdonald28 View Post
The original poster wanted to know why a strategy that backtested good should not be used in live trading ?
He was referring to a video that you made about automated strategies.

Refer to the below webinar:

5 Ways to incorrectly build a system

Diversification is the only free lunch
Follow me on Twitter Reply With Quote
  #16 (permalink)
 
tturner86's Avatar
 tturner86 
Portland, Oregon
 
Experience: Intermediate
Platform: F-16CM-40
Trading: GBU-39
Posts: 6,191 since Sep 2013
Thanks Given: 10,459
Thanks Received: 12,695


shanemcdonald28 View Post
The original poster wanted to know why a strategy that backtested good should not be used in live trading ?
He was referring to a video that you made about automated strategies.

Done.


tturner86 View Post
Past performance is not indicative of future results.


kevinkdog View Post
There are ways to trick the backtest software. Also, if you heavily optimize, your results will never be the same going forward.


Hulk View Post
Other factors are that fills and slippage are not accounted for correctly in most backtests. The strategy may report an order filled at the price determined by the strategy but in reality, the order may not get filled (if limit order) or may slip against the trade by a tick or more thereby invalidating the strategy results. So you are looking at trades that the strategy reports that a trade entry got filled but in reality you may not get the fill or it may report that a trade exit got filled but in reality it may not etc. You could set some backtesting software to report fills only if the entry price is pierced but that leads to the same result - price may not pierce but in reality you could get filled etc.


Visit my NexusFi Trade Journal Reply With Quote
  #17 (permalink)
 
Big Mike's Avatar
 Big Mike 
Manta, Ecuador
Site Administrator
Developer
Swing Trader
 
Experience: Advanced
Platform: Custom solution
Broker: IBKR
Trading: Stocks & Futures
Frequency: Every few days
Duration: Weeks
Posts: 50,399 since Jun 2009
Thanks Given: 33,175
Thanks Received: 101,541


shanemcdonald28 View Post
The original poster wanted to know why a strategy that backtested good should not be used in live trading ?
He was referring to a video that you made about automated strategies.

There are many reasons. I suggest watching all the webinars in the archive about automated trading, they answer the question and provide many hours of detail, something I am not going to do right now in this post.

Mike

We're here to help: just ask the community or contact our Help Desk

Quick Links: Change your Username or Register as a Vendor
Searching for trading reviews? Review this list
Lifetime Elite Membership: Sign-up for only $149 USD
Exclusive money saving offers from our Site Sponsors: Browse Offers
Report problems with the site: Using the NexusFi changelog thread
Follow me on Twitter Visit my NexusFi Trade Journal Reply With Quote
  #18 (permalink)
 tulanch 
Salt Lake City, UT
 
Experience: Intermediate
Platform: SC, NT, MT
Broker: AMP
Trading: NQ ES YM Bonds
Posts: 265 since Mar 2010
Thanks Given: 50
Thanks Received: 386

The past has occurred and reviewing it is of itself a non-real "constant". Back testing is vastly different than the live market.

Who really knows what's gona happen on the right hand side of the chart - if you did this game would be over.

We can therefore only hypothesize based on past behavior with some best/educated "guess" indication of direction based on volume, price, orders flow, news, etc... In back testing you know what's gona happen.

Further more what something (notice I did not say just someone) categorizes as a "buy" may and is in fact justifiably a "sell" from another point of view (someone buys from a seller and someone sells to a buyer) - think of time frames, chart types, fractals...it absolutely a multidimensional dynamic environment. Again in back testing this is fixed.

While it may on the onset appear to be similar to some form of higher math, curve fit, 5th order differential equation, etc...the fact is movement still relies some one/thing/computer to determine it's time to pull the trigger. The entire system "can" and does change state instantly. Watch it, you will see things just happen. For example someone with a large interest needs cash and does not care, they just want out - they sell a sload, what happens? My view is how fast can you react when you see these things. These occurrences don't just pop up in back testing.

Now think about the word "instantly"... with todays computers and networks, things happen so fast that a booked limit order placed in Chicago on a local hosted server running on a direct 10gig connection to the CME can fill before you on your DSL line in Los Angeles even know it was booked. Depending on your back testing data granularity, you may or may not see things at this sub-second level.

Very importantly, there is never the same exact players in the market at the exact same levels, at the exact same times - ever. In back testing this aspect is a constant.

All of this is why something that works in back testing does not work in the live market as it did in back testing.

This does not mean that back testing is bad, just that nothing in this game is ever exactly the same, the only constant is change.

Reply With Quote
  #19 (permalink)
 ecredic 
Lauro de Freitas, Bahia, Brasi
 
Experience: Intermediate
Platform: HomeBroker and Amibroker
Broker: Rico.com.vc/Tradezone
Trading: Bonds, Stocks, Commodity Futures
Posts: 6 since Dec 2010
Thanks Given: 37
Thanks Received: 3

HI Big Mike and folks.

Long time out.

About the discussion, a had not the time to whatch the webinars yet, but one thing tha puzzles me and I never read any coment about, but found ruge diference between life and backtest results acording to the data you use on the backtest. I got huge diference using diferent data sources and another huge diference in results using data adjusted for dividends a data not adjusted.

I think when you use a adjusted data, the result of your backtest reflects two efects: it's like you received all the duvidends and the price is distorted.

Follow me on Twitter Reply With Quote
  #20 (permalink)
Momozo
fortaleza + ceara/brazil
 
Posts: 23 since Jun 2014
Thanks Given: 26
Thanks Received: 4


For a backtest to be reliable it need to simulate live trading conditions.

The simplest example I can give is the use of a moving avarage as some kind of resistance.

Let's say you want to buy when prices are above the MA and it touchs the same MA. The buy price will be at the exactly point where price touch the MA.

In such conditions, if you tell the back test to calculate "previous low > MA AND actual low <= MA", the buy price registered in the backtest will be out of reality. You must tell the system to estimate the exactly point where the MA will be when price touch it.

Then it would be a reliable backtest.

Reply With Quote




Last Updated on July 14, 2014


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts