I am completely new to using NT software and I was running through some youtube videos to try and get a hang of programming. I came across a video made by Big Mike where he programs a simple strategy. He then backtests the strategy and it proves to be profitable. However he says not to bother with this strategy, as it is not actually profitable in real practice. How can this be? I know some indicators can have lagging signals, but wouldnt running a backtest also take the lagging signals into consideration? Do lagging signals even matter in this case or can successfully-backtested strategies not be profitable for other reasons? Thanks for the explanation in advance.
Can you help answer these questions from other members on futures io?
Other factors are that fills and slippage are not accounted for correctly in most backtests. The strategy may report an order filled at the price determined by the strategy but in reality, the order may not get filled (if limit order) or may slip against the trade by a tick or more thereby invalidating the strategy results. So you are looking at trades that the strategy reports that a trade entry got filled but in reality you may not get the fill or it may report that a trade exit got filled but in reality it may not etc. You could set some backtesting software to report fills only if the entry price is pierced but that leads to the same result - price may not pierce but in reality you could get filled etc.
Also consider being filled...on Tradestation there is an option to add slippage and a number or percentage of shares before being filled. This would provide a little more real world analysis.
If the backtest was preceded by an optimization process to select strategy parameters, then over fitting is quite likely a reason why backtest results would not be at all reflective of live results.
Volatility of price action is not recorded in the data used for backtesting. Wild swings up or down can trigger many signals in live trading, but these same signals may not fire at all in backtesting.
Considering this, I do find decent backtesting with Multicharts if you set the option to use tick data. It does not work on specialty bars like renko or line break and other specialty bars. But I do find it to be very good on time and tick based charts.
Well if the data being used for backtesting only has the open, close, low and high data, then the buy and sell signals will be totally different than live tick by tick data .
In another scenario, a strategy that uses volume rate of change, would not trigger buy and sell signals in backtesting in the same way as live data.
Also, volatility expansion type signals will fire on live data much differently than backtesting.
A macd crossover on a daily chart would not matter much I suppose.
Site Administrator Swing Trader Data Scientist & DevOps
Manta, Ecuador
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
Posts: 49,790 since Jun 2009
Thanks: 32,314 given,
97,615
received
I am saying ...
This is false.
This is false.
So use tick data instead of minute or daily data. It's simple, if you want more accuracy then increase the resolution of your data.
This is false.
False.
My point being, I don't agree with anything you've said. Now anyone that reads my posts know that I routinely talk about all the pitfalls of backtesting, so don't misconstrue my meaning. But what you've described here is not accurate, not unless you are purposely making it inaccurate by using minute or daily data.
Now that you have negated everything I have said about backtesting, can you offer an answer for the original question that was posted ?
The original poster wanted to know why a strategy that backtested good should not be used in live trading ?
He was referring to a video that you made about automated strategies.
Site Administrator Swing Trader Data Scientist & DevOps
Manta, Ecuador
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
Posts: 49,790 since Jun 2009
Thanks: 32,314 given,
97,615
received
There are many reasons. I suggest watching all the webinars in the archive about automated trading, they answer the question and provide many hours of detail, something I am not going to do right now in this post.
The past has occurred and reviewing it is of itself a non-real "constant". Back testing is vastly different than the live market.
Who really knows what's gona happen on the right hand side of the chart - if you did this game would be over.
We can therefore only hypothesize based on past behavior with some best/educated "guess" indication of direction based on volume, price, orders flow, news, etc... In back testing you know what's gona happen.
Further more what something (notice I did not say just someone) categorizes as a "buy" may and is in fact justifiably a "sell" from another point of view (someone buys from a seller and someone sells to a buyer) - think of time frames, chart types, fractals...it absolutely a multidimensional dynamic environment. Again in back testing this is fixed.
While it may on the onset appear to be similar to some form of higher math, curve fit, 5th order differential equation, etc...the fact is movement still relies some one/thing/computer to determine it's time to pull the trigger. The entire system "can" and does change state instantly. Watch it, you will see things just happen. For example someone with a large interest needs cash and does not care, they just want out - they sell a sload, what happens? My view is how fast can you react when you see these things. These occurrences don't just pop up in back testing.
Now think about the word "instantly"... with todays computers and networks, things happen so fast that a booked limit order placed in Chicago on a local hosted server running on a direct 10gig connection to the CME can fill before you on your DSL line in Los Angeles even know it was booked. Depending on your back testing data granularity, you may or may not see things at this sub-second level.
Very importantly, there is never the same exact players in the market at the exact same levels, at the exact same times - ever. In back testing this aspect is a constant.
All of this is why something that works in back testing does not work in the live market as it did in back testing.
This does not mean that back testing is bad, just that nothing in this game is ever exactly the same, the only constant is change.
About the discussion, a had not the time to whatch the webinars yet, but one thing tha puzzles me and I never read any coment about, but found ruge diference between life and backtest results acording to the data you use on the backtest. I got huge diference using diferent data sources and another huge diference in results using data adjusted for dividends a data not adjusted.
I think when you use a adjusted data, the result of your backtest reflects two efects: it's like you received all the duvidends and the price is distorted.
For a backtest to be reliable it need to simulate live trading conditions.
The simplest example I can give is the use of a moving avarage as some kind of resistance.
Let's say you want to buy when prices are above the MA and it touchs the same MA. The buy price will be at the exactly point where price touch the MA.
In such conditions, if you tell the back test to calculate "previous low > MA AND actual low <= MA", the buy price registered in the backtest will be out of reality. You must tell the system to estimate the exactly point where the MA will be when price touch it.
Broker: AMP/NinjaTrader/CQG/Kinetick/Tickdata, Inc.
Trading: ES. ZB
Posts: 7 since Jul 2010
Thanks: 4 given,
8
received
No mechanical system--totally objective, will be profitable in the future.
They may "work" for a while and you'll think you are going to be rich.
(They would be profitable for longer in the past--1980s and 90s.)
Think that's not true? Then name one.
By the way, why are so many systems offered for sale versus actually trading them and getting rich?
Excalibur Testing Software by Futures Truth was one of the first full-featured backtesting platforms. The software development process started in 1985 on a CROMEMCO 3 computer utilizing FORTRAN IV. The computer had a whopping 64k of RAM and used 5-1/4-inch floppy drives. Trader John Hill spent nearly $20,000 for the computer and hard drive — an enormous sum for a machine with the fraction of the computing power available today for a few hundred bucks.
Wayne Andrews, a close friend of Hill’s and a computer scientist, was instrumental in helping Hill acquire the CROMEMCO, as well as providing his own personal tick level price data for Hill to use.
I do have a little experience.
The following 2 users say Thank You to wcandrews for this post:
I'd agree completely with your first sentence, if you add the word "forever" to the end of it. Mechanical systems can work for a good amount of time before they don't.
Your point is a very good one - traders should assume that at some point their strategy will stop working, and build risk management / position sizing etc. with that in mind. That way, they will avoid being wiped out by a strategy that stops working.
Some of the worst blowouts I have heard about were from traders who stubbornly held on to a method, when the market was telling them the method no longer worked.
The following user says Thank You to kevinkdog for this post:
I believe it is because most are developed the wrong way. Too many rules, too much optimizing, improper data sampling and data mining, improper accounting of slippage and commission, bad assumptions about order fills,etc. Garbage In, Garbage Out as they say.
The end result is a great backtest that can sell the system to the unwary public, but will never work in the real world.
The following 2 users say Thank You to kevinkdog for this post:
Broker: AMP/NinjaTrader/CQG/Kinetick/Tickdata, Inc.
Trading: ES. ZB
Posts: 7 since Jul 2010
Thanks: 4 given,
8
received
"The end result is a great backtest that can sell the system to the unwary public, but will never work in the real world. "
Very true. Years ago there used to be a "Club 3000." Systems used to cost $3000. Big Mike probably knows what I'm
talking about. The club critiqued the various scams, oops, I meant systems.
I am 72 and several years ago I finally figured out the "secret"--it's all about money (and also power).
Apparently there is an endless supply of "marks."
Sorry for the cynicism.
The following 2 users say Thank You to wcandrews for this post:
For what my view is worth, I think in the context of the poster's question, this is the most accurate answer. The issues experienced lie mostly with the software.
The following 4 users say Thank You to artemiso for this post:
These are situations I have observed during live trading.
These are conclusions I have come to after comparing backtesting with live results. I observed more slippage at certain times than others.
The exchanges use sophisticated algorithms to fill all of the orders and this can push the orders around wildly in the lineup.
Do you think tick data being backtested demonstrates accurately how those orders are being filled ?
I dont think it does.
