Caveat - I am a discretionary trader so what follows may be worthless:
I'm not familiar with the internal working of Multicharts and perhaps the detailed flow of the back-test engine is explained in the Help file but I couldn't find it.
So - Let's assume Multicharts runs back-testing serially, that is, it processes the data received from disk, halts the data, builds a bar or not, runs your "black-box" process, resumes retrieving data from the disk, and repeats until an order is generated.
Further assume I'm running on a slow machine and it takes Multcharts (MC) a full minute to process whatever I do, but the decision to buy or sell is based on the close being at the high of the three minute bar generated by MC. I'm making the assumption that when back-testing, MC serially processes the on-disk "tick" data, produces a 3 minute bar, stops the data stream, runs through the my holy grail process, and makes a buy "at-the-market" decision. So a buy decision exists. Then the MC back-test engine allows the next stored tick(s) to be obtained from the historic back-test file, builds a new 3 minute bar, and then buys on the first available tick which is the open of the newly-built three minute bar. So I always buy on the first tick after processing is complete. There is no latency between the signal to buy and the buy.
In the real world the decision is made some time after the "decision" bar has been generated, no matter how fast the machine or how elegant the code. I don't care how quickly or slowly MC runs, the execution of the buy in real time will always be some finite, but variable time after the open of the next 3 minute bar, because the close-of-bar to open-of-next-bar is just one trade away and in a fast market that is milliseconds. As others have pointed out there is a great deal of latency involved starting with the exchange data source to the the broker, to your computer, the computer process, back to your broker and to the exchange order processing engine. Thus, back-testing results will always be different than any real time results because no matter what I do I cannot consistently get the opening tick of the next bar, but I always get the opening tick in back-testing. (An end-of-day trading signal to buy-on-the open is an exception to this deficiency).
So, in your back-testing code, you must include a mechanism to delay your buy or sell signal some (variable) period of time to more accurately simulate your system's results. You'd then disable the delay when running real time against a broker's live simulator or live account.
Obviously, when dealing with the futures market, limit orders outside the market would be placed in a queue, further complicating back-testing.
Thanks for your comments, Jack. You're basically agreeing with what Khalaad said in Post#58. And you're both right.
But I still want the basic problem of different backtesting results with the SAME code, one period of time being done 'live', and then the same period of time done historically, with the same code merely being re-compiled, to be fixed. The results from each should agree.
This is a MultiCharts (/TradeStation) problem -- not mine. I don't promote Bar Magnification, etc. MultiCharts does. This problem needs to be fixed so that their platforms are indeed credible.
I realize I'm coming down hard on MultiCharts. I am doing so only to try to force them to see this problem. I'm sure they already know it. Perhaps it's not correctable. But I would think that if data is received tick by tick, then that same data can be sent tick by tick so that programs can work in an exactly sequential manner. That's all I want. And if programs work in an exactly sequential manner, then I think that the historical backtesting results should be exactly equal to the 'live' testing results. (Note: I'm NOT saying Live Trading results.)
If the two periods can then exactly agree, then 'workarounds' for Real Live Trading can then be programmed into the code. This is what I'm after.
I think I understand what you want (above). What does "live" mean to you? How would Multicharts(MC) "know" the delay caused by your connection latency, where your order would be in a queue, and whether you could be filled based on the bid/offer size at the time of your orders arrival at the exchange.
As I posted, I don't know how the MC back-testing engine works. However, and once again I probably don't know what I'm saying - MC needs three processes (threads?) running simultaneously, one that reads the time-stamped tick data stored on your disk and feeds it to another thread running your "holy grail" process. Thus the signal produced by your "system" would be traded against a constantly running data stream, making it more realistic. Further, the back-testing engine would need a "latency" input feature so that after your grail processed the data, the third thread, the "broker/exchange execution" process would have its execution delayed by the time it would take for your order to be transmitted to your broker, processed by your broker, and then forwarded to the exchange. That latency factor would be unique (and probably jitter a bit) for each user and broker. For a reversal system, MC would need to have a user adjustable order acknowledged/completed latency factor available so that your system could not reverse "illegally."
Once MC worked out that process, they'd need to scale everything appropriately, so that back-testing could run at a high speed, rather than in the actual elapsed time of the stored tick stream.
Thanks for your input, Jack. But MultiCharts doesn't have to simulate anything "live". Just take the data, tick by tick, and send it to me, tick by tick. With that, * I * can do the experimenting to convert the data to Real Time Live conditions. That operation is a big guess. The tick-by-tick stuff is real. And that is NOT what I am getting. The different results that I get between the so-called 'live' quotes I receive when watching the program work in a simulated mode, and then closing and reloading (or re-compiling) the same program with supposedly the same data is what tells me that I'm NOT getting tick-by-tick data as it occurs throughout the day and for the historical backtesting.
I guess I'm having a difficult time making myself clear on this. I can't seem to put it in a more understandable way (yet). Maybe tomorrow I'll be able to. I'm off to bed.
I understand that you don't want to share your strategy and I feel for your backtesting problems, I can't imagine how frustrating these have to be. That being said, can you perhaps post an 'cleaned out' example of your strategy (with order logic) and Data1 and Data2 references?
If SPTrading's suggestion is right, and there is some lookahead bias, we can help you find a way around this, and I also suppose that that also helps to get your point stronger across to MultiCharts. This would also open the possibility to further investigate this issue, because, if you experience wildly different backtest results when changing a semicolon and have a big gap between backtest results and real-time results, this is a hugely serious issue, which I think also the community should know about, and should be put forth as concrete as possible, so that MultiCharts can fix this as soon as possible.
I'm not saying you're wrong in your criticism, but the discussion in this topic is getting somewhat abstract with our assumptions about MultiCharts' backtest assumptions (also see page 199 and beyond of the MultiCharts manual for their take on this) - not that that shouldn't be discussed, but that won't allow for much concrete help or problem analysis.
Just thinking aloud.
The following user says Thank You to Jura for this post:
One possible reason for differences between live trading results and historical results (or just reloading the data in the same day) is that your tick data is no longer accurate or kept in exact sequential order. Open up the quotemanager and examine (edit data) for yesterdays trading. There should be a tickid for every tick within one second and the number should be in consecutive order. CME only timestamps the tick data down to the second and NOT millisecond or microsecond. So data providers assign a number to every tick that happens in the same second, that way it is kept in exact sequence. Examine a time period that is off hours and activity is slow. The tickid should be in exact sequential order with no skips in numbers (starts over every new second). Now examine the first second of market open action (8:30 exchange time), there will probably be skips in the tick id, but the main thing is that they are in sequential order. Do a free trial to DTN IQFeed. and compare your tick data to theirs. I know their data is recorded almost perfectly and then kept in the exact sequence. If your stored data is much different then what they are showing, then you have one huge problem for backtesting.
The following 2 users say Thank You to tradersheldon for this post:
OK, Jura. Your post convinced me to post the basics of my code. I'll change a few things, but keep the logic the same. It's really very simple. It has 2 data series (and 2 graphs). Both graphs use 2-cent range bars. (It also works well with 40-second bars and 12-Tick bars [for CL here] The upper graph is the current month contract of CL or ES. The lower graph is a contract that is farther out. The signals are derived from the lower graph and all trading is done off of the upper graph, where the Bid/Ask spread is minimal.
I've gone through 2 expiration dates with this program. I change the contracts a set number of days before the actual expiration day. MultiCharts is also set for Bar Magnifier On with 1 Tick as the minimum move. IntraBar Order Generation is also set to On.
Here's an example of the code:
As of 9:40 AM Mountain Time, the CL results since Midnight is + 8718 Net on 367 R.T.'s. The ES results is + 2660 Net on 19 R.T.'s. I have not reloaded, reset, or re-compiled anything.
The following 2 users say Thank You to bobbakerr for this post: