I have a question to ask someone familiar with the internals of NT7. I have read where they are storing bid, ask, and last tick in separate databases with a resolution of “One second.” If this is true it seems impossible to build historical data in the proper sequence. Seems like MC and NT can’t seem to get their act together regarding this vital piece of information.
You son't get an answer here. That said, remember that there are also market replay files that they already creaste from 6.5 upward. I can not imagine hey store things with a 1 second resolution there - this would mean in a 1:1 time replay you get "blocks" of updates every second, which TOTALLY alters the feeling of something like tape reading and would - in the sense of being an epic idiotical programming thing - make market replay totally unusuble as training tool for many things. Heck, you possibly could not even see anything on development of the order book. So, I think they actually store a higher resolution at least there, but do not expose it via API.
Storing historical data in separate files for ticks, bid / ask and with a 1 second resolution would mean any stream reconstruction is simply impossible.
See my previous post with regards to NT timestamp resolution.
I don't use the historical data so can't comment on that but I know from discussions with the NT support team that market replay data is timestamped with 1 sec granularity when it is recorded. This becomes a massive problem because the L1 and L2 data streams are recorded into separate files. In doing this the original sequencing of the L1 and L2 data messages is permanently lost. When the market replay engine comes to play the data back the only thing it has to work with when trying to intersplice the L1 and L2 data is the timestamps. But because these have only 1 sec resolution the engine replays ALL the L1 data for a particular 1 second period first and then replays ALL the L2 data for the particular 1 second period. The data is not correctly interspliced.
You probably won't notice the effect with your eye but try feeding market replay data into an algo strategy and during fast markets the result is a complete mess.
If you want to see the effect yourself just create an NT indicator using the attached code. All this code does is dump out each market data message to the Output window. Attach the indicator to a chart, replay some fast data (e.g. around a major news release) and watch the unsynchronised chaos unfold.
Why is this important? Well, if your algos are basing their decisions on a combination of both L1 and L2 data and those data streams are out of synch then those decisions are going to be heavily flawed.
I know that the NT development team monitor this thread because they have PMed me regarding my previous comments, so if anything I have said here is wrong then please step in and correct me...
Ok, I am reinstating my statement Ninja is written by people with no idea waht they do. WHOW. Did they ever get a sanity check on that? You are right, it deal any recorded data totally useless. But I think you underestimate the gravity... it should be visible with the eye when you run a DOM in replay - especially during active times, you should see bid/ask move out of sync with actual trades, T&S should be a total fuckup as the bid/ask coloring will obviously be totally whacky during active trading, too.
The idea to have a 1 second timestamp was stupid to start with (there is nothing lost going lower). But then storing L2 separate from the tick data makes it something I really have to write up in a nice blog post I would fire anyone responsible for such a great architecutral decision.
I have never used any of NT's visual toolset so can't comment on output via the DOM or T&S windows. I use NT purely as a data replay engine, pumping data to my algos via their API. To be fair to NT I can see why they split out the data into 2 files. The L2 data files are large and a lot of their customers probably aren't interested in anything beyond the top of the book. However, the 1 second granularity thing is dangerous and I personally think that people should be made aware of the potential consequences. It can lead to backtest results during fast markets being wildly different to reality.
Favorite Futures: Gameplay Klownbine® Trading of Globex
Posts: 1,276 since Jul 2009
Thanks: 1,227 given,
NT bid ask problem
If, at the time of adding a trade price to the Last file, NT also recorded the bid and ask prices, it would reduce the problems caused by the coarse timestamping.
I suggested that they do this by adding four decimal points to the Volume field. The first two would be the offset in tick size units between the last and the bid, and the second two would be the offset in tick size units from the last to the ask. Then there would be no need for bid and ask files anymore.
The Volume indicator could easily be set to ignore the decimal portion of the stored values.
I posted this in the NT forum and they ignored it. They are truly in their own world.....