After reading a lot in this great forum I still have some questions that I wish someone can help me with.
Currently I am connected to CQG data feed and I wish to record Level 1 and Level 2 data for analysis purposes.
I will start with the ES futures.
So these are my questions:
1. Is there a good research or statistics about the L1 and L2 impact on price movements?
2. If I use OnMarketDepth to record each L2 change should I face any issues? (miss some data, sync issues, etc.)
3. If I am recording data into one csv file from OnMarketData and from OnMarketDepth should I expect any issues?
4. Does CQG do any filtering to the data?
5. What does filtering data mean? Aggregating market orders?
6. Does NT do any filtering to the data?
7. The fact that NT doesn't supply the ms timestamp means that the data can arrive out of order?
8. If I use my own computer's ms timestamp will it be enough to sync between the L1 and L2?very
9. Do I need a very fast computer to record the data?
10. Will charting or using other softwares on my computer may interfere with the recording process?
11. Should I use my internet connection only for recording? (and not for downloading at the same time for instance)
11. What is the minimum bandwidth requirement for catching all the events properly?
12. Is WIFI enough or should I use a cable?
I know these are a lot of questions but I am sure others will benefit from them as well.
-> Historical Data: You can download backfill for minute and tick data from various historical data providers. Time stamps do not come with subsecond granularity. Analysis of trade data, level 1 and level 2 data is not possible, as it is not synchronized.
-> Replay Data: NinjaTrader 7 offers free replay data - including level 1 and level 2 - which can be downloaded for many instruments. Simply download via File -> Utilities -> Downloade Replay Data and select instrument and date.
If you wish to go beyond the features that are offered by NinjaTrader 7, I suggest to install the GOM recorder, which records and stores trade data, bid and ask data in a different format.
Sorry for the late reply but I had to consult my fellow traders that went over GOMI's recorder code and used it.
As they (and I) see it, the recorded data is correctly synced if written from a feed. Gomi just added the ms for use in other indicators he publishes. If I do not need the ms my recording will be sufficient.
So this leads us back to my original set of questions except 8. Can anyone please help me with some of them?
I had some experiments with it and discovered that there are more complications with this task. Sometimes the L2 collection is not synced correctly and if you build the order book yourself you have the take into consideration that small disconnects can cause sync issues as well. Also if you use ms timestamps like GOMrecorder you will get issues with that also because the computer syncs its clock every now and then which can create earlier timestamps to new events. You ca find threads here that address some clock syncing issues.
The bottom line is that it is possible but requires more programming than I originally expected and may result in some out of sync L2 data.