I'm new to trading futures. I have a subscription to IQFeed's data feed for the E-Mini futures, and I also have an Interactive Brokers account. I've been comparing the data feeds from each one in MultCharts. I'm seeing wild discrepancies between them, which surprises and worries me. Can anyone explain why I'm seeing this?
I've heard complaints in the past about how Interactive Brokers doesn't provide "real" tick data. However, I'm seeing major differences between daily and minute charts, not just tick charts.
For example, the attached screenshot of a chart shows daily bars for the current E-Mini S&P 500 contract, ESZ11. IB is on top, and IQFeed is below.
There are wild differences between the reported volume and trading prices on a day by day basis. For example, they report the following daily values for November 1, 2011:
For earlier days in the contract, there are entire day's worth of data that is missing from the IB data.
Likewise, the IQFeed data shows the opening price for today as being 1257.50, which matches the official opening price shown on the CME Group's website (also 1257.50). In contrast, the IB data gives today's open as 1261.75, which is way off.
I'm also seeing some differences between minute bars, but those differences aren't quite as stark. For example, the reported closing prices might be off by a tick in either direction, or the volume off by a few contracts.
This might be a newbie question, but are these types of discrepancies unusual? I can understand why tick data might not always be precise, but I can't understand why there should be such major differences even between even daily bars. (I had assumed that daily data, at least, was based on some sort of official data from the exchange.)
Also, is it possible to trust either vendor's data, at least for minute and daily bars? I'm guessing that the IQFeed data is more accurate, given the above. However, does this mean "mostly accurate," or "highly, highly accurate"?
The reason for the small difference is that Interactive Brokers does not have genuine data as published by the exchange for the trading day, but builds the daily bars from intraday data. This leads to the following differences
(a) IB uses the close, not the settlement price as determined over the settlement period. There can be an important difference between the close and the settlement price, depending on the trading hours.
(b) CME and NYMEX have trading days that cover 2 calendar days (6 per year). There should be only one daily bar for the trading day, but Interactive Brokers will supply a daily bar for each calendar day
(c) Volume corrections made by the exchange are not taken into account.
The data that you have found for Interactive Brokers is the open, high, low, close and volume for the regular session.
Extended session for ES : 3:30 PM CT until 3:15 PM CT (except for later start Sunday evening)
Regular session for ES: 8:30 AM CT until 3:15PM CT
You probably got the daily data for the regular session of ES, because you used a RTH session template with MultiCharts.
All data is correct, but you need to know
-> the difference between the regular and the extended session
-> the difference between the close and the settlement price
-> the diffenrence between a trading day and a calendar day
-> how MultiCharts builds the daily data which is supplied by your data feed
Then you will be fine.
The following 4 users say Thank You to Fat Tails for this post:
DTN/IQ feed will supply you with genuine tick data as provided by the exchange. This data comes with a time stamp, so even if your system clock is a bit off, you will have your charts displayed correctly. You should not find any difference between historical and real-time data if you update your charts.
Interactive Brokers supplies low cost data, which is not identical with the data supplied by the exchange. These are the differences:
(a) IB data has no timestamps for real-time ticks, so MultiCharts will need to timestamp the data when it is arriving.
(b) IB does not supply real ticks, but the ticks are condensed, which means you will get fewer ticks. This even has an advantage as your software is less likely to freeze during a news release, that is if Interactive Brokers still supplies ticks in a timely manner.
(c) IB does not have historical backfill for tick-data. If you want to have a history, you need to store real-time tick data. However IB does offer backfill for minute data and daily data. For futures the daily data never shows the settlement price, but the close.
(d) The availability of Interactive Broker's servers for market data is good, but you will observe occasional (a few times per year) downtimes of 20 minutes and longer during the market hours. According to my experience the reliability of DTN/IQ feed is better.
You get what you pay for.
The following 3 users say Thank You to Fat Tails for this post:
imo the IB data is wrong. not because of RTH or ETH. because it looks like it uses just the ES dec contract and not a continuous contract. that makes a big difference. especially if your're looking a long term charts with a long look back period.
attached picture from kinetick with normal contract and continuous contract. you can see the difference pretty good with the highs and lows. the high of may 2nd is about 1256 with dec contract and about 1273 with continuous contract. huge difference.
I know many are using this data for longer term charting, but it makes it just about worthless.
I am sorry, but this is not true. Each type of contract has it advantages and disadvantages. You can use
-> single month contract data
-> merge backadjusted data
-> ratio backadjusted data
-> continous (spliced) contracts
None of those is wrong or correct, it really depends what you want to do with it.
The basis for all types of contracts are single month contracts. This is the only genuine source of data. All other contracts are artificial constructs. Some data providers - such as DTN/IQ - allow you to download both single month and continous contract data.
Single month contract data: Best, if you do not need data that goes back beyond the last rollover data.
Merge-backadjusted data: Best, if you are a swing trader and keep position several months, but not longer than a year.
-> The advantage of merge-backadjusted data is that rollover gains and losses are correctly reflected by price data. It is the only type of contract that can be used for backtesting strategies over a longer time period.
-> Merge-backadjusted contracts are built from single month contract data. I use NinjaTrader to build merge-backadjusted contracts from intraday data from Interactive Brokers. This is perfect for backtesting
-> Merge-backadjusted data does not correctly reflect absolute price leves, price may in some extreme cases move into negative territory. The most critical cases are commodities with an in-built contango or backwardation or total return index futures, which have an upward bias.
Continuous contracts: Correctly shows price trends over a longer period, as it does not suffer from the distortion produced by merge-backadjusted data
-> continuous futures is built from artificial data, you cannot enter or exit a position at a given time, because the price is not a market price
-> continous futures cannot be used for backtesting
Single month contracts are not false. They are the only genuine source of data, and with backadjustment you can easily use them over 1 or 2 years to evaluate trading strategies.
You would certainly not look at the May high by taking the December contract of ES..... If you correctly backadjust your data for the rollover gaps, the merge-backadjusted contract will show you a high of 1362, while the continuous contract gets you 1373.
If you sold that high and bought back your position yesterday at the close at 1257.50 , the price difference of 104.50 points on the merge-backadjusted contract would exactly reflect your gain (excluding commissions for rolling the position). The price difference on the continuous contract does not mean anything.
The next question is whether or not the continuous contract is better in following the price of the cash market. So let us have a look and compare teh values for May 2, and yesterday
-> S&P 500: May 2 High : 1370.58 November 7 Close: 1261.12 -> difference 109.46
-> ES continuous contract: May 2 High : 1373.00 November 7 Close: 1257.50 -> difference 115.50
-> ES merge-backadjusted contract: May 2 High : 1362.00 November 7 Close: 1257.50 -> 104.50
Although in general the continuous contract is better at tracking the cash market, there is no ssignificant difference in this case. But have a look at CL. You will enjoy it.
The following 2 users say Thank You to Fat Tails for this post:
Thanks to everyone for the responses. They're very helpful! I'll have to study this further.
Concerning the IB feed, the data is what IB returns when MultiCharts requests daily data -- it doesn't appear to depend on whether I'm using RTH, etc. As a workaround, I've found that I can create pseudo-daily bars by using very large "minute" bars. E.g., if I want a "daily" bar just for the e-mini daily trading session, I can create a chart with sessions from 08:30 to 15:15 Central time, and then create a minute bar with a period of 405 minutes -- the number of minutes between 08:30 and 15:15.