NexusFi: Find Your Edge


Home Menu

 





Bad daily/minute data for E-Mini?


Discussion in Brokers

Updated
      Top Posters
    1. looks_one Fat Tails with 3 posts (9 thanks)
    2. looks_two Xyzzy with 2 posts (0 thanks)
    3. looks_3 Silvester17 with 2 posts (1 thanks)
    4. looks_4 Quick Summary with 1 posts (0 thanks)
    1. trending_up 5,088 views
    2. thumb_up 10 thanks given
    3. group 2 followers
    1. forum 7 posts
    2. attach_file 2 attachments




 
Search this Thread

Bad daily/minute data for E-Mini?

  #1 (permalink)
 Xyzzy 
Seattle, WA
 
Experience: Advanced
Platform: MultiCharts
Broker: IB/IQFeed
Trading: ES
Posts: 24 since Dec 2010
Thanks Given: 27
Thanks Received: 22

All,

I'm new to trading futures. I have a subscription to IQFeed's data feed for the E-Mini futures, and I also have an Interactive Brokers account. I've been comparing the data feeds from each one in MultCharts. I'm seeing wild discrepancies between them, which surprises and worries me. Can anyone explain why I'm seeing this?

I've heard complaints in the past about how Interactive Brokers doesn't provide "real" tick data. However, I'm seeing major differences between daily and minute charts, not just tick charts.

For example, the attached screenshot of a chart shows daily bars for the current E-Mini S&P 500 contract, ESZ11. IB is on top, and IQFeed is below.

There are wild differences between the reported volume and trading prices on a day by day basis. For example, they report the following daily values for November 1, 2011:

IB -- O: 1214.75, H: 1229.50, L: 1209.50, C:1224.00, V: 2,988388
IQ -- O: 1247.50, H: 1248.00, L: 1208.50, C:1224.50, V: 3,666,184

For earlier days in the contract, there are entire day's worth of data that is missing from the IB data.

Likewise, the IQFeed data shows the opening price for today as being 1257.50, which matches the official opening price shown on the CME Group's website (also 1257.50). In contrast, the IB data gives today's open as 1261.75, which is way off.

I'm also seeing some differences between minute bars, but those differences aren't quite as stark. For example, the reported closing prices might be off by a tick in either direction, or the volume off by a few contracts.

This might be a newbie question, but are these types of discrepancies unusual? I can understand why tick data might not always be precise, but I can't understand why there should be such major differences even between even daily bars. (I had assumed that daily data, at least, was based on some sort of official data from the exchange.)

Also, is it possible to trust either vendor's data, at least for minute and daily bars? I'm guessing that the IQFeed data is more accurate, given the above. However, does this mean "mostly accurate," or "highly, highly accurate"?

Thanks much in advance.

Attached Thumbnails
Click image for larger version

Name:	IB v. IQ.jpg
Views:	215
Size:	259.8 KB
ID:	54310  
Started this thread Reply With Quote

Can you help answer these questions
from other members on NexusFi?
Are there any eval firms that allow you to sink to your …
Traders Hideout
ZombieSqueeze
Platforms and Indicators
NexusFi Journal Challenge - April 2024
Feedback and Announcements
NT7 Indicator Script Troubleshooting - Camarilla Pivots
NinjaTrader
New Micros: Ultra 10-Year & Ultra T-Bond -- Live Now
Treasury Notes and Bonds
 
Best Threads (Most Thanked)
in the last 7 days on NexusFi
Get funded firms 2023/2024 - Any recommendations or word …
61 thanks
Funded Trader platforms
39 thanks
NexusFi site changelog and issues/problem reporting
26 thanks
GFIs1 1 DAX trade per day journal
18 thanks
The Program
18 thanks
  #3 (permalink)
 
Fat Tails's Avatar
 Fat Tails 
Berlin, Europe
Market Wizard
 
Experience: Advanced
Platform: NinjaTrader, MultiCharts
Broker: Interactive Brokers
Trading: Keyboard
Posts: 9,888 since Mar 2010
Thanks Given: 4,242
Thanks Received: 27,102



Xyzzy View Post
All,

For example, the attached screenshot of a chart shows daily bars for the current E-Mini S&P 500 contract, ESZ11. IB is on top, and IQFeed is below.

There are wild differences between the reported volume and trading prices on a day by day basis. For example, they report the following daily values for November 1, 2011:

IB -- O: 1214.75, H: 1229.50, L: 1209.50, C:1224.00, V: 2,988388
IQ -- O: 1247.50, H: 1248.00, L: 1208.50, C:1224.50, V: 3,666,184


I do not have IQ feed, but I have Kinetick with NinjaTrader which is essentially the same. I find the same data as you did

IQ -- O: 1247.50, H: 1248.00, L: 1208.50, C:1224.50, V: 3,666,184

IQ uses the ETH open, high and low and the settlement price as published by CME. So if you are in doubt, you can go to the CMEGroup website and check it.


The data taken from Interactive Brokers is

IB -- O: 1247.50, H: 1248.00, L: 1208.50, C:1224.00, V: 3,665,002

The reason for the small difference is that Interactive Brokers does not have genuine data as published by the exchange for the trading day, but builds the daily bars from intraday data. This leads to the following differences

(a) IB uses the close, not the settlement price as determined over the settlement period. There can be an important difference between the close and the settlement price, depending on the trading hours.
(b) CME and NYMEX have trading days that cover 2 calendar days (6 per year). There should be only one daily bar for the trading day, but Interactive Brokers will supply a daily bar for each calendar day
(c) Volume corrections made by the exchange are not taken into account.

The data that you have found for Interactive Brokers is the open, high, low, close and volume for the regular session.

IB -- O: 1214.75, H: 1229.50, L: 1209.50, C:1224.00, V: 2,988388

This is the difference:

Extended session for ES : 3:30 PM CT until 3:15 PM CT (except for later start Sunday evening)
Regular session for ES: 8:30 AM CT until 3:15PM CT

You probably got the daily data for the regular session of ES, because you used a RTH session template with MultiCharts.


Conclusion:

All data is correct, but you need to know

-> the difference between the regular and the extended session
-> the difference between the close and the settlement price
-> the diffenrence between a trading day and a calendar day
-> how MultiCharts builds the daily data which is supplied by your data feed

Then you will be fine.

Reply With Quote
  #4 (permalink)
 
Fat Tails's Avatar
 Fat Tails 
Berlin, Europe
Market Wizard
 
Experience: Advanced
Platform: NinjaTrader, MultiCharts
Broker: Interactive Brokers
Trading: Keyboard
Posts: 9,888 since Mar 2010
Thanks Given: 4,242
Thanks Received: 27,102


Xyzzy View Post
Also, is it possible to trust either vendor's data, at least for minute and daily bars? I'm guessing that the IQFeed data is more accurate, given the above. However, does this mean "mostly accurate," or "highly, highly accurate"?
Thanks much in advance.

DTN/IQ feed will supply you with genuine tick data as provided by the exchange. This data comes with a time stamp, so even if your system clock is a bit off, you will have your charts displayed correctly. You should not find any difference between historical and real-time data if you update your charts.

Interactive Brokers supplies low cost data, which is not identical with the data supplied by the exchange. These are the differences:

(a) IB data has no timestamps for real-time ticks, so MultiCharts will need to timestamp the data when it is arriving.
(b) IB does not supply real ticks, but the ticks are condensed, which means you will get fewer ticks. This even has an advantage as your software is less likely to freeze during a news release, that is if Interactive Brokers still supplies ticks in a timely manner.
(c) IB does not have historical backfill for tick-data. If you want to have a history, you need to store real-time tick data. However IB does offer backfill for minute data and daily data. For futures the daily data never shows the settlement price, but the close.
(d) The availability of Interactive Broker's servers for market data is good, but you will observe occasional (a few times per year) downtimes of 20 minutes and longer during the market hours. According to my experience the reliability of DTN/IQ feed is better.

You get what you pay for.

Reply With Quote
Thanked by:
  #5 (permalink)
 
Silvester17's Avatar
 Silvester17 
Columbus, OH
Market Wizard
 
Experience: None
Platform: NT 8, TOS
Trading: ES
Posts: 3,603 since Aug 2009
Thanks Given: 5,139
Thanks Received: 11,527


Xyzzy View Post
All,

I'm new to trading futures. I have a subscription to IQFeed's data feed for the E-Mini futures, and I also have an Interactive Brokers account. I've been comparing the data feeds from each one in MultCharts. I'm seeing wild discrepancies between them, which surprises and worries me. Can anyone explain why I'm seeing this?

I've heard complaints in the past about how Interactive Brokers doesn't provide "real" tick data. However, I'm seeing major differences between daily and minute charts, not just tick charts.

For example, the attached screenshot of a chart shows daily bars for the current E-Mini S&P 500 contract, ESZ11. IB is on top, and IQFeed is below.

There are wild differences between the reported volume and trading prices on a day by day basis. For example, they report the following daily values for November 1, 2011:

IB -- O: 1214.75, H: 1229.50, L: 1209.50, C:1224.00, V: 2,988388
IQ -- O: 1247.50, H: 1248.00, L: 1208.50, C:1224.50, V: 3,666,184

For earlier days in the contract, there are entire day's worth of data that is missing from the IB data.

Likewise, the IQFeed data shows the opening price for today as being 1257.50, which matches the official opening price shown on the CME Group's website (also 1257.50). In contrast, the IB data gives today's open as 1261.75, which is way off.

I'm also seeing some differences between minute bars, but those differences aren't quite as stark. For example, the reported closing prices might be off by a tick in either direction, or the volume off by a few contracts.

This might be a newbie question, but are these types of discrepancies unusual? I can understand why tick data might not always be precise, but I can't understand why there should be such major differences even between even daily bars. (I had assumed that daily data, at least, was based on some sort of official data from the exchange.)

Also, is it possible to trust either vendor's data, at least for minute and daily bars? I'm guessing that the IQFeed data is more accurate, given the above. However, does this mean "mostly accurate," or "highly, highly accurate"?

Thanks much in advance.

I would like to add a few things.

imo the IB data is wrong. not because of RTH or ETH. because it looks like it uses just the ES dec contract and not a continuous contract. that makes a big difference. especially if your're looking a long term charts with a long look back period.

attached picture from kinetick with normal contract and continuous contract. you can see the difference pretty good with the highs and lows. the high of may 2nd is about 1256 with dec contract and about 1273 with continuous contract. huge difference.

I know many are using this data for longer term charting, but it makes it just about worthless.

made some comments in this thread:


Attached Thumbnails
Click image for larger version

Name:	2011-11-08_1208.png
Views:	169
Size:	96.7 KB
ID:	54345  
Reply With Quote
Thanked by:
  #6 (permalink)
 
Fat Tails's Avatar
 Fat Tails 
Berlin, Europe
Market Wizard
 
Experience: Advanced
Platform: NinjaTrader, MultiCharts
Broker: Interactive Brokers
Trading: Keyboard
Posts: 9,888 since Mar 2010
Thanks Given: 4,242
Thanks Received: 27,102


Silvester17 View Post
I would like to add a few things.

imo the IB data is wrong. not because of RTH or ETH. because it looks like it uses just the ES dec contract and not a continuous contract. that makes a big difference. especially if your're looking a long term charts with a long look back period.

I am sorry, but this is not true. Each type of contract has it advantages and disadvantages. You can use

-> single month contract data
-> merge backadjusted data
-> ratio backadjusted data
-> continous (spliced) contracts

None of those is wrong or correct, it really depends what you want to do with it.

The basis for all types of contracts are single month contracts. This is the only genuine source of data. All other contracts are artificial constructs. Some data providers - such as DTN/IQ - allow you to download both single month and continous contract data.

Single month contract data: Best, if you do not need data that goes back beyond the last rollover data.

Merge-backadjusted data: Best, if you are a swing trader and keep position several months, but not longer than a year.

-> The advantage of merge-backadjusted data is that rollover gains and losses are correctly reflected by price data. It is the only type of contract that can be used for backtesting strategies over a longer time period.
-> Merge-backadjusted contracts are built from single month contract data. I use NinjaTrader to build merge-backadjusted contracts from intraday data from Interactive Brokers. This is perfect for backtesting
-> Merge-backadjusted data does not correctly reflect absolute price leves, price may in some extreme cases move into negative territory. The most critical cases are commodities with an in-built contango or backwardation or total return index futures, which have an upward bias.

Continuous contracts: Correctly shows price trends over a longer period, as it does not suffer from the distortion produced by merge-backadjusted data

-> continuous futures is built from artificial data, you cannot enter or exit a position at a given time, because the price is not a market price
-> continous futures cannot be used for backtesting

Single month contracts are not false. They are the only genuine source of data, and with backadjustment you can easily use them over 1 or 2 years to evaluate trading strategies.


Silvester17 View Post
attached picture from kinetick with normal contract and continuous contract. you can see the difference pretty good with the highs and lows. the high of may 2nd is about 1256 with dec contract and about 1273 with continuous contract. huge difference.

I know many are using this data for longer term charting, but it makes it just about worthless.

You would certainly not look at the May high by taking the December contract of ES..... If you correctly backadjust your data for the rollover gaps, the merge-backadjusted contract will show you a high of 1362, while the continuous contract gets you 1373.

If you sold that high and bought back your position yesterday at the close at 1257.50 , the price difference of 104.50 points on the merge-backadjusted contract would exactly reflect your gain (excluding commissions for rolling the position). The price difference on the continuous contract does not mean anything.

The next question is whether or not the continuous contract is better in following the price of the cash market. So let us have a look and compare teh values for May 2, and yesterday

-> S&P 500: May 2 High : 1370.58 November 7 Close: 1261.12 -> difference 109.46
-> ES continuous contract: May 2 High : 1373.00 November 7 Close: 1257.50 -> difference 115.50
-> ES merge-backadjusted contract: May 2 High : 1362.00 November 7 Close: 1257.50 -> 104.50

Although in general the continuous contract is better at tracking the cash market, there is no ssignificant difference in this case. But have a look at CL. You will enjoy it.

Reply With Quote
Thanked by:
  #7 (permalink)
 
Silvester17's Avatar
 Silvester17 
Columbus, OH
Market Wizard
 
Experience: None
Platform: NT 8, TOS
Trading: ES
Posts: 3,603 since Aug 2009
Thanks Given: 5,139
Thanks Received: 11,527


Fat Tails View Post
I am sorry, but this is not true. Each type of contract has it advantages and disadvantages. You can use

-> single month contract data
-> merge backadjusted data
-> ratio backadjusted data
-> continous (spliced) contracts

None of those is wrong or correct, it really depends what you want to do with it.

The basis for all types of contracts are single month contracts. This is the only genuine source of data. All other contracts are artificial constructs. Some data providers - such as DTN/IQ - allow you to download both single month and continous contract data.

Single month contract data: Best, if you do not need data that goes back beyond the last rollover data.

Merge-backadjusted data: Best, if you are a swing trader and keep position several months, but not longer than a year.

-> The advantage of merge-backadjusted data is that rollover gains and losses are correctly reflected by price data. It is the only type of contract that can be used for backtesting strategies over a longer time period.
-> Merge-backadjusted contracts are built from single month contract data. I use NinjaTrader to build merge-backadjusted contracts from intraday data from Interactive Brokers. This is perfect for backtesting
-> Merge-backadjusted data does not correctly reflect absolute price leves, price may in some extreme cases move into negative territory. The most critical cases are commodities with an in-built contango or backwardation or total return index futures, which have an upward bias.

Continuous contracts: Correctly shows price trends over a longer period, as it does not suffer from the distortion produced by merge-backadjusted data

-> continuous futures is built from artificial data, you cannot enter or exit a position at a given time, because the price is not a market price
-> continous futures cannot be used for backtesting

Single month contracts are not false. They are the only genuine source of data, and with backadjustment you can easily use them over 1 or 2 years to evaluate trading strategies.



You would certainly not look at the May high by taking the December contract of ES..... If you correctly backadjust your data for the rollover gaps, the merge-backadjusted contract will show you a high of 1362, while the continuous contract gets you 1373.

If you sold that high and bought back your position yesterday at the close at 1257.50 , the price difference of 104.50 points on the merge-backadjusted contract would exactly reflect your gain (excluding commissions for rolling the position). The price difference on the continuous contract does not mean anything.

The next question is whether or not the continuous contract is better in following the price of the cash market. So let us have a look and compare teh values for May 2, and yesterday

-> S&P 500: May 2 High : 1370.58 November 7 Close: 1261.12 -> difference 109.46
-> ES continuous contract: May 2 High : 1373.00 November 7 Close: 1257.50 -> difference 115.50
-> ES merge-backadjusted contract: May 2 High : 1362.00 November 7 Close: 1257.50 -> 104.50

Although in general the continuous contract is better at tracking the cash market, there is no ssignificant difference in this case. But have a look at CL. You will enjoy it.


of course I do not disagree with what you're saying.

but for any kind of analysis over a longer term, I would never use a single month contract. (I know ES is quarterly). for me 10 points could make the difference between a winning or a losing trade.

anyway you're still the best.

Reply With Quote
  #8 (permalink)
 Xyzzy 
Seattle, WA
 
Experience: Advanced
Platform: MultiCharts
Broker: IB/IQFeed
Trading: ES
Posts: 24 since Dec 2010
Thanks Given: 27
Thanks Received: 22

Thanks to everyone for the responses. They're very helpful! I'll have to study this further.

Concerning the IB feed, the data is what IB returns when MultiCharts requests daily data -- it doesn't appear to depend on whether I'm using RTH, etc. As a workaround, I've found that I can create pseudo-daily bars by using very large "minute" bars. E.g., if I want a "daily" bar just for the e-mini daily trading session, I can create a chart with sessions from 08:30 to 15:15 Central time, and then create a minute bar with a period of 405 minutes -- the number of minutes between 08:30 and 15:15.

Started this thread Reply With Quote




Last Updated on November 8, 2011


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts