Bad daily/minute data for E-Mini? - Reviews of Brokers and Data Feeds | futures io social day trading
futures io futures trading


Bad daily/minute data for E-Mini?
Updated: Views / Replies:2,956 / 7
Created: by Xyzzy Attachments:2

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 
Thread Tools Search this Thread
 

Bad daily/minute data for E-Mini?

  #1 (permalink)
Elite Member
Seattle, WA
 
Futures Experience: Intermediate
Platform: MultiCharts
Broker/Data: IB/IQFeed
Favorite Futures: ES
 
Posts: 20 since Dec 2010
Thanks: 27 given, 7 received

Bad daily/minute data for E-Mini?

All,

I'm new to trading futures. I have a subscription to IQFeed's data feed for the E-Mini futures, and I also have an Interactive Brokers account. I've been comparing the data feeds from each one in MultCharts. I'm seeing wild discrepancies between them, which surprises and worries me. Can anyone explain why I'm seeing this?

I've heard complaints in the past about how Interactive Brokers doesn't provide "real" tick data. However, I'm seeing major differences between daily and minute charts, not just tick charts.

For example, the attached screenshot of a chart shows daily bars for the current E-Mini S&P 500 contract, ESZ11. IB is on top, and IQFeed is below.

There are wild differences between the reported volume and trading prices on a day by day basis. For example, they report the following daily values for November 1, 2011:

IB -- O: 1214.75, H: 1229.50, L: 1209.50, C:1224.00, V: 2,988388
IQ -- O: 1247.50, H: 1248.00, L: 1208.50, C:1224.50, V: 3,666,184

For earlier days in the contract, there are entire day's worth of data that is missing from the IB data.

Likewise, the IQFeed data shows the opening price for today as being 1257.50, which matches the official opening price shown on the CME Group's website (also 1257.50). In contrast, the IB data gives today's open as 1261.75, which is way off.

I'm also seeing some differences between minute bars, but those differences aren't quite as stark. For example, the reported closing prices might be off by a tick in either direction, or the volume off by a few contracts.

This might be a newbie question, but are these types of discrepancies unusual? I can understand why tick data might not always be precise, but I can't understand why there should be such major differences even between even daily bars. (I had assumed that daily data, at least, was based on some sort of official data from the exchange.)

Also, is it possible to trust either vendor's data, at least for minute and daily bars? I'm guessing that the IQFeed data is more accurate, given the above. However, does this mean "mostly accurate," or "highly, highly accurate"?

Thanks much in advance.

Attached Thumbnails
Bad daily/minute data for E-Mini?-ib-v.-iq.jpg  
Reply With Quote
 
  #2 (permalink)
Quick Summary
Quick Summary Post

Quick Summary is created and edited by users like you... Add FAQ's, Links and other Relevant Information by clicking the edit button in the lower right hand corner of this message.

 
  #3 (permalink)
Elite Member
Berlin, Europe
 
Futures Experience: Advanced
Platform: NinjaTrader, MultiCharts
Broker/Data: Interactive Brokers
Favorite Futures: Keyboard
 
Fat Tails's Avatar
 
Posts: 9,653 since Mar 2010
Thanks: 4,226 given, 25,601 received
Forum Reputation: Legendary



Xyzzy View Post
All,

For example, the attached screenshot of a chart shows daily bars for the current E-Mini S&P 500 contract, ESZ11. IB is on top, and IQFeed is below.

There are wild differences between the reported volume and trading prices on a day by day basis. For example, they report the following daily values for November 1, 2011:

IB -- O: 1214.75, H: 1229.50, L: 1209.50, C:1224.00, V: 2,988388
IQ -- O: 1247.50, H: 1248.00, L: 1208.50, C:1224.50, V: 3,666,184


I do not have IQ feed, but I have Kinetick with NinjaTrader which is essentially the same. I find the same data as you did

IQ -- O: 1247.50, H: 1248.00, L: 1208.50, C:1224.50, V: 3,666,184

IQ uses the ETH open, high and low and the settlement price as published by CME. So if you are in doubt, you can go to the CMEGroup website and check it.


The data taken from Interactive Brokers is

IB -- O: 1247.50, H: 1248.00, L: 1208.50, C:1224.00, V: 3,665,002

The reason for the small difference is that Interactive Brokers does not have genuine data as published by the exchange for the trading day, but builds the daily bars from intraday data. This leads to the following differences

(a) IB uses the close, not the settlement price as determined over the settlement period. There can be an important difference between the close and the settlement price, depending on the trading hours.
(b) CME and NYMEX have trading days that cover 2 calendar days (6 per year). There should be only one daily bar for the trading day, but Interactive Brokers will supply a daily bar for each calendar day
(c) Volume corrections made by the exchange are not taken into account.

The data that you have found for Interactive Brokers is the open, high, low, close and volume for the regular session.

IB -- O: 1214.75, H: 1229.50, L: 1209.50, C:1224.00, V: 2,988388

This is the difference:

Extended session for ES : 3:30 PM CT until 3:15 PM CT (except for later start Sunday evening)
Regular session for ES: 8:30 AM CT until 3:15PM CT

You probably got the daily data for the regular session of ES, because you used a RTH session template with MultiCharts.


Conclusion:

All data is correct, but you need to know

-> the difference between the regular and the extended session
-> the difference between the close and the settlement price
-> the diffenrence between a trading day and a calendar day
-> how MultiCharts builds the daily data which is supplied by your data feed

Then you will be fine.

Reply With Quote
The following 4 users say Thank You to Fat Tails for this post:
 
  #4 (permalink)
Elite Member
Berlin, Europe
 
Futures Experience: Advanced
Platform: NinjaTrader, MultiCharts
Broker/Data: Interactive Brokers
Favorite Futures: Keyboard
 
Fat Tails's Avatar
 
Posts: 9,653 since Mar 2010
Thanks: 4,226 given, 25,601 received
Forum Reputation: Legendary


Xyzzy View Post
Also, is it possible to trust either vendor's data, at least for minute and daily bars? I'm guessing that the IQFeed data is more accurate, given the above. However, does this mean "mostly accurate," or "highly, highly accurate"?
Thanks much in advance.

DTN/IQ feed will supply you with genuine tick data as provided by the exchange. This data comes with a time stamp, so even if your system clock is a bit off, you will have your charts displayed correctly. You should not find any difference between historical and real-time data if you update your charts.

Interactive Brokers supplies low cost data, which is not identical with the data supplied by the exchange. These are the differences:

(a) IB data has no timestamps for real-time ticks, so MultiCharts will need to timestamp the data when it is arriving.
(b) IB does not supply real ticks, but the ticks are condensed, which means you will get fewer ticks. This even has an advantage as your software is less likely to freeze during a news release, that is if Interactive Brokers still supplies ticks in a timely manner.
(c) IB does not have historical backfill for tick-data. If you want to have a history, you need to store real-time tick data. However IB does offer backfill for minute data and daily data. For futures the daily data never shows the settlement price, but the close.
(d) The availability of Interactive Broker's servers for market data is good, but you will observe occasional (a few times per year) downtimes of 20 minutes and longer during the market hours. According to my experience the reliability of DTN/IQ feed is better.

You get what you pay for.

Reply With Quote
The following 3 users say Thank You to Fat Tails for this post:
 
  #5 (permalink)
Market Wizard
Columbus, OH
 
Futures Experience: None
Platform: NT 8, TOS
Favorite Futures: ES
 
Silvester17's Avatar
 
Posts: 3,332 since Aug 2009
Thanks: 4,579 given, 10,313 received


Xyzzy View Post
All,

I'm new to trading futures. I have a subscription to IQFeed's data feed for the E-Mini futures, and I also have an Interactive Brokers account. I've been comparing the data feeds from each one in MultCharts. I'm seeing wild discrepancies between them, which surprises and worries me. Can anyone explain why I'm seeing this?

I've heard complaints in the past about how Interactive Brokers doesn't provide "real" tick data. However, I'm seeing major differences between daily and minute charts, not just tick charts.

For example, the attached screenshot of a chart shows daily bars for the current E-Mini S&P 500 contract, ESZ11. IB is on top, and IQFeed is below.

There are wild differences between the reported volume and trading prices on a day by day basis. For example, they report the following daily values for November 1, 2011:

IB -- O: 1214.75, H: 1229.50, L: 1209.50, C:1224.00, V: 2,988388
IQ -- O: 1247.50, H: 1248.00, L: 1208.50, C:1224.50, V: 3,666,184

For earlier days in the contract, there are entire day's worth of data that is missing from the IB data.

Likewise, the IQFeed data shows the opening price for today as being 1257.50, which matches the official opening price shown on the CME Group's website (also 1257.50). In contrast, the IB data gives today's open as 1261.75, which is way off.

I'm also seeing some differences between minute bars, but those differences aren't quite as stark. For example, the reported closing prices might be off by a tick in either direction, or the volume off by a few contracts.

This might be a newbie question, but are these types of discrepancies unusual? I can understand why tick data might not always be precise, but I can't understand why there should be such major differences even between even daily bars. (I had assumed that daily data, at least, was based on some sort of official data from the exchange.)

Also, is it possible to trust either vendor's data, at least for minute and daily bars? I'm guessing that the IQFeed data is more accurate, given the above. However, does this mean "mostly accurate," or "highly, highly accurate"?

Thanks much in advance.

I would like to add a few things.

imo the IB data is wrong. not because of RTH or ETH. because it looks like it uses just the ES dec contract and not a continuous contract. that makes a big difference. especially if your're looking a long term charts with a long look back period.

attached picture from kinetick with normal contract and continuous contract. you can see the difference pretty good with the highs and lows. the high of may 2nd is about 1256 with dec contract and about 1273 with continuous contract. huge difference.

I know many are using this data for longer term charting, but it makes it just about worthless.

made some comments in this thread:

https://futures.io/ninjatrader/14618-simple-moving-average-difference-ninja-tos-thinkorswim-trading-platform.html#post166386

Attached Thumbnails
Bad daily/minute data for E-Mini?-2011-11-08_1208.png  
Reply With Quote
The following user says Thank You to Silvester17 for this post:
 
  #6 (permalink)
Elite Member
Berlin, Europe
 
Futures Experience: Advanced
Platform: NinjaTrader, MultiCharts
Broker/Data: Interactive Brokers
Favorite Futures: Keyboard
 
Fat Tails's Avatar
 
Posts: 9,653 since Mar 2010
Thanks: 4,226 given, 25,601 received
Forum Reputation: Legendary


Silvester17 View Post
I would like to add a few things.

imo the IB data is wrong. not because of RTH or ETH. because it looks like it uses just the ES dec contract and not a continuous contract. that makes a big difference. especially if your're looking a long term charts with a long look back period.

I am sorry, but this is not true. Each type of contract has it advantages and disadvantages. You can use

-> single month contract data
-> merge backadjusted data
-> ratio backadjusted data
-> continous (spliced) contracts

None of those is wrong or correct, it really depends what you want to do with it.

The basis for all types of contracts are single month contracts. This is the only genuine source of data. All other contracts are artificial constructs. Some data providers - such as DTN/IQ - allow you to download both single month and continous contract data.

Single month contract data: Best, if you do not need data that goes back beyond the last rollover data.

Merge-backadjusted data: Best, if you are a swing trader and keep position several months, but not longer than a year.

-> The advantage of merge-backadjusted data is that rollover gains and losses are correctly reflected by price data. It is the only type of contract that can be used for backtesting strategies over a longer time period.
-> Merge-backadjusted contracts are built from single month contract data. I use NinjaTrader to build merge-backadjusted contracts from intraday data from Interactive Brokers. This is perfect for backtesting
-> Merge-backadjusted data does not correctly reflect absolute price leves, price may in some extreme cases move into negative territory. The most critical cases are commodities with an in-built contango or backwardation or total return index futures, which have an upward bias.

Continuous contracts: Correctly shows price trends over a longer period, as it does not suffer from the distortion produced by merge-backadjusted data

-> continuous futures is built from artificial data, you cannot enter or exit a position at a given time, because the price is not a market price
-> continous futures cannot be used for backtesting

Single month contracts are not false. They are the only genuine source of data, and with backadjustment you can easily use them over 1 or 2 years to evaluate trading strategies.


Silvester17 View Post
attached picture from kinetick with normal contract and continuous contract. you can see the difference pretty good with the highs and lows. the high of may 2nd is about 1256 with dec contract and about 1273 with continuous contract. huge difference.

I know many are using this data for longer term charting, but it makes it just about worthless.

You would certainly not look at the May high by taking the December contract of ES..... If you correctly backadjust your data for the rollover gaps, the merge-backadjusted contract will show you a high of 1362, while the continuous contract gets you 1373.

If you sold that high and bought back your position yesterday at the close at 1257.50 , the price difference of 104.50 points on the merge-backadjusted contract would exactly reflect your gain (excluding commissions for rolling the position). The price difference on the continuous contract does not mean anything.

The next question is whether or not the continuous contract is better in following the price of the cash market. So let us have a look and compare teh values for May 2, and yesterday

-> S&P 500: May 2 High : 1370.58 November 7 Close: 1261.12 -> difference 109.46
-> ES continuous contract: May 2 High : 1373.00 November 7 Close: 1257.50 -> difference 115.50
-> ES merge-backadjusted contract: May 2 High : 1362.00 November 7 Close: 1257.50 -> 104.50

Although in general the continuous contract is better at tracking the cash market, there is no ssignificant difference in this case. But have a look at CL. You will enjoy it.

Reply With Quote
The following 2 users say Thank You to Fat Tails for this post:
 
  #7 (permalink)
Market Wizard
Columbus, OH
 
Futures Experience: None
Platform: NT 8, TOS
Favorite Futures: ES
 
Silvester17's Avatar
 
Posts: 3,332 since Aug 2009
Thanks: 4,579 given, 10,313 received


Fat Tails View Post
I am sorry, but this is not true. Each type of contract has it advantages and disadvantages. You can use

-> single month contract data
-> merge backadjusted data
-> ratio backadjusted data
-> continous (spliced) contracts

None of those is wrong or correct, it really depends what you want to do with it.

The basis for all types of contracts are single month contracts. This is the only genuine source of data. All other contracts are artificial constructs. Some data providers - such as DTN/IQ - allow you to download both single month and continous contract data.

Single month contract data: Best, if you do not need data that goes back beyond the last rollover data.

Merge-backadjusted data: Best, if you are a swing trader and keep position several months, but not longer than a year.

-> The advantage of merge-backadjusted data is that rollover gains and losses are correctly reflected by price data. It is the only type of contract that can be used for backtesting strategies over a longer time period.
-> Merge-backadjusted contracts are built from single month contract data. I use NinjaTrader to build merge-backadjusted contracts from intraday data from Interactive Brokers. This is perfect for backtesting
-> Merge-backadjusted data does not correctly reflect absolute price leves, price may in some extreme cases move into negative territory. The most critical cases are commodities with an in-built contango or backwardation or total return index futures, which have an upward bias.

Continuous contracts: Correctly shows price trends over a longer period, as it does not suffer from the distortion produced by merge-backadjusted data

-> continuous futures is built from artificial data, you cannot enter or exit a position at a given time, because the price is not a market price
-> continous futures cannot be used for backtesting

Single month contracts are not false. They are the only genuine source of data, and with backadjustment you can easily use them over 1 or 2 years to evaluate trading strategies.



You would certainly not look at the May high by taking the December contract of ES..... If you correctly backadjust your data for the rollover gaps, the merge-backadjusted contract will show you a high of 1362, while the continuous contract gets you 1373.

If you sold that high and bought back your position yesterday at the close at 1257.50 , the price difference of 104.50 points on the merge-backadjusted contract would exactly reflect your gain (excluding commissions for rolling the position). The price difference on the continuous contract does not mean anything.

The next question is whether or not the continuous contract is better in following the price of the cash market. So let us have a look and compare teh values for May 2, and yesterday

-> S&P 500: May 2 High : 1370.58 November 7 Close: 1261.12 -> difference 109.46
-> ES continuous contract: May 2 High : 1373.00 November 7 Close: 1257.50 -> difference 115.50
-> ES merge-backadjusted contract: May 2 High : 1362.00 November 7 Close: 1257.50 -> 104.50

Although in general the continuous contract is better at tracking the cash market, there is no ssignificant difference in this case. But have a look at CL. You will enjoy it.


of course I do not disagree with what you're saying.

but for any kind of analysis over a longer term, I would never use a single month contract. (I know ES is quarterly). for me 10 points could make the difference between a winning or a losing trade.

anyway you're still the best.


Last edited by Silvester17; November 9th, 2011 at 11:02 AM.
Reply With Quote
 
  #8 (permalink)
Elite Member
Seattle, WA
 
Futures Experience: Intermediate
Platform: MultiCharts
Broker/Data: IB/IQFeed
Favorite Futures: ES
 
Posts: 20 since Dec 2010
Thanks: 27 given, 7 received

Thanks to everyone for the responses. They're very helpful! I'll have to study this further.

Concerning the IB feed, the data is what IB returns when MultiCharts requests daily data -- it doesn't appear to depend on whether I'm using RTH, etc. As a workaround, I've found that I can create pseudo-daily bars by using very large "minute" bars. E.g., if I want a "daily" bar just for the e-mini daily trading session, I can create a chart with sessions from 08:30 to 15:15 Central time, and then create a minute bar with a period of 405 minutes -- the number of minutes between 08:30 and 15:15.

Reply With Quote

Reply



futures io > > > > Bad daily/minute data for E-Mini?

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Jigsaw Trading: TBA

Elite only

FuturesTrader71: TBA

Elite only

NinjaTrader: TBA

Jan 18

RandBots: TBA

Jan 23

GFF Brokers & CME Group: Futures & Bitcoin

Elite only

Adam Grimes: TBA

Elite only

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Searching for HSI and MHI Minute or Tick Data for Market Replay - Please help! wgreenie NinjaTrader 8 July 9th, 2017 09:59 AM
Historical CL Tick Data and Minute Data Big Mike The Elite Circle 112 July 26th, 2014 07:07 PM
2000-2009 continuous contract minute data for CL, 6E, & ES now available cunparis The Elite Circle 20 December 28th, 2010 01:02 PM
CL Minute data requested kenz987 Reviews of Brokers and Data Feeds 1 August 20th, 2010 03:09 PM
SPY Minute data needed roonius Reviews of Brokers and Data Feeds 13 August 15th, 2010 08:02 AM


All times are GMT -4. The time now is 02:32 AM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-14 in 0.18 seconds with 20 queries on phoenix via your IP 54.221.76.68