inconsistent results from different data sources - Emini Index Futures Trading | futures io social day trading
futures io futures trading


inconsistent results from different data sources
Updated: Views / Replies:1,350 / 13
Created: by mitty Attachments:2

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 100,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

 
 2  
 
Thread Tools
 

inconsistent results from different data sources

 
Trading Apprentice
Christchurch, New Zealand
 
Futures Experience: Beginner
Platform: Multicharts
Broker/Data: Interactive Brokers/IQ Feed
Favorite Futures: E-mini S&P
 
Posts: 4 since Mar 2018
Thanks: 0 given, 0 received

inconsistent results from different data sources

Hi - I am having trouble getting consistent results from different data sources (IQ feed CME/Globex and IB Globex).

I am currently using IQ feed to backtest some ES futures strategies in Multicharts. When I test the Globex data from IQ feed vs data from Interactive Brokers over the same period the results differ significantly. Various metrics can be out by as much as 2-3x. Anyone have any idea what might be causing this?

Also, what is the difference between the straight CME data vs the Globex data from IQ Feed? the two also give significantly different results when tested under the same strategies.

 
 
Quick Summary
Quick Summary Post

Quick Summary is created and edited by users like you... Add FAQ's, Links and other Relevant Information by clicking the edit button in the lower right hand corner of this message.

 
 
Elite Member
New York, NY
 
Futures Experience: Beginner
Platform: thinkorswim
Broker/Data: TD Ameritrade
Favorite Futures: Stocks
 
Posts: 975 since Jul 2012
Thanks: 659 given, 1,992 received


IB data has a known issue. It aggregates the data in windowed snapshots, so you don't see everything that happened inbetween snapshots..

IQF data is lossless from what I understand.

CME's (DataMine's) data preserves the original message trees, which allows you to construct a more accurate simulation than IQF's data, which undergoes some modifications before it reaches you.

The following 3 users say Thank You to artemiso for this post:
 
 
Elite Member
Sliema+Sliema/Malta
 
Futures Experience: None
Platform: Multicharts, Proprietary
Broker/Data: IB,IQFeed,Barcharts.com
Favorite Futures: Stocks,Futures
 
Posts: 40 since Mar 2014
Thanks: 87 given, 17 received

This is a common problem. It also depends on how well your data source filters bad ticks. Multicharts does not deal with this on its own. IQFeed is unfiltered but it also does not do a great job of correcting bad data. I also have a second feed that is much better at cleaning historical data but is problematic in other areas.

Keep in mind that your real time data might not match your historical data either regardless of who the provider is. Honestly, I don't think there is much you can do about it. Don't put too much faith in your back tests unless they correlate well with your forward tests.

 
 
Trading Apprentice
Christchurch, New Zealand
 
Futures Experience: Beginner
Platform: Multicharts
Broker/Data: Interactive Brokers/IQ Feed
Favorite Futures: E-mini S&P
 
Posts: 4 since Mar 2018
Thanks: 0 given, 0 received

I maybe should have been more specific, I am using the continuous backadjusted futures contract from IQ Feed and a custom futures contract for data from IB as they do not allow the continuous contracts to be queried through IB gateway yet or something.

I am aware of IBs snapshot but thought this only applies to really small timeframes (eg 1 second) and would have a minimal effect on the hour bars my system is built to trade on.

also, as far as the CME Vs globex data issue, IQ Feed appears to offer both CME data and CME_GLOBEX and was wondering if anyone could explain what the difference was? I presume regular CME data includes open out cry but the globex data is electronic only?

 
 
 Vendor: www.abctradinggroup.com 
Hamburg Germany
 
Futures Experience: Advanced
Platform: Multicharts, Tradestation
Broker/Data: DTN IQ
Favorite Futures: ES
 
Posts: 1,846 since Apr 2013
Thanks: 303 given, 1,130 received
Forum Reputation: Legendary

mitty,

I'd start with comparing the actual number of bars you get for the two instruments to make sure they are same.

You could also switch to the continuous contract for IQFeed to rule out any differences caused by the backadjustment or compare your system's results on the backadjusted contract versus the continuous contract both using IQFeed data.

Regards,

ABCTG

 
 
Elite Member
Berlin, Europe
 
Futures Experience: Advanced
Platform: NinjaTrader, MultiCharts
Broker/Data: Interactive Brokers
Favorite Futures: Keyboard
 
Fat Tails's Avatar
 
Posts: 9,732 since Mar 2010
Thanks: 4,231 given, 26,006 received

The data from Interactive Brokers is not suited for backtesting. There is a number of differences, notably


1. Daily data

DTN/IQ supplies to the official data published by CME. The bar close represents the daily settlement price calculated as an average from all transactions between 3:14:30 and 3:15:00 PM CT (different method used for the last business day of the month). Interactive Brokers aggregates intraday data to build daily bars. The bar close represents the last traded price prior to 4:00 PM CT.

2. Minute data

HIstorical minute data should be identical. However, Interactive Brokers does not supply genuine real-time volume. This means that with data supplied from Interactive Brokers volume indicators will not work correctly on your minute charts. Historical backfill supplied by IB shows correct volume. Volume figures will change when you replace real-time data with historical backfill.

3. Tick data

Interactive Brokers does not supply historical tick data. Real-time tick data is snapshot data, not showing all transactions and not showing correct volume.


Comparing backtests

When you use daily data or tick data, back tests will not match. When you use volume for your backtest, there will be minor discrepancies resulting from slightly different data for historical back fill.

In case that you find major differences when backtesting strategies on minute data, please look at the following issues

- are there any holes in your data?
- how are contract merged, what rollover offsets are used?

You should be able to match backtests on minute data taken from DTN/IQ and IB. Please pay attention to all minor details.

The following 2 users say Thank You to Fat Tails for this post:
 
 
Trading for Fun
Oslo + Norway
 
Futures Experience: Intermediate
Platform: NinjaTrader
Favorite Futures: ES
 
Posts: 141 since May 2016
Thanks: 144 given, 52 received

Is IQFeed still considered a reliable and good datafeed...?

It's what I've been and am using, but I'm not quite up to date about what else there is.

 
 
Elite Member
Sliema+Sliema/Malta
 
Futures Experience: None
Platform: Multicharts, Proprietary
Broker/Data: IB,IQFeed,Barcharts.com
Favorite Futures: Stocks,Futures
 
Posts: 40 since Mar 2014
Thanks: 87 given, 17 received

IQFeed has been working well and they have an excellent set of indexes that work in real time. It is however not without problems. I think it depends a lot on which platform you use so you can't just look at the feed alone.

I use Multicharts which has no way to deal with bad ticks except by manually correcting them. This is very frustrating. It also leads to incorrect back tests on intraday data. I have attached a snapshot of SPY and an indicator that tracks volatility. It gets completely distorted by these bad ticks which don't get corrected. You will have to decide if this is important to you.

regards,

Alex

Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).

 
 
 Vendor: www.abctradinggroup.com 
Hamburg Germany
 
Futures Experience: Advanced
Platform: Multicharts, Tradestation
Broker/Data: DTN IQ
Favorite Futures: ES
 
Posts: 1,846 since Apr 2013
Thanks: 303 given, 1,130 received
Forum Reputation: Legendary


Alex,

depending on what causes these "bad" ticks, you might have a chance to filter them by enabling the "Filter extended trades supplied by the exchange" option in the data feed settings for IQ (open the MC Quotemanager and
go to -> Tools -> Data Sources, select IQFeed and click on "Settings" to get to the window, where you can enable this option).

Regards,

ABCTG


 



futures io > > > > inconsistent results from different data sources
Thread Tools



Upcoming Webinars and Events (4:30PM ET unless noted)
 

Fundraiser: Feeding Families in Need (ends Dec 17)

December
 

Battlestations! Show us your trading desks w/GFF Brokers ($750 in Prizes)

January
 

An Afternoon With Morad Askar (aka FuturesTrader71)

Elite only
 

TBA w/Brett Steenbarger

Elite only
 

Traders Tax Advice w/Ryan Curran CPA

Elite only
 

Black Swan Events w/Adam Grimes

Elite only
 

Trading Preparedness w/GFF Brokers

Jan 22
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Cannot open charts or connect to data sources kieran MultiCharts 9 November 11th, 2015 03:25 AM
Inconsistent performance w/ different range of data asobi TradeStation 3 August 25th, 2015 10:08 AM
Will trading multiple pairs make my trading results inconsistent ?? sharpshoota Currency Futures 10 August 11th, 2015 10:54 AM
R - Financial data sources available in R Big Mike Matlab, R project and Python 9 August 6th, 2015 10:14 PM
Sources/Vendors for tick data RM99 Trading Reviews and Vendors 6 July 30th, 2011 10:56 PM


Tags
abc, average, back test, backadjust, backadjusted, backfill, backtest, backtesting, brokers, cme, continuous contract, continuous contracts, data, data feed, data issue, data source, datafeed, dow, dtn_iqfeed, ema, es, es futures, futures, globex, historical data, indexes, index_futures, indicator, indicators, information, interactive brokers, intraday, iq feed, iqfeed, market replay, market_replay, minute data, multicharts, ninjatrader, nt7, portfolio, real time, replay data, rollover, simulation, spy, strategies, strategy, switch, tick data, volatility, volume

All times are GMT -4. The time now is 12:44 AM. (this page content is cached, log in for real-time version)

Copyright © 2018 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432 WhatsApp Business, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2018-12-18 in 0.15 seconds with 33 queries on phoenix