NexusFi: Find Your Edge


Home Menu

 





Best data feed for backtesting?


Discussion in Brokers

Updated
      Top Posters
    1. looks_one NetTecture with 3 posts (5 thanks)
    2. looks_two Corinne with 3 posts (0 thanks)
    3. looks_3 Lampert with 2 posts (2 thanks)
    4. looks_4 kiloguru with 2 posts (0 thanks)
      Best Posters
    1. looks_one GoldStandard with 3 thanks per post
    2. looks_two program with 2 thanks per post
    3. looks_3 NetTecture with 1.7 thanks per post
    4. looks_4 Lampert with 1 thanks per post
    1. trending_up 12,140 views
    2. thumb_up 13 thanks given
    3. group 8 followers
    1. forum 15 posts
    2. attach_file 0 attachments




 
Search this Thread

Best data feed for backtesting?

  #11 (permalink)
 NetTecture 
Szczecin
 
Experience: Intermediate
Platform: Ninja, writing own now
Posts: 211 since Mar 2010

Still Nanex, but besids needing a programmer make sure you aer preapred to HANDLE that

I ahve CME from December 2010 to now -around 600gigabyte tape files, pre prosssing.

Equity 2008 till now? 4000gigabyte.

OPRA to that - 40.000 gigabyte

THE best feed, but nothing you can deal with on a small home computer with a large CPU only - you need storage arrays with decent IO performance. Enterprise level setup. Even the CME data is not trivial to process - you may want toparalellize data extraction (obviously), but reading multiple files even from a 4 disc RAID 5 slows seriously down once you do that for 6-8 files (been there, seen that - in my case it is easier to copy them one by one to a local SSD, then process local, then delete from SSD ...).

Reply With Quote

Can you help answer these questions
from other members on NexusFi?
My NT8 Volume Profile Split by Asian/Euro/Open
NinjaTrader
New Micros: Ultra 10-Year & Ultra T-Bond -- Live Now
Treasury Notes and Bonds
ZombieSqueeze
Platforms and Indicators
Deepmoney LLM
Elite Quantitative GenAI/LLM
Better Renko Gaps
The Elite Circle
 
  #12 (permalink)
 
program's Avatar
 program 
CA USA
 
Experience: Intermediate
Platform: MultiCharts
Broker: DDT+land/Rithmic/IQ
Trading: S&P500,6E,YM
Posts: 99 since Nov 2010
Thanks Given: 127
Thanks Received: 64


NetTecture View Post
Still Nanex, but besids needing a programmer make sure you aer preapred to HANDLE that

I ahve CME from December 2010 to now -around 600gigabyte tape files, pre prosssing.

Equity 2008 till now? 4000gigabyte.

OPRA to that - 40.000 gigabyte

THE best feed, but nothing you can deal with on a small home computer with a large CPU only - you need storage arrays with decent IO performance. Enterprise level setup. Even the CME data is not trivial to process - you may want toparalellize data extraction (obviously), but reading multiple files even from a 4 disc RAID 5 slows seriously down once you do that for 6-8 files (been there, seen that - in my case it is easier to copy them one by one to a local SSD, then process local, then delete from SSD ...).

Wow!!!! thats way over my head...

Corinne View Post
Hi,

This is my first post on futures.io (formerly BMT). Xyzzy's post pretty much described my situation...developing autotrading strategy using Multicharts for backtesting, trading through IB and am looking for a quality data provider with historical minute data going back +10 years.

I've been trying out DTN IQFeed for a couple of weeks and I finally figured out that IQFeed does not provide weekend data. I first noticed the weekend data gaps because the backtesting results always look unreasonable on the weekends. Has anyone also come across this a problem?

Is the top recommended data feed provider still Nanex/NXcore?

What are some other quality data feed provider I can explore...long historical minute data with weekend data and reasonably priced? Metastock? eSignal? FXCM?

Thanks.

the only one i can think of that fills your historical data needs would be tradestation.... otherwise i'd suggest sticking with IQ and downloading ASCII files. I downloaded 10+ years of ES tick data right here from futures.io (formerly BMT). you need to become an elite member though. But i can honestly say it being an elite member was easily one of the ten best investments i've made.


I have approximate answers, possible beliefs and different degrees of certainty of different things. But I'm not absolutely sure of anything and there are many things i don't know anything about.
Reply With Quote
Thanked by:
  #13 (permalink)
 Corinne 
Toronto, Ontario, Canada
 
Experience: Intermediate
Platform: NinjaTrader
Trading: forex, Eminis
Posts: 4 since Jul 2012
Thanks Given: 9
Thanks Received: 0



;244512
Still Nanex, but besids needing a programmer make sure you aer preapred to HANDLE that

Thanks, NetTecture. I’m not ready for Nanex yet.


program View Post
the only one i can think of that fills your historical data needs would be tradestation.... otherwise i'd suggest sticking with IQ and downloading ASCII files. I downloaded 10+ years of ES tick data right here from futures.io (formerly BMT). you need to become an elite member though. But i can honestly say it being an elite member was easily one of the ten best investments i've made.

Thanks, program. I’ll explore your data suggestions and futures.io (formerly BMT) elite membership as well.

A friend of mine found this last night: histdata dot com. Has anyone heard of or used their data? I’m going to look into it today.

Reply With Quote
  #14 (permalink)
 
Lampert's Avatar
 Lampert 
Calgary, Canada
 
Experience: Intermediate
Platform: Multicharts
Broker: IB, IQFeed
Trading: GC
Posts: 76 since Nov 2011
Thanks Given: 73
Thanks Received: 142


Corinne View Post
Hi,

This is my first post on futures.io (formerly BMT). Xyzzy's post pretty much described my situation...developing autotrading strategy using Multicharts for backtesting, trading through IB and am looking for a quality data provider with historical minute data going back +10 years.

I've been trying out DTN IQFeed for a couple of weeks and I finally figured out that IQFeed does not provide weekend data. I first noticed the weekend data gaps because the backtesting results always look unreasonable on the weekends. Has anyone also come across this a problem?

Is the top recommended data feed provider still Nanex/NXcore?

What are some other quality data feed provider I can explore...long historical minute data with weekend data and reasonably priced? Metastock? eSignal? FXCM?

Thanks.

Corinne - I have the same setup as you (IQFeed-Multicharts-IB). I assume you are interested in forex because equities and futures do not have weekend data. Here are a few comments:
  • The historical data in the futures.io (formerly BMT) elite section is mostly tick data for several different futures (ES, CL, TF, 6E). There is no forex data.
  • IQFeed provides forex data from two sources - FXCM and Tenfore (which is now owned by Morningstar)
  • The FXCM 1 minute data from IQFeed goes back seven years (to Feb 2005)
  • FXCM only provides data from 5:15 pm Sunday to 5 pm Friday - see reference page (no weekend data)
  • Tenfore data may be more extensive as this is a premium service from IQFeed ($50/month). You would have to ask IQFeed for details.
Personally I am very happy with IQFeed data (realtime and historical). Because the FXCM historical data is essentially free with the IQFeed basic service, you might consider simply modifying your backtesting to exclude the weekend periods.

Len

Reply With Quote
Thanked by:
  #15 (permalink)
 Corinne 
Toronto, Ontario, Canada
 
Experience: Intermediate
Platform: NinjaTrader
Trading: forex, Eminis
Posts: 4 since Jul 2012
Thanks Given: 9
Thanks Received: 0


Lampert View Post
Personally I am very happy with IQFeed data (realtime and historical). Because the FXCM historical data is essentially free with the IQFeed basic service, you might consider simply modifying your backtesting to exclude the weekend periods.

Thanks for the info, Len. I'm thinking about excluding weekends in my backtesting too. There shouldn't be much, if any, effect on the day trades but what if I have an open swing trade that I carry over the weekend?

I also wonder how that would affect the backtest results. Wouldn't the weekend data gap somewhat distorts many of the indicator values e.g. moving averages, adx...?

Reply With Quote
  #16 (permalink)
 
Lampert's Avatar
 Lampert 
Calgary, Canada
 
Experience: Intermediate
Platform: Multicharts
Broker: IB, IQFeed
Trading: GC
Posts: 76 since Nov 2011
Thanks Given: 73
Thanks Received: 142


Corinne View Post
Thanks for the info, Len. I'm thinking about excluding weekends in my backtesting too. There shouldn't be much, if any, effect on the day trades but what if I have an open swing trade that I carry over the weekend?

I also wonder how that would affect the backtest results. Wouldn't the weekend data gap somewhat distorts many of the indicator values e.g. moving averages, adx...?

Corinne - Yes price gaps over the weekend will affect open positions and indicators. If this is significant with your strategy, you can choose to delay opening any positions after the weekend until the indicators are reasonable and to close all positions by the end of the week.

Len

Reply With Quote
Thanked by:




Last Updated on July 26, 2012


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts