Free (or cheap) daily historical (EOD) data which could be retrieved programmatically - Reviews of Brokers and Data Feeds | futures io social day trading
futures io futures trading


Free (or cheap) daily historical (EOD) data which could be retrieved programmatically
Updated: Views / Replies:4,937 / 12
Created: by Nicolas11 Attachments:10

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 10  
 
Thread Tools Search this Thread
 

Free (or cheap) daily historical (EOD) data which could be retrieved programmatically

  #1 (permalink)
Elite Member
near Paris, France
 
Futures Experience: Beginner
Platform: -
Favorite Futures: -
 
Nicolas11's Avatar
 
Posts: 1,070 since Aug 2011
Thanks: 2,232 given, 1,729 received

Free (or cheap) daily historical (EOD) data which could be retrieved programmatically

Hi,

In this thread, I propose to list resources for free (or cheap) daily historical (EOD) data which could be retrieved programmatically.

In the following messages, links will be provided in relation with:
- Yahoo Finance
- Google Finance
- Quandl
- Sierra Chart

Feel free to contribute!

Nicolas


Last edited by Nicolas11; April 16th, 2014 at 06:22 PM.
Reply With Quote
The following 2 users say Thank You to Nicolas11 for this post:
 
  #2 (permalink)
Quick Summary
Quick Summary Post

Quick Summary is created and edited by users like you... Add FAQ's, Links and other Relevant Information by clicking the edit button in the lower right hand corner of this message.

 
  #3 (permalink)
Elite Member
near Paris, France
 
Futures Experience: Beginner
Platform: -
Favorite Futures: -
 
Nicolas11's Avatar
 
Posts: 1,070 since Aug 2011
Thanks: 2,232 given, 1,729 received


Yahoo Finance (free) has been addressed in this thread, with examples of codes in R and Java: https://futures.io/brokers-data-feeds/31382-yahoo-finance-historical-daily-data-retrieved-programmatically.html

Google Finance (free) has been addressed in this thread, with examples of codes in R and Java: https://futures.io/brokers-data-feeds/31385-google-finance-historical-daily-data-retrieved-programmatically.html

Quandl (free) has been addressed in this thread, with examples of codes in R and Java: https://futures.io/vendors-product-reviews/29501-quandl-www-quandl-com.html


Last edited by Nicolas11; April 16th, 2014 at 05:49 PM.
Reply With Quote
The following 3 users say Thank You to Nicolas11 for this post:
 
  #4 (permalink)
Elite Member
near Paris, France
 
Futures Experience: Beginner
Platform: -
Favorite Futures: -
 
Nicolas11's Avatar
 
Posts: 1,070 since Aug 2011
Thanks: 2,232 given, 1,729 received

Sierra Chart is not free.

Packages 3 and 5 include:
- historical data for many exchanges: Sierra Chart Historical Data Service - Sierra Chart
- historical data for FOREX and CFD: Sierra Chart Real Time and Historical Forex and CFD Data Service - Sierra Chart

The interest is that the data is stored on the computer as plain files, directly readable (you can open them with Notepad).

So, we would first need to open SC and update the charts.

Then, it is possible to close SC, and access the information from R (example here: https://futures.io/elite-circle/29755-r-import-sierra-chart-data-price-indicators-into-r.html#post372645) or any other programming language, for instance Java.

This is the end of the series. Do not hesitate to add other ideas.

Nicolas

Reply With Quote
The following user says Thank You to Nicolas11 for this post:
 
  #5 (permalink)
Elite Member
near Paris, France
 
Futures Experience: Beginner
Platform: -
Favorite Futures: -
 
Nicolas11's Avatar
 
Posts: 1,070 since Aug 2011
Thanks: 2,232 given, 1,729 received

As reminded by ratfink in another trade, Kinetic EOD data provided though NinjaTrader is another free source of data, which can be accessed programmatically from outside through .ntd files.

Reply With Quote
The following user says Thank You to Nicolas11 for this post:
 
  #6 (permalink)
Patient Discipline
Bellingham, WA USA
 
Futures Experience: Beginner
Platform: NT
Broker/Data: Mirus (Broker), Continuum (Data), Dorman (Clearing)
Favorite Futures: Futures
 
aventeren's Avatar
 
Posts: 202 since Mar 2013
Thanks: 426 given, 191 received

Nicolas--

I've been looking for daily price data, too--albeit for my use within NT.

I've built a tool for NT that uses .NET to make a WebRequest to Quandl for Historical Contract data (here is Quandl's wheat landing site: https://www.quandl.com/c/futures/cme-wheat-futures), at which point I parse out the data into a NT7 compatible import format (NinjaTrader Version 7), and then write the cleaned up data to a properly named text file (i.e., ZW 12-59.Last.txt)--at which point this text file can be imported into NT7. I built it to have an end date, so you can download all of the historical contract data within a time span (i.e., in the case of wheat, 1959 - 2014) by contract (i.e., F, G, H, J, etc). It's pretty cool.

Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


But, and here is the big but(t) with Quandl, it's tough to know how good this data is. For instance, you can click on their Continuous Contracts to see a merged chart that uses all of the data (presumably) within the Historical Contract data files, and the data gaps jump right out at you. For instance, here are various Continuous Contracts that show the gaps.

Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).

Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).

Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).

Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).

Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


So as you can see, a number of the daily Quandl futures contracts have data gaps that would impact a backtest (I looked at ES, GC, ZW, ZC, ZS, 6A, 6B, 6C, 6E, 6J, 6L, 6M, 6N, 6R, 6S, CL, HG, NG, NQ, PL, SI, YM, TF, ZN, ZF, ZT and SP). And the real challenge is that I haven't run across a tool that would allow us to determine whether the free data that you've mentioned here (i.e., in your first post)--plus the various tick sharing threads on futures.io (formerly BMT) (i.e., CL, ES, QCollector, etc)--is any good. So to the extent that we can think about and collaborate on some sort of tool/standard that we could use as a community to confirm various data sources, it would seem to me that we need to be very careful.

Here is an idea I've been thinking about with regard to how to address data cleanliness.

Either futures.io (formerly BMT) purchases or a current futures.io (formerly BMT) members offers up a subscription to a historical data service (i.e., Bloomberg, NANEX, etc) that either BigMike or the futures.io (formerly BMT) member offering up the subscription could then use to test various data files against (I'm thinking like some sort of a grader with a report card per run). The report card could report what the discrepancies were in the submitted data and the subscription feed--presumably so that the submitted data could be edited to bring the data in line with the subscription data. This edited data could then be posted in a new forum thread called "Data" or something by instrument (i.e., CL, ZW, ES, etc). The idea here is that the community is out collecting data, but futures.io (formerly BMT) has a standard by which collected data is graded against so that futures.io (formerly BMT) Elite Members (and heck, I could see creating a new membership beyond Elite for this) knew that the data we were using was sound.

However in the mean time, I think given that there is not some sort of a data quality standard on futures.io (formerly BMT) yet, we need to be careful about using the free data that's posted on the site and elsewhere. For instance, I wrote my Quandl Data Grabber indicator to download a ton of Quandl daily data into NT7 format for the express purposes of posting everything I downloaded on futures.io (formerly BMT). But then I looked at the data gaps and I'm pausing on posting the historical contract data, as I don't want others to be using this data until it's in a better shape.

In any event, those are my current thoughts on free data. I'd be interested to hear your thoughts when you have a moment.

All best,

Aventeren

Reply With Quote
The following 2 users say Thank You to aventeren for this post:
 
  #7 (permalink)
Trading for Fun
Buffalo, NY
 
Futures Experience: None
Platform: Python, Excel, R, Ninjatrader
Broker/Data: IB quandl
Favorite Futures: Low frequency trading
 
Posts: 64 since Jul 2014
Thanks: 4 given, 37 received

Eat your oatmeal, it is good for you
anaconda/ipython notebook

%matplotlib inline
import Quandl
mpl.rc('figure', figsize=(10, 7))

start = "2007-01-01"
end = "2014-09-01"
token = "YOUR TOKEN HERE"

data = Quandl.get("OFDP/FUTURE_O1",trim_start = start,trim_end = end, authtoken=token)

data['Settle'].plot()

Attached Thumbnails
Free (or cheap) daily historical (EOD) data which could be retrieved programmatically-quanoats.jpg  
Reply With Quote
 
  #8 (permalink)
Trading for Fun
Buffalo, NY
 
Futures Experience: None
Platform: Python, Excel, R, Ninjatrader
Broker/Data: IB quandl
Favorite Futures: Low frequency trading
 
Posts: 64 since Jul 2014
Thanks: 4 given, 37 received


aventeren View Post
So as you can see, a number of the daily Quandl futures contracts have data gaps that would impact a backtest (I looked at ES, GC, ZW, ZC, ZS, 6A, 6B, 6C, 6E, 6J, 6L, 6M, 6N, 6R, 6S, CL, HG, NG, NQ, PL, SI, YM, TF, ZN, ZF, ZT and SP).


I think those charts above look scarier than they are. If you are grabbing data in python and leaving it as a pandas dataframe a NaN isn't going to mess up any kind of backtesting. A missing value isn't the price going to zero like how it looks on those charts.


I think to get all hung up on bad ticks is to view the market too deterministically anyway. No strategy is failing because it uses quandl or yahoo instead of bloomberg. That is a pretty clear self deception IMO.


Last edited by NoiseTrader716; September 16th, 2014 at 11:07 PM.
Reply With Quote
 
  #9 (permalink)
Patient Discipline
Bellingham, WA USA
 
Futures Experience: Beginner
Platform: NT
Broker/Data: Mirus (Broker), Continuum (Data), Dorman (Clearing)
Favorite Futures: Futures
 
aventeren's Avatar
 
Posts: 202 since Mar 2013
Thanks: 426 given, 191 received

You may be right. If nothing else, hopefully our discussion here will prompt others to at least give their data a cursory once over instead of blindly assuming that it is sound.

I'm currently working on incorporating some rollover date and offset functionality to my Quandl grabber (based on volume, open interest or volume and open interest). We can't programmatically define new expiries via NScript, so the indicator will just print the expiries and their rollover (roll into) dates and the corresponding offsets, which will then need to be manually created and entered in to NT.

I wish I had the time to go down the R rabbit hole (i.e., other languages beyond NT7), but I just need to stay focused on .NET right now.

Thanks for your thoughts.

Reply With Quote
 
  #10 (permalink)
Trading for Fun
Buffalo, NY
 
Futures Experience: None
Platform: Python, Excel, R, Ninjatrader
Broker/Data: IB quandl
Favorite Futures: Low frequency trading
 
Posts: 64 since Jul 2014
Thanks: 4 given, 37 received


I guess I view things along the lines if you had a database of 10,000 poker hands with the money that exchanged hands on each one, even if you deleted 5000 of the hands you would still have a great sample of the process at work. If you had 3 companies recording the poker hands, there would be no value in tracking down the reason for the discrepancy on hand #7877 with all 3 companies had the players holding different cards.

I think most trading backtesting is fools gold because like in the above example you could test for what happens when a player gets a certain hand over a certain number of deals and if they win or lose but that tells you absolutely nothing about what is going to happen on the next hand in the future. A deck of cards is surely more deterministic than the markets but we tend to think in terms of the opposite.

All you really need to know for R is how to get data in and then transform it to the input the function you want to use takes.

Reply With Quote
The following user says Thank You to NoiseTrader716 for this post:

Reply



futures io > > > > Free (or cheap) daily historical (EOD) data which could be retrieved programmatically

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Jigsaw Trading: TBA

Elite only

FuturesTrader71: TBA

Elite only

NinjaTrader: TBA

Jan 18

RandBots: TBA

Jan 23

GFF Brokers & CME Group: Futures & Bitcoin

Elite only

Adam Grimes: TBA

Elite only

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Google Finance - historical daily data retrieved programmatically Nicolas11 Reviews of Brokers and Data Feeds 5 April 16th, 2014 05:14 PM
Yahoo Finance - historical daily data retrieved programmatically Nicolas11 Reviews of Brokers and Data Feeds 5 April 16th, 2014 03:10 PM
daily OHLC data with time stamp from historical data edassery NinjaTrader 4 January 4th, 2014 07:44 AM
Who tried the free EOD futures data for NinjaTrader? Fat Tails Reviews of Brokers and Data Feeds 2 May 21st, 2010 08:07 PM
Free Daily (EOD) continuous eMini ES data? sharp2be Reviews of Brokers and Data Feeds 2 May 2nd, 2010 11:43 PM


All times are GMT -4. The time now is 06:23 AM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-14 in 0.14 seconds with 20 queries on phoenix via your IP 54.160.245.121