AmiBroker IQFeed historical tick data - AmiBroker | futures io social day trading
futures io futures trading


AmiBroker IQFeed historical tick data
Updated: Views / Replies:7,156 / 29
Created: by Big Mike Attachments:0

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 
Thread Tools Search this Thread
 

AmiBroker IQFeed historical tick data

  #11 (permalink)
Banned: Previously banned user, now re-banned
London/UK
 
Futures Experience: Advanced
Platform: AmiBroker
Favorite Futures: Any
 
Posts: 56 since Apr 2013
Thanks: 0 given, 12 received


Big Mike View Post
Since all other platforms I've ever used have properly backfilled 120 days of tick data from IQFeed, the issue is NOT IQFeed but rather AmiBroker's implementation.

I don't know how IQfeeds plaftorm works and how it looks like but why does IQFeed's own historical download platform not download full 120 days? Can you explain that? So no AB, Sierra chart etc involved. And why does Sierra Chart mention a limit during market hours? That's kind of a contradiction.

Reply With Quote
 
  #12 (permalink)
Banned: Previously banned user, now re-banned
London/UK
 
Futures Experience: Advanced
Platform: AmiBroker
Favorite Futures: Any
 
Posts: 56 since Apr 2013
Thanks: 0 given, 12 received


Big Mike View Post
Now I am being told just to display 120 days of tick data for a SINGLE CHART, I would need 256 GB of RAM.

I'm not sure if you are mixing up RAM with bars here but

2 million ticks/day * 120 days = 240 000 000 bars
240 million bars * 40 bytes are 9.6 GB not 256 GB.

256GB would be 6.4 billion bars.

Reply With Quote
 
  #13 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,214 received



badabingbadaboom View Post
I don't know how IQfeeds plaftorm works and how it looks like but why does IQFeed's own historical download platform not download full 120 days? Can you explain that? So no AB, Sierra chart etc involved. And why does Sierra Chart mention a limit during market hours? That's kind of a contradiction.

Look, I think we can safely stop corresponding. I was hoping it was user error when I created the thread. It is not user error, but AmiBroker error. There is no need to have this conversation both in the thread, and with Tomasz at AmiBroker.

To briefly answer your question, which I have already told AmiBroker, the issue is with requesting the data all at once. You should not do this, but rather break it into chunks. If AmiBroker had tested the 120 days of tick data backfill, they should have noticed there was a problem with their implementation prior to me bringing it to their attention.

While I was securing the proof that IQFeed even had 120 days of tick data, because Tomasz initially told me I was "dreaming in fantasy land" if I thought they did, I also talked to IQFeed about the problem and the first thing they commented on was that most likely the issue was AmiBroker not splitting the request into chunks. It is a common thing.

As for market hours, I do not know what you are implying. It is well known IQFeed has a market hours and after market hours policy. I am not confused by this and it has nothing to do with the problem.


badabingbadaboom View Post
I'm not sure if you are mixing up RAM with bars here but

2 million ticks/day * 120 days = 240 000 000 bars
240 million bars * 40 bytes are 9.6 GB not 256 GB.

256GB would be 6.4 billion bars.

You'd have to ask Tomasz. It is his figure.

This is my final reply in this thread until there is progress in one way or another from AmiBroker.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #14 (permalink)
Banned: Previously banned user, now re-banned
London/UK
 
Futures Experience: Advanced
Platform: AmiBroker
Favorite Futures: Any
 
Posts: 56 since Apr 2013
Thanks: 0 given, 12 received


Big Mike View Post
As for market hours, I do not know what you are implying. It is well known IQFeed has a market hours and after market hours policy. I am not confused by this and it has nothing to do with the problem.

I only mentioned what has been tried by others. For example I have tried with another user who uses IQfeed on 5 minute timeframe to download tick bars and failed to download complete 120 days or rather the complete 10 million maximum tick bars been set. Also that other befriended user has sent a request to IQfeed today and they replied

Quoting 
Hello,

Iqfeed tick data is limited to 7 days during market hours and after market hours is 120 days.



Thanks,



Justin Siekman

Customer Service Representative

DTN

DTNIQ/IQFEED - 800/779-7299 or 402/255-8787

And if IQFeeds own application also doesn't download full 120 days without any other software involved then there indeed seems to be a problem with their download. Again I can't confirm the last one as I haven't used it myself but it has been told to me today because I was interested in that issue myself.



Big Mike View Post
You should not do this, but rather break it into chunks.

I'm not sure why and how I should do this? AB is not open source and therefore I don't have access to AB's source code.


Last edited by badabingbadaboom; July 2nd, 2013 at 03:34 PM.
Reply With Quote
 
  #15 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,214 received

After some more email exchanges with Tomasz, the conclusion is that he has reproduced my issue but says that no one else has ever asked for this -- so he deems it is low priority and says it will take "months at least" for him to fix it.

I cannot help but feel like the fix should not take months. I think he is mad at me personally...

Anyway, in his defense he offered a refund of AmiBroker over the problem. I told him I've had it for only 1 day and would really like more time with it, 120 days of IQFeed backfill is not the only thing that is important to me, so if it does other things well then there would be no need for a refund.

I am still hopeful that it is as optimized and powerful as I have read. More time will tell.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #16 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,214 received


badabingbadaboom View Post
I only mentioned what has been tried by others. For example I have tried with another user who uses IQfeed on 5 minute timeframe to download tick bars and failed to download complete 120 days or rather the complete 10 million maximum tick bars been set. Also that other befriended user has sent a request to IQfeed today and they replied

Again, the IQFeed policy limit is well known and has nothing at all to do with my issue.


Quoting 
And if IQFeeds own application also doesn't download full 120 days without any other software involved then there indeed seems to be a problem with their download. Again I can't confirm the last one as I haven't used it myself but it has been told to me today because I was interested in that issue myself.

Yes, I spoke to Jay at IQFeed and he confirmed that Tomasz found a bug and IQFeed will patch it shortly. But I think you are missing the point. My understanding of the good practices is to retrieve the backfill in chunks instead of making a single request. AmiBroker seems to make a single request, resulting in timeouts, overflow errors, or similar. All other charting programs do not have this problem, so from my perspective it seems like IQFeed has no issue filling the backfill requests with other programs and AmiBroker should be able to do the same.

In the end, he told me a "patch" would be coming -- in several months.

There is no need to continue going back and forth.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
The following user says Thank You to Big Mike for this post:
 
  #17 (permalink)
Elite Member
Paris, France
 
Futures Experience: Advanced
Platform: Amibroker
Favorite Futures: Futures, Stocks
 
Posts: 96 since Jun 2011
Thanks: 118 given, 102 received

In my opinion, you've spotted the weakest thing is AB i.e. handling tick data.

Even if you had successfully downloaded the data your backtests would have taken ages to run (but maybe still quicker than MC or Ninja).

AB has many, many other great features and I really hope to see more constructive threads about it at futures.io (formerly BMT) in the future.

Reply With Quote
 
  #18 (permalink)
Elite Member
Karlsruhe, Germany
 
Futures Experience: Intermediate
Platform: MultiCharts, AmiBroker
Broker/Data: IB/IQFeed
Favorite Futures: FDAX, ES
 
medias's Avatar
 
Posts: 60 since Jul 2009
Thanks: 30 given, 49 received

Workaround suggestion: QCollector with offline database

I guess you need the (long) history of 120 days of tickdata for backtesting purposes.

So you could build an offline database and import tickdata that is downloaded with QCollector. Use an offline database as you should not mix online (realtime) datafeed with imported data.

The import process can be automated on both sides: Qcollector and Amibroker provide convenient ways to do that.
In Amibroker use this approach:
Tools > Customize... [Tab:Tools] > [Button: New]
Command: #import
Arguments: your path to local stored tickfile
Initial directory: your path to local format file.

See "Update US Symbol list and category" as a starting point. Of course this example only loads symbols and categories, but you should get the point.

Once done, you can import data with a single mouse-click.

But be warned: in online mode (with real data) use a separate database. Never mix manual backfills with realtime data.
BTW: You can start multiple instances of Amibroker simultaneously.

Best regards,
medias.

Reply With Quote
The following 2 users say Thank You to medias for this post:
 
  #19 (permalink)
Elite Member
Munich Germany
 
Futures Experience: Beginner
Platform: MultiCharts .NET,MultiCharts,Amibroker
Broker/Data: Interactive Brokers IQFeed
Favorite Futures: Stocks,Index CFDs
 
Posts: 149 since Aug 2012
Thanks: 81 given, 77 received

Try to find a clue why


badabingbadaboom View Post
I'm not sure if you are mixing up RAM with bars here but

2 million ticks/day * 120 days = 240 000 000 bars
240 million bars * 40 bytes are 9.6 GB not 256 GB.

256GB would be 6.4 billion bars.

Do you know how those 40 bytes are generated ?
I think
1 double has 8 bytes, right ?
So 5 values in a bar structure OHCLV but Date is missing
Therefore a bar would be structured in the following way.
4 bytes for a int for the Volume maybe also a short 2 bytes would do
4 double with 8 bytes
and the rest for the Date Time

This leads to the conclusion that for storing tickdata
a tick structure of 8 bytes (volume and Date Time) and 8 bytes for Price is needed
2 million ticks/day * 120days * 16 bytes =3.84 GB

But I am not sure if that is needed in the RAM because there is always only a limited lookback period.
Even if I will display the complete data by shrinking my timescale in an chart, i guess that the chart application will reduce the needed data in an intelligent way. Because the eye cannot see every tick and the indicator calculation is maybe only done on slices. I think that is the way a good program should be reduce data.

Maybe that could be the reason why it would take that long for Tomasz to implement that feature.
Because maybe Amibroker has no strategy to deal in the above discribed way with data.

Reply With Quote
 
  #20 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,214 received



medias View Post
I guess you need the (long) history of 120 days of tickdata for backtesting purposes.

So you could build an offline database and import tickdata that is downloaded with QCollector. Use an offline database as you should not mix online (realtime) datafeed with imported data.

The import process can be automated on both sides: Qcollector and Amibroker provide convenient ways to do that.
In Amibroker use this approach:
Tools > Customize... [Tab:Tools] > [Button: New]
Command: #import
Arguments: your path to local stored tickfile
Initial directory: your path to local format file.

See "Update US Symbol list and category" as a starting point. Of course this example only loads symbols and categories, but you should get the point.

Once done, you can import data with a single mouse-click.

But be warned: in online mode (with real data) use a separate database. Never mix manual backfills with realtime data.
BTW: You can start multiple instances of Amibroker simultaneously.

Best regards,
medias.

Hi,

Yes the 120 days of IQFeed was just a fast simple test, I wasn't expecting to hit a brick wall. 120 days of tick data is not useful by itself, it's just useful for backfill. I have my main MySQL databases with years and years of tick data for many instruments, which I would need to import to do any useful testing.

I have not yet started that process, and will likely make a separate thread to discuss it.

One thing in your reply concerns me and caught my attention however --- you said "never mix realtime with historical", why is that? This is a concern, because my typical behavior is to import the data from my database into the platform, then use the platform to backfill and track data going forward -- both realtime and backfill as needed. Can you go into more detail on this?

Cool, I did not know you could run multiple concurrent instances of AmiBroker so that is good. I also had meant to do some research about what the licensing allows --- is it a single user license, but can be deployed on multiple pieces of hardware (ie: my development workstation + my production server) concurrently, as a single user? Or would it require multiple licenses. Need to check on that still.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote

Reply



futures io > > > > AmiBroker IQFeed historical tick data

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Linda Bradford Raschke: Reading The Tape

Elite only

Adam Grimes: TBA

Elite only

NinjaTrader: TBA

January

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Historical CL Tick Data and Minute Data Big Mike The Elite Circle 112 July 26th, 2014 07:07 PM
Data Feed: How do you confirm Accuracy vs Historical tick data provided. FAUGH45568 Reviews of Brokers and Data Feeds 2 February 14th, 2013 07:04 PM
Historical YM Tick Data and Minute Data?? Hermo The Elite Circle 3 December 1st, 2012 02:56 PM
MultiCharts and IQfeed alignment with 120 days of Tick Data??? benharrell Reviews of Brokers and Data Feeds 36 June 11th, 2011 04:06 PM
Platform that can go tick by tick through historical data acdrew0 Platforms and Indicators 1 October 21st, 2010 01:10 PM


All times are GMT -4. The time now is 03:58 AM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-11 in 0.17 seconds with 19 queries on phoenix via your IP 54.226.113.250