NexusFi: Find Your Edge


Home Menu

 





AmiBroker IQFeed historical tick data


Discussion in Platforms and Indicators

Updated
      Top Posters
    1. looks_one badabingbadaboom with 11 posts (3 thanks)
    2. looks_two Big Mike with 10 posts (3 thanks)
    3. looks_3 medias with 2 posts (5 thanks)
    4. looks_4 prouser with 2 posts (0 thanks)
      Best Posters
    1. looks_one medias with 2.5 thanks per post
    2. looks_two bob7123 with 1 thanks per post
    3. looks_3 badabingbadaboom with 0.3 thanks per post
    4. looks_4 Big Mike with 0.3 thanks per post
    1. trending_up 14,764 views
    2. thumb_up 12 thanks given
    3. group 9 followers
    1. forum 29 posts
    2. attach_file 0 attachments




 
Search this Thread

AmiBroker IQFeed historical tick data

  #11 (permalink)
 badabingbadaboom 
London/UK
 
Posts: 56 since Apr 2013


Big Mike View Post
Since all other platforms I've ever used have properly backfilled 120 days of tick data from IQFeed, the issue is NOT IQFeed but rather AmiBroker's implementation.

I don't know how IQfeeds plaftorm works and how it looks like but why does IQFeed's own historical download platform not download full 120 days? Can you explain that? So no AB, Sierra chart etc involved. And why does Sierra Chart mention a limit during market hours? That's kind of a contradiction.

Reply With Quote

Can you help answer these questions
from other members on NexusFi?
Request for MACD with option to use different MAs for fa …
NinjaTrader
ZombieSqueeze
Platforms and Indicators
NexusFi Journal Challenge - May 2024
Feedback and Announcements
 
Best Threads (Most Thanked)
in the last 7 days on NexusFi
Retail Trading As An Industry
66 thanks
NexusFi site changelog and issues/problem reporting
46 thanks
Battlestations: Show us your trading desks!
35 thanks
GFIs1 1 DAX trade per day journal
32 thanks
What percentage per day is possible? [Poll]
25 thanks

  #12 (permalink)
 badabingbadaboom 
London/UK
 
Posts: 56 since Apr 2013


Big Mike View Post
Now I am being told just to display 120 days of tick data for a SINGLE CHART, I would need 256 GB of RAM.

I'm not sure if you are mixing up RAM with bars here but

2 million ticks/day * 120 days = 240 000 000 bars
240 million bars * 40 bytes are 9.6 GB not 256 GB.

256GB would be 6.4 billion bars.

Reply With Quote
  #13 (permalink)
 
Big Mike's Avatar
 Big Mike 
Manta, Ecuador
Site Administrator
Developer
Swing Trader
 
Experience: Advanced
Platform: Custom solution
Broker: IBKR
Trading: Stocks & Futures
Frequency: Every few days
Duration: Weeks
Posts: 50,319 since Jun 2009
Thanks Given: 33,139
Thanks Received: 101,474



badabingbadaboom View Post
I don't know how IQfeeds plaftorm works and how it looks like but why does IQFeed's own historical download platform not download full 120 days? Can you explain that? So no AB, Sierra chart etc involved. And why does Sierra Chart mention a limit during market hours? That's kind of a contradiction.

Look, I think we can safely stop corresponding. I was hoping it was user error when I created the thread. It is not user error, but AmiBroker error. There is no need to have this conversation both in the thread, and with Tomasz at AmiBroker.

To briefly answer your question, which I have already told AmiBroker, the issue is with requesting the data all at once. You should not do this, but rather break it into chunks. If AmiBroker had tested the 120 days of tick data backfill, they should have noticed there was a problem with their implementation prior to me bringing it to their attention.

While I was securing the proof that IQFeed even had 120 days of tick data, because Tomasz initially told me I was "dreaming in fantasy land" if I thought they did, I also talked to IQFeed about the problem and the first thing they commented on was that most likely the issue was AmiBroker not splitting the request into chunks. It is a common thing.

As for market hours, I do not know what you are implying. It is well known IQFeed has a market hours and after market hours policy. I am not confused by this and it has nothing to do with the problem.


badabingbadaboom View Post
I'm not sure if you are mixing up RAM with bars here but

2 million ticks/day * 120 days = 240 000 000 bars
240 million bars * 40 bytes are 9.6 GB not 256 GB.

256GB would be 6.4 billion bars.

You'd have to ask Tomasz. It is his figure.

This is my final reply in this thread until there is progress in one way or another from AmiBroker.

Mike

We're here to help: just ask the community or contact our Help Desk

Quick Links: Change your Username or Register as a Vendor
Searching for trading reviews? Review this list
Lifetime Elite Membership: Sign-up for only $149 USD
Exclusive money saving offers from our Site Sponsors: Browse Offers
Report problems with the site: Using the NexusFi changelog thread
Follow me on Twitter Visit my NexusFi Trade Journal Started this thread Reply With Quote
  #14 (permalink)
 badabingbadaboom 
London/UK
 
Posts: 56 since Apr 2013


Big Mike View Post
As for market hours, I do not know what you are implying. It is well known IQFeed has a market hours and after market hours policy. I am not confused by this and it has nothing to do with the problem.

I only mentioned what has been tried by others. For example I have tried with another user who uses IQfeed on 5 minute timeframe to download tick bars and failed to download complete 120 days or rather the complete 10 million maximum tick bars been set. Also that other befriended user has sent a request to IQfeed today and they replied

Quoting 
Hello,

Iqfeed tick data is limited to 7 days during market hours and after market hours is 120 days.



Thanks,



Justin Siekman

Customer Service Representative

DTN

DTNIQ/IQFEED - 800/779-7299 or 402/255-8787

And if IQFeeds own application also doesn't download full 120 days without any other software involved then there indeed seems to be a problem with their download. Again I can't confirm the last one as I haven't used it myself but it has been told to me today because I was interested in that issue myself.



Big Mike View Post
You should not do this, but rather break it into chunks.

I'm not sure why and how I should do this? AB is not open source and therefore I don't have access to AB's source code.

Reply With Quote
  #15 (permalink)
 
Big Mike's Avatar
 Big Mike 
Manta, Ecuador
Site Administrator
Developer
Swing Trader
 
Experience: Advanced
Platform: Custom solution
Broker: IBKR
Trading: Stocks & Futures
Frequency: Every few days
Duration: Weeks
Posts: 50,319 since Jun 2009
Thanks Given: 33,139
Thanks Received: 101,474

After some more email exchanges with Tomasz, the conclusion is that he has reproduced my issue but says that no one else has ever asked for this -- so he deems it is low priority and says it will take "months at least" for him to fix it.

I cannot help but feel like the fix should not take months. I think he is mad at me personally...

Anyway, in his defense he offered a refund of AmiBroker over the problem. I told him I've had it for only 1 day and would really like more time with it, 120 days of IQFeed backfill is not the only thing that is important to me, so if it does other things well then there would be no need for a refund.

I am still hopeful that it is as optimized and powerful as I have read. More time will tell.

Mike

We're here to help: just ask the community or contact our Help Desk

Quick Links: Change your Username or Register as a Vendor
Searching for trading reviews? Review this list
Lifetime Elite Membership: Sign-up for only $149 USD
Exclusive money saving offers from our Site Sponsors: Browse Offers
Report problems with the site: Using the NexusFi changelog thread
Follow me on Twitter Visit my NexusFi Trade Journal Started this thread Reply With Quote
  #16 (permalink)
 
Big Mike's Avatar
 Big Mike 
Manta, Ecuador
Site Administrator
Developer
Swing Trader
 
Experience: Advanced
Platform: Custom solution
Broker: IBKR
Trading: Stocks & Futures
Frequency: Every few days
Duration: Weeks
Posts: 50,319 since Jun 2009
Thanks Given: 33,139
Thanks Received: 101,474


badabingbadaboom View Post
I only mentioned what has been tried by others. For example I have tried with another user who uses IQfeed on 5 minute timeframe to download tick bars and failed to download complete 120 days or rather the complete 10 million maximum tick bars been set. Also that other befriended user has sent a request to IQfeed today and they replied

Again, the IQFeed policy limit is well known and has nothing at all to do with my issue.


Quoting 
And if IQFeeds own application also doesn't download full 120 days without any other software involved then there indeed seems to be a problem with their download. Again I can't confirm the last one as I haven't used it myself but it has been told to me today because I was interested in that issue myself.

Yes, I spoke to Jay at IQFeed and he confirmed that Tomasz found a bug and IQFeed will patch it shortly. But I think you are missing the point. My understanding of the good practices is to retrieve the backfill in chunks instead of making a single request. AmiBroker seems to make a single request, resulting in timeouts, overflow errors, or similar. All other charting programs do not have this problem, so from my perspective it seems like IQFeed has no issue filling the backfill requests with other programs and AmiBroker should be able to do the same.

In the end, he told me a "patch" would be coming -- in several months.

There is no need to continue going back and forth.

Mike

We're here to help: just ask the community or contact our Help Desk

Quick Links: Change your Username or Register as a Vendor
Searching for trading reviews? Review this list
Lifetime Elite Membership: Sign-up for only $149 USD
Exclusive money saving offers from our Site Sponsors: Browse Offers
Report problems with the site: Using the NexusFi changelog thread
Follow me on Twitter Visit my NexusFi Trade Journal Started this thread Reply With Quote
The following user says Thank You to Big Mike for this post:
  #17 (permalink)
 TheTrend 
Paris, France
 
Experience: Advanced
Platform: Amibroker
Trading: Futures, Stocks
Posts: 95 since Jun 2011
Thanks Given: 118
Thanks Received: 105

In my opinion, you've spotted the weakest thing is AB i.e. handling tick data.

Even if you had successfully downloaded the data your backtests would have taken ages to run (but maybe still quicker than MC or Ninja).

AB has many, many other great features and I really hope to see more constructive threads about it at futures.io (formerly BMT) in the future.

Reply With Quote
  #18 (permalink)
 
medias's Avatar
 medias 
Karlsruhe, Germany
 
Experience: Advanced
Platform: MultiCharts, AmiBroker
Broker: IB/IQFeed
Trading: Stocks, Emini ES
Posts: 60 since Jul 2009
Thanks Given: 30
Thanks Received: 49

I guess you need the (long) history of 120 days of tickdata for backtesting purposes.

So you could build an offline database and import tickdata that is downloaded with QCollector. Use an offline database as you should not mix online (realtime) datafeed with imported data.

The import process can be automated on both sides: Qcollector and Amibroker provide convenient ways to do that.
In Amibroker use this approach:
Tools > Customize... [Tab:Tools] > [Button: New]
Command: #import
Arguments: your path to local stored tickfile
Initial directory: your path to local format file.

See "Update US Symbol list and category" as a starting point. Of course this example only loads symbols and categories, but you should get the point.

Once done, you can import data with a single mouse-click.

But be warned: in online mode (with real data) use a separate database. Never mix manual backfills with realtime data.
BTW: You can start multiple instances of Amibroker simultaneously.

Best regards,
medias.

Reply With Quote
The following 2 users say Thank You to medias for this post:
  #19 (permalink)
 FB2012 
Munich Germany
 
Experience: Beginner
Platform: MultiCharts .NET,MultiCharts,Amibroker
Broker: Interactive Brokers IQFeed
Trading: Stocks,Index CFDs
Posts: 149 since Aug 2012
Thanks Given: 81
Thanks Received: 84


badabingbadaboom View Post
I'm not sure if you are mixing up RAM with bars here but

2 million ticks/day * 120 days = 240 000 000 bars
240 million bars * 40 bytes are 9.6 GB not 256 GB.

256GB would be 6.4 billion bars.

Do you know how those 40 bytes are generated ?
I think
1 double has 8 bytes, right ?
So 5 values in a bar structure OHCLV but Date is missing
Therefore a bar would be structured in the following way.
4 bytes for a int for the Volume maybe also a short 2 bytes would do
4 double with 8 bytes
and the rest for the Date Time

This leads to the conclusion that for storing tickdata
a tick structure of 8 bytes (volume and Date Time) and 8 bytes for Price is needed
2 million ticks/day * 120days * 16 bytes =3.84 GB

But I am not sure if that is needed in the RAM because there is always only a limited lookback period.
Even if I will display the complete data by shrinking my timescale in an chart, i guess that the chart application will reduce the needed data in an intelligent way. Because the eye cannot see every tick and the indicator calculation is maybe only done on slices. I think that is the way a good program should be reduce data.

Maybe that could be the reason why it would take that long for Tomasz to implement that feature.
Because maybe Amibroker has no strategy to deal in the above discribed way with data.

Visit my NexusFi Trade Journal Reply With Quote
  #20 (permalink)
 
Big Mike's Avatar
 Big Mike 
Manta, Ecuador
Site Administrator
Developer
Swing Trader
 
Experience: Advanced
Platform: Custom solution
Broker: IBKR
Trading: Stocks & Futures
Frequency: Every few days
Duration: Weeks
Posts: 50,319 since Jun 2009
Thanks Given: 33,139
Thanks Received: 101,474



medias View Post
I guess you need the (long) history of 120 days of tickdata for backtesting purposes.

So you could build an offline database and import tickdata that is downloaded with QCollector. Use an offline database as you should not mix online (realtime) datafeed with imported data.

The import process can be automated on both sides: Qcollector and Amibroker provide convenient ways to do that.
In Amibroker use this approach:
Tools > Customize... [Tab:Tools] > [Button: New]
Command: #import
Arguments: your path to local stored tickfile
Initial directory: your path to local format file.

See "Update US Symbol list and category" as a starting point. Of course this example only loads symbols and categories, but you should get the point.

Once done, you can import data with a single mouse-click.

But be warned: in online mode (with real data) use a separate database. Never mix manual backfills with realtime data.
BTW: You can start multiple instances of Amibroker simultaneously.

Best regards,
medias.

Hi,

Yes the 120 days of IQFeed was just a fast simple test, I wasn't expecting to hit a brick wall. 120 days of tick data is not useful by itself, it's just useful for backfill. I have my main MySQL databases with years and years of tick data for many instruments, which I would need to import to do any useful testing.

I have not yet started that process, and will likely make a separate thread to discuss it.

One thing in your reply concerns me and caught my attention however --- you said "never mix realtime with historical", why is that? This is a concern, because my typical behavior is to import the data from my database into the platform, then use the platform to backfill and track data going forward -- both realtime and backfill as needed. Can you go into more detail on this?

Cool, I did not know you could run multiple concurrent instances of AmiBroker so that is good. I also had meant to do some research about what the licensing allows --- is it a single user license, but can be deployed on multiple pieces of hardware (ie: my development workstation + my production server) concurrently, as a single user? Or would it require multiple licenses. Need to check on that still.

Mike

We're here to help: just ask the community or contact our Help Desk

Quick Links: Change your Username or Register as a Vendor
Searching for trading reviews? Review this list
Lifetime Elite Membership: Sign-up for only $149 USD
Exclusive money saving offers from our Site Sponsors: Browse Offers
Report problems with the site: Using the NexusFi changelog thread
Follow me on Twitter Visit my NexusFi Trade Journal Started this thread Reply With Quote





Last Updated on May 9, 2015


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts