AmiBroker IQFeed historical tick data - AmiBroker | futures io social day trading
futures io futures trading


AmiBroker IQFeed historical tick data
Updated: Views / Replies:7,178 / 29
Created: by Big Mike Attachments:0

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors – all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you don’t need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 
Thread Tools Search this Thread
 

AmiBroker IQFeed historical tick data

  #21 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,240 since Jun 2009
Thanks: 29,353 given, 83,234 received


FB2012 View Post
Do you know how those 40 bytes are generated ?
I think
1 double has 8 bytes, right ?
So 5 values in a bar structure OHCLV but Date is missing
Therefore a bar would be structured in the following way.
4 bytes for a int for the Volume maybe also a short 2 bytes would do
4 double with 8 bytes
and the rest for the Date Time

This leads to the conclusion that for storing tickdata
a tick structure of 8 bytes (volume and Date Time) and 8 bytes for Price is needed
2 million ticks/day * 120days * 16 bytes =3.84 GB

But I am not sure if that is needed in the RAM because there is always only a limited lookback period.
Even if I will display the complete data by shrinking my timescale in an chart, i guess that the chart application will reduce the needed data in an intelligent way. Because the eye cannot see every tick and the indicator calculation is maybe only done on slices. I think that is the way a good program should be reduce data.

Maybe that could be the reason why it would take that long for Tomasz to implement that feature.
Because maybe Amibroker has no strategy to deal in the above discribed way with data.

It is interesting you mention this, because I was confused by the "lookback" or whatever you want to call it on a chart. I asked Tomasz how I could limit a chart to only 'x' bars or 'x' days, or to specify a specific period such as 1/1/2011 - 1/1/2012, and if I understood his answer correctly you cannot do this. Apparently it loads all or nothing in the entire database, so the only way to split it up is by having multiple databases? I haven't questioned him further on this but that was my understanding.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #22 (permalink)
Elite Member
Karlsruhe, Germany
 
Futures Experience: Intermediate
Platform: MultiCharts, AmiBroker
Broker/Data: IB/IQFeed
Favorite Futures: FDAX, ES
 
medias's Avatar
 
Posts: 60 since Jul 2009
Thanks: 30 given, 49 received


Big Mike View Post
Hi,

Yes the 120 days of IQFeed was just a fast simple test, I wasn't expecting to hit a brick wall. 120 days of tick data is not useful by itself, it's just useful for backfill. I have my main MySQL databases with years and years of tick data for many instruments, which I would need to import to do any useful testing.

I have not yet started that process, and will likely make a separate thread to discuss it.

One thing in your reply concerns me and caught my attention however --- you said "never mix realtime with historical", why is that? This is a concern, because my typical behavior is to import the data from my database into the platform, then use the platform to backfill and track data going forward -- both realtime and backfill as needed. Can you go into more detail on this?

Cool, I did not know you could run multiple concurrent instances of AmiBroker so that is good. I also had meant to do some research about what the licensing allows --- is it a single user license, but can be deployed on multiple pieces of hardware (ie: my development workstation + my production server) concurrently, as a single user? Or would it require multiple licenses. Need to check on that still.

Mike

Let me just quote the official Amibroker docs:
"Things you should NOT do, or you should do very carefully

You should note the fact that when you are using data plugin then the plugin controls the quotation database (see Understanding database concepts article), therefore you should NOT import quotes from ASCII files (this includes AmiQuote) for symbols that are already present in the real-time database.

If you do, the plugin will eventually overwrite your imports with the real-time data or your database will become corrupted (if you import end-of-day data over intraday database)."
For further details see How to work with Real-Time data plugins

You should be fine when you start with an empty database (no symbols) and start with importing offline data (and symbols). Then go on with your realtime data, but never fill gaps with imported data again. Connect regularly, probably use an "backfill exploration" (you can google that). There is no need to use that exploration (as Amibroker fills dataholes in a background task), but it "makes me feel safe" to have control over that process. With that exploration (with all your symbols selected) you don't have to open charts or "Realtime Quote" Panel.

One more thing ;-) : At the beginning you have to decide in which resolution you want to build your database (and how many lookback periods (=bars or ticks) you want to have.

I struggled at the beginning to find the best answer for myself. For now, I have 2 databases: 5sec Resolution (discretionary day-trading, 17280 bars, probably overkill, 5000 should be enough) and EOD (enough bars to go back to 1998, that is enough for my EOD backtasting).

Of course: with the 5sec database you can show all other longer timeframes (15 sec, 1min, etc...)

For your purposes (doing backtesting on longer periods with tickdata), your database would have a different setup.

High likely, I am also going t build a separate (offline 5sec) database with huge lookback periods to do some testing on my discretionary rules. But I have to do some homework first to translate my rules to a mechanical approach. But this is a different topic...

The answer to your license question can be found here:
Single license use on multiple computers?


Best regards,
medias

Reply With Quote
The following 3 users say Thank You to medias for this post:
 
  #23 (permalink)
Banned: Previously banned user, now re-banned
London/UK
 
Futures Experience: Advanced
Platform: AmiBroker
Favorite Futures: Any
 
Posts: 56 since Apr 2013
Thanks: 0 given, 12 received



FB2012 View Post
Do you know how those 40 bytes are generated ?
I think
1 double has 8 bytes, right ?
So 5 values in a bar structure OHCLV but Date is missing
Therefore a bar would be structured in the following way.
4 bytes for a int for the Volume maybe also a short 2 bytes would do
4 double with 8 bytes
and the rest for the Date Time

This leads to the conclusion that for storing tickdata
a tick structure of 8 bytes (volume and Date Time) and 8 bytes for Price is needed
2 million ticks/day * 120days * 16 bytes =3.84 GB

But I am not sure if that is needed in the RAM because there is always only a limited lookback period.
Even if I will display the complete data by shrinking my timescale in an chart, i guess that the chart application will reduce the needed data in an intelligent way. Because the eye cannot see every tick and the indicator calculation is maybe only done on slices. I think that is the way a good program should be reduce data.

Maybe that could be the reason why it would take that long for Tomasz to implement that feature.
Because maybe Amibroker has no strategy to deal in the above discribed way with data.


No, I don't know but if Tomasz mentions that it is 40 bytes then it is 40 bytes.

Don't take it the wrong way but if you wanna try to be smarter than someone with a Ph.D. and M.Sc. degree in Computer Science and Telecommunication like him then you should contact and argue with him. But IMO, "I am no sure" is not a good basis to question something. You and me don't know how and why it works the way it works. There is a term that nowadays there are a lot of so called Internet "experts" while on the other side Tomasz is one of the smartest guys I know and a real hardware and software expert. So I rather think he knows best why it's this and that. And the software speaks for itself. Just my two cents.

FYI, I'm using long tick data history for analysis myself and had no serious problems to far. AmiBroker makes optimal or full use of the hardware. The only limit is the hardware. It's a high performance software with very reliable results. For example you won't see different analysis results after each run.
On the other side, for example Ninjatrader 7 tick data import (millions of bars for example) and analysis takes ages and it didn't even wanna chart the tick data. And it has multiple other flaws that are not present in AB.

What feature do you mean to implement? You mean the chunked download? Well, it would not be related to AmiBroker itself but rather to the IQfeed data plugin. AFAIK IQfeeds own test download application was role model (not sure on this). So there is no AmiBroker design flaw, IMO. So as for implementing suggestions ... there is a feedback center and AFAIK there is a development schedule like everywhere else too. Some things have higher priority than others. But that's my outside perspective as user with limited knowledge about the internals.


Last edited by badabingbadaboom; July 8th, 2013 at 01:48 PM.
Reply With Quote
 
  #24 (permalink)
Banned: Previously banned user, now re-banned
London/UK
 
Futures Experience: Advanced
Platform: AmiBroker
Favorite Futures: Any
 
Posts: 56 since Apr 2013
Thanks: 0 given, 12 received


Big Mike View Post
It is interesting you mention this, because I was confused by the "lookback" or whatever you want to call it on a chart. I asked Tomasz how I could limit a chart to only 'x' bars or 'x' days, or to specify a specific period such as 1/1/2011 - 1/1/2012, and if I understood his answer correctly you cannot do this. Apparently it loads all or nothing in the entire database, so the only way to split it up is by having multiple databases? I haven't questioned him further on this but that was my understanding.

Mike

I guess by "lookback" you mean number of bars in data base settings? It is only for realtime plugins.

And for charting and also for analysis there is a feature called Quick AFL AmiBroker Knowledge Base » QuickAFL facts

That one can be turned off in preferences or in a code by using SetBarsRequired( sbrall, sbrall ); or SetBarsRequired( -2, -2 );

Not sure what you wanna split. A symbol? Like let's say ES from 2010 to 2011 and from 2012 to 2013?

Reply With Quote
 
  #25 (permalink)
Banned: Previously banned user, now re-banned
London/UK
 
Futures Experience: Advanced
Platform: AmiBroker
Favorite Futures: Any
 
Posts: 56 since Apr 2013
Thanks: 0 given, 12 received

As for mixed data (EOD/intraday)


Quoting 
Allow mixed EOD/Intraday data - it allows to work with database that has a mixture of intraday and EOD data in one data file. If this is turned on then in intraday modes EOD bars are removed on-the-fly and in daily mode EOD bars are displayed instead of time compressed intraday or if there is no EOD bar for corresponding day then intraday bars are compressed as usual.
This mode works in conjunction with new versions of plugins that allow mixed data. As of June 2008 Mixed mode is now supported by IQFeed plugin, eSignal plugin (1.7.0 or higher) plugins only. Mixed mode allows intraday plus very long daily histories in one database.

http://www.amibroker.com/guide/w_dbsettings.html‎

Reply With Quote
 
  #26 (permalink)
Elite Member
Amsterdam, The Netherlands
 
Futures Experience: Intermediate
Platform: Sierra, IRT, ToS, Ninja
Broker/Data: Stage 5
Favorite Futures: NQ
 
bob7123's Avatar
 
Posts: 588 since Oct 2011
Thanks: 109 given, 1,909 received


Big Mike View Post
After some more email exchanges with Tomasz, the conclusion is that he has reproduced my issue but says that no one else has ever asked for this -- so he deems it is low priority and says it will take "months at least" for him to fix it.

I cannot help but feel like the fix should not take months. I think he is mad at me personally...

Anyway, in his defense he offered a refund of AmiBroker over the problem. I told him I've had it for only 1 day and would really like more time with it, 120 days of IQFeed backfill is not the only thing that is important to me, so if it does other things well then there would be no need for a refund.

I am still hopeful that it is as optimized and powerful as I have read. More time will tell.

Mike

Small world. Guess what's holding up the next release of BaBAR.

I just got off a chat with IQFeed who gave me the link to this thread. I didn't realize you were having trouble too.

I've been like "Horton hears a Who" since April. (What is it with me and elephants...) Anyway, IQFeed is finally acknowledging they have a problem with large history requests in their v5 client.

I am releasing the next version of BaBAR with a workaround where it actually makes up to 120 single day requests. I am not sure how easy that is for others to do, and if it isn't, then the next best things I can tell you is that
  • This is mostly a problem with SPY and ES due to the very high number of ticks

  • It takes a request of more than about 28 days of those instruments to blow up my app. If I keep it below that, it seems to work OK

  • IQFeed will have this resolved in v5.1. Release date not scheduled but they are expecting a beta in a few weeks. Maybe fire it up in a VM when it comes out and see if it fixes your issue?

-Bob

Reply With Quote
The following user says Thank You to bob7123 for this post:
 
  #27 (permalink)
Membership Temporarily Revoked
Zurich/Switzerland
 
Futures Experience: Advanced
Platform: -
Favorite Futures: -
 
Posts: 79 since Oct 2014
Thanks: 0 given, 33 received

AmiBroker uses in-memory database.

See this thread for example https://groups.yahoo.com/neo/groups/amibroker/conversations/messages/175796


Toby Riddell, Mar 16, 2013
I have some experience architecting & managing infrastructure used by hedge funds for systematic trading, including storage of tick data (database sizes in the tens of TB size range). The preferred solutions are things like StreamBase or kdb+, or they write their own. A large memory footprint is necessary, the data is slurped off disk once and then processed in-memory. Even with a very high performance disk I/O system (possibly using arrays of SSDs), you still want to do as little disk I/O as possible (i.e. do it once to load the data into RAM).

SQL databases are used for managing things like lists of products traded, mapping symbologies between trading venues, trade reconciliation, etc., etc. - smaller scale, more general problems where a general-purpose query language is useful/necessary.

Just wanted to offer another perspective...

To which Tomasz Janeczko replied


Quoting 
Hello,

Precisely.
Exactly the same approach ( load once and keep in RAM, as little IO as possible) is used by AmiBroker's own database.

Best regards,
Tomasz Janeczko
amibroker.com

As well as


Quoting 
AmiBroker is designed for speed, hence it uses in-RAM database.
In-memory database - Wikipedia, the free encyclopedia

In each system there are tradeoffs between speed, memory usage, etc.

You could declare virtual memory in windows to be hundreds of gigabytes
and then Windows would start doing what you are suggesting here - swap/commit data
to the disk. Net result would be orders of magnitude drop in performance.

...

Best regards,
Tomasz Janeczko
amibroker.com

and others. You may read that entire thread.



FB2012 View Post
...

But I am not sure if that is needed in the RAM because there is always only a limited lookback period.
Even if I will display the complete data by shrinking my timescale in an chart, i guess that the chart application will reduce the needed data in an intelligent way. Because the eye cannot see every tick and the indicator calculation is maybe only done on slices. I think that is the way a good program should be reduce data.

Maybe that could be the reason why it would take that long for Tomasz to implement that feature.
Because maybe Amibroker has no strategy to deal in the above discribed way with data.

There already is QuickAFL feature.
Another one is QuickData


Quoting 
CHANGES FOR VERSION 5.63.0 (as compared to 5.62.0)

1. QuickData for charts implemented. QuickData is a new feature that greatly decreases time required to prepare compressed/filtered data for real-time charts.

2. ...
..


Reply With Quote
 
  #28 (permalink)
Novice
Hong Kong China
 
Futures Experience: None
Platform: NT
Favorite Futures: Stocks
 
Posts: 7 since Sep 2014
Thanks: 22 given, 0 received


medias View Post
I guess you need the (long) history of 120 days of tickdata for backtesting purposes.

So you could build an offline database and import tickdata that is downloaded with QCollector. Use an offline database as you should not mix online (realtime) datafeed with imported data.

The import process can be automated on both sides: Qcollector and Amibroker provide convenient ways to do that.
In Amibroker use this approach:
Tools > Customize... [Tab:Tools] > [Button: New]
Command: #import
Arguments: your path to local stored tickfile
Initial directory: your path to local format file.

See "Update US Symbol list and category" as a starting point. Of course this example only loads symbols and categories, but you should get the point.

Once done, you can import data with a single mouse-click.

But be warned: in online mode (with real data) use a separate database. Never mix manual backfills with realtime data.
BTW: You can start multiple instances of Amibroker simultaneously.

Best regards,
medias.

Hello you look like an experienced ab user. I'm a new user to ab and facing manual backfills vs realtime data issue with IB data feed.

I think I can manually download much more data than ab's IB plugin. Would you mind elaborate more on why must we use a separate data for manual backfills and realtime data? I've read the document of ab about it but I cannot find the exact reason why.......

One closest reason I saw is that a corruption may happen if realtime data writes to the database at the same time when manually importing data on the same symbol. So it means if I manage to disconnect the data feed plugin while importing the manual backfills........and never import any backfills when data feed plugin is online, it should be fine. Right?

Maybe I'm not leaving, maybe I'm going home
Reply With Quote
 
  #29 (permalink)
Elite Member
Singapore
 
Futures Experience: Intermediate
Platform: NinjaTrader,TradeStation, OX
Broker/Data: CQG
Favorite Futures: ES,FX,Oil,Stock,Options, GC, 6E, TF
 
Posts: 11 since Feb 2013
Thanks: 4 given, 2 received

NinjaTrader download data convert to AmiBroker

Anyone know how to convert data downloaded in NT [NTH1 formate] into Amibroker data?

Reply With Quote
 
  #30 (permalink)
Membership Temporarily Revoked
Zurich/Switzerland
 
Futures Experience: Advanced
Platform: -
Favorite Futures: -
 
Posts: 79 since Oct 2014
Thanks: 0 given, 33 received



SamYeotl View Post
Anyone know how to convert data downloaded in NT [NTH1 formate] into Amibroker data?

Is your question related to IQFeed or what are you referring to?
As for ASCII import to AmiBroker. You don't need to convert data as AmiBroker can import literally any data.
The only thing you possibly need to take care of is that NT uses forward looking timestamps, AFAIR.
AmiBroker Knowledge Base » How to correct forward looking timestamps

Reply With Quote

Reply



futures io > > > > AmiBroker IQFeed historical tick data

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Jigsaw Trading: TBA

Elite only

FuturesTrader71: TBA

Elite only

NinjaTrader: TBA

Jan 18

RandBots: TBA

Jan 23

GFF Brokers & CME Group: Futures & Bitcoin

Elite only

Adam Grimes: TBA

Elite only

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Historical CL Tick Data and Minute Data Big Mike The Elite Circle 112 July 26th, 2014 07:07 PM
Data Feed: How do you confirm Accuracy vs Historical tick data provided. FAUGH45568 Reviews of Brokers and Data Feeds 2 February 14th, 2013 07:04 PM
Historical YM Tick Data and Minute Data?? Hermo The Elite Circle 3 December 1st, 2012 02:56 PM
MultiCharts and IQfeed alignment with 120 days of Tick Data??? benharrell Reviews of Brokers and Data Feeds 36 June 11th, 2011 04:06 PM
Platform that can go tick by tick through historical data acdrew0 Platforms and Indicators 1 October 21st, 2010 01:10 PM


All times are GMT -4. The time now is 09:53 PM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-15 in 0.18 seconds with 19 queries on phoenix via your IP 54.91.171.137