NexusFi: Find Your Edge


Home Menu

 





AmiBroker IQFeed historical tick data


Discussion in Platforms and Indicators

Updated
      Top Posters
    1. looks_one badabingbadaboom with 11 posts (3 thanks)
    2. looks_two Big Mike with 10 posts (3 thanks)
    3. looks_3 prouser with 2 posts (0 thanks)
    4. looks_4 medias with 2 posts (5 thanks)
      Best Posters
    1. looks_one medias with 2.5 thanks per post
    2. looks_two bob7123 with 1 thanks per post
    3. looks_3 Big Mike with 0.3 thanks per post
    4. looks_4 badabingbadaboom with 0.3 thanks per post
    1. trending_up 14,768 views
    2. thumb_up 12 thanks given
    3. group 9 followers
    1. forum 29 posts
    2. attach_file 0 attachments




 
Search this Thread

AmiBroker IQFeed historical tick data

  #21 (permalink)
 
Big Mike's Avatar
 Big Mike 
Manta, Ecuador
Site Administrator
Developer
Swing Trader
 
Experience: Advanced
Platform: Custom solution
Broker: IBKR
Trading: Stocks & Futures
Frequency: Every few days
Duration: Weeks
Posts: 50,322 since Jun 2009
Thanks Given: 33,143
Thanks Received: 101,477


FB2012 View Post
Do you know how those 40 bytes are generated ?
I think
1 double has 8 bytes, right ?
So 5 values in a bar structure OHCLV but Date is missing
Therefore a bar would be structured in the following way.
4 bytes for a int for the Volume maybe also a short 2 bytes would do
4 double with 8 bytes
and the rest for the Date Time

This leads to the conclusion that for storing tickdata
a tick structure of 8 bytes (volume and Date Time) and 8 bytes for Price is needed
2 million ticks/day * 120days * 16 bytes =3.84 GB

But I am not sure if that is needed in the RAM because there is always only a limited lookback period.
Even if I will display the complete data by shrinking my timescale in an chart, i guess that the chart application will reduce the needed data in an intelligent way. Because the eye cannot see every tick and the indicator calculation is maybe only done on slices. I think that is the way a good program should be reduce data.

Maybe that could be the reason why it would take that long for Tomasz to implement that feature.
Because maybe Amibroker has no strategy to deal in the above discribed way with data.

It is interesting you mention this, because I was confused by the "lookback" or whatever you want to call it on a chart. I asked Tomasz how I could limit a chart to only 'x' bars or 'x' days, or to specify a specific period such as 1/1/2011 - 1/1/2012, and if I understood his answer correctly you cannot do this. Apparently it loads all or nothing in the entire database, so the only way to split it up is by having multiple databases? I haven't questioned him further on this but that was my understanding.

Mike

We're here to help: just ask the community or contact our Help Desk

Quick Links: Change your Username or Register as a Vendor
Searching for trading reviews? Review this list
Lifetime Elite Membership: Sign-up for only $149 USD
Exclusive money saving offers from our Site Sponsors: Browse Offers
Report problems with the site: Using the NexusFi changelog thread
Follow me on Twitter Visit my NexusFi Trade Journal Started this thread Reply With Quote

Can you help answer these questions
from other members on NexusFi?
My NT8 Volume Profile Split by Asian/Euro/Open
NinjaTrader
Request for MACD with option to use different MAs for fa …
NinjaTrader
ZombieSqueeze
Platforms and Indicators
NexusFi Journal Challenge - April 2024
Feedback and Announcements
 
Best Threads (Most Thanked)
in the last 7 days on NexusFi
Retail Trading As An Industry
58 thanks
Battlestations: Show us your trading desks!
52 thanks
NexusFi site changelog and issues/problem reporting
48 thanks
GFIs1 1 DAX trade per day journal
31 thanks
What percentage per day is possible? [Poll]
31 thanks

  #22 (permalink)
 
medias's Avatar
 medias 
Karlsruhe, Germany
 
Experience: Advanced
Platform: MultiCharts, AmiBroker
Broker: IB/IQFeed
Trading: Stocks, Emini ES
Posts: 60 since Jul 2009
Thanks Given: 30
Thanks Received: 49


Big Mike View Post
Hi,

Yes the 120 days of IQFeed was just a fast simple test, I wasn't expecting to hit a brick wall. 120 days of tick data is not useful by itself, it's just useful for backfill. I have my main MySQL databases with years and years of tick data for many instruments, which I would need to import to do any useful testing.

I have not yet started that process, and will likely make a separate thread to discuss it.

One thing in your reply concerns me and caught my attention however --- you said "never mix realtime with historical", why is that? This is a concern, because my typical behavior is to import the data from my database into the platform, then use the platform to backfill and track data going forward -- both realtime and backfill as needed. Can you go into more detail on this?

Cool, I did not know you could run multiple concurrent instances of AmiBroker so that is good. I also had meant to do some research about what the licensing allows --- is it a single user license, but can be deployed on multiple pieces of hardware (ie: my development workstation + my production server) concurrently, as a single user? Or would it require multiple licenses. Need to check on that still.

Mike

Let me just quote the official Amibroker docs:
"Things you should NOT do, or you should do very carefully

You should note the fact that when you are using data plugin then the plugin controls the quotation database (see Understanding database concepts article), therefore you should NOT import quotes from ASCII files (this includes AmiQuote) for symbols that are already present in the real-time database.

If you do, the plugin will eventually overwrite your imports with the real-time data or your database will become corrupted (if you import end-of-day data over intraday database)."
For further details see How to work with Real-Time data plugins

You should be fine when you start with an empty database (no symbols) and start with importing offline data (and symbols). Then go on with your realtime data, but never fill gaps with imported data again. Connect regularly, probably use an "backfill exploration" (you can google that). There is no need to use that exploration (as Amibroker fills dataholes in a background task), but it "makes me feel safe" to have control over that process. With that exploration (with all your symbols selected) you don't have to open charts or "Realtime Quote" Panel.

One more thing ;-) : At the beginning you have to decide in which resolution you want to build your database (and how many lookback periods (=bars or ticks) you want to have.

I struggled at the beginning to find the best answer for myself. For now, I have 2 databases: 5sec Resolution (discretionary day-trading, 17280 bars, probably overkill, 5000 should be enough) and EOD (enough bars to go back to 1998, that is enough for my EOD backtasting).

Of course: with the 5sec database you can show all other longer timeframes (15 sec, 1min, etc...)

For your purposes (doing backtesting on longer periods with tickdata), your database would have a different setup.

High likely, I am also going t build a separate (offline 5sec) database with huge lookback periods to do some testing on my discretionary rules. But I have to do some homework first to translate my rules to a mechanical approach. But this is a different topic...

The answer to your license question can be found here:
Single license use on multiple computers?


Best regards,
medias

Reply With Quote
The following 3 users say Thank You to medias for this post:
  #23 (permalink)
 badabingbadaboom 
London/UK
 
Posts: 56 since Apr 2013



FB2012 View Post
Do you know how those 40 bytes are generated ?
I think
1 double has 8 bytes, right ?
So 5 values in a bar structure OHCLV but Date is missing
Therefore a bar would be structured in the following way.
4 bytes for a int for the Volume maybe also a short 2 bytes would do
4 double with 8 bytes
and the rest for the Date Time

This leads to the conclusion that for storing tickdata
a tick structure of 8 bytes (volume and Date Time) and 8 bytes for Price is needed
2 million ticks/day * 120days * 16 bytes =3.84 GB

But I am not sure if that is needed in the RAM because there is always only a limited lookback period.
Even if I will display the complete data by shrinking my timescale in an chart, i guess that the chart application will reduce the needed data in an intelligent way. Because the eye cannot see every tick and the indicator calculation is maybe only done on slices. I think that is the way a good program should be reduce data.

Maybe that could be the reason why it would take that long for Tomasz to implement that feature.
Because maybe Amibroker has no strategy to deal in the above discribed way with data.


No, I don't know but if Tomasz mentions that it is 40 bytes then it is 40 bytes.

Don't take it the wrong way but if you wanna try to be smarter than someone with a Ph.D. and M.Sc. degree in Computer Science and Telecommunication like him then you should contact and argue with him. But IMO, "I am no sure" is not a good basis to question something. You and me don't know how and why it works the way it works. There is a term that nowadays there are a lot of so called Internet "experts" while on the other side Tomasz is one of the smartest guys I know and a real hardware and software expert. So I rather think he knows best why it's this and that. And the software speaks for itself. Just my two cents.

FYI, I'm using long tick data history for analysis myself and had no serious problems to far. AmiBroker makes optimal or full use of the hardware. The only limit is the hardware. It's a high performance software with very reliable results. For example you won't see different analysis results after each run.
On the other side, for example Ninjatrader 7 tick data import (millions of bars for example) and analysis takes ages and it didn't even wanna chart the tick data. And it has multiple other flaws that are not present in AB.

What feature do you mean to implement? You mean the chunked download? Well, it would not be related to AmiBroker itself but rather to the IQfeed data plugin. AFAIK IQfeeds own test download application was role model (not sure on this). So there is no AmiBroker design flaw, IMO. So as for implementing suggestions ... there is a feedback center and AFAIK there is a development schedule like everywhere else too. Some things have higher priority than others. But that's my outside perspective as user with limited knowledge about the internals.

Reply With Quote
  #24 (permalink)
 badabingbadaboom 
London/UK
 
Posts: 56 since Apr 2013


Big Mike View Post
It is interesting you mention this, because I was confused by the "lookback" or whatever you want to call it on a chart. I asked Tomasz how I could limit a chart to only 'x' bars or 'x' days, or to specify a specific period such as 1/1/2011 - 1/1/2012, and if I understood his answer correctly you cannot do this. Apparently it loads all or nothing in the entire database, so the only way to split it up is by having multiple databases? I haven't questioned him further on this but that was my understanding.

Mike

I guess by "lookback" you mean number of bars in data base settings? It is only for realtime plugins.

And for charting and also for analysis there is a feature called Quick AFL AmiBroker Knowledge Base » QuickAFL facts

That one can be turned off in preferences or in a code by using SetBarsRequired( sbrall, sbrall ); or SetBarsRequired( -2, -2 );

Not sure what you wanna split. A symbol? Like let's say ES from 2010 to 2011 and from 2012 to 2013?

Reply With Quote
  #25 (permalink)
 badabingbadaboom 
London/UK
 
Posts: 56 since Apr 2013

As for mixed data (EOD/intraday)


Quoting 
Allow mixed EOD/Intraday data - it allows to work with database that has a mixture of intraday and EOD data in one data file. If this is turned on then in intraday modes EOD bars are removed on-the-fly and in daily mode EOD bars are displayed instead of time compressed intraday or if there is no EOD bar for corresponding day then intraday bars are compressed as usual.
This mode works in conjunction with new versions of plugins that allow mixed data. As of June 2008 Mixed mode is now supported by IQFeed plugin, eSignal plugin (1.7.0 or higher) plugins only. Mixed mode allows intraday plus very long daily histories in one database.

https://www.amibroker.com/guide/w_dbsettings.html‎

Reply With Quote
  #26 (permalink)
 
bob7123's Avatar
 bob7123 
Amsterdam, The Netherlands
Legendary Market Wizard
 
Experience: Intermediate
Platform: Ninja, IRT, ToS
Broker: EdgeClear, LMAX
Trading: NQ
Posts: 668 since Oct 2011
Thanks Given: 134
Thanks Received: 2,682


Big Mike View Post
After some more email exchanges with Tomasz, the conclusion is that he has reproduced my issue but says that no one else has ever asked for this -- so he deems it is low priority and says it will take "months at least" for him to fix it.

I cannot help but feel like the fix should not take months. I think he is mad at me personally...

Anyway, in his defense he offered a refund of AmiBroker over the problem. I told him I've had it for only 1 day and would really like more time with it, 120 days of IQFeed backfill is not the only thing that is important to me, so if it does other things well then there would be no need for a refund.

I am still hopeful that it is as optimized and powerful as I have read. More time will tell.

Mike

Small world. Guess what's holding up the next release of BaBAR.

I just got off a chat with IQFeed who gave me the link to this thread. I didn't realize you were having trouble too.

I've been like "Horton hears a Who" since April. (What is it with me and elephants...) Anyway, IQFeed is finally acknowledging they have a problem with large history requests in their v5 client.

I am releasing the next version of BaBAR with a workaround where it actually makes up to 120 single day requests. I am not sure how easy that is for others to do, and if it isn't, then the next best things I can tell you is that
  • This is mostly a problem with SPY and ES due to the very high number of ticks

  • It takes a request of more than about 28 days of those instruments to blow up my app. If I keep it below that, it seems to work OK

  • IQFeed will have this resolved in v5.1. Release date not scheduled but they are expecting a beta in a few weeks. Maybe fire it up in a VM when it comes out and see if it fixes your issue?

-Bob

Follow me on Twitter Reply With Quote
The following user says Thank You to bob7123 for this post:
  #27 (permalink)
 prouser 
Zurich/Switzerland
 
Posts: 79 since Oct 2014

AmiBroker uses in-memory database.

See this thread for example https://www.yahoo.com/


Toby Riddell, Mar 16, 2013
I have some experience architecting & managing infrastructure used by hedge funds for systematic trading, including storage of tick data (database sizes in the tens of TB size range). The preferred solutions are things like StreamBase or kdb+, or they write their own. A large memory footprint is necessary, the data is slurped off disk once and then processed in-memory. Even with a very high performance disk I/O system (possibly using arrays of SSDs), you still want to do as little disk I/O as possible (i.e. do it once to load the data into RAM).

SQL databases are used for managing things like lists of products traded, mapping symbologies between trading venues, trade reconciliation, etc., etc. - smaller scale, more general problems where a general-purpose query language is useful/necessary.

Just wanted to offer another perspective...

To which Tomasz Janeczko replied


Quoting 
Hello,

Precisely.
Exactly the same approach ( load once and keep in RAM, as little IO as possible) is used by AmiBroker's own database.

Best regards,
Tomasz Janeczko
amibroker.com

As well as


Quoting 
AmiBroker is designed for speed, hence it uses in-RAM database.
In-memory database - Wikipedia, the free encyclopedia

In each system there are tradeoffs between speed, memory usage, etc.

You could declare virtual memory in windows to be hundreds of gigabytes
and then Windows would start doing what you are suggesting here - swap/commit data
to the disk. Net result would be orders of magnitude drop in performance.

...

Best regards,
Tomasz Janeczko
amibroker.com

and others. You may read that entire thread.



FB2012 View Post
...

But I am not sure if that is needed in the RAM because there is always only a limited lookback period.
Even if I will display the complete data by shrinking my timescale in an chart, i guess that the chart application will reduce the needed data in an intelligent way. Because the eye cannot see every tick and the indicator calculation is maybe only done on slices. I think that is the way a good program should be reduce data.

Maybe that could be the reason why it would take that long for Tomasz to implement that feature.
Because maybe Amibroker has no strategy to deal in the above discribed way with data.

There already is QuickAFL feature.
Another one is QuickData


Quoting 
CHANGES FOR VERSION 5.63.0 (as compared to 5.62.0)

1. QuickData for charts implemented. QuickData is a new feature that greatly decreases time required to prepare compressed/filtered data for real-time charts.

2. ...
..


Reply With Quote
  #28 (permalink)
 a8d7e8 
Hong Kong China
 
Experience: None
Platform: NT
Trading: Stocks
Posts: 7 since Sep 2014
Thanks Given: 22
Thanks Received: 0


medias View Post
I guess you need the (long) history of 120 days of tickdata for backtesting purposes.

So you could build an offline database and import tickdata that is downloaded with QCollector. Use an offline database as you should not mix online (realtime) datafeed with imported data.

The import process can be automated on both sides: Qcollector and Amibroker provide convenient ways to do that.
In Amibroker use this approach:
Tools > Customize... [Tab:Tools] > [Button: New]
Command: #import
Arguments: your path to local stored tickfile
Initial directory: your path to local format file.

See "Update US Symbol list and category" as a starting point. Of course this example only loads symbols and categories, but you should get the point.

Once done, you can import data with a single mouse-click.

But be warned: in online mode (with real data) use a separate database. Never mix manual backfills with realtime data.
BTW: You can start multiple instances of Amibroker simultaneously.

Best regards,
medias.

Hello you look like an experienced ab user. I'm a new user to ab and facing manual backfills vs realtime data issue with IB data feed.

I think I can manually download much more data than ab's IB plugin. Would you mind elaborate more on why must we use a separate data for manual backfills and realtime data? I've read the document of ab about it but I cannot find the exact reason why.......

One closest reason I saw is that a corruption may happen if realtime data writes to the database at the same time when manually importing data on the same symbol. So it means if I manage to disconnect the data feed plugin while importing the manual backfills........and never import any backfills when data feed plugin is online, it should be fine. Right?

Maybe I'm not leaving, maybe I'm going home
Reply With Quote
  #29 (permalink)
 SamYeotl 
Singapore
 
Experience: Intermediate
Platform: NinjaTrader,TradeStation, OX
Broker: CQG
Trading: ES,FX,Oil,Stock,Options, GC, 6E, TF
Posts: 11 since Feb 2013
Thanks Given: 4
Thanks Received: 2

Anyone know how to convert data downloaded in NT [NTH1 formate] into Amibroker data?

Reply With Quote
  #30 (permalink)
 prouser 
Zurich/Switzerland
 
Posts: 79 since Oct 2014



SamYeotl View Post
Anyone know how to convert data downloaded in NT [NTH1 formate] into Amibroker data?

Is your question related to IQFeed or what are you referring to?
As for ASCII import to AmiBroker. You don't need to convert data as AmiBroker can import literally any data.
The only thing you possibly need to take care of is that NT uses forward looking timestamps, AFAIR.
AmiBroker Knowledge Base » How to correct forward looking timestamps

Reply With Quote





Last Updated on May 9, 2015


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts