Time Series database for tick and trade data. - futures io
futures io futures trading



Time Series database for tick and trade data.


Discussion in Platforms and Indicators

Updated by ab456
      Top Posters
    1. looks_one gregid with 9 posts (10 thanks)
    2. looks_two Jasonnator with 7 posts (8 thanks)
    3. looks_3 stocksharp with 4 posts (4 thanks)
    4. looks_4 ClutchAce with 3 posts (0 thanks)
      Best Posters
    1. looks_one artemiso with 1.3 thanks per post
    2. looks_two gregid with 1.1 thanks per post
    3. looks_3 Jasonnator with 1.1 thanks per post
    4. looks_4 stocksharp with 1.0 thanks per post
    1. trending_up 11,300 views
    2. thumb_up 34 thanks given
    3. group 23 followers
    1. forum 40 replies
    2. attach_file 2 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 100,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

Time Series database for tick and trade data.

(login for full post details)
  #1 (permalink)
Wrocław, Poland
 
Experience: Intermediate
Platform: NinjaTrader, Racket
Trading: Ockham's razor
 
gregid's Avatar
 
Posts: 651 since Aug 2009
Thanks: 320 given, 618 received

Majority of custom solutions for storing tick and trade data utilize standard RDBMS (MS SQL Server, MySQL, etc.) which I have done myself in the past. I believe this is mostly due to familiarity with relational dbs and maturity of the solutions.

Still I keep wondering about the solutions more suitable to the task and started exploring Time Series databases. I have no previous experience with any, and was wondering what are your views on this approach and if anyone here has any experience with them and/or recommendations.

Some open source Time Series databases I am aware of:
Druid | Interactive Analytics at Scale
InfluxDB - Open Source Time Series, Metrics, and Analytics Database
Blueflood
OpenTSDB - A Distributed, Scalable Monitoring System
https://github.com/kairosdb/kairosdb
https://github.com/OpenNMS/newts
https://code.google.com/archive/p/tsdb
https://github.com/gar1t/tsdb
https://github.com/dustin/seriesly
https://github.com/ezhulenev/marketdb
https://github.com/graphite-project/ceres
https://github.com/prometheus/prometheus
https://github.com/akumuli/Akumuli
https://github.com/despegar/khronus

Reply With Quote
The following 3 users say Thank You to gregid for this post:

Can you help answer these questions
from other members on futures io?
Degiro Negative Balance Advice
Traders Hideout
Capping Size in Tradestation
TradeStation
QDEL
Stocks and ETFs
Do you know any futures platform that allows grid automation like shown on this vid?
Platforms and Indicators
Zytrade Brokerage Review (www.zytrade.com)
Brokers
 
Best Threads (Most Thanked)
in the last 7 days on futures io
Finally Turning the Corner, tha "its 80% Psychology" thing...
35 thanks
Voter Sentiment in upcoming US election and effect on market after election results
25 thanks
Lost & losing hope
24 thanks
Coronavirus COVID-19
12 thanks
The Journey
10 thanks
 
(login for full post details)
  #3 (permalink)
Rio de Janeiro / Barcelona
 
Experience: Advanced
Platform: NinjaTrader & Excel
Broker: IB
Trading: EWZ
 
Posts: 125 since Apr 2013
Thanks: 89 given, 94 received



gregid View Post
Majority of custom solutions for storing tick and trade data utilize standard RDBMS (MS SQL Server, MySQL, etc.) which I have done myself in the past. I believe this is mostly due to familiarity with relational dbs and maturity of the solutions.

Still I keep wondering about the solutions more suitable to the task and started exploring Time Series databases. I have no previous experience with any, and was wondering what are your views on this approach and if anyone here has any experience with them and/or recommendations.

Some open source Time Series databases I am aware of:
Druid | Interactive Analytics at Scale
InfluxDB - Open Source Time Series, Metrics, and Analytics Database
Blueflood
OpenTSDB - A Distributed, Scalable Monitoring System
https://github.com/kairosdb/kairosdb
https://github.com/OpenNMS/newts
https://code.google.com/archive/p/tsdb
https://github.com/gar1t/tsdb
https://github.com/dustin/seriesly
https://github.com/ezhulenev/marketdb
https://github.com/graphite-project/ceres
https://github.com/prometheus/prometheus
https://github.com/akumuli/Akumuli
https://github.com/despegar/khronus

I have started to build a security master data db. I has now only eod from yahoo, but will add data from IB and add fundamentals with help by different APIs and also with help of the Python tool called Scrapy. I do not think it is so important what database you choose. If you later down the road decide to change it is not that big issue to have someone converting it. I use MySQL, as I hire programmers to build my database, I prefer to use a well known database so it's easier to find good programmers to a reasonable price. I use Python for the same reason as well and also Python has a lots of free scripts such as wrappers to APIs of the fundamental databases I need the data from. Such as Yahoo,ECB,FRED,etc.

Reply With Quote
The following user says Thank You to donedge for this post:
 
(login for full post details)
  #4 (permalink)
Site Moderator
 
 
sam028's Avatar
 
Posts: 3,641 since Jun 2009
Thanks: 3,769 given, 4,447 received


donedge View Post
I have started to build a security master data db. I has now only eod from yahoo, but will add data from IB and add fundamentals with help by different APIs and also with help of the Python tool called Scrapy. I do not think it is so important what database you choose. If you later down the road decide to change it is not that big issue to have someone converting it. I use MySQL, as I hire programmers to build my database, I prefer to use a well known database so it's easier to find good programmers to a reasonable price. I use Python for the same reason as well and also Python has a lots of free scripts such as wrappers to APIs of the fundamental databases I need the data from. Such as Yahoo,ECB,FRED,etc.

MySQL is fine with a "limited" amount of rows, with billions and billions of row per table things can become more difficult (slow). This is were a noSQL data base can help.
I may need such kind of data bases soon and already take a quick look but didn't had a "This is the DB type I need for my databases" (storing tick data with bid/ask).
FYI Big Mike is using MariaDB (with TokuDB as its engine) and it's seems to works with dozen of billions rows
(served by fast hardware).
So I'm do not have answers for @gregid but will follow this thread to find some .

Success requires no deodorant! (Sun Tzu)
Follow me on Twitter Reply With Quote
The following user says Thank You to sam028 for this post:
 
(login for full post details)
  #5 (permalink)
Jacksonville, Florida United States
 
Experience: Intermediate
Platform: NT8 + Custom
Broker: NT Brokerage, Kinetick, IQFeed, Interactive Brokers
Trading: ES
 
Jasonnator's Avatar
 
Posts: 108 since Dec 2014
Thanks: 30 given, 95 received

Have you looked at QDMS at Software - QUSMA? That may not be a bad place to start with something custom. It's written in C#, open source, and uses MariaDB.

Jason

Reply With Quote
The following 2 users say Thank You to Jasonnator for this post:
 
(login for full post details)
  #6 (permalink)
Wrocław, Poland
 
Experience: Intermediate
Platform: NinjaTrader, Racket
Trading: Ockham's razor
 
gregid's Avatar
 
Posts: 651 since Aug 2009
Thanks: 320 given, 618 received

Thanks guys for your input.

Just as @sam028 mentioned the reason for exploring NoSQL solutions is the usual degrading performance of the relational databases for huge amount of data. I agree with the argument for large knowledge base in the case of SQL and from my own experience I've found SQL dbs to be entirely sufficient for EOD data.
Once you take into consideration tick + bid/ask then the reasoning for a search for better solution becomes more understandable. It's not that I don't want to look at RDBMS - I use them my entire career - it is about exploring what else is out there.

In regards to QUSMA it looks like a nice analytics tool where - from my cursory look - the storage is expected to be local MySQL or MS SQL Server, so it doesn't address the alternative storage issue.
Still a very nice find I am about to explore!

Reply With Quote
 
(login for full post details)
  #7 (permalink)
Jacksonville, Florida United States
 
Experience: Intermediate
Platform: NT8 + Custom
Broker: NT Brokerage, Kinetick, IQFeed, Interactive Brokers
Trading: ES
 
Jasonnator's Avatar
 
Posts: 108 since Dec 2014
Thanks: 30 given, 95 received

I was using MongoDB with MongoVUE a few years ago when I was doing some neural network stuff that needed speedy I/O. That combo may be worth a look as well since it is noSQL, simple, and free.

Reply With Quote
The following user says Thank You to Jasonnator for this post:
 
(login for full post details)
  #8 (permalink)
Spain
 
Experience: Intermediate
Platform: NinjaTrader
Trading: Options
 
Posts: 13 since Oct 2009
Thanks: 11 given, 16 received

What about Tea Files? I use it and works fine....

Reply With Quote
The following 2 users say Thank You to Optiondreamer for this post:
 
(login for full post details)
  #9 (permalink)
Wrocław, Poland
 
Experience: Intermediate
Platform: NinjaTrader, Racket
Trading: Ockham's razor
 
gregid's Avatar
 
Posts: 651 since Aug 2009
Thanks: 320 given, 618 received


Optiondreamer View Post
What about Tea Files? I use it and works fine....

Tea Files was not on my radar so thanks for chiming in! Will definitely take a look.

Reply With Quote
 
(login for full post details)
  #10 (permalink)
Legendary Market Wizard
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
 
Posts: 1,094 since Jul 2012
Thanks: 747 given, 2,312 received



donedge View Post
I do not think it is so important what database you choose.

Golden advice here.

Reply With Quote


futures io Trading Community Platforms and Indicators > [Other]       Time Series database for tick and trade data.


May 21, 2019


Upcoming Webinars and Events
 

NinjaTrader (special webinar #400 w/giveaways)

Oct TBD
 

Bookmap

Oct TBD
 

SharkIndicators

Oct TBD
 

Carley Garner

Oct TBD
 

Ironbeam

Oct TBD
 

Zytrade

Oct TBD
 

GruttePier

Oct TBD
     



Copyright © 2020 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts