[Other] Time Series database for tick and trade data. - Platforms and Indicators | futures.io
futures io futures trading


Time Series database for tick and trade data.
Started: by gregid Views / Replies:4,027 / 36
Last Reply: Attachments:2

Welcome to futures io.

Welcome, Guest!

This forum was established to help traders (especially futures traders) by openly sharing indicators, strategies, methods, trading journals and discussing the psychology of trading.

We are fundamentally different than most other trading forums:
  • We work extremely hard to keep things positive on our forums.
  • We do not tolerate rude behavior, trolling, or vendor advertising in posts.
  • We firmly believe in openness and encourage sharing. The holy grail is within you, it is not something tangible you can download.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.


You'll need to register in order to view the content of the threads and start contributing to our community. It's free and simple, and we will never resell your private information.

-- Big Mike

Reply
 2  
 
Thread Tools Search this Thread
 

Time Series database for tick and trade data.

  #1 (permalink)
Elite Member
Wrocław, Poland
 
Futures Experience: Intermediate
Platform: NinjaTrader, Racket
Favorite Futures: Ockham's razor
 
gregid's Avatar
 
Posts: 651 since Aug 2009
Thanks: 321 given, 601 received

Time Series database for tick and trade data.

Majority of custom solutions for storing tick and trade data utilize standard RDBMS (MS SQL Server, MySQL, etc.) which I have done myself in the past. I believe this is mostly due to familiarity with relational dbs and maturity of the solutions.

Still I keep wondering about the solutions more suitable to the task and started exploring Time Series databases. I have no previous experience with any, and was wondering what are your views on this approach and if anyone here has any experience with them and/or recommendations.

Some open source Time Series databases I am aware of:
Druid | Interactive Analytics at Scale
InfluxDB - Open Source Time Series, Metrics, and Analytics Database
Blueflood
OpenTSDB - A Distributed, Scalable Monitoring System
https://github.com/kairosdb/kairosdb
https://github.com/OpenNMS/newts
https://code.google.com/p/tsdb/
https://github.com/gar1t/tsdb
https://github.com/dustin/seriesly
https://github.com/ezhulenev/marketdb
https://github.com/graphite-project/ceres
https://github.com/prometheus/prometheus
https://github.com/akumuli/Akumuli
https://github.com/despegar/khronus

Reply With Quote
The following 3 users say Thank You to gregid for this post:
 
  #2 (permalink)
Quick Summary
Quick Summary Post

Quick Summary is created and edited by users like you... Add FAQ's, Links and other Relevant Information by clicking the edit button in the lower right hand corner of this message.

 
  #3 (permalink)
Elite Member
Stockholm
 
Futures Experience: Advanced
Platform: NinjaTrader,Sierra Chart, Python
Broker/Data: IB
Favorite Futures: ES & Stocks
 
Posts: 86 since Apr 2013
Thanks: 80 given, 69 received



gregid View Post
Majority of custom solutions for storing tick and trade data utilize standard RDBMS (MS SQL Server, MySQL, etc.) which I have done myself in the past. I believe this is mostly due to familiarity with relational dbs and maturity of the solutions.

Still I keep wondering about the solutions more suitable to the task and started exploring Time Series databases. I have no previous experience with any, and was wondering what are your views on this approach and if anyone here has any experience with them and/or recommendations.

Some open source Time Series databases I am aware of:
Druid | Interactive Analytics at Scale
InfluxDB - Open Source Time Series, Metrics, and Analytics Database
Blueflood
OpenTSDB - A Distributed, Scalable Monitoring System
https://github.com/kairosdb/kairosdb
https://github.com/OpenNMS/newts
https://code.google.com/p/tsdb/
https://github.com/gar1t/tsdb
https://github.com/dustin/seriesly
https://github.com/ezhulenev/marketdb
https://github.com/graphite-project/ceres
https://github.com/prometheus/prometheus
https://github.com/akumuli/Akumuli
https://github.com/despegar/khronus

I have started to build a security master data db. I has now only eod from yahoo, but will add data from IB and add fundamentals with help by different APIs and also with help of the Python tool called Scrapy. I do not think it is so important what database you choose. If you later down the road decide to change it is not that big issue to have someone converting it. I use MySQL, as I hire programmers to build my database, I prefer to use a well known database so it's easier to find good programmers to a reasonable price. I use Python for the same reason as well and also Python has a lots of free scripts such as wrappers to APIs of the fundamental databases I need the data from. Such as Yahoo,ECB,FRED,etc.

Reply With Quote
The following user says Thank You to donedge for this post:
 
  #4 (permalink)
Administrator: Retired Backtester
 Vendor: speedytradingservers.com 
Rennes France
 
Futures Experience: Advanced
Platform: NinjaTrader
Broker/Data: IB/Kinetick
Favorite Futures: Futures
 
sam028's Avatar
 
Posts: 3,301 since Jun 2009
Thanks: 3,508 given, 3,900 received


donedge View Post
I have started to build a security master data db. I has now only eod from yahoo, but will add data from IB and add fundamentals with help by different APIs and also with help of the Python tool called Scrapy. I do not think it is so important what database you choose. If you later down the road decide to change it is not that big issue to have someone converting it. I use MySQL, as I hire programmers to build my database, I prefer to use a well known database so it's easier to find good programmers to a reasonable price. I use Python for the same reason as well and also Python has a lots of free scripts such as wrappers to APIs of the fundamental databases I need the data from. Such as Yahoo,ECB,FRED,etc.

MySQL is fine with a "limited" amount of rows, with billions and billions of row per table things can become more difficult (slow). This is were a noSQL data base can help.
I may need such kind of data bases soon and already take a quick look but didn't had a "This is the DB type I need for my databases" (storing tick data with bid/ask).
FYI Big Mike is using MariaDB (with TokuDB as its engine) and it's seems to works with dozen of billions rows
(served by fast hardware).
So I'm do not have answers for @gregid but will follow this thread to find some .

Success requires no deodorant! (Sun Tzu)
Reply With Quote
The following user says Thank You to sam028 for this post:
 
  #5 (permalink)
Elite Member
Jacksonville, Florida United States
 
Futures Experience: Intermediate
Platform: Fully custom
Broker/Data: Optimus Futures, Interactive Brokers
Favorite Futures: Profitable ones
 
Jasonnator's Avatar
 
Posts: 67 since Dec 2014
Thanks: 18 given, 41 received

Futures Edge on FIO
Have you looked at QDMS at Software - QUSMA? That may not be a bad place to start with something custom. It's written in C#, open source, and uses MariaDB.

Jason

Reply With Quote
The following 2 users say Thank You to Jasonnator for this post:
 
  #6 (permalink)
Elite Member
Wrocław, Poland
 
Futures Experience: Intermediate
Platform: NinjaTrader, Racket
Favorite Futures: Ockham's razor
 
gregid's Avatar
 
Posts: 651 since Aug 2009
Thanks: 321 given, 601 received

Thanks guys for your input.

Just as @sam028 mentioned the reason for exploring NoSQL solutions is the usual degrading performance of the relational databases for huge amount of data. I agree with the argument for large knowledge base in the case of SQL and from my own experience I've found SQL dbs to be entirely sufficient for EOD data.
Once you take into consideration tick + bid/ask then the reasoning for a search for better solution becomes more understandable. It's not that I don't want to look at RDBMS - I use them my entire career - it is about exploring what else is out there.

In regards to QUSMA it looks like a nice analytics tool where - from my cursory look - the storage is expected to be local MySQL or MS SQL Server, so it doesn't address the alternative storage issue.
Still a very nice find I am about to explore!

Reply With Quote
 
  #7 (permalink)
Elite Member
Jacksonville, Florida United States
 
Futures Experience: Intermediate
Platform: Fully custom
Broker/Data: Optimus Futures, Interactive Brokers
Favorite Futures: Profitable ones
 
Jasonnator's Avatar
 
Posts: 67 since Dec 2014
Thanks: 18 given, 41 received

I was using MongoDB with MongoVUE a few years ago when I was doing some neural network stuff that needed speedy I/O. That combo may be worth a look as well since it is noSQL, simple, and free.

Reply With Quote
The following user says Thank You to Jasonnator for this post:
 
  #8 (permalink)
Elite Member
Spain
 
Futures Experience: Intermediate
Platform: NinjaTrader
Favorite Futures: Options
 
Posts: 12 since Oct 2009
Thanks: 10 given, 15 received

What about Tea Files? I use it and works fine....

Reply With Quote
The following 2 users say Thank You to Optiondreamer for this post:
 
  #9 (permalink)
Elite Member
Wrocław, Poland
 
Futures Experience: Intermediate
Platform: NinjaTrader, Racket
Favorite Futures: Ockham's razor
 
gregid's Avatar
 
Posts: 651 since Aug 2009
Thanks: 321 given, 601 received


Optiondreamer View Post
What about Tea Files? I use it and works fine....

Tea Files was not on my radar so thanks for chiming in! Will definitely take a look.

Reply With Quote
 
  #10 (permalink)
Elite Member
Manchester, NH
 
Futures Experience: Beginner
Platform: thinkorswim
Broker/Data: TD Ameritrade
Favorite Futures: Stocks
 
Posts: 884 since Jul 2012
Thanks: 595 given, 1,761 received



donedge View Post
I do not think it is so important what database you choose.

Golden advice here.

Reply With Quote

Reply



futures io > > > [Other]       Time Series database for tick and trade data.

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)
 

Spring Grains Outlook w/Sean Lusk @ Walsh Trading

Elite only

FIO Video Journal Challenge featuring NinjaTrader ($2,000+ of prizes)

April

Process above all else w/Anthony Crudele @ Futures Radio Show

Elite only

Machine Learning - Quantitative Trading w/Martin Froehler @ Quantiacs

Elite only

Market Dynamics w/Peter Davies @ Jigsaw Trading

Elite only

Ask Me Anything w/Patrick Rooney @ Trading Technologies

Apr 18

Ask Me Anything w/FuturesTrader71

Apr 19

Machine Learning w/Kris Longmore

Elite only

Market Analysis w/Dave Forss

Apr 25

Introducing iSystems with Stage 5 Trading

Apr 27
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
CTA - Series 3, Series 7, Series 9, Series 10, Series 56 NASD exams certifications Big Mike Traders Hideout 41 January 15th, 2015 09:27 AM
Tick data for ES a few years in time elitetradernyc NinjaTrader Programming 2 May 6th, 2013 04:23 PM
DataBase of TICK Data bomberone1 The Elite Circle 4 June 17th, 2012 09:31 PM
Is the 6E 12.50 a tick? Best time to trade it Texas time? skyfly Currency Futures 7 August 7th, 2011 09:49 AM
PLT_NTDataCollect store bid ask tick data to its local database? rcabri Vendors and Product Reviews 3 December 27th, 2010 10:00 AM


All times are GMT -4. The time now is 06:56 PM.

no new posts
Page generated 2017-03-29 in 0.14 seconds with 20 queries on phoenix via your IP 54.157.3.92