Majority of custom solutions for storing tick and trade data utilize standard RDBMS (MS SQL Server, MySQL, etc.) which I have done myself in the past. I believe this is mostly due to familiarity with relational dbs and maturity of the solutions.
Still I keep wondering about the solutions more suitable to the task and started exploring Time Series databases. I have no previous experience with any, and was wondering what are your views on this approach and if anyone here has any experience with them and/or recommendations.
I have started to build a security master data db. I has now only eod from yahoo, but will add data from IB and add fundamentals with help by different APIs and also with help of the Python tool called Scrapy. I do not think it is so important what database you choose. If you later down the road decide to change it is not that big issue to have someone converting it. I use MySQL, as I hire programmers to build my database, I prefer to use a well known database so it's easier to find good programmers to a reasonable price. I use Python for the same reason as well and also Python has a lots of free scripts such as wrappers to APIs of the fundamental databases I need the data from. Such as Yahoo,ECB,FRED,etc.
The following user says Thank You to donedge for this post:
MySQL is fine with a "limited" amount of rows, with billions and billions of row per table things can become more difficult (slow). This is were a noSQL data base can help.
I may need such kind of data bases soon and already take a quick look but didn't had a "This is the DB type I need for my databases" (storing tick data with bid/ask).
FYI Big Mike is using MariaDB (with TokuDB as its engine) and it's seems to works with dozen of billions rows
(served by fast hardware).
So I'm do not have answers for @gregid but will follow this thread to find some .
Success requires no deodorant! (Sun Tzu)
The following user says Thank You to sam028 for this post:
Broker: NT Brokerage, Kinetick, IQFeed, Interactive Brokers
Trading: ES
Posts: 159 since Dec 2014
Thanks: 40 given,
161
received
Have you looked at QDMS at Software - QUSMA? That may not be a bad place to start with something custom. It's written in C#, open source, and uses MariaDB.
Jason
The following 2 users say Thank You to Jasonnator for this post:
Just as @sam028 mentioned the reason for exploring NoSQL solutions is the usual degrading performance of the relational databases for huge amount of data. I agree with the argument for large knowledge base in the case of SQL and from my own experience I've found SQL dbs to be entirely sufficient for EOD data.
Once you take into consideration tick + bid/ask then the reasoning for a search for better solution becomes more understandable. It's not that I don't want to look at RDBMS - I use them my entire career - it is about exploring what else is out there.
In regards to QUSMA it looks like a nice analytics tool where - from my cursory look - the storage is expected to be local MySQL or MS SQL Server, so it doesn't address the alternative storage issue.
Still a very nice find I am about to explore!
Broker: NT Brokerage, Kinetick, IQFeed, Interactive Brokers
Trading: ES
Posts: 159 since Dec 2014
Thanks: 40 given,
161
received
I was using MongoDB with MongoVUE a few years ago when I was doing some neural network stuff that needed speedy I/O. That combo may be worth a look as well since it is noSQL, simple, and free.
The following user says Thank You to Jasonnator for this post:
.
- it is about exploring what else is out there.