NexusFi: Find Your Edge


Home Menu

 





Time Series database for tick and trade data.


Discussion in Platforms and Indicators

Updated
      Top Posters
    1. looks_one gregid with 9 posts (10 thanks)
    2. looks_two Jasonnator with 7 posts (8 thanks)
    3. looks_3 stocksharp with 4 posts (4 thanks)
    4. looks_4 artemiso with 3 posts (5 thanks)
      Best Posters
    1. looks_one artemiso with 1.7 thanks per post
    2. looks_two gregid with 1.1 thanks per post
    3. looks_3 Jasonnator with 1.1 thanks per post
    4. looks_4 stocksharp with 1 thanks per post
    1. trending_up 20,946 views
    2. thumb_up 35 thanks given
    3. group 25 followers
    1. forum 40 posts
    2. attach_file 2 attachments




 
Search this Thread

Time Series database for tick and trade data.

  #11 (permalink)
 donedge 
Rio de Janeiro / Barcelona
 
Experience: Advanced
Platform: NinjaTrader & Excel
Broker: IB
Trading: EWZ
Posts: 125 since Apr 2013
Thanks Given: 89
Thanks Received: 96


artemiso View Post
Golden advice here.



People who want to store tick data maybe there can be advantages to use a special database. The issue with store tick data is difficult to find a data source that can supplies many years of tick data.. I need more history that that so daily/minute data is more than enough for my purpose and then I can get 20-50 years of history..

Reply With Quote

Can you help answer these questions
from other members on NexusFi?
New Micros: Ultra 10-Year & Ultra T-Bond -- Live Now
Treasury Notes and Bonds
Are there any eval firms that allow you to sink to your …
Traders Hideout
Online prop firm The Funded Trader (TFT) going under?
Traders Hideout
Deepmoney LLM
Elite Quantitative GenAI/LLM
Build trailing stop for micro index(s)
Psychology and Money Management
 
Best Threads (Most Thanked)
in the last 7 days on NexusFi
Get funded firms 2023/2024 - Any recommendations or word …
60 thanks
Funded Trader platforms
43 thanks
NexusFi site changelog and issues/problem reporting
24 thanks
GFIs1 1 DAX trade per day journal
22 thanks
The Program
19 thanks
  #12 (permalink)
Darthtrader4beta
western NY
 
Posts: 42 since Apr 2015
Thanks Given: 9
Thanks Received: 26


Optiondreamer View Post
What about Tea Files? I use it and works fine....

Looks pretty cool.
If you google TeaFile vs hdf5 there is a nice discussion on this topic on nuclear phynance and the creator of TeaFile chimes in.

That same thread they mention scidb Paradigm4: Home. That has quant finance in mind but might be overkill for an individual.

As far as not being important what you decide on I also think there is a technological fun factor not to be dismissed.
Some people get a kick out of maximizing the performance of their car engines but any engine that runs can easily go the speed limit. To say you don't need to bother maxing out the performance of the engine while true completely misses the point.

Reply With Quote
  #13 (permalink)
 
gregid's Avatar
 gregid 
Wrocław, Poland
 
Experience: Intermediate
Platform: NinjaTrader, Racket
Trading: Ockham's razor
Posts: 650 since Aug 2009
Thanks Given: 320
Thanks Received: 623



Darthtrader4beta View Post
Looks pretty cool.
If you google TeaFile vs hdf5 there is a nice discussion on this topic on nuclear phynance and the creator of TeaFile chimes in.

That same thread they mention scidb Paradigm4: Home. That has quant finance in mind but might be overkill for an individual.

As far as not being important what you decide on I also think there is a technological fun factor not to be dismissed.
Some people get a kick out of maximizing the performance of their car engines but any engine that runs can easily go the speed limit. To say you don't need to bother maxing out the performance of the engine while true completely misses the point.

True that! Some do it with their cars, others with computer rigs, yet others with their software, and algorithms (strategies anyone? ) After all - isn't it how the progress is made? Especially if it brings joy to your life

[Comments on the lack of social life are not welcomed ]

Started this thread Reply With Quote
  #14 (permalink)
 
gregid's Avatar
 gregid 
Wrocław, Poland
 
Experience: Intermediate
Platform: NinjaTrader, Racket
Trading: Ockham's razor
Posts: 650 since Aug 2009
Thanks Given: 320
Thanks Received: 623

SciDB is at the moment my most serious contender for the next project. For "array type" languages like python+numpy+pandas or julia this looks to be the most natural way of working with persistent data.

Here a presentation on timeseries and SciDB:


Another interesting project from the python world:
Ecosystem — Blaze 0.7.3 documentation

Started this thread Reply With Quote
  #15 (permalink)
 i960 
San Francisco, CA
 
Experience: Advanced
Platform: SC, eSignal
Broker: IB
Trading: Spreads
Posts: 46 since Jan 2015
Thanks Given: 44
Thanks Received: 43

Some obvious well-known contenders that come to mind:

RRD
BDB

When it comes to efficiency/overhead, a row-based database (which most SQL accessed ones are) is not ideal for tick data.

Reply With Quote
  #16 (permalink)
Darthtrader4beta
western NY
 
Posts: 42 since Apr 2015
Thanks Given: 9
Thanks Received: 26

damn, SciDB and python in that video looks like the sex machine.

Reply With Quote
  #17 (permalink)
femto
Paris, France
 
Posts: 3 since Sep 2016
Thanks Given: 0
Thanks Received: 3

Hi,

sorry for seeing this thread so late!

What is your opinion about Arctic ?
Search "arctic datastore"

Thanks

Reply With Quote
Thanked by:
  #18 (permalink)
femto
Paris, France
 
Posts: 3 since Sep 2016
Thanks Given: 0
Thanks Received: 3

An other option may be Feather, it's "fast, interoperable binary data frame storage for Python, R, and more powered by Apache Arrow."

Any opinion about this one also?

Reply With Quote
Thanked by:
  #19 (permalink)
 
gregid's Avatar
 gregid 
Wrocław, Poland
 
Experience: Intermediate
Platform: NinjaTrader, Racket
Trading: Ockham's razor
Posts: 650 since Aug 2009
Thanks Given: 320
Thanks Received: 623

From a cursory look Feather looks more like an alternative to hdf5.
Arctic looks like a more mature solution though. Nice slideshow about arctic:
Arctic: High-performance IoT and financial data storage with Python a?

Started this thread Reply With Quote
  #20 (permalink)
 
Jasonnator's Avatar
 Jasonnator 
Denver, Colorado United States
 
Experience: Intermediate
Platform: NT8 + Custom
Broker: NT Brokerage, Kinetick, IQFeed, Interactive Brokers
Trading: ES
Posts: 159 since Dec 2014
Thanks Given: 40
Thanks Received: 166


Set up PostgreSQL according to the attachment and you're set.

Attached Thumbnails
Time Series database for tick and trade data.-nosql-way-postgresql.pdf   Time Series database for tick and trade data.-superfast-postgresql.pdf  
Reply With Quote
Thanked by:




Last Updated on May 21, 2019


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts