[Other] Time Series database for tick and trade data. - Platforms and Indicators | futures io social trading
futures io futures trading


Time Series database for tick and trade data.
Updated: Views / Replies:4,358 / 36
Created: by gregid Attachments:2

Welcome to futures io.

Welcome, Guest!

This forum was established to help traders (especially futures traders) by openly sharing indicators, strategies, methods, trading journals and discussing the psychology of trading.

We are fundamentally different than most other trading forums:
  • We work extremely hard to keep things positive on our forums.
  • We do not tolerate rude behavior, trolling, or vendor advertising in posts.
  • We firmly believe in openness and encourage sharing. The holy grail is within you, it is not something tangible you can download.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.


You'll need to register in order to view the content of the threads and start contributing to our community. It's free and simple, and we will never resell your private information.

-- Big Mike

Reply
 2  
 
Thread Tools Search this Thread
 

Time Series database for tick and trade data.

  #11 (permalink)
Elite Member
Stockholm
 
Futures Experience: Advanced
Platform: NinjaTrader,Sierra Chart, Python
Broker/Data: IB
Favorite Futures: ES & Stocks
 
Posts: 86 since Apr 2013
Thanks: 80 given, 71 received


artemiso View Post
Golden advice here.



People who want to store tick data maybe there can be advantages to use a special database. The issue with store tick data is difficult to find a data source that can supplies many years of tick data.. I need more history that that so daily/minute data is more than enough for my purpose and then I can get 20-50 years of history..

Reply With Quote
 
  #12 (permalink)
Trading Apprentice
western NY
 
Futures Experience: Intermediate
Platform: NinjaTrader, Python
Favorite Futures: equities, bonds, currency, commodities
 
Posts: 42 since Apr 2015
Thanks: 9 given, 25 received


Optiondreamer View Post
What about Tea Files? I use it and works fine....

Looks pretty cool.
If you google TeaFile vs hdf5 there is a nice discussion on this topic on nuclear phynance and the creator of TeaFile chimes in.

That same thread they mention scidb Paradigm4: Home. That has quant finance in mind but might be overkill for an individual.

As far as not being important what you decide on I also think there is a technological fun factor not to be dismissed.
Some people get a kick out of maximizing the performance of their car engines but any engine that runs can easily go the speed limit. To say you don't need to bother maxing out the performance of the engine while true completely misses the point.


Last edited by Darthtrader4beta; May 3rd, 2015 at 03:39 PM.
Reply With Quote
 
  #13 (permalink)
Elite Member
Wrocław, Poland
 
Futures Experience: Intermediate
Platform: NinjaTrader, Racket
Favorite Futures: Ockham's razor
 
gregid's Avatar
 
Posts: 651 since Aug 2009
Thanks: 321 given, 603 received



Darthtrader4beta View Post
Looks pretty cool.
If you google TeaFile vs hdf5 there is a nice discussion on this topic on nuclear phynance and the creator of TeaFile chimes in.

That same thread they mention scidb Paradigm4: Home. That has quant finance in mind but might be overkill for an individual.

As far as not being important what you decide on I also think there is a technological fun factor not to be dismissed.
Some people get a kick out of maximizing the performance of their car engines but any engine that runs can easily go the speed limit. To say you don't need to bother maxing out the performance of the engine while true completely misses the point.

True that! Some do it with their cars, others with computer rigs, yet others with their software, and algorithms (strategies anyone? ) After all - isn't it how the progress is made? Especially if it brings joy to your life

[Comments on the lack of social life are not welcomed ]

Reply With Quote
 
  #14 (permalink)
Elite Member
Wrocław, Poland
 
Futures Experience: Intermediate
Platform: NinjaTrader, Racket
Favorite Futures: Ockham's razor
 
gregid's Avatar
 
Posts: 651 since Aug 2009
Thanks: 321 given, 603 received

SciDB is at the moment my most serious contender for the next project. For "array type" languages like python+numpy+pandas or julia this looks to be the most natural way of working with persistent data.

Here a presentation on timeseries and SciDB:



Another interesting project from the python world:
Ecosystem — Blaze 0.7.3 documentation

Reply With Quote
The following 5 users say Thank You to gregid for this post:
 
  #15 (permalink)
Elite Member
San Francisco, CA
 
Futures Experience: Advanced
Platform: SC, eSignal
Broker/Data: IB
Favorite Futures: Spreads
 
Posts: 44 since Jan 2015
Thanks: 43 given, 36 received

Some obvious well-known contenders that come to mind:

RRD
BDB

When it comes to efficiency/overhead, a row-based database (which most SQL accessed ones are) is not ideal for tick data.

Reply With Quote
 
  #16 (permalink)
Trading Apprentice
western NY
 
Futures Experience: Intermediate
Platform: NinjaTrader, Python
Favorite Futures: equities, bonds, currency, commodities
 
Posts: 42 since Apr 2015
Thanks: 9 given, 25 received

damn, SciDB and python in that video looks like the sex machine.

Reply With Quote
 
  #17 (permalink)
Trading Apprentice
Paris, France
 
Futures Experience: Intermediate
Platform: MetaTrader
Favorite Futures: Currency Futures
 
Posts: 2 since Sep 2016
Thanks: 0 given, 2 received

Hi,

sorry for seeing this thread so late!

What is your opinion about Arctic ?
Search "arctic datastore"

Thanks

Reply With Quote
The following user says Thank You to femto for this post:
 
  #18 (permalink)
Trading Apprentice
Paris, France
 
Futures Experience: Intermediate
Platform: MetaTrader
Favorite Futures: Currency Futures
 
Posts: 2 since Sep 2016
Thanks: 0 given, 2 received

An other option may be Feather, it's "fast, interoperable binary data frame storage for Python, R, and more powered by Apache Arrow."

Any opinion about this one also?

Reply With Quote
The following user says Thank You to femto for this post:
 
  #19 (permalink)
Elite Member
Wrocław, Poland
 
Futures Experience: Intermediate
Platform: NinjaTrader, Racket
Favorite Futures: Ockham's razor
 
gregid's Avatar
 
Posts: 651 since Aug 2009
Thanks: 321 given, 603 received

From a cursory look Feather looks more like an alternative to hdf5.
Arctic looks like a more mature solution though. Nice slideshow about arctic:
Arctic: High-performance IoT and financial data storage with Python a?

Reply With Quote
 
  #20 (permalink)
Elite Member
Jacksonville, Florida United States
 
Futures Experience: Intermediate
Platform: Fully custom
Broker/Data: Optimus Futures, Interactive Brokers
Favorite Futures: Profitable ones
 
Jasonnator's Avatar
 
Posts: 67 since Dec 2014
Thanks: 18 given, 41 received

PostgreSQL


Set up PostgreSQL according to the attachment and you're set.

Attached Thumbnails
Time Series database for tick and trade data.-nosql-way-postgresql.pdf   Time Series database for tick and trade data.-superfast-postgresql.pdf  

Last edited by Jasonnator; September 29th, 2016 at 08:25 AM. Reason: add Superfast PostgreSQL pdf
Reply With Quote
The following user says Thank You to Jasonnator for this post:

Reply



futures io > > > [Other]       Time Series database for tick and trade data.

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

FIO Journal Challenge featuring NinjaTrader ($2,000+ of prizes)

May
 

Prototyping Python Strategies (part 2: Backtesting) w/Ran Aroussi [special time: 1PM ET]

Elite only
 

Portfolio Diversification w/Brendon Delate @ Diversify Portfolio

Elite only

An Afternoon with FIO member Softsoap (being rescheduled)

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
CTA - Series 3, Series 7, Series 9, Series 10, Series 56 NASD exams certifications Big Mike Traders Hideout 41 January 15th, 2015 09:27 AM
Tick data for ES a few years in time elitetradernyc NinjaTrader Programming 2 May 6th, 2013 04:23 PM
DataBase of TICK Data bomberone1 The Elite Circle 4 June 17th, 2012 09:31 PM
Is the 6E 12.50 a tick? Best time to trade it Texas time? skyfly Currency Futures 7 August 7th, 2011 09:49 AM
PLT_NTDataCollect store bid ask tick data to its local database? rcabri Vendors and Product Reviews 3 December 27th, 2010 10:00 AM


All times are GMT -4. The time now is 08:05 AM.

Copyright © 2017 by futures io, s.a. Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-05-24 in 0.11 seconds with 20 queries on phoenix via your IP 54.224.153.234