Tick Database Storage - Platforms and Indicators | futures io social day trading
futures io futures trading


Tick Database Storage
Updated: Views / Replies:17,918 / 40
Created: by MXASJ Attachments:3

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 3  
 
Thread Tools Search this Thread
 

Tick Database Storage

  #11 (permalink)
Elite Member
Asia
 
Futures Experience: Beginner
Platform: NinjaTrader, TOS
 
Posts: 798 since Jun 2009
Thanks: 109 given, 786 received

This outputs to the Output Window OK, going back the number of days specified in "Days to load" when you start the strategy. Next hurdle is getting to write a CSV file to disk:

 
Code
                            
#region Using declarations
using System;
using System.ComponentModel;
using System.Diagnostics;
using System.Drawing;
using System.Drawing.Drawing2D;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Data;
using NinjaTrader.Indicator;
using NinjaTrader.Gui.Chart;
using NinjaTrader.Strategy;
using System.IO;
#endregion
// This namespace holds all strategies and is required. Do not change it.
namespace NinjaTrader.Strategy
{
/// <summary>
/// Enter the description of your strategy here
/// </summary>
[Description("Strategy to write historical data from BarSeries[0]")]
public class 
DataWriter Strategy
{
#region Variables
 
#endregion
/// <summary>
/// This method is used to configure the strategy and is called once before any strategy method is called.
/// </summary>
protected override void Initialize()
{
CalculateOnBarClose true;
BarsRequired 20;
 
}
/// <summary>
/// Called on each bar update event (incoming tick)
/// </summary>
protected override void OnBarUpdate()
{
PrintWithTimeStamp(Open[0] + "," High[0] + "," Low[0] + ","Close[0]);
}
#region Properties
 
#endregion
}

Having a bit of agro with StreamWriter. This one only writes one line, though the Output Window looks OK. Here is the code;

 
Code
                            
{
//Print the data to the Output Window 
PrintWithTimeStamp(Open[0] + "," High[0] + "," Low[0] + "," Close[0] + "," Volume[0]);
 
//Write the data to PathFileName
 
using (StreamWriter sw File.CreateText(PathFileName))
{
sw.WriteLine(Time[0] + "," Open[0] + "," High[0] + "," Low[0] + "," Close[0] + "," Volume[0]);
sw.Close();
}

I've attached the strat as it stands now. NT7.

Attached Files
Register to download File Type: zip DataWriter.zip (1.1 KB, 30 views)

Last edited by MXASJ; January 29th, 2010 at 04:55 AM.
Reply With Quote
 
  #12 (permalink)
Elite Member
Asia
 
Futures Experience: Beginner
Platform: NinjaTrader, TOS
 
Posts: 798 since Jun 2009
Thanks: 109 given, 786 received

Too late to edit the bad code above, so here is the good code. Its based on something that was already available on the Ninja support site (Duh!). My next step is to offer the option of changing the DateTime format.

This should make my workflow a bit simpler for now at least.

Indicator ExportChartData and sample output attached (NT7).

EDIT: I've completed the indicator (mostly) and posted it in the downloads section under Ninja Trader/Misc. Hope it helps.

Attached Files
Register to download File Type: txt ES_03_10_5min.txt (78.1 KB, 52 views)

Last edited by MXASJ; January 30th, 2010 at 08:54 AM.
Reply With Quote
 
  #13 (permalink)
Trading Apprentice
Portland, Oregon
 
Futures Experience: Beginner
Platform: Ninjatrader
 
Posts: 27 since Jan 2010
Thanks: 3 given, 2 received



Big Mike View Post
Right, it uses OnMarketData and etc, it does not convert the existing database.

I used to run it on a separate box, a Quad Core Q6600, it would come close to maxing out NT 6.5 single-core/thread, it would be much better to do it on NT7. The SQL db server should be local if possible as well to reduce latency. I was recording for a half dozen instruments.

Mike

Big Mike, what's the granularity of the ticks you are storing? Are you storing OHLC or the individual transactions, like this:

7:24:47.664433 [TRADE] ESH0.CME 1 @ 1096.75
7:24:47.664433 [VOLUME] ESH0.CME 115924 @ 0.0
7:24:47.664623 [BID] ESH0.CME 176 @ 1096.75
7:24:48.065112 [TRADE] ESH0.CME 1 @ 1096.75
7:24:48.065112 [VOLUME] ESH0.CME 115925 @ 0.0
7:24:48.06566 [BID] ESH0.CME 175 @ 1096.75

I spent about 5 minutes looking for a way to do this in NinjaTrader, but was not able to figure it out. I am currently logging to text files, but I plan to store binary files by day. Maybe eventually I will move everything to something like Berkeley DB. Whatever supports the most data compression and reliability.

Reply With Quote
 
  #14 (permalink)
Elite Member
Chicago
 
Futures Experience: Intermediate
Platform: NinjaTrader
Broker/Data: Mirus/Zen-FIre
Favorite Futures: GC, ES
 
Posts: 73 since Mar 2010
Thanks: 40 given, 31 received

If I may add my $.02. Saving tickdata is an immensely expensive (data storage, latency and resource HOG) endeavor. Saving to the database, even with a buffer, would make any relational database grow exponentially fast. Further, querying that data with SQL would take forever and is time-prohibitive. I worked on a project where we listened to tick data and saved them to binary-files (similar to how NT saves tick data in NT7) using a buffer (like BM suggested). We then created binary read applications that would be able to translate and chart the data as needed (timebound bars, tick bars, etc).

I would recommend simply opening up a bunch of charts and have NT do all of the heavy lifting with regards to listening and saving the data to binary (have numerous NT charts open but minimized maybe?) and have it store the data in its binary format. Then create some sort of binary-reader that will be able to translate the saved binary files for later use (either in matlab, r or anything else).

Does that make sense? I guess my question is, if you simply open up a 1tick chart on NT, will the data be saved? I don't have a broker-connected NT at the moment, so I'm unable to test this.

Reply With Quote
 
  #15 (permalink)
Elite Member
Asia
 
Futures Experience: Beginner
Platform: NinjaTrader, TOS
 
Posts: 798 since Jun 2009
Thanks: 109 given, 786 received

There is a 4 Gig limit on the size of the SQL CE database engine that NT7 uses, so that would need to be taken into consideration ie you couldn't record tick data of the whole CME Group product list for very long on a single instance of NT ..

A side project I haven't looked into yet is sync between SQL CE and SQL, but I'm looking at that more for getting an overview of multiple NT users positions than for storing tick data.

I'm leaning towards the "buy" camp in the "buy vs. build" for historical tick data if 1 second timestamps are OK.

phyzfer if you need live futures data for development (not trading) try this:

Zen-Fire. The Ultimate Trading Solution.

That will get you login credentials for 30 days or so which you can use with your existing NT installation.

Reply With Quote
 
  #16 (permalink)
Elite Member
Chicago
 
Futures Experience: Intermediate
Platform: NinjaTrader
Broker/Data: Mirus/Zen-FIre
Favorite Futures: GC, ES
 
Posts: 73 since Mar 2010
Thanks: 40 given, 31 received

Doh! 4GB? I don't understand. I was under the impression if you opened up a 1-tick chart in NT, that it will save the data into the \Documents\NinjaTrader 7\db\tick folder. I guess that isn't correct then. Right?

If not, then going off of your strategy code, I'd create a binary-writer to register to the contracts and write to disk (via a buffer + some compression algorithms). Then the fun part. We'd create an interface (separate C# app) that would find, decompress and return the data in whatever format your calling program (matlab, r, NT even) would need.

I'd love to help with something like this. I think keeping tick data in a relational database makes no sense whatsoever. Querying the data will take forever and the db size will grow significantly.

Reply With Quote
 
  #17 (permalink)
Elite Member
Asia
 
Futures Experience: Beginner
Platform: NinjaTrader, TOS
 
Posts: 798 since Jun 2009
Thanks: 109 given, 786 received

Where I get lost is if SQL CE is writing binaries to the file system and the DB is merely referencing those binaries... do the binaries themselves count towards the 4 Gig limit or are only the references (which should only be a few bytes) counted.

I'll be honest I'm waaaaaaaaay out of my depth on this one.

Reply With Quote
 
  #18 (permalink)
Elite Member
Chicago
 
Futures Experience: Intermediate
Platform: NinjaTrader
Broker/Data: Mirus/Zen-FIre
Favorite Futures: GC, ES
 
Posts: 73 since Mar 2010
Thanks: 40 given, 31 received

Personally I don't THINK that NT uses the (SQL CE) database for any market data storage. I think they use it mostly for internal data (account performance, ATM strategies, etc).

I was under the impression that NT saved all Market Data in the Documents\NinjaTrader7\db folder. If you take a look, you'll see a Tick, Minute, Day subfolders that contain binary files (.ntd file extension). How'd these folders get populated?

Personally I'd want NT to save all tick-data for a product to disk whenever any chart is open.

Reply With Quote
 
  #19 (permalink)
Elite Member
Chicago
 
Futures Experience: Intermediate
Platform: NinjaTrader
Broker/Data: Mirus/Zen-FIre
Favorite Futures: GC, ES
 
Posts: 73 since Mar 2010
Thanks: 40 given, 31 received

I just got this from the NT's help document:

NinjaTrader Converts Real-Time Data into Historical Data
NinjaTrader stores real-time incoming tick data to your local PC if you have a Chart or Market Analyzer (must have an indicator column added) window open. This data can then be used as historical data. For example, if you open a chart and let it run all day long, the data collected today, will be available as historical data when you open the same chart tomorrow.


I take this to mean that NT is saving tick data on the machine (in the location I specified above) and makes that data available to NT. We'd simply need a way to 'convert' it back and traverse the data. I'd also like to place some sort of backup strategy in place on the machine that is running this data collection.





A colleague of mine suggested I look into this for storing market data:
HDF Group - HDF5

Has anyone used HDF5?

thanks!

Reply With Quote
 
  #20 (permalink)
Elite Member
Near the BEuTiFULL Horse Shoe
 
Futures Experience: Beginner
Platform: NinjaTrader
Broker/Data: MBTrading Dukascopy ZenFire
Favorite Futures: $EURUSD when it is trending
 
Trader.Jon's Avatar
 
Posts: 500 since Jul 2009
Thanks: 401 given, 176 received


At one time I was talking with a programmer to do this type of project for me: he stopped answering my emails ... cant think why he would do that LOL@me

"" The TREE Data Server captures real-time financial data from one or several datafeed services, archives data in a historical database, and makes both live and archived data available to client applications. The system can be used in real-time charting, an ATS, tick feed simulator, etc. or any situation in which multiple clients need real-time access to the tick stream or archived tick data. In addition, the archived tick stream and tick data are available for offline data analysis and backtesting. ""

Support for the code is one person ... but he does eventually answer email
tree.sourceforge.net | TREE Data Server

TJ

Reply With Quote

Reply



futures io > > > Tick Database Storage

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Jigsaw Trading: TBA

Elite only

FuturesTrader71: TBA

Elite only

NinjaTrader: TBA

Jan 18

RandBots: TBA

Jan 23

GFF Brokers & CME Group: Futures & Bitcoin

Elite only

Adam Grimes: TBA

Elite only

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
U.S. Strategic Reserve Oil May Be Put in Storage by Traders kbit News and Current Events 1 July 6th, 2011 08:28 PM
Database for NT bomberone1 NinjaTrader Programming 6 April 29th, 2011 10:11 AM
PLT_NTDataCollect store bid ask tick data to its local database? rcabri Trading Reviews and Vendors 3 December 27th, 2010 11:00 AM
Tick Data Storage timmyb The Elite Circle 9 June 16th, 2010 05:44 AM
My 2 cents in mysql database iwannatoscript NinjaTrader Programming 5 March 7th, 2010 09:56 AM


All times are GMT -4. The time now is 11:11 AM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-13 in 0.14 seconds with 20 queries on phoenix via your IP 54.227.51.103