Too late to edit the bad code above, so here is the good code. Its based on something that was already available on the Ninja support site (Duh!). My next step is to offer the option of changing the DateTime format.
This should make my workflow a bit simpler for now at least.
Indicator ExportChartData and sample output attached (NT7).
EDIT: I've completed the indicator (mostly) and posted it in the downloads section under Ninja Trader/Misc. Hope it helps.
Last edited by MXASJ; January 30th, 2010 at 07:54 AM.
I spent about 5 minutes looking for a way to do this in NinjaTrader, but was not able to figure it out. I am currently logging to text files, but I plan to store binary files by day. Maybe eventually I will move everything to something like Berkeley DB. Whatever supports the most data compression and reliability.
If I may add my $.02. Saving tickdata is an immensely expensive (data storage, latency and resource HOG) endeavor. Saving to the database, even with a buffer, would make any relational database grow exponentially fast. Further, querying that data with SQL would take forever and is time-prohibitive. I worked on a project where we listened to tick data and saved them to binary-files (similar to how NT saves tick data in NT7) using a buffer (like BM suggested). We then created binary read applications that would be able to translate and chart the data as needed (timebound bars, tick bars, etc).
I would recommend simply opening up a bunch of charts and have NT do all of the heavy lifting with regards to listening and saving the data to binary (have numerous NT charts open but minimized maybe?) and have it store the data in its binary format. Then create some sort of binary-reader that will be able to translate the saved binary files for later use (either in matlab, r or anything else).
Does that make sense? I guess my question is, if you simply open up a 1tick chart on NT, will the data be saved? I don't have a broker-connected NT at the moment, so I'm unable to test this.
There is a 4 Gig limit on the size of the SQL CE database engine that NT7 uses, so that would need to be taken into consideration ie you couldn't record tick data of the whole CME Group product list for very long on a single instance of NT ..
A side project I haven't looked into yet is sync between SQL CE and SQL, but I'm looking at that more for getting an overview of multiple NT users positions than for storing tick data.
I'm leaning towards the "buy" camp in the "buy vs. build" for historical tick data if 1 second timestamps are OK.
phyzfer if you need live futures data for development (not trading) try this:
Doh! 4GB? I don't understand. I was under the impression if you opened up a 1-tick chart in NT, that it will save the data into the \Documents\NinjaTrader 7\db\tick folder. I guess that isn't correct then. Right?
If not, then going off of your strategy code, I'd create a binary-writer to register to the contracts and write to disk (via a buffer + some compression algorithms). Then the fun part. We'd create an interface (separate C# app) that would find, decompress and return the data in whatever format your calling program (matlab, r, NT even) would need.
I'd love to help with something like this. I think keeping tick data in a relational database makes no sense whatsoever. Querying the data will take forever and the db size will grow significantly.
Where I get lost is if SQL CE is writing binaries to the file system and the DB is merely referencing those binaries... do the binaries themselves count towards the 4 Gig limit or are only the references (which should only be a few bytes) counted.
I'll be honest I'm waaaaaaaaay out of my depth on this one.
Personally I don't THINK that NT uses the (SQL CE) database for any market data storage. I think they use it mostly for internal data (account performance, ATM strategies, etc).
I was under the impression that NT saved all Market Data in the Documents\NinjaTrader7\db folder. If you take a look, you'll see a Tick, Minute, Day subfolders that contain binary files (.ntd file extension). How'd these folders get populated?
Personally I'd want NT to save all tick-data for a product to disk whenever any chart is open.
NinjaTrader Converts Real-Time Data into Historical Data NinjaTrader stores real-time incoming tick data to your local PC if you have a Chart or Market Analyzer (must have an indicator column added) window open. This data can then be used as historical data. For example, if you open a chart and let it run all day long, the data collected today, will be available as historical data when you open the same chart tomorrow.
I take this to mean that NT is saving tick data on the machine (in the location I specified above) and makes that data available to NT. We'd simply need a way to 'convert' it back and traverse the data. I'd also like to place some sort of backup strategy in place on the machine that is running this data collection.
A colleague of mine suggested I look into this for storing market data: HDF Group - HDF5
At one time I was talking with a programmer to do this type of project for me: he stopped answering my emails ... cant think why he would do that LOL@me
"" The TREE Data Server captures real-time financial data from one or several datafeed services, archives data in a historical database, and makes both live and archived data available to client applications. The system can be used in real-time charting, an ATS, tick feed simulator, etc. or any situation in which multiple clients need real-time access to the tick stream or archived tick data. In addition, the archived tick stream and tick data are available for offline data analysis and backtesting. ""