NTD File Specification - NinjaTrader Programming | futures io social day trading
futures io futures trading


NTD File Specification
Updated: Views / Replies:9,003 / 21
Created: by mrjoe Attachments:8

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 8  
 
Thread Tools Search this Thread
 

NTD File Specification

  #1 (permalink)
Elite Member
Bratislava, Slovakia
 
Futures Experience: Intermediate
Platform: NinjaTrader
Broker/Data: AMP/CQG
Favorite Futures: NQ, 6E
 
mrjoe's Avatar
 
Posts: 21 since Oct 2010
Thanks: 30 given, 60 received

NTD File Specification

This is reverse engineered NTD file format specification. This file format is used to store NinjaTrader 7 (possibly v6 too) tick data.

I do not have any official file specification from Ninja guys so information contained
in this document may not be perfect. If you find any errors in this document please
post comments.


My primary language is not english so watch out for grammar errors .

--- START ---

LE - little endian
BE - big endian

Address : Description
0x00 : Price multiplier stored as IEEE754 double precision 64-bit. For some reason is stored as negative number.
0x0C : Record count [4B UINT LE] including first record.
0x10 : Price of the first record in the NTD file [8b IEEE754 LE].
0x18,0x20,0x28 : same data as 0x10 (don't know why)
0x30 : Serialized .NET DateTime structure of the first record [8b MS-TICK].
0x38 : Volume of the first record [8B UINT LE].
0x40 : Beginning of the body part (starts with the 2nd record).

Body structure
MASK [1B]
DELTA TIME [1B-3B UINT BE]
DELTA PRICE [1B,2B,4B UINT BE]
VOLUME [1B,2B,4B,8B UINT BE]

Size can vary depending on the mask settings. Order of the fields do not change.


MASK
Defines size of the remaining three parts of the record.
MASK bits are ordered as 01234567.

b: 0 - do not know
b: 1,2,3 - VOLUME size
001 - 1B
110 - 2B
111 - 4B
010 - 8B
b: 4,5 - DELTA PRICE
00 - 0B (price field left out)
01 - 1B
10 - 2B
11 - 4B
b: 6,7 - DELTA TIME
00 - 0B (time field left out)
01 - 1B
10 - 2B
11 - 3B


Working with DELTA values:
Delta values are added to previous record value.

Consider following time values:
20081208 001609
20081208 001619
20081208 001629
20081208 001639
20081208 001639

You can clearly see 10 second delta (excluding last line). Binary value at offset 0x30 is then
0x80eac54de126cb08 for first record. Other record will have following delta time values 0x0a, 0x0a, 0x0a.
Last record will have delta value excluded.

Same applies for price changes. But there is one trick involved in computing price value. Price can move
to higher or lower values. So we need positive and negative values for price delta. This is accomplished by
splitting value interval into two parts. First part is for the negative delta and second part is for the
positive delta.

Example:
First record price is 1557.25, for the second is 1557.50 and for the third is 1557.00
At offset 0x10 will be value 0x0000000000559840
record 2 delta is: 0x81 (1557.25 + ((+1) * 0.25) = 1557.50)
record 3 delta is: 0x78 (1557.50 + ((-2) * 0.25) = 1557.00)

List of delta "zero" values:
size : delta
1B : 0x80
2B : 0x4000
4B : 0x40000000

Attached Thumbnails
NTD File Specification-schema.gif  
Reply With Quote
The following 17 users say Thank You to mrjoe for this post:
 
  #2 (permalink)
Quick Summary
Quick Summary Post

Quick Summary is created and edited by users like you... Add FAQ's, Links and other Relevant Information by clicking the edit button in the lower right hand corner of this message.

 
  #3 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,165 since Jun 2009
Thanks: 29,276 given, 82,773 received


Thanks for this. Been a while since I used NT, remind me -- is this the replay data that contains L1 and L2 (bid/ask/DOM) when 'record historical data' is ticked? Or is it just L1 data?

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #4 (permalink)
Elite Member
Bratislava, Slovakia
 
Futures Experience: Intermediate
Platform: NinjaTrader
Broker/Data: AMP/CQG
Favorite Futures: NQ, 6E
 
mrjoe's Avatar
 
Posts: 21 since Oct 2010
Thanks: 30 given, 60 received

Unfortunately this is not market replay data. It is not possible to reconstruct the level 2 data from ntd files. You can get ask/bid/last data, but they are stored in 3 separate files which are not synchronized together.

Reply With Quote
The following user says Thank You to mrjoe for this post:
 
  #5 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,165 since Jun 2009
Thanks: 29,276 given, 82,773 received


mrjoe View Post
Unfortunately this is not market replay data. It is not possible to reconstruct the level 2 data from ntd files. You can get ask/bid/last data, but they are stored in 3 separate files which are not synchronized together.

Got it.

So maybe you will work on reverse engineering NTM? hehe

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #6 (permalink)
Elite Member
Bratislava, Slovakia
 
Futures Experience: Intermediate
Platform: NinjaTrader
Broker/Data: AMP/CQG
Favorite Futures: NQ, 6E
 
mrjoe's Avatar
 
Posts: 21 since Oct 2010
Thanks: 30 given, 60 received

NTM adn NT2 files are too complex and hard do visually "debug" in NinjaTrader because I cannot directly see data changes when I make a modification in the file.
I tried do decompile NinjaTrader bytecode but I think it is packed by Remotesoft .NET Obfuscator for which I do not have unpacker.


Last edited by mrjoe; December 29th, 2010 at 03:03 PM.
Reply With Quote
 
  #7 (permalink)
Elite Member
Paris
 
Futures Experience: None
Platform: NinjaTrader
 
Posts: 1,168 since Oct 2009
Thanks: 226 given, 4,052 received


mrjoe View Post
b: 1,2,3 - VOLUME size
001 - 1B
110 - 2B
111 - 4B
010 - 8B

I found another funky value : 011, in which case the next value is a 1B, but you need to multiply it by 100 to get the correct volume.
Clever way to store multiples of 100 with less bytes.

Reply With Quote
The following 4 users say Thank You to gomi for this post:
 
  #8 (permalink)
Trading Apprentice
New York
 
Futures Experience: None
Platform: NinjaTrader
Favorite Futures: 10016
 
Posts: 1 since May 2011
Thanks: 0 given, 1 received

Would you happen to have a code snippet (in any language) that shows how you are generating the .ntd files? That would be awesome

Reply With Quote
The following user says Thank You to trapezoid for this post:
 
  #9 (permalink)
Elite Member
Paris
 
Futures Experience: None
Platform: NinjaTrader
 
Posts: 1,168 since Oct 2009
Thanks: 226 given, 4,052 received

I only read them.
Code is here :
https://futures.io/elite-circle/6802-gomrecorder-2-a-37.html#post112912

Reply With Quote
The following 2 users say Thank You to gomi for this post:
 
  #10 (permalink)
Elite Member
Paris
 
Futures Experience: None
Platform: NinjaTrader
 
Posts: 1,168 since Oct 2009
Thanks: 226 given, 4,052 received



gomi View Post
I found another funky value : 011, in which case the next value is a 1B, but you need to multiply it by 100 to get the correct volume.
Clever way to store multiples of 100 with less bytes.

Found 2 other multipliers : 100 , multiplier 500 ; 101, multiplier 1000.

Reply With Quote
The following 4 users say Thank You to gomi for this post:

Reply



futures io > > > > > NTD File Specification

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)
 

RandBots: AutoTrading Bots to diversify your trading

Oct 24
 

TopstepTrader: Profiting Without Risking Your Own Money

Oct 26

Building Open Source Indicators for NT8 w/Fat Tails @ Lizard Indicators

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Video file cabinet kbit Off-Topic 21 October 3rd, 2011 03:52 AM
CS file greatsky NinjaTrader 6 December 23rd, 2010 10:27 AM
zip file downloads ollie NinjaTrader Programming 5 July 30th, 2010 09:11 AM


All times are GMT -4. The time now is 11:21 PM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-10-19 in 0.13 seconds with 20 queries on phoenix via your IP 54.166.232.243