NexusFi: Find Your Edge


Home Menu

 





NTD File Specification


Discussion in NinjaTrader

Updated
      Top Posters
    1. looks_one gomi with 3 posts (10 thanks)
    2. looks_two mrjoe with 3 posts (18 thanks)
    3. looks_3 intermarkettrade with 3 posts (0 thanks)
    4. looks_4 Nicolas11 with 2 posts (22 thanks)
      Best Posters
    1. looks_one Nicolas11 with 11 thanks per post
    2. looks_two dalebru with 9 thanks per post
    3. looks_3 mrjoe with 6 thanks per post
    4. looks_4 gomi with 3.3 thanks per post
    1. trending_up 17,867 views
    2. thumb_up 77 thanks given
    3. group 16 followers
    1. forum 22 posts
    2. attach_file 8 attachments




 
Search this Thread

NTD File Specification

  #1 (permalink)
 
mrjoe's Avatar
 mrjoe 
Bratislava, Slovakia
 
Experience: Intermediate
Platform: NinjaTrader
Broker: AMP/CQG
Trading: NQ, 6E
Posts: 21 since Oct 2010
Thanks Given: 30
Thanks Received: 62

This is reverse engineered NTD file format specification. This file format is used to store NinjaTrader 7 (possibly v6 too) tick data.

I do not have any official file specification from Ninja guys so information contained
in this document may not be perfect. If you find any errors in this document please
post comments.


My primary language is not english so watch out for grammar errors .

--- START ---

LE - little endian
BE - big endian

Address : Description
0x00 : Price multiplier stored as IEEE754 double precision 64-bit. For some reason is stored as negative number.
0x0C : Record count [4B UINT LE] including first record.
0x10 : Price of the first record in the NTD file [8b IEEE754 LE].
0x18,0x20,0x28 : same data as 0x10 (don't know why)
0x30 : Serialized .NET DateTime structure of the first record [8b MS-TICK].
0x38 : Volume of the first record [8B UINT LE].
0x40 : Beginning of the body part (starts with the 2nd record).

Body structure
MASK [1B]
DELTA TIME [1B-3B UINT BE]
DELTA PRICE [1B,2B,4B UINT BE]
VOLUME [1B,2B,4B,8B UINT BE]

Size can vary depending on the mask settings. Order of the fields do not change.


MASK
Defines size of the remaining three parts of the record.
MASK bits are ordered as 01234567.

b: 0 - do not know
b: 1,2,3 - VOLUME size
001 - 1B
110 - 2B
111 - 4B
010 - 8B
b: 4,5 - DELTA PRICE
00 - 0B (price field left out)
01 - 1B
10 - 2B
11 - 4B
b: 6,7 - DELTA TIME
00 - 0B (time field left out)
01 - 1B
10 - 2B
11 - 3B


Working with DELTA values:
Delta values are added to previous record value.

Consider following time values:
20081208 001609
20081208 001619
20081208 001629
20081208 001639
20081208 001639

You can clearly see 10 second delta (excluding last line). Binary value at offset 0x30 is then
0x80eac54de126cb08 for first record. Other record will have following delta time values 0x0a, 0x0a, 0x0a.
Last record will have delta value excluded.

Same applies for price changes. But there is one trick involved in computing price value. Price can move
to higher or lower values. So we need positive and negative values for price delta. This is accomplished by
splitting value interval into two parts. First part is for the negative delta and second part is for the
positive delta.

Example:
First record price is 1557.25, for the second is 1557.50 and for the third is 1557.00
At offset 0x10 will be value 0x0000000000559840
record 2 delta is: 0x81 (1557.25 + ((+1) * 0.25) = 1557.50)
record 3 delta is: 0x78 (1557.50 + ((-2) * 0.25) = 1557.00)

List of delta "zero" values:
size : delta
1B : 0x80
2B : 0x4000
4B : 0x40000000

Attached Thumbnails
Click image for larger version

Name:	schema.GIF
Views:	1157
Size:	35.0 KB
ID:	27424  
Started this thread Reply With Quote

Can you help answer these questions
from other members on NexusFi?
Better Renko Gaps
The Elite Circle
Futures True Range Report
The Elite Circle
Build trailing stop for micro index(s)
Psychology and Money Management
Exit Strategy
NinjaTrader
The space time continuum and the dynamics of a financial …
Emini and Emicro Index
 
  #3 (permalink)
 
Big Mike's Avatar
 Big Mike 
Manta, Ecuador
Site Administrator
Developer
Swing Trader
 
Experience: Advanced
Platform: Custom solution
Broker: IBKR
Trading: Stocks & Futures
Frequency: Every few days
Duration: Weeks
Posts: 50,399 since Jun 2009
Thanks Given: 33,175
Thanks Received: 101,541


Thanks for this. Been a while since I used NT, remind me -- is this the replay data that contains L1 and L2 (bid/ask/DOM) when 'record historical data' is ticked? Or is it just L1 data?

Mike

We're here to help: just ask the community or contact our Help Desk

Quick Links: Change your Username or Register as a Vendor
Searching for trading reviews? Review this list
Lifetime Elite Membership: Sign-up for only $149 USD
Exclusive money saving offers from our Site Sponsors: Browse Offers
Report problems with the site: Using the NexusFi changelog thread
Follow me on Twitter Visit my NexusFi Trade Journal Reply With Quote
  #4 (permalink)
 
mrjoe's Avatar
 mrjoe 
Bratislava, Slovakia
 
Experience: Intermediate
Platform: NinjaTrader
Broker: AMP/CQG
Trading: NQ, 6E
Posts: 21 since Oct 2010
Thanks Given: 30
Thanks Received: 62

Unfortunately this is not market replay data. It is not possible to reconstruct the level 2 data from ntd files. You can get ask/bid/last data, but they are stored in 3 separate files which are not synchronized together.

Started this thread Reply With Quote
Thanked by:
  #5 (permalink)
 
Big Mike's Avatar
 Big Mike 
Manta, Ecuador
Site Administrator
Developer
Swing Trader
 
Experience: Advanced
Platform: Custom solution
Broker: IBKR
Trading: Stocks & Futures
Frequency: Every few days
Duration: Weeks
Posts: 50,399 since Jun 2009
Thanks Given: 33,175
Thanks Received: 101,541


mrjoe View Post
Unfortunately this is not market replay data. It is not possible to reconstruct the level 2 data from ntd files. You can get ask/bid/last data, but they are stored in 3 separate files which are not synchronized together.

Got it.

So maybe you will work on reverse engineering NTM? hehe

Mike

We're here to help: just ask the community or contact our Help Desk

Quick Links: Change your Username or Register as a Vendor
Searching for trading reviews? Review this list
Lifetime Elite Membership: Sign-up for only $149 USD
Exclusive money saving offers from our Site Sponsors: Browse Offers
Report problems with the site: Using the NexusFi changelog thread
Follow me on Twitter Visit my NexusFi Trade Journal Reply With Quote
  #6 (permalink)
 
mrjoe's Avatar
 mrjoe 
Bratislava, Slovakia
 
Experience: Intermediate
Platform: NinjaTrader
Broker: AMP/CQG
Trading: NQ, 6E
Posts: 21 since Oct 2010
Thanks Given: 30
Thanks Received: 62

NTM adn NT2 files are too complex and hard do visually "debug" in NinjaTrader because I cannot directly see data changes when I make a modification in the file.
I tried do decompile NinjaTrader bytecode but I think it is packed by Remotesoft .NET Obfuscator for which I do not have unpacker.

Started this thread Reply With Quote
  #7 (permalink)
 gomi 
Paris
Market Wizard
 
Experience: None
Platform: NinjaTrader
Posts: 1,270 since Oct 2009
Thanks Given: 282
Thanks Received: 4,505


mrjoe View Post
b: 1,2,3 - VOLUME size
001 - 1B
110 - 2B
111 - 4B
010 - 8B

I found another funky value : 011, in which case the next value is a 1B, but you need to multiply it by 100 to get the correct volume.
Clever way to store multiples of 100 with less bytes.

Reply With Quote
  #8 (permalink)
trapezoid
New York
 
Posts: 1 since May 2011
Thanks Given: 0
Thanks Received: 1

Would you happen to have a code snippet (in any language) that shows how you are generating the .ntd files? That would be awesome

Reply With Quote
Thanked by:
  #9 (permalink)
 gomi 
Paris
Market Wizard
 
Experience: None
Platform: NinjaTrader
Posts: 1,270 since Oct 2009
Thanks Given: 282
Thanks Received: 4,505

I only read them.
Code is here :

Reply With Quote
Thanked by:
  #10 (permalink)
 gomi 
Paris
Market Wizard
 
Experience: None
Platform: NinjaTrader
Posts: 1,270 since Oct 2009
Thanks Given: 282
Thanks Received: 4,505



gomi View Post
I found another funky value : 011, in which case the next value is a 1B, but you need to multiply it by 100 to get the correct volume.
Clever way to store multiples of 100 with less bytes.

Found 2 other multipliers : 100 , multiplier 500 ; 101, multiplier 1000.

Reply With Quote




Last Updated on December 23, 2021


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts