NTD File Specification - NinjaTrader Programming | futures.io
futures.io futures trading


NTD File Specification
Started: by mrjoe Views / Replies:7,846 / 17
Last Reply: Attachments:8

Welcome to futures.io.

Welcome, Guest!

This forum was established to help traders (especially futures traders) by openly sharing indicators, strategies, methods, trading journals and discussing the psychology of trading.

We are fundamentally different than most other trading forums:
  • We work extremely hard to keep things positive on our forums.
  • We do not tolerate rude behavior, trolling, or vendor advertising in posts.
  • We firmly believe in openness and encourage sharing. The holy grail is within you, it is not something tangible you can download.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.


You'll need to register in order to view the content of the threads and start contributing to our community. It's free and simple, and we will never resell your private information.

-- Big Mike

Reply
 8  
 
Thread Tools Search this Thread
 

NTD File Specification

  #1 (permalink)
Elite Member
Bratislava, Slovakia
 
Futures Experience: Intermediate
Platform: NinjaTrader
Broker/Data: AMP/CQG
Favorite Futures: NQ, 6E
 
mrjoe's Avatar
 
Posts: 21 since Oct 2010
Thanks: 30 given, 60 received

NTD File Specification

This is reverse engineered NTD file format specification. This file format is used to store NinjaTrader 7 (possibly v6 too) tick data.

I do not have any official file specification from Ninja guys so information contained
in this document may not be perfect. If you find any errors in this document please
post comments.


My primary language is not english so watch out for grammar errors .

--- START ---

LE - little endian
BE - big endian

Address : Description
0x00 : Price multiplier stored as IEEE754 double precision 64-bit. For some reason is stored as negative number.
0x0C : Record count [4B UINT LE] including first record.
0x10 : Price of the first record in the NTD file [8b IEEE754 LE].
0x18,0x20,0x28 : same data as 0x10 (don't know why)
0x30 : Serialized .NET DateTime structure of the first record [8b MS-TICK].
0x38 : Volume of the first record [8B UINT LE].
0x40 : Beginning of the body part (starts with the 2nd record).

Body structure
MASK [1B]
DELTA TIME [1B-3B UINT BE]
DELTA PRICE [1B,2B,4B UINT BE]
VOLUME [1B,2B,4B,8B UINT BE]

Size can vary depending on the mask settings. Order of the fields do not change.


MASK
Defines size of the remaining three parts of the record.
MASK bits are ordered as 01234567.

b: 0 - do not know
b: 1,2,3 - VOLUME size
001 - 1B
110 - 2B
111 - 4B
010 - 8B
b: 4,5 - DELTA PRICE
00 - 0B (price field left out)
01 - 1B
10 - 2B
11 - 4B
b: 6,7 - DELTA TIME
00 - 0B (time field left out)
01 - 1B
10 - 2B
11 - 3B


Working with DELTA values:
Delta values are added to previous record value.

Consider following time values:
20081208 001609
20081208 001619
20081208 001629
20081208 001639
20081208 001639

You can clearly see 10 second delta (excluding last line). Binary value at offset 0x30 is then
0x80eac54de126cb08 for first record. Other record will have following delta time values 0x0a, 0x0a, 0x0a.
Last record will have delta value excluded.

Same applies for price changes. But there is one trick involved in computing price value. Price can move
to higher or lower values. So we need positive and negative values for price delta. This is accomplished by
splitting value interval into two parts. First part is for the negative delta and second part is for the
positive delta.

Example:
First record price is 1557.25, for the second is 1557.50 and for the third is 1557.00
At offset 0x10 will be value 0x0000000000559840
record 2 delta is: 0x81 (1557.25 + ((+1) * 0.25) = 1557.50)
record 3 delta is: 0x78 (1557.50 + ((-2) * 0.25) = 1557.00)

List of delta "zero" values:
size : delta
1B : 0x80
2B : 0x4000
4B : 0x40000000

Attached Thumbnails
NTD File Specification-schema.gif  
Reply With Quote
The following 17 users say Thank You to mrjoe for this post:
 
  #2 (permalink)
Quick Summary
Quick Summary Post

Quick Summary is created and edited by users like you... Add FAQ's, Links and other Relevant Information by clicking the edit button in the lower right hand corner of this message.

 
  #3 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 45,656 since Jun 2009
Thanks: 29,044 given, 80,670 received


Thanks for this. Been a while since I used NT, remind me -- is this the replay data that contains L1 and L2 (bid/ask/DOM) when 'record historical data' is ticked? Or is it just L1 data?

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #4 (permalink)
Elite Member
Bratislava, Slovakia
 
Futures Experience: Intermediate
Platform: NinjaTrader
Broker/Data: AMP/CQG
Favorite Futures: NQ, 6E
 
mrjoe's Avatar
 
Posts: 21 since Oct 2010
Thanks: 30 given, 60 received

Unfortunately this is not market replay data. It is not possible to reconstruct the level 2 data from ntd files. You can get ask/bid/last data, but they are stored in 3 separate files which are not synchronized together.

Reply With Quote
The following user says Thank You to mrjoe for this post:
 
  #5 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 45,656 since Jun 2009
Thanks: 29,044 given, 80,670 received

Futures Edge on FIO

Which products do you trade?

 

mrjoe View Post
Unfortunately this is not market replay data. It is not possible to reconstruct the level 2 data from ntd files. You can get ask/bid/last data, but they are stored in 3 separate files which are not synchronized together.

Got it.

So maybe you will work on reverse engineering NTM? hehe

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #6 (permalink)
Elite Member
Bratislava, Slovakia
 
Futures Experience: Intermediate
Platform: NinjaTrader
Broker/Data: AMP/CQG
Favorite Futures: NQ, 6E
 
mrjoe's Avatar
 
Posts: 21 since Oct 2010
Thanks: 30 given, 60 received

NTM adn NT2 files are too complex and hard do visually "debug" in NinjaTrader because I cannot directly see data changes when I make a modification in the file.
I tried do decompile NinjaTrader bytecode but I think it is packed by Remotesoft .NET Obfuscator for which I do not have unpacker.


Last edited by mrjoe; December 29th, 2010 at 04:03 PM.
Reply With Quote
 
  #7 (permalink)
Elite Member
Paris
 
Futures Experience: None
Platform: NinjaTrader
 
Posts: 1,140 since Oct 2009
Thanks: 208 given, 3,924 received


mrjoe View Post
b: 1,2,3 - VOLUME size
001 - 1B
110 - 2B
111 - 4B
010 - 8B

I found another funky value : 011, in which case the next value is a 1B, but you need to multiply it by 100 to get the correct volume.
Clever way to store multiples of 100 with less bytes.

Reply With Quote
The following 4 users say Thank You to gomi for this post:
 
  #8 (permalink)
Trading Apprentice
New York
 
Futures Experience: None
Platform: NinjaTrader
Favorite Futures: 10016
 
Posts: 1 since May 2011
Thanks: 0 given, 1 received

Would you happen to have a code snippet (in any language) that shows how you are generating the .ntd files? That would be awesome

Reply With Quote
The following user says Thank You to trapezoid for this post:
 
  #9 (permalink)
Elite Member
Paris
 
Futures Experience: None
Platform: NinjaTrader
 
Posts: 1,140 since Oct 2009
Thanks: 208 given, 3,924 received

I only read them.
Code is here :
https://futures.io/elite-circle/6802-gomrecorder-2-a-37.html#post112912

Reply With Quote
The following 2 users say Thank You to gomi for this post:
 
  #10 (permalink)
Elite Member
Paris
 
Futures Experience: None
Platform: NinjaTrader
 
Posts: 1,140 since Oct 2009
Thanks: 208 given, 3,924 received



gomi View Post
I found another funky value : 011, in which case the next value is a 1B, but you need to multiply it by 100 to get the correct volume.
Clever way to store multiples of 100 with less bytes.

Found 2 other multipliers : 100 , multiplier 500 ; 101, multiplier 1000.

Reply With Quote
The following 4 users say Thank You to gomi for this post:

Reply



futures.io > > > > > NTD File Specification

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

2016 True Edge Awards: Voting is open until Jan 21!

Now
 

An Afternoon with FIO member Massive l

Elite only
 

Anthony Drager: Pulling the trigger with confidence

Elite only
 

Leo Murphy: The Art and Science of Technical Analysis

Elite only
 

FuturesTrader71: Ask Me Anything

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Video file cabinet kbit Off-Topic 21 October 3rd, 2011 04:52 AM
CS file greatsky NinjaTrader 6 December 23rd, 2010 11:27 AM
zip file downloads ollie NinjaTrader Programming 5 July 30th, 2010 10:11 AM


All times are GMT -4. The time now is 12:47 AM.

no new posts
Page generated 2017-01-18 in 0.13 seconds with 20 queries on phoenix via your IP 54.163.149.151