NTD File Specification - futures io
futures io



NTD File Specification


Discussion in NinjaTrader

Updated
      Top Posters
    1. looks_one mrjoe with 3 posts (18 thanks)
    2. looks_two gomi with 3 posts (10 thanks)
    3. looks_3 intermarkettrade with 3 posts (0 thanks)
    4. looks_4 Big Mike with 2 posts (0 thanks)
      Best Posters
    1. looks_one Nicolas11 with 11 thanks per post
    2. looks_two dalebru with 9 thanks per post
    3. looks_3 mrjoe with 6 thanks per post
    4. looks_4 gomi with 3.3 thanks per post
    1. trending_up 13,559 views
    2. thumb_up 76 thanks given
    3. group 15 followers
    1. forum 21 posts
    2. attach_file 8 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 125,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

NTD File Specification

(login for full post details)
  #1 (permalink)
 mrjoe 
Bratislava, Slovakia
 
Experience: Intermediate
Platform: NinjaTrader
Broker: AMP/CQG
Trading: NQ, 6E
 
mrjoe's Avatar
 
Posts: 21 since Oct 2010
Thanks: 30 given, 61 received

This is reverse engineered NTD file format specification. This file format is used to store NinjaTrader 7 (possibly v6 too) tick data.

I do not have any official file specification from Ninja guys so information contained
in this document may not be perfect. If you find any errors in this document please
post comments.


My primary language is not english so watch out for grammar errors .

--- START ---

LE - little endian
BE - big endian

Address : Description
0x00 : Price multiplier stored as IEEE754 double precision 64-bit. For some reason is stored as negative number.
0x0C : Record count [4B UINT LE] including first record.
0x10 : Price of the first record in the NTD file [8b IEEE754 LE].
0x18,0x20,0x28 : same data as 0x10 (don't know why)
0x30 : Serialized .NET DateTime structure of the first record [8b MS-TICK].
0x38 : Volume of the first record [8B UINT LE].
0x40 : Beginning of the body part (starts with the 2nd record).

Body structure
MASK [1B]
DELTA TIME [1B-3B UINT BE]
DELTA PRICE [1B,2B,4B UINT BE]
VOLUME [1B,2B,4B,8B UINT BE]

Size can vary depending on the mask settings. Order of the fields do not change.


MASK
Defines size of the remaining three parts of the record.
MASK bits are ordered as 01234567.

b: 0 - do not know
b: 1,2,3 - VOLUME size
001 - 1B
110 - 2B
111 - 4B
010 - 8B
b: 4,5 - DELTA PRICE
00 - 0B (price field left out)
01 - 1B
10 - 2B
11 - 4B
b: 6,7 - DELTA TIME
00 - 0B (time field left out)
01 - 1B
10 - 2B
11 - 3B


Working with DELTA values:
Delta values are added to previous record value.

Consider following time values:
20081208 001609
20081208 001619
20081208 001629
20081208 001639
20081208 001639

You can clearly see 10 second delta (excluding last line). Binary value at offset 0x30 is then
0x80eac54de126cb08 for first record. Other record will have following delta time values 0x0a, 0x0a, 0x0a.
Last record will have delta value excluded.

Same applies for price changes. But there is one trick involved in computing price value. Price can move
to higher or lower values. So we need positive and negative values for price delta. This is accomplished by
splitting value interval into two parts. First part is for the negative delta and second part is for the
positive delta.

Example:
First record price is 1557.25, for the second is 1557.50 and for the third is 1557.00
At offset 0x10 will be value 0x0000000000559840
record 2 delta is: 0x81 (1557.25 + ((+1) * 0.25) = 1557.50)
record 3 delta is: 0x78 (1557.50 + ((-2) * 0.25) = 1557.00)

List of delta "zero" values:
size : delta
1B : 0x80
2B : 0x4000
4B : 0x40000000

Attached Thumbnails
Click image for larger version

Name:	schema.GIF
Views:	881
Size:	35.0 KB
ID:	27424  
Started this thread Reply With Quote
The following 17 users say Thank You to mrjoe for this post:

Journal Challenge April 2021 results:
Competing for $1800 in prizes from Jigsaw
looks_oneMaking a Living with the Microsby sstheo
(616 thanks from 61 posts)
looks_twoSalao's Journalby Salao
(160 thanks from 29 posts)
looks_3Learning to Profit - A journey in algorithms and optionsby Syntax
(115 thanks from 27 posts)
looks_4Deetee’s DAX Trading Journal (time based)by Deetee
(94 thanks from 30 posts)
looks_5Maybe a little bit different journalby Malykubo
(53 thanks from 32 posts)
 
Best Threads (Most Thanked)
in the last 7 days on futures io
I finally blew up an account
404 thanks
The Crude Dude Oil Trading System
81 thanks
Spoo-nalysis ES e-mini futures S&P 500
64 thanks
The tiyfTradePlanFactory indicator
25 thanks
Building a Crypto Mining Rig
19 thanks
 
(login for full post details)
  #3 (permalink)
 Big Mike 
Site Administrator
Swing Trader
Data Scientist & DevOps
Manta, Ecuador
 
Experience: Advanced
Platform: Custom solution
Trading: Futures & Crypto
 
Big Mike's Avatar
 
Posts: 50,068 since Jun 2009
Thanks: 32,534 given, 98,492 received


Thanks for this. Been a while since I used NT, remind me -- is this the replay data that contains L1 and L2 (bid/ask/DOM) when 'record historical data' is ticked? Or is it just L1 data?

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/

Visit other sites? Please spread the word about your experience with our community!
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #4 (permalink)
 mrjoe 
Bratislava, Slovakia
 
Experience: Intermediate
Platform: NinjaTrader
Broker: AMP/CQG
Trading: NQ, 6E
 
mrjoe's Avatar
 
Posts: 21 since Oct 2010
Thanks: 30 given, 61 received

Unfortunately this is not market replay data. It is not possible to reconstruct the level 2 data from ntd files. You can get ask/bid/last data, but they are stored in 3 separate files which are not synchronized together.

Started this thread Reply With Quote
The following user says Thank You to mrjoe for this post:
 
(login for full post details)
  #5 (permalink)
 Big Mike 
Site Administrator
Swing Trader
Data Scientist & DevOps
Manta, Ecuador
 
Experience: Advanced
Platform: Custom solution
Trading: Futures & Crypto
 
Big Mike's Avatar
 
Posts: 50,068 since Jun 2009
Thanks: 32,534 given, 98,492 received


mrjoe View Post
Unfortunately this is not market replay data. It is not possible to reconstruct the level 2 data from ntd files. You can get ask/bid/last data, but they are stored in 3 separate files which are not synchronized together.

Got it.

So maybe you will work on reverse engineering NTM? hehe

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/

Visit other sites? Please spread the word about your experience with our community!
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #6 (permalink)
 mrjoe 
Bratislava, Slovakia
 
Experience: Intermediate
Platform: NinjaTrader
Broker: AMP/CQG
Trading: NQ, 6E
 
mrjoe's Avatar
 
Posts: 21 since Oct 2010
Thanks: 30 given, 61 received

NTM adn NT2 files are too complex and hard do visually "debug" in NinjaTrader because I cannot directly see data changes when I make a modification in the file.
I tried do decompile NinjaTrader bytecode but I think it is packed by Remotesoft .NET Obfuscator for which I do not have unpacker.

Started this thread Reply With Quote
 
(login for full post details)
  #7 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received


mrjoe View Post
b: 1,2,3 - VOLUME size
001 - 1B
110 - 2B
111 - 4B
010 - 8B

I found another funky value : 011, in which case the next value is a 1B, but you need to multiply it by 100 to get the correct volume.
Clever way to store multiples of 100 with less bytes.

Reply With Quote
The following 4 users say Thank You to gomi for this post:
 
(login for full post details)
  #8 (permalink)
trapezoid
New York
 
 
Posts: 1 since May 2011
Thanks: 0 given, 1 received

Would you happen to have a code snippet (in any language) that shows how you are generating the .ntd files? That would be awesome

Reply With Quote
The following user says Thank You to trapezoid for this post:
 
(login for full post details)
  #9 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

I only read them.
Code is here :

Reply With Quote
The following 2 users say Thank You to gomi for this post:
 
(login for full post details)
  #10 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received



gomi View Post
I found another funky value : 011, in which case the next value is a 1B, but you need to multiply it by 100 to get the correct volume.
Clever way to store multiples of 100 with less bytes.

Found 2 other multipliers : 100 , multiplier 500 ; 101, multiplier 1000.

Reply With Quote
The following 4 users say Thank You to gomi for this post:
 
(login for full post details)
  #11 (permalink)
dom993
OTTAWA ON/CANADA
 
 
Posts: 11 since May 2012
Thanks: 0 given, 12 received

Would you guys be able to generate NTM files from historical tick data (say, in the Ninja .txt format) ? I do not need L1 info, this is just to do tick-by-tick replay for real-time testing of trading strategies (CalculateBarOnClose=false)

Thanks in advance
D.

Reply With Quote
 
(login for full post details)
  #12 (permalink)
 dalebru 
Indianapolis/IN
 
Experience: Intermediate
Platform: SC, NT, ToS, my own
Broker: Stage5, InteractiveBrokers
Trading: ES
 
Posts: 67 since Jan 2013
Thanks: 1,030 given, 110 received

Thanks to MrJoe for the Tick file specification. Here is the additional information needed to read day or minute files, along with some code for reading them.

First, note that under "Address" MrJoe has:
0x10 : Price of the first record in the NTD file [8b IEEE754 LE].
0x18,0x20,0x28 : same data as 0x10 (don't know why)
For day and minute bars
the Open is at 0x10,
the High is at 0x18
the Low is at 0x20
the Close at 0x28

For subsequent records after the header:
The Open is calculated the same as for tick files, from the first 1-byte mask
But for Day and Minute files, there is a second mask byte (mask2) for reading DeltaPrices for High, Low, Close
Bits 01234567
0,1 bitmask for bytes to read for Low relative to Open (must be 0 or negative)
2,3 bitmask for bytes to read for High relative to Open (must be 0 or positive)
4,5 not used
6,7 bitmask for bytes to read for Close relative to Low (must be 0 or positive)
For each pair of bits:
00 no data
01 1B BE (Big Endian)
10 2B BE
11 4B BE

Rules, starting with Open
High is equal to Open or higher (data bits 2,3)
Low is equal to Open or lower (data bits 0,1)
Close is equal to Low or higher (data bits 6,7)

Delta Prices are in the following order: High, Low, Close
1. The first DeltaPrice is for the High, a positive number of ticks ABOVE the Open from the header record
2. The second DeltaPrice is for the Low, a positive number representing the number of ticks BELOW the Open
3. The third DeltaPrice is for the Close, a positive number representing the number of ticks ABOVE the CLOSE.

To use the attached file BruReadNtd.cs, make a test solution and make a test calls. Program.cs is also attached as an example for calling it.

Attached Files
Register to download File Type: cs BruReadNtd.cs (27.7 KB, 141 views)
Register to download File Type: cs Program.cs (1.2 KB, 124 views)
Reply With Quote
The following 18 users say Thank You to dalebru for this post:
 
(login for full post details)
  #13 (permalink)
 Nicolas11 
near Paris, France
 
Experience: Beginner
Platform: -
Trading: -
 
Nicolas11's Avatar
 
Posts: 1,071 since Aug 2011
Thanks: 2,232 given, 1,755 received

For some purpose, I needed to be able to read .ntd daily files.
The above great work of mrjoe, gomi and dalebru allowed me understanding the structure of these files
In the enclosed document, I have illustrated their findings with AAPL quotes for 2014.
Thanks again to them,

Nicolas

Attached Thumbnails
NTD File Specification-2014-05-17-ntd-files-structure-illustration-content.pdf  
Visit my futures io Trade Journal Reply With Quote
The following 10 users say Thank You to Nicolas11 for this post:
 
(login for full post details)
  #14 (permalink)
 Nicolas11 
near Paris, France
 
Experience: Beginner
Platform: -
Trading: -
 
Nicolas11's Avatar
 
Posts: 1,071 since Aug 2011
Thanks: 2,232 given, 1,755 received

The below R code implements the above specification and allows reading .ntd daily files.

It provides a function, called ntdFileToXts, which accepts two arguments:
a) directory containing the daily .ntd files for a given security
b) a flag "VERBOSE" (true = all comments written on the console)
It returns an xts element containing the series of quotes for the given security.
I do not pretend that the code is fully optimized. But I hope that it is clear.

Let's take the example of AAPL daily quotes provided by (free) Kinetic End of Day.

On my computer, related .ntd files (containing data from 1990 to 2014) are stored on:
C:\Users\Nicolas\Documents\NinjaTrader 7\db\day\AAPL\ directory.

The 3 following lines use the above-mentioned function to read those files, and generate a xts file:
 
Code
directory <- "C:\\Users\\Nicolas\\Documents\\NinjaTrader 7\\db\\day\\AAPL\\"
ts.daily <- ntdFileToXts(directory, FALSE)
chartSeries(ts.daily, theme="white")
We obtain the following chart:


Let's focus on 2014 :
 
Code
chartSeries(ts.daily["2014"], theme="white")
Chart:


It really looks the same as NT chart:


In order to be sure that there was no mistake in the reading of the data by the R code, I have compared the xts element to the file produced by NT's export function: 100% identical.

I have tested it also with MSFT, ^SP500, CL ##-## and ES ##-##

Nicolas

 
Code
library(xts)
library(BMS)
library(quantmod)

ntdFileToXts <- function(directory, VERBOSE) {
  # This function reads all .ntd files in a given directory, and returns the corresponding xts series
  
  cat("\nReading from .nft files within ",directory,":\n", sep="")
  
  # 1. Some auxiliary functions:
  # ----------------------------
  
  skip <- function(length) {
    readBin(file.con, raw(), n=length)
  }
  
  read.as.numeric.LE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    readBin(raw.vector, numeric(), n=1, endian="little")
  }
  
  read.as.numeric.BE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    readBin(raw.vector, numeric(), n=1, endian="big")
  }
  
  read.as.unsigned.integer.LE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    as.numeric(paste("0x", paste(rev(raw.vector), collapse=''), sep=""))
  }
  
  read.as.unsigned.integer.BE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    as.numeric(paste("0x", paste(raw.vector, collapse=''), sep=""))
  }
  
  read.as.binary.mask <- function() {
    raw.vector <- readBin(file.con, raw(), n=1)
    hex2bin(toString(raw.vector[1]))
  }
  
  # 2. Creation of the vectors which will temporarily store quotes,
  #    before the creation of the xts
  # ---------------------------------------------------------------
  
  all.date <- as.Date(vector("integer", 1))
  all.open <- vector("numeric", 1) 
  all.high <- vector("numeric", 1) 
  all.low <- vector("numeric", 1) 
  all.close <- vector("numeric", 1) 
  all.volume <- vector("integer", 1) 
  
  # 3. For each ntd file of the directory...
  # ---------------------------------------- 
  
  files <- list.files(path = directory, pattern = "*.ntd")
  files <- files[order(files)]
  index <- 0
  for (file.name0 in files) {
    
    full.file.name <- paste(directory, file.name0, sep="")
    cat("Reading", full.file.name, "...\n")
    file.con <- file(full.file.name, "rb")
    
    
    # 4. Reading of the first record
    # ------------------------------
    
    index = index + 1 
    
    if (VERBOSE) cat("\nRecord #1 (specific structure):\n")
    
    # tick size
    tick.size <--read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Tick size:", tick.size, "\n")
    
    # skip 4 bytes
    skip(4)
    
    # nb of records
    nb.of.records <- read.as.unsigned.integer.LE(4) 
    if (VERBOSE) cat("   Nb of records:", nb.of.records, "\n")
    
    # open
    open <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Open:", open, "\n")
    all.open[index] <- open
    
    # high
    high <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   High:", high, "\n")
    all.high[index] <- high
    
    #low
    low <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Low:", low, "\n")
    all.low[index] <- low
    
    # close
    close <- read.as.numeric.LE(8)  
    if (VERBOSE) cat("   Close:", close, "\n")
    all.close[index] <- close
    
    # date
    date <- as.Date(as.POSIXct(read.as.unsigned.integer.LE(8)/10000000, origin="1-01-01"))
    if (VERBOSE) cat("   Date:", format(date, format="%Y-%m-%d"), "\n")
    all.date[index] <- date
    
    # volume
    volume <- read.as.unsigned.integer.LE(8)
    if (VERBOSE) cat("   Volume:", format(volume, big.mark=","), "\n") 
    all.volume[index] <- volume
    
    # 5. Reading of record #2 and the subsequent ones
    # -----------------------------------------------
    
    for (record.number in 2:nb.of.records) { # 
      
      index <- index + 1
      
      if (VERBOSE) cat("\nRecord #", record.number, ":\n", sep="")
      
      # mask 1
      mask1 <- read.as.binary.mask()
      if (VERBOSE) cat("   Mask 1:", mask1, "\n")
      
      # volume mask within mask 1
      volume.mask <- mask1[2:4]
      if (all(volume.mask == c(0,0,1))) {
        volume.length <- 1
        volume.multiplier <- 1
      } else if (all(volume.mask == c(0,1,1))) {
        volume.length <- 1
        volume.multiplier <- 100
      } else if (all(volume.mask == c(1,0,0))) { 
        volume.length <- 1
        volume.multiplier <- 500    
      } else if (all(volume.mask == c(1,0,1))) { 
        volume.length <- 1
        volume.multiplier <- 1000    
      } else if (all(volume.mask == c(1,1,0))) {
        volume.length <- 2
        volume.multiplier <- 1
      } else if (all(volume.mask == c(1,1,1))) {
        volume.length <- 4
        volume.multiplier <- 1
      } else if (all(volume.mask == c(0,1,0))) {
        volume.length <- 8
        volume.multiplier <- 1
      } else {
        cat("ERROR: volume mask not recognized: ", volume.mask)
      }
      
      # open mask within mask 1
      open.mask <- mask1[5:6]
      if (all(open.mask == c(0,0))) {
        open.length <- 0
      } else if (all(open.mask == c(0,1))) {
        open.length <- 1
        open.floor <- 128
      } else if (all(open.mask == c(1,0))) {
        open.length <- 2
        open.floor <- 16384
      } else if (all(open.mask == c(1,1))) {
        open.length <- 4
        open.floor <- 1073741824
      } else {
        cat("ERROR: open mask not recognized: ", open.mask)
      }
      
      # delta time mask within mask 1
      delta.time.mask <- mask1[7:8]
      if (all(delta.time.mask == c(0,0))) {
        delta.time.length <- 0
      } else if (all(delta.time.mask == c(0,1))) {
        delta.time.length <- 1
      } else if (all(delta.time.mask == c(1,0))) {
        delta.time.length <- 2
      } else if (all(delta.time.mask == c(1,1))) {
        delta.time.length <- 4
      } else {
        cat("ERROR: delta time mask not recognized: ", delta.time.mask)
      }
      
      #mask 2
      mask2 <- read.as.binary.mask()
      if (VERBOSE) cat("   Mask 2:", mask1, "\n")
      
      # low mask within mask 2
      low.mask <- mask2[1:2]
      if (all(low.mask == c(0,0))) {
        low.length <- 0
      } else if (all(low.mask == c(0,1))) {
        low.length <- 1
      } else if (all(low.mask == c(1,0))) {
        low.length <- 2
      } else if (all(low.mask == c(1,1))) {
        low.length <- 4
      } else {
        cat("ERROR: low mask not recognized: ", low.mask)
      }
      
      # high mask within mask 2
      high.mask <- mask2[3:4]
      if (all(high.mask == c(0,0))) {
        high.length <- 0
      } else if (all(high.mask == c(0,1))) {
        high.length <- 1
      } else if (all(high.mask == c(1,0))) {
        high.length <- 2
      } else if (all(high.mask == c(1,1))) {
        high.length <- 4
      } else {
        cat("ERROR: high mask not recognized: ", high.mask)
      }
      
      # close mask within mask 2
      close.mask <- mask2[7:8]
      if (all(close.mask == c(0,0))) {
        close.length <- 0
      } else if (all(close.mask == c(0,1))) {
        close.length <- 1
      } else if (all(close.mask == c(1,0))) {
        close.length <- 2
      } else if (all(close.mask == c(1,1))) {
        close.length <- 4
      } else {
        cat("ERROR: close mask not recognized: ", close.mask)
      }
      
      # delta time and date
      delta.time <- ifelse(delta.time.length == 0, 1, read.as.unsigned.integer.BE(delta.time.length))
      if (VERBOSE) cat("   Delta time:", delta.time) 
      date <- date + delta.time;
      all.date[index] <- date
      if (VERBOSE) cat(" => Date:", format(date, format="%Y-%m-%d"), "\n")
      
      # open delta and open
      open.delta <- ifelse(open.length == 0, 0, read.as.unsigned.integer.BE(open.length)-open.floor)
      if (VERBOSE) cat("   Open delta:", open.delta)
      open <- open + open.delta * tick.size
      if (VERBOSE) cat(" => Open:", open, "\n")
      all.open[index] <- open
      
      # high delta and high
      high.delta <- ifelse(high.length == 0, 0, read.as.unsigned.integer.BE(high.length))
      if (VERBOSE) cat("   High delta:", high.delta)
      high <- open + high.delta * tick.size
      if (VERBOSE) cat(" => High:", high, "\n")
      all.high[index] <- high
      
      # low delta and low
      low.delta <- ifelse(low.length == 0, 0, read.as.unsigned.integer.BE(low.length))         
      if (VERBOSE) cat("   Low delta:", low.delta)
      low <- open - low.delta * tick.size
      if (VERBOSE) cat(" => Low:", low, "\n")
      all.low[index] <- low
      
      # close delta and close
      close.delta <- ifelse(close.length == 0, 0, read.as.unsigned.integer.BE(close.length))
      if (VERBOSE) cat("   Close delta:", close.delta)
      close <- low + close.delta * tick.size
      if (VERBOSE) cat(" => Close:", close, "\n")
      all.close[index] <- close
      
      # volume
      volume <- volume.multiplier * read.as.unsigned.integer.BE(volume.length)
      if (VERBOSE) cat("   Volume:", format(volume, big.mark=","), "\n") 
      all.volume[index] <- volume
    }
    
    close(file.con)
  }
  
  # 6. Creation of xts
  # ------------------
  
  quotes <- cbind(all.open, all.high, all.low, all.close, all.volume)
  colnames(quotes) <- c("open", "high", "low", "close", "volume")
  xts(x=quotes, order.by=all.date)
}

Visit my futures io Trade Journal Reply With Quote
The following 12 users say Thank You to Nicolas11 for this post:
 
(login for full post details)
  #15 (permalink)
 BMMA14 
Montreal,Quebec/Canada
 
Experience: Intermediate
Platform: NinjaTrader
Trading: Forex
 
Posts: 17 since Jun 2014
Thanks: 5 given, 15 received

very good code. Thanks for sharing

Reply With Quote
The following user says Thank You to BMMA14 for this post:
 
(login for full post details)
  #16 (permalink)
 BMMA14 
Montreal,Quebec/Canada
 
Experience: Intermediate
Platform: NinjaTrader
Trading: Forex
 
Posts: 17 since Jun 2014
Thanks: 5 given, 15 received


Nicolas11 View Post
For some purpose, I needed to be able to read .ntd daily files.
The above great work of mrjoe, gomi and dalebru allowed me understanding the structure of these files
In the enclosed document, I have illustrated their findings with AAPL quotes for 2014.
Thanks again to them,

Nicolas

Good and well organized document for the work done by those wonderful members. Keep up the good work. Thanks.

Reply With Quote
The following user says Thank You to BMMA14 for this post:
 
(login for full post details)
  #17 (permalink)
 bboyle1234 
Australia
 
Experience: Advanced
Platform: NinjaTrader
Trading: CL
 
Posts: 1 since Feb 2013
Thanks: 6 given, 1 received

Thanks for sharing everybody, here's what I created for my own use based on your ideas.

https://github.com/bboyle1234/NTDFileReader

Reply With Quote
The following 4 users say Thank You to bboyle1234 for this post:
 
(login for full post details)
  #18 (permalink)
testingnew
Delhi+India
 
 
Posts: 1 since Jul 2016
Thanks: 1 given, 1 received


Nicolas11 View Post
The below R code implements the above specification and allows reading .ntd daily files.

It provides a function, called ntdFileToXts, which accepts two arguments:
a) directory containing the daily .ntd files for a given security
b) a flag "VERBOSE" (true = all comments written on the console)
It returns an xts element containing the series of quotes for the given security.
I do not pretend that the code is fully optimized. But I hope that it is clear.

Let's take the example of AAPL daily quotes provided by (free) Kinetic End of Day.

On my computer, related .ntd files (containing data from 1990 to 2014) are stored on:
C:\Users\Nicolas\Documents\NinjaTrader 7\db\day\AAPL\ directory.

The 3 following lines use the above-mentioned function to read those files, and generate a xts file:
 
Code
directory <- "C:\\Users\\Nicolas\\Documents\\NinjaTrader 7\\db\\day\\AAPL\\"
ts.daily <- ntdFileToXts(directory, FALSE)
chartSeries(ts.daily, theme="white")
We obtain the following chart:


Let's focus on 2014 :
 
Code
chartSeries(ts.daily["2014"], theme="white")
Chart:


It really looks the same as NT chart:


In order to be sure that there was no mistake in the reading of the data by the R code, I have compared the xts element to the file produced by NT's export function: 100% identical.

I have tested it also with MSFT, ^SP500, CL ##-## and ES ##-##

Nicolas

 
Code
library(xts)
library(BMS)
library(quantmod)

ntdFileToXts <- function(directory, VERBOSE) {
  # This function reads all .ntd files in a given directory, and returns the corresponding xts series
  
  cat("\nReading from .nft files within ",directory,":\n", sep="")
  
  # 1. Some auxiliary functions:
  # ----------------------------
  
  skip <- function(length) {
    readBin(file.con, raw(), n=length)
  }
  
  read.as.numeric.LE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    readBin(raw.vector, numeric(), n=1, endian="little")
  }
  
  read.as.numeric.BE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    readBin(raw.vector, numeric(), n=1, endian="big")
  }
  
  read.as.unsigned.integer.LE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    as.numeric(paste("0x", paste(rev(raw.vector), collapse=''), sep=""))
  }
  
  read.as.unsigned.integer.BE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    as.numeric(paste("0x", paste(raw.vector, collapse=''), sep=""))
  }
  
  read.as.binary.mask <- function() {
    raw.vector <- readBin(file.con, raw(), n=1)
    hex2bin(toString(raw.vector[1]))
  }
  
  # 2. Creation of the vectors which will temporarily store quotes,
  #    before the creation of the xts
  # ---------------------------------------------------------------
  
  all.date <- as.Date(vector("integer", 1))
  all.open <- vector("numeric", 1) 
  all.high <- vector("numeric", 1) 
  all.low <- vector("numeric", 1) 
  all.close <- vector("numeric", 1) 
  all.volume <- vector("integer", 1) 
  
  # 3. For each ntd file of the directory...
  # ---------------------------------------- 
  
  files <- list.files(path = directory, pattern = "*.ntd")
  files <- files[order(files)]
  index <- 0
  for (file.name0 in files) {
    
    full.file.name <- paste(directory, file.name0, sep="")
    cat("Reading", full.file.name, "...\n")
    file.con <- file(full.file.name, "rb")
    
    
    # 4. Reading of the first record
    # ------------------------------
    
    index = index + 1 
    
    if (VERBOSE) cat("\nRecord #1 (specific structure):\n")
    
    # tick size
    tick.size <--read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Tick size:", tick.size, "\n")
    
    # skip 4 bytes
    skip(4)
    
    # nb of records
    nb.of.records <- read.as.unsigned.integer.LE(4) 
    if (VERBOSE) cat("   Nb of records:", nb.of.records, "\n")
    
    # open
    open <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Open:", open, "\n")
    all.open[index] <- open
    
    # high
    high <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   High:", high, "\n")
    all.high[index] <- high
    
    #low
    low <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Low:", low, "\n")
    all.low[index] <- low
    
    # close
    close <- read.as.numeric.LE(8)  
    if (VERBOSE) cat("   Close:", close, "\n")
    all.close[index] <- close
    
    # date
    date <- as.Date(as.POSIXct(read.as.unsigned.integer.LE(8)/10000000, origin="1-01-01"))
    if (VERBOSE) cat("   Date:", format(date, format="%Y-%m-%d"), "\n")
    all.date[index] <- date
    
    # volume
    volume <- read.as.unsigned.integer.LE(8)
    if (VERBOSE) cat("   Volume:", format(volume, big.mark=","), "\n") 
    all.volume[index] <- volume
    
    # 5. Reading of record #2 and the subsequent ones
    # -----------------------------------------------
    
    for (record.number in 2:nb.of.records) { # 
      
      index <- index + 1
      
      if (VERBOSE) cat("\nRecord #", record.number, ":\n", sep="")
      
      # mask 1
      mask1 <- read.as.binary.mask()
      if (VERBOSE) cat("   Mask 1:", mask1, "\n")
      
      # volume mask within mask 1
      volume.mask <- mask1[2:4]
      if (all(volume.mask == c(0,0,1))) {
        volume.length <- 1
        volume.multiplier <- 1
      } else if (all(volume.mask == c(0,1,1))) {
        volume.length <- 1
        volume.multiplier <- 100
      } else if (all(volume.mask == c(1,0,0))) { 
        volume.length <- 1
        volume.multiplier <- 500    
      } else if (all(volume.mask == c(1,0,1))) { 
        volume.length <- 1
        volume.multiplier <- 1000    
      } else if (all(volume.mask == c(1,1,0))) {
        volume.length <- 2
        volume.multiplier <- 1
      } else if (all(volume.mask == c(1,1,1))) {
        volume.length <- 4
        volume.multiplier <- 1
      } else if (all(volume.mask == c(0,1,0))) {
        volume.length <- 8
        volume.multiplier <- 1
      } else {
        cat("ERROR: volume mask not recognized: ", volume.mask)
      }
      
      # open mask within mask 1
      open.mask <- mask1[5:6]
      if (all(open.mask == c(0,0))) {
        open.length <- 0
      } else if (all(open.mask == c(0,1))) {
        open.length <- 1
        open.floor <- 128
      } else if (all(open.mask == c(1,0))) {
        open.length <- 2
        open.floor <- 16384
      } else if (all(open.mask == c(1,1))) {
        open.length <- 4
        open.floor <- 1073741824
      } else {
        cat("ERROR: open mask not recognized: ", open.mask)
      }
      
      # delta time mask within mask 1
      delta.time.mask <- mask1[7:8]
      if (all(delta.time.mask == c(0,0))) {
        delta.time.length <- 0
      } else if (all(delta.time.mask == c(0,1))) {
        delta.time.length <- 1
      } else if (all(delta.time.mask == c(1,0))) {
        delta.time.length <- 2
      } else if (all(delta.time.mask == c(1,1))) {
        delta.time.length <- 4
      } else {
        cat("ERROR: delta time mask not recognized: ", delta.time.mask)
      }
      
      #mask 2
      mask2 <- read.as.binary.mask()
      if (VERBOSE) cat("   Mask 2:", mask1, "\n")
      
      # low mask within mask 2
      low.mask <- mask2[1:2]
      if (all(low.mask == c(0,0))) {
        low.length <- 0
      } else if (all(low.mask == c(0,1))) {
        low.length <- 1
      } else if (all(low.mask == c(1,0))) {
        low.length <- 2
      } else if (all(low.mask == c(1,1))) {
        low.length <- 4
      } else {
        cat("ERROR: low mask not recognized: ", low.mask)
      }
      
      # high mask within mask 2
      high.mask <- mask2[3:4]
      if (all(high.mask == c(0,0))) {
        high.length <- 0
      } else if (all(high.mask == c(0,1))) {
        high.length <- 1
      } else if (all(high.mask == c(1,0))) {
        high.length <- 2
      } else if (all(high.mask == c(1,1))) {
        high.length <- 4
      } else {
        cat("ERROR: high mask not recognized: ", high.mask)
      }
      
      # close mask within mask 2
      close.mask <- mask2[7:8]
      if (all(close.mask == c(0,0))) {
        close.length <- 0
      } else if (all(close.mask == c(0,1))) {
        close.length <- 1
      } else if (all(close.mask == c(1,0))) {
        close.length <- 2
      } else if (all(close.mask == c(1,1))) {
        close.length <- 4
      } else {
        cat("ERROR: close mask not recognized: ", close.mask)
      }
      
      # delta time and date
      delta.time <- ifelse(delta.time.length == 0, 1, read.as.unsigned.integer.BE(delta.time.length))
      if (VERBOSE) cat("   Delta time:", delta.time) 
      date <- date + delta.time;
      all.date[index] <- date
      if (VERBOSE) cat(" => Date:", format(date, format="%Y-%m-%d"), "\n")
      
      # open delta and open
      open.delta <- ifelse(open.length == 0, 0, read.as.unsigned.integer.BE(open.length)-open.floor)
      if (VERBOSE) cat("   Open delta:", open.delta)
      open <- open + open.delta * tick.size
      if (VERBOSE) cat(" => Open:", open, "\n")
      all.open[index] <- open
      
      # high delta and high
      high.delta <- ifelse(high.length == 0, 0, read.as.unsigned.integer.BE(high.length))
      if (VERBOSE) cat("   High delta:", high.delta)
      high <- open + high.delta * tick.size
      if (VERBOSE) cat(" => High:", high, "\n")
      all.high[index] <- high
      
      # low delta and low
      low.delta <- ifelse(low.length == 0, 0, read.as.unsigned.integer.BE(low.length))         
      if (VERBOSE) cat("   Low delta:", low.delta)
      low <- open - low.delta * tick.size
      if (VERBOSE) cat(" => Low:", low, "\n")
      all.low[index] <- low
      
      # close delta and close
      close.delta <- ifelse(close.length == 0, 0, read.as.unsigned.integer.BE(close.length))
      if (VERBOSE) cat("   Close delta:", close.delta)
      close <- low + close.delta * tick.size
      if (VERBOSE) cat(" => Close:", close, "\n")
      all.close[index] <- close
      
      # volume
      volume <- volume.multiplier * read.as.unsigned.integer.BE(volume.length)
      if (VERBOSE) cat("   Volume:", format(volume, big.mark=","), "\n") 
      all.volume[index] <- volume
    }
    
    close(file.con)
  }
  
  # 6. Creation of xts
  # ------------------
  
  quotes <- cbind(all.open, all.high, all.low, all.close, all.volume)
  colnames(quotes) <- c("open", "high", "low", "close", "volume")
  xts(x=quotes, order.by=all.date)
}


Hi Nicolas,

I am getting this error.

 
Code
Reading from .nft files within C:\Users\Anil\Documents\NinjaTrader 7\db\minute\SBINEQ\:
Error in as.Date.numeric(vector("integer", 1)) : 
  'origin' must be supplied

Please pardon me as I am not quite familiar with R.

I have also attached one of the file in my NT db.

EDIT: IT WORKED. Apparently the xts package was not installed. :-P

Attached Files
Register to download File Type: zip 20160609.Last.zip (2.4 KB, 13 views)
Reply With Quote
The following user says Thank You to testingnew for this post:
 
(login for full post details)
  #19 (permalink)
intermarkettrade
rome,italy,italy
 
 
Posts: 3 since Aug 2010
Thanks: 1 given, 0 received


Nicolas11 View Post
The below R code implements the above specification and allows reading .ntd daily files.

It provides a function, called ntdFileToXts, which accepts two arguments:
a) directory containing the daily .ntd files for a given security
b) a flag "VERBOSE" (true = all comments written on the console)
It returns an xts element containing the series of quotes for the given security.
I do not pretend that the code is fully optimized. But I hope that it is clear.

Let's take the example of AAPL daily quotes provided by (free) Kinetic End of Day.

On my computer, related .ntd files (containing data from 1990 to 2014) are stored on:
C:\Users\Nicolas\Documents\NinjaTrader 7\db\day\AAPL\ directory.

The 3 following lines use the above-mentioned function to read those files, and generate a xts file:
 
Code
directory <- "C:\\Users\\Nicolas\\Documents\\NinjaTrader 7\\db\\day\\AAPL\\"
ts.daily <- ntdFileToXts(directory, FALSE)
chartSeries(ts.daily, theme="white")
We obtain the following chart:


Let's focus on 2014 :
 
Code
chartSeries(ts.daily["2014"], theme="white")
Chart:


It really looks the same as NT chart:


In order to be sure that there was no mistake in the reading of the data by the R code, I have compared the xts element to the file produced by NT's export function: 100% identical.

I have tested it also with MSFT, ^SP500, CL ##-## and ES ##-##

Nicolas

 
Code
library(xts)
library(BMS)
library(quantmod)

ntdFileToXts <- function(directory, VERBOSE) {
  # This function reads all .ntd files in a given directory, and returns the corresponding xts series
  
  cat("\nReading from .nft files within ",directory,":\n", sep="")
  
  # 1. Some auxiliary functions:
  # ----------------------------
  
  skip <- function(length) {
    readBin(file.con, raw(), n=length)
  }
  
  read.as.numeric.LE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    readBin(raw.vector, numeric(), n=1, endian="little")
  }
  
  read.as.numeric.BE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    readBin(raw.vector, numeric(), n=1, endian="big")
  }
  
  read.as.unsigned.integer.LE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    as.numeric(paste("0x", paste(rev(raw.vector), collapse=''), sep=""))
  }
  
  read.as.unsigned.integer.BE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    as.numeric(paste("0x", paste(raw.vector, collapse=''), sep=""))
  }
  
  read.as.binary.mask <- function() {
    raw.vector <- readBin(file.con, raw(), n=1)
    hex2bin(toString(raw.vector[1]))
  }
  
  # 2. Creation of the vectors which will temporarily store quotes,
  #    before the creation of the xts
  # ---------------------------------------------------------------
  
  all.date <- as.Date(vector("integer", 1))
  all.open <- vector("numeric", 1) 
  all.high <- vector("numeric", 1) 
  all.low <- vector("numeric", 1) 
  all.close <- vector("numeric", 1) 
  all.volume <- vector("integer", 1) 
  
  # 3. For each ntd file of the directory...
  # ---------------------------------------- 
  
  files <- list.files(path = directory, pattern = "*.ntd")
  files <- files[order(files)]
  index <- 0
  for (file.name0 in files) {
    
    full.file.name <- paste(directory, file.name0, sep="")
    cat("Reading", full.file.name, "...\n")
    file.con <- file(full.file.name, "rb")
    
    
    # 4. Reading of the first record
    # ------------------------------
    
    index = index + 1 
    
    if (VERBOSE) cat("\nRecord #1 (specific structure):\n")
    
    # tick size
    tick.size <--read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Tick size:", tick.size, "\n")
    
    # skip 4 bytes
    skip(4)
    
    # nb of records
    nb.of.records <- read.as.unsigned.integer.LE(4) 
    if (VERBOSE) cat("   Nb of records:", nb.of.records, "\n")
    
    # open
    open <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Open:", open, "\n")
    all.open[index] <- open
    
    # high
    high <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   High:", high, "\n")
    all.high[index] <- high
    
    #low
    low <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Low:", low, "\n")
    all.low[index] <- low
    
    # close
    close <- read.as.numeric.LE(8)  
    if (VERBOSE) cat("   Close:", close, "\n")
    all.close[index] <- close
    
    # date
    date <- as.Date(as.POSIXct(read.as.unsigned.integer.LE(8)/10000000, origin="1-01-01"))
    if (VERBOSE) cat("   Date:", format(date, format="%Y-%m-%d"), "\n")
    all.date[index] <- date
    
    # volume
    volume <- read.as.unsigned.integer.LE(8)
    if (VERBOSE) cat("   Volume:", format(volume, big.mark=","), "\n") 
    all.volume[index] <- volume
    
    # 5. Reading of record #2 and the subsequent ones
    # -----------------------------------------------
    
    for (record.number in 2:nb.of.records) { # 
      
      index <- index + 1
      
      if (VERBOSE) cat("\nRecord #", record.number, ":\n", sep="")
      
      # mask 1
      mask1 <- read.as.binary.mask()
      if (VERBOSE) cat("   Mask 1:", mask1, "\n")
      
      # volume mask within mask 1
      volume.mask <- mask1[2:4]
      if (all(volume.mask == c(0,0,1))) {
        volume.length <- 1
        volume.multiplier <- 1
      } else if (all(volume.mask == c(0,1,1))) {
        volume.length <- 1
        volume.multiplier <- 100
      } else if (all(volume.mask == c(1,0,0))) { 
        volume.length <- 1
        volume.multiplier <- 500    
      } else if (all(volume.mask == c(1,0,1))) { 
        volume.length <- 1
        volume.multiplier <- 1000    
      } else if (all(volume.mask == c(1,1,0))) {
        volume.length <- 2
        volume.multiplier <- 1
      } else if (all(volume.mask == c(1,1,1))) {
        volume.length <- 4
        volume.multiplier <- 1
      } else if (all(volume.mask == c(0,1,0))) {
        volume.length <- 8
        volume.multiplier <- 1
      } else {
        cat("ERROR: volume mask not recognized: ", volume.mask)
      }
      
      # open mask within mask 1
      open.mask <- mask1[5:6]
      if (all(open.mask == c(0,0))) {
        open.length <- 0
      } else if (all(open.mask == c(0,1))) {
        open.length <- 1
        open.floor <- 128
      } else if (all(open.mask == c(1,0))) {
        open.length <- 2
        open.floor <- 16384
      } else if (all(open.mask == c(1,1))) {
        open.length <- 4
        open.floor <- 1073741824
      } else {
        cat("ERROR: open mask not recognized: ", open.mask)
      }
      
      # delta time mask within mask 1
      delta.time.mask <- mask1[7:8]
      if (all(delta.time.mask == c(0,0))) {
        delta.time.length <- 0
      } else if (all(delta.time.mask == c(0,1))) {
        delta.time.length <- 1
      } else if (all(delta.time.mask == c(1,0))) {
        delta.time.length <- 2
      } else if (all(delta.time.mask == c(1,1))) {
        delta.time.length <- 4
      } else {
        cat("ERROR: delta time mask not recognized: ", delta.time.mask)
      }
      
      #mask 2
      mask2 <- read.as.binary.mask()
      if (VERBOSE) cat("   Mask 2:", mask1, "\n")
      
      # low mask within mask 2
      low.mask <- mask2[1:2]
      if (all(low.mask == c(0,0))) {
        low.length <- 0
      } else if (all(low.mask == c(0,1))) {
        low.length <- 1
      } else if (all(low.mask == c(1,0))) {
        low.length <- 2
      } else if (all(low.mask == c(1,1))) {
        low.length <- 4
      } else {
        cat("ERROR: low mask not recognized: ", low.mask)
      }
      
      # high mask within mask 2
      high.mask <- mask2[3:4]
      if (all(high.mask == c(0,0))) {
        high.length <- 0
      } else if (all(high.mask == c(0,1))) {
        high.length <- 1
      } else if (all(high.mask == c(1,0))) {
        high.length <- 2
      } else if (all(high.mask == c(1,1))) {
        high.length <- 4
      } else {
        cat("ERROR: high mask not recognized: ", high.mask)
      }
      
      # close mask within mask 2
      close.mask <- mask2[7:8]
      if (all(close.mask == c(0,0))) {
        close.length <- 0
      } else if (all(close.mask == c(0,1))) {
        close.length <- 1
      } else if (all(close.mask == c(1,0))) {
        close.length <- 2
      } else if (all(close.mask == c(1,1))) {
        close.length <- 4
      } else {
        cat("ERROR: close mask not recognized: ", close.mask)
      }
      
      # delta time and date
      delta.time <- ifelse(delta.time.length == 0, 1, read.as.unsigned.integer.BE(delta.time.length))
      if (VERBOSE) cat("   Delta time:", delta.time) 
      date <- date + delta.time;
      all.date[index] <- date
      if (VERBOSE) cat(" => Date:", format(date, format="%Y-%m-%d"), "\n")
      
      # open delta and open
      open.delta <- ifelse(open.length == 0, 0, read.as.unsigned.integer.BE(open.length)-open.floor)
      if (VERBOSE) cat("   Open delta:", open.delta)
      open <- open + open.delta * tick.size
      if (VERBOSE) cat(" => Open:", open, "\n")
      all.open[index] <- open
      
      # high delta and high
      high.delta <- ifelse(high.length == 0, 0, read.as.unsigned.integer.BE(high.length))
      if (VERBOSE) cat("   High delta:", high.delta)
      high <- open + high.delta * tick.size
      if (VERBOSE) cat(" => High:", high, "\n")
      all.high[index] <- high
      
      # low delta and low
      low.delta <- ifelse(low.length == 0, 0, read.as.unsigned.integer.BE(low.length))         
      if (VERBOSE) cat("   Low delta:", low.delta)
      low <- open - low.delta * tick.size
      if (VERBOSE) cat(" => Low:", low, "\n")
      all.low[index] <- low
      
      # close delta and close
      close.delta <- ifelse(close.length == 0, 0, read.as.unsigned.integer.BE(close.length))
      if (VERBOSE) cat("   Close delta:", close.delta)
      close <- low + close.delta * tick.size
      if (VERBOSE) cat(" => Close:", close, "\n")
      all.close[index] <- close
      
      # volume
      volume <- volume.multiplier * read.as.unsigned.integer.BE(volume.length)
      if (VERBOSE) cat("   Volume:", format(volume, big.mark=","), "\n") 
      all.volume[index] <- volume
    }
    
    close(file.con)
  }
  
  # 6. Creation of xts
  # ------------------
  
  quotes <- cbind(all.open, all.high, all.low, all.close, all.volume)
  colnames(quotes) <- c("open", "high", "low", "close", "volume")
  xts(x=quotes, order.by=all.date)
}


___________________________________________________

Hi , I have very appreciate and thank you for your superb R code to convert Ninja .ntd file to XTS .

But my ignorance is very "cool" :-) and becouse I need to convert tick Ninja .ntd file to XTS , appear many errors like:

volume mask not recognized: 0 0 0Error in read.as.unsigned.integer.BE(volume.length) converted from warning) NAs introduced by coercion


Any help...?

Many many thansk and hope in your answer

Kind regards by Giuseppe from Italy: stickyman:

Reply With Quote
 
(login for full post details)
  #20 (permalink)
intermarkettrade
rome,italy,italy
 
 
Posts: 3 since Aug 2010
Thanks: 1 given, 0 received


dalebru View Post
Thanks to MrJoe for the Tick file specification. Here is the additional information needed to read day or minute files, along with some code for reading them.

First, note that under "Address" MrJoe has:
0x10 : Price of the first record in the NTD file [8b IEEE754 LE].
0x18,0x20,0x28 : same data as 0x10 (don't know why)
For day and minute bars
the Open is at 0x10,
the High is at 0x18
the Low is at 0x20
the Close at 0x28

For subsequent records after the header:
The Open is calculated the same as for tick files, from the first 1-byte mask
But for Day and Minute files, there is a second mask byte (mask2) for reading DeltaPrices for High, Low, Close
Bits 01234567
0,1 bitmask for bytes to read for Low relative to Open (must be 0 or negative)
2,3 bitmask for bytes to read for High relative to Open (must be 0 or positive)
4,5 not used
6,7 bitmask for bytes to read for Close relative to Low (must be 0 or positive)
For each pair of bits:
00 no data
01 1B BE (Big Endian)
10 2B BE
11 4B BE

Rules, starting with Open
High is equal to Open or higher (data bits 2,3)
Low is equal to Open or lower (data bits 0,1)
Close is equal to Low or higher (data bits 6,7)

Delta Prices are in the following order: High, Low, Close
1. The first DeltaPrice is for the High, a positive number of ticks ABOVE the Open from the header record
2. The second DeltaPrice is for the Low, a positive number representing the number of ticks BELOW the Open
3. The third DeltaPrice is for the Close, a positive number representing the number of ticks ABOVE the CLOSE.

To use the attached file BruReadNtd.cs, make a test solution and make a test calls. Program.cs is also attached as an example for calling it.

_______________________________________________________

Hi dalebru, thanks fro your ninjascripts. I'm trying to convert .ntd tick file to json .

can you help me how to get tick data from your BruReadNtd script? The BruReadNtd.ReadAndCacheNtd return only a bool ...

exscuse my incompetence...

Thanks for your help

Reply With Quote
 
(login for full post details)
  #21 (permalink)
 dalebru 
Indianapolis/IN
 
Experience: Intermediate
Platform: SC, NT, ToS, my own
Broker: Stage5, InteractiveBrokers
Trading: ES
 
Posts: 67 since Jan 2013
Thanks: 1,030 given, 110 received


intermarkettrade View Post
_______________________________________________________

Hi dalebru, thanks fro your ninjascripts. I'm trying to convert .ntd tick file to json .

can you help me how to get tick data from your BruReadNtd script? The BruReadNtd.ReadAndCacheNtd return only a bool ...

exscuse my incompetence...

Thanks for your help

I'm not sure I understand. You might take a look at ReadFile(), which returns a list of bars:
private static List<BruBar> ReadFile(DataType dataType, string filename)
If you use DataType.Tick, each "bar" in the list will represent a tick.

Reply With Quote
 
(login for full post details)
  #22 (permalink)
intermarkettrade
rome,italy,italy
 
 
Posts: 3 since Aug 2010
Thanks: 1 given, 0 received


dalebru View Post
I'm not sure I understand. You might take a look at ReadFile(), which returns a list of bars:
private static List<BruBar> ReadFile(DataType dataType, string filename)
If you use DataType.Tick, each "bar" in the list will represent a tick.

___________________________________________________

Hi dalebru, GREAT help, I have use your indications..and grasp tick data from .ntd.

Many many thanks

Kind regards

Reply With Quote


futures io Trading Community Platforms and Indicators NinjaTrader > NTD File Specification


Last Updated on March 11, 2017


Upcoming Webinars and Events

NinjaTrader Indicator Challenge!

Ongoing

Journal Challenge w/$1,800 in prizes!

May 7

The Cold Hard Truth: Maybe I Am Not Good Enough w/Chris Gray @ Earn2Trade

Elite only
     



Copyright © 2021 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada), info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts