NTD File Specification - NinjaTrader Programming | futures io social day trading
futures io futures trading


NTD File Specification
Updated: Views / Replies:8,729 / 21
Created: by mrjoe Attachments:8

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 8  
 
Thread Tools Search this Thread
 

NTD File Specification

  #11 (permalink)
Trading Apprentice
OTTAWA ON/CANADA
 
Futures Experience: Intermediate
Platform: Ensign, NinjaTrader
Favorite Futures: CL
 
Posts: 11 since May 2012
Thanks: 0 given, 12 received

Would you guys be able to generate NTM files from historical tick data (say, in the Ninja .txt format) ? I do not need L1 info, this is just to do tick-by-tick replay for real-time testing of trading strategies (CalculateBarOnClose=false)

Thanks in advance
D.

Reply With Quote
 
  #12 (permalink)
DabbaDo
Indianapolis/IN
 
Futures Experience: Intermediate
Platform: SC, NT, ToS, my own
Broker/Data: Stage5, InteractiveBrokers
Favorite Futures: ES
 
Posts: 56 since Jan 2013
Thanks: 950 given, 91 received

Day and Minute File Specification

Thanks to MrJoe for the Tick file specification. Here is the additional information needed to read day or minute files, along with some code for reading them.

First, note that under "Address" MrJoe has:
0x10 : Price of the first record in the NTD file [8b IEEE754 LE].
0x18,0x20,0x28 : same data as 0x10 (don't know why)
For day and minute bars
the Open is at 0x10,
the High is at 0x18
the Low is at 0x20
the Close at 0x28

For subsequent records after the header:
The Open is calculated the same as for tick files, from the first 1-byte mask
But for Day and Minute files, there is a second mask byte (mask2) for reading DeltaPrices for High, Low, Close
Bits 01234567
0,1 bitmask for bytes to read for Low relative to Open (must be 0 or negative)
2,3 bitmask for bytes to read for High relative to Open (must be 0 or positive)
4,5 not used
6,7 bitmask for bytes to read for Close relative to Low (must be 0 or positive)
For each pair of bits:
00 no data
01 1B BE (Big Endian)
10 2B BE
11 4B BE

Rules, starting with Open
High is equal to Open or higher (data bits 2,3)
Low is equal to Open or lower (data bits 0,1)
Close is equal to Low or higher (data bits 6,7)

Delta Prices are in the following order: High, Low, Close
1. The first DeltaPrice is for the High, a positive number of ticks ABOVE the Open from the header record
2. The second DeltaPrice is for the Low, a positive number representing the number of ticks BELOW the Open
3. The third DeltaPrice is for the Close, a positive number representing the number of ticks ABOVE the CLOSE.

To use the attached file BruReadNtd.cs, make a test solution and make a test calls. Program.cs is also attached as an example for calling it.

Attached Files
Register to download File Type: cs BruReadNtd.cs (27.7 KB, 111 views)
Register to download File Type: cs Program.cs (1.2 KB, 99 views)

Last edited by dalebru; July 21st, 2013 at 09:20 AM.
Reply With Quote
The following 16 users say Thank You to dalebru for this post:
 
  #13 (permalink)
Elite Member
near Paris, France
 
Futures Experience: Beginner
Platform: -
Favorite Futures: -
 
Nicolas11's Avatar
 
Posts: 1,070 since Aug 2011
Thanks: 2,232 given, 1,727 received


For some purpose, I needed to be able to read .ntd daily files.
The above great work of mrjoe, gomi and dalebru allowed me understanding the structure of these files
In the enclosed document, I have illustrated their findings with AAPL quotes for 2014.
Thanks again to them,

Nicolas

Attached Thumbnails
NTD File Specification-2014-05-17-ntd-files-structure-illustration-content.pdf  
Reply With Quote
The following 9 users say Thank You to Nicolas11 for this post:
 
  #14 (permalink)
Elite Member
near Paris, France
 
Futures Experience: Beginner
Platform: -
Favorite Futures: -
 
Nicolas11's Avatar
 
Posts: 1,070 since Aug 2011
Thanks: 2,232 given, 1,727 received

The below R code implements the above specification and allows reading .ntd daily files.

It provides a function, called ntdFileToXts, which accepts two arguments:
a) directory containing the daily .ntd files for a given security
b) a flag "VERBOSE" (true = all comments written on the console)
It returns an xts element containing the series of quotes for the given security.
I do not pretend that the code is fully optimized. But I hope that it is clear.

Let's take the example of AAPL daily quotes provided by (free) Kinetic End of Day.

On my computer, related .ntd files (containing data from 1990 to 2014) are stored on:
C:\Users\Nicolas\Documents\NinjaTrader 7\db\day\AAPL\ directory.

The 3 following lines use the above-mentioned function to read those files, and generate a xts file:
 
Code
directory <- "C:\\Users\\Nicolas\\Documents\\NinjaTrader 7\\db\\day\\AAPL\\"
ts.daily <- ntdFileToXts(directory, FALSE)
chartSeries(ts.daily, theme="white")
We obtain the following chart:
Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


Let's focus on 2014 :
 
Code
chartSeries(ts.daily["2014"], theme="white")
Chart:
Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


It really looks the same as NT chart:
Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


In order to be sure that there was no mistake in the reading of the data by the R code, I have compared the xts element to the file produced by NT's export function: 100% identical.

I have tested it also with MSFT, ^SP500, CL ##-## and ES ##-##

Nicolas

 
Code
library(xts)
library(BMS)
library(quantmod)

ntdFileToXts <- function(directory, VERBOSE) {
  # This function reads all .ntd files in a given directory, and returns the corresponding xts series
  
  cat("\nReading from .nft files within ",directory,":\n", sep="")
  
  # 1. Some auxiliary functions:
  # ----------------------------
  
  skip <- function(length) {
    readBin(file.con, raw(), n=length)
  }
  
  read.as.numeric.LE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    readBin(raw.vector, numeric(), n=1, endian="little")
  }
  
  read.as.numeric.BE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    readBin(raw.vector, numeric(), n=1, endian="big")
  }
  
  read.as.unsigned.integer.LE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    as.numeric(paste("0x", paste(rev(raw.vector), collapse=''), sep=""))
  }
  
  read.as.unsigned.integer.BE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    as.numeric(paste("0x", paste(raw.vector, collapse=''), sep=""))
  }
  
  read.as.binary.mask <- function() {
    raw.vector <- readBin(file.con, raw(), n=1)
    hex2bin(toString(raw.vector[1]))
  }
  
  # 2. Creation of the vectors which will temporarily store quotes,
  #    before the creation of the xts
  # ---------------------------------------------------------------
  
  all.date <- as.Date(vector("integer", 1))
  all.open <- vector("numeric", 1) 
  all.high <- vector("numeric", 1) 
  all.low <- vector("numeric", 1) 
  all.close <- vector("numeric", 1) 
  all.volume <- vector("integer", 1) 
  
  # 3. For each ntd file of the directory...
  # ---------------------------------------- 
  
  files <- list.files(path = directory, pattern = "*.ntd")
  files <- files[order(files)]
  index <- 0
  for (file.name0 in files) {
    
    full.file.name <- paste(directory, file.name0, sep="")
    cat("Reading", full.file.name, "...\n")
    file.con <- file(full.file.name, "rb")
    
    
    # 4. Reading of the first record
    # ------------------------------
    
    index = index + 1 
    
    if (VERBOSE) cat("\nRecord #1 (specific structure):\n")
    
    # tick size
    tick.size <--read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Tick size:", tick.size, "\n")
    
    # skip 4 bytes
    skip(4)
    
    # nb of records
    nb.of.records <- read.as.unsigned.integer.LE(4) 
    if (VERBOSE) cat("   Nb of records:", nb.of.records, "\n")
    
    # open
    open <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Open:", open, "\n")
    all.open[index] <- open
    
    # high
    high <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   High:", high, "\n")
    all.high[index] <- high
    
    #low
    low <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Low:", low, "\n")
    all.low[index] <- low
    
    # close
    close <- read.as.numeric.LE(8)  
    if (VERBOSE) cat("   Close:", close, "\n")
    all.close[index] <- close
    
    # date
    date <- as.Date(as.POSIXct(read.as.unsigned.integer.LE(8)/10000000, origin="1-01-01"))
    if (VERBOSE) cat("   Date:", format(date, format="%Y-%m-%d"), "\n")
    all.date[index] <- date
    
    # volume
    volume <- read.as.unsigned.integer.LE(8)
    if (VERBOSE) cat("   Volume:", format(volume, big.mark=","), "\n") 
    all.volume[index] <- volume
    
    # 5. Reading of record #2 and the subsequent ones
    # -----------------------------------------------
    
    for (record.number in 2:nb.of.records) { # 
      
      index <- index + 1
      
      if (VERBOSE) cat("\nRecord #", record.number, ":\n", sep="")
      
      # mask 1
      mask1 <- read.as.binary.mask()
      if (VERBOSE) cat("   Mask 1:", mask1, "\n")
      
      # volume mask within mask 1
      volume.mask <- mask1[2:4]
      if (all(volume.mask == c(0,0,1))) {
        volume.length <- 1
        volume.multiplier <- 1
      } else if (all(volume.mask == c(0,1,1))) {
        volume.length <- 1
        volume.multiplier <- 100
      } else if (all(volume.mask == c(1,0,0))) { 
        volume.length <- 1
        volume.multiplier <- 500    
      } else if (all(volume.mask == c(1,0,1))) { 
        volume.length <- 1
        volume.multiplier <- 1000    
      } else if (all(volume.mask == c(1,1,0))) {
        volume.length <- 2
        volume.multiplier <- 1
      } else if (all(volume.mask == c(1,1,1))) {
        volume.length <- 4
        volume.multiplier <- 1
      } else if (all(volume.mask == c(0,1,0))) {
        volume.length <- 8
        volume.multiplier <- 1
      } else {
        cat("ERROR: volume mask not recognized: ", volume.mask)
      }
      
      # open mask within mask 1
      open.mask <- mask1[5:6]
      if (all(open.mask == c(0,0))) {
        open.length <- 0
      } else if (all(open.mask == c(0,1))) {
        open.length <- 1
        open.floor <- 128
      } else if (all(open.mask == c(1,0))) {
        open.length <- 2
        open.floor <- 16384
      } else if (all(open.mask == c(1,1))) {
        open.length <- 4
        open.floor <- 1073741824
      } else {
        cat("ERROR: open mask not recognized: ", open.mask)
      }
      
      # delta time mask within mask 1
      delta.time.mask <- mask1[7:8]
      if (all(delta.time.mask == c(0,0))) {
        delta.time.length <- 0
      } else if (all(delta.time.mask == c(0,1))) {
        delta.time.length <- 1
      } else if (all(delta.time.mask == c(1,0))) {
        delta.time.length <- 2
      } else if (all(delta.time.mask == c(1,1))) {
        delta.time.length <- 4
      } else {
        cat("ERROR: delta time mask not recognized: ", delta.time.mask)
      }
      
      #mask 2
      mask2 <- read.as.binary.mask()
      if (VERBOSE) cat("   Mask 2:", mask1, "\n")
      
      # low mask within mask 2
      low.mask <- mask2[1:2]
      if (all(low.mask == c(0,0))) {
        low.length <- 0
      } else if (all(low.mask == c(0,1))) {
        low.length <- 1
      } else if (all(low.mask == c(1,0))) {
        low.length <- 2
      } else if (all(low.mask == c(1,1))) {
        low.length <- 4
      } else {
        cat("ERROR: low mask not recognized: ", low.mask)
      }
      
      # high mask within mask 2
      high.mask <- mask2[3:4]
      if (all(high.mask == c(0,0))) {
        high.length <- 0
      } else if (all(high.mask == c(0,1))) {
        high.length <- 1
      } else if (all(high.mask == c(1,0))) {
        high.length <- 2
      } else if (all(high.mask == c(1,1))) {
        high.length <- 4
      } else {
        cat("ERROR: high mask not recognized: ", high.mask)
      }
      
      # close mask within mask 2
      close.mask <- mask2[7:8]
      if (all(close.mask == c(0,0))) {
        close.length <- 0
      } else if (all(close.mask == c(0,1))) {
        close.length <- 1
      } else if (all(close.mask == c(1,0))) {
        close.length <- 2
      } else if (all(close.mask == c(1,1))) {
        close.length <- 4
      } else {
        cat("ERROR: close mask not recognized: ", close.mask)
      }
      
      # delta time and date
      delta.time <- ifelse(delta.time.length == 0, 1, read.as.unsigned.integer.BE(delta.time.length))
      if (VERBOSE) cat("   Delta time:", delta.time) 
      date <- date + delta.time;
      all.date[index] <- date
      if (VERBOSE) cat(" => Date:", format(date, format="%Y-%m-%d"), "\n")
      
      # open delta and open
      open.delta <- ifelse(open.length == 0, 0, read.as.unsigned.integer.BE(open.length)-open.floor)
      if (VERBOSE) cat("   Open delta:", open.delta)
      open <- open + open.delta * tick.size
      if (VERBOSE) cat(" => Open:", open, "\n")
      all.open[index] <- open
      
      # high delta and high
      high.delta <- ifelse(high.length == 0, 0, read.as.unsigned.integer.BE(high.length))
      if (VERBOSE) cat("   High delta:", high.delta)
      high <- open + high.delta * tick.size
      if (VERBOSE) cat(" => High:", high, "\n")
      all.high[index] <- high
      
      # low delta and low
      low.delta <- ifelse(low.length == 0, 0, read.as.unsigned.integer.BE(low.length))         
      if (VERBOSE) cat("   Low delta:", low.delta)
      low <- open - low.delta * tick.size
      if (VERBOSE) cat(" => Low:", low, "\n")
      all.low[index] <- low
      
      # close delta and close
      close.delta <- ifelse(close.length == 0, 0, read.as.unsigned.integer.BE(close.length))
      if (VERBOSE) cat("   Close delta:", close.delta)
      close <- low + close.delta * tick.size
      if (VERBOSE) cat(" => Close:", close, "\n")
      all.close[index] <- close
      
      # volume
      volume <- volume.multiplier * read.as.unsigned.integer.BE(volume.length)
      if (VERBOSE) cat("   Volume:", format(volume, big.mark=","), "\n") 
      all.volume[index] <- volume
    }
    
    close(file.con)
  }
  
  # 6. Creation of xts
  # ------------------
  
  quotes <- cbind(all.open, all.high, all.low, all.close, all.volume)
  colnames(quotes) <- c("open", "high", "low", "close", "volume")
  xts(x=quotes, order.by=all.date)
}


Last edited by Nicolas11; May 18th, 2014 at 02:38 PM.
Reply With Quote
The following 11 users say Thank You to Nicolas11 for this post:
 
  #15 (permalink)
Elite Member
Montreal,Quebec/Canada
 
Futures Experience: Intermediate
Platform: NinjaTrader
Favorite Futures: Forex
 
Posts: 17 since Jun 2014
Thanks: 5 given, 15 received

Thanks

very good code. Thanks for sharing

Reply With Quote
The following user says Thank You to BMMA14 for this post:
 
  #16 (permalink)
Elite Member
Montreal,Quebec/Canada
 
Futures Experience: Intermediate
Platform: NinjaTrader
Favorite Futures: Forex
 
Posts: 17 since Jun 2014
Thanks: 5 given, 15 received


Nicolas11 View Post
For some purpose, I needed to be able to read .ntd daily files.
The above great work of mrjoe, gomi and dalebru allowed me understanding the structure of these files
In the enclosed document, I have illustrated their findings with AAPL quotes for 2014.
Thanks again to them,

Nicolas

Good and well organized document for the work done by those wonderful members. Keep up the good work. Thanks.

Reply With Quote
The following user says Thank You to BMMA14 for this post:
 
  #17 (permalink)
Elite Member
Australia
 
Futures Experience: Advanced
Platform: NinjaTrader
Favorite Futures: CL
 
Posts: 1 since Feb 2013
Thanks: 6 given, 1 received

Thanks for sharing everybody, here's what I created for my own use based on your ideas.

https://github.com/bboyle1234/NTDFileReader

Reply With Quote
The following 4 users say Thank You to bboyle1234 for this post:
 
  #18 (permalink)
Trading Apprentice
Delhi+India
 
Futures Experience: Beginner
Platform: Ninjatrader
Favorite Futures: Gold GC
 
Posts: 1 since Jul 2016
Thanks: 1 given, 1 received

Error


Nicolas11 View Post
The below R code implements the above specification and allows reading .ntd daily files.

It provides a function, called ntdFileToXts, which accepts two arguments:
a) directory containing the daily .ntd files for a given security
b) a flag "VERBOSE" (true = all comments written on the console)
It returns an xts element containing the series of quotes for the given security.
I do not pretend that the code is fully optimized. But I hope that it is clear.

Let's take the example of AAPL daily quotes provided by (free) Kinetic End of Day.

On my computer, related .ntd files (containing data from 1990 to 2014) are stored on:
C:\Users\Nicolas\Documents\NinjaTrader 7\db\day\AAPL\ directory.

The 3 following lines use the above-mentioned function to read those files, and generate a xts file:
 
Code
directory <- "C:\\Users\\Nicolas\\Documents\\NinjaTrader 7\\db\\day\\AAPL\\"
ts.daily <- ntdFileToXts(directory, FALSE)
chartSeries(ts.daily, theme="white")
We obtain the following chart:
Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


Let's focus on 2014 :
 
Code
chartSeries(ts.daily["2014"], theme="white")
Chart:
Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


It really looks the same as NT chart:
Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


In order to be sure that there was no mistake in the reading of the data by the R code, I have compared the xts element to the file produced by NT's export function: 100% identical.

I have tested it also with MSFT, ^SP500, CL ##-## and ES ##-##

Nicolas

 
Code
library(xts)
library(BMS)
library(quantmod)

ntdFileToXts <- function(directory, VERBOSE) {
  # This function reads all .ntd files in a given directory, and returns the corresponding xts series
  
  cat("\nReading from .nft files within ",directory,":\n", sep="")
  
  # 1. Some auxiliary functions:
  # ----------------------------
  
  skip <- function(length) {
    readBin(file.con, raw(), n=length)
  }
  
  read.as.numeric.LE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    readBin(raw.vector, numeric(), n=1, endian="little")
  }
  
  read.as.numeric.BE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    readBin(raw.vector, numeric(), n=1, endian="big")
  }
  
  read.as.unsigned.integer.LE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    as.numeric(paste("0x", paste(rev(raw.vector), collapse=''), sep=""))
  }
  
  read.as.unsigned.integer.BE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    as.numeric(paste("0x", paste(raw.vector, collapse=''), sep=""))
  }
  
  read.as.binary.mask <- function() {
    raw.vector <- readBin(file.con, raw(), n=1)
    hex2bin(toString(raw.vector[1]))
  }
  
  # 2. Creation of the vectors which will temporarily store quotes,
  #    before the creation of the xts
  # ---------------------------------------------------------------
  
  all.date <- as.Date(vector("integer", 1))
  all.open <- vector("numeric", 1) 
  all.high <- vector("numeric", 1) 
  all.low <- vector("numeric", 1) 
  all.close <- vector("numeric", 1) 
  all.volume <- vector("integer", 1) 
  
  # 3. For each ntd file of the directory...
  # ---------------------------------------- 
  
  files <- list.files(path = directory, pattern = "*.ntd")
  files <- files[order(files)]
  index <- 0
  for (file.name0 in files) {
    
    full.file.name <- paste(directory, file.name0, sep="")
    cat("Reading", full.file.name, "...\n")
    file.con <- file(full.file.name, "rb")
    
    
    # 4. Reading of the first record
    # ------------------------------
    
    index = index + 1 
    
    if (VERBOSE) cat("\nRecord #1 (specific structure):\n")
    
    # tick size
    tick.size <--read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Tick size:", tick.size, "\n")
    
    # skip 4 bytes
    skip(4)
    
    # nb of records
    nb.of.records <- read.as.unsigned.integer.LE(4) 
    if (VERBOSE) cat("   Nb of records:", nb.of.records, "\n")
    
    # open
    open <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Open:", open, "\n")
    all.open[index] <- open
    
    # high
    high <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   High:", high, "\n")
    all.high[index] <- high
    
    #low
    low <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Low:", low, "\n")
    all.low[index] <- low
    
    # close
    close <- read.as.numeric.LE(8)  
    if (VERBOSE) cat("   Close:", close, "\n")
    all.close[index] <- close
    
    # date
    date <- as.Date(as.POSIXct(read.as.unsigned.integer.LE(8)/10000000, origin="1-01-01"))
    if (VERBOSE) cat("   Date:", format(date, format="%Y-%m-%d"), "\n")
    all.date[index] <- date
    
    # volume
    volume <- read.as.unsigned.integer.LE(8)
    if (VERBOSE) cat("   Volume:", format(volume, big.mark=","), "\n") 
    all.volume[index] <- volume
    
    # 5. Reading of record #2 and the subsequent ones
    # -----------------------------------------------
    
    for (record.number in 2:nb.of.records) { # 
      
      index <- index + 1
      
      if (VERBOSE) cat("\nRecord #", record.number, ":\n", sep="")
      
      # mask 1
      mask1 <- read.as.binary.mask()
      if (VERBOSE) cat("   Mask 1:", mask1, "\n")
      
      # volume mask within mask 1
      volume.mask <- mask1[2:4]
      if (all(volume.mask == c(0,0,1))) {
        volume.length <- 1
        volume.multiplier <- 1
      } else if (all(volume.mask == c(0,1,1))) {
        volume.length <- 1
        volume.multiplier <- 100
      } else if (all(volume.mask == c(1,0,0))) { 
        volume.length <- 1
        volume.multiplier <- 500    
      } else if (all(volume.mask == c(1,0,1))) { 
        volume.length <- 1
        volume.multiplier <- 1000    
      } else if (all(volume.mask == c(1,1,0))) {
        volume.length <- 2
        volume.multiplier <- 1
      } else if (all(volume.mask == c(1,1,1))) {
        volume.length <- 4
        volume.multiplier <- 1
      } else if (all(volume.mask == c(0,1,0))) {
        volume.length <- 8
        volume.multiplier <- 1
      } else {
        cat("ERROR: volume mask not recognized: ", volume.mask)
      }
      
      # open mask within mask 1
      open.mask <- mask1[5:6]
      if (all(open.mask == c(0,0))) {
        open.length <- 0
      } else if (all(open.mask == c(0,1))) {
        open.length <- 1
        open.floor <- 128
      } else if (all(open.mask == c(1,0))) {
        open.length <- 2
        open.floor <- 16384
      } else if (all(open.mask == c(1,1))) {
        open.length <- 4
        open.floor <- 1073741824
      } else {
        cat("ERROR: open mask not recognized: ", open.mask)
      }
      
      # delta time mask within mask 1
      delta.time.mask <- mask1[7:8]
      if (all(delta.time.mask == c(0,0))) {
        delta.time.length <- 0
      } else if (all(delta.time.mask == c(0,1))) {
        delta.time.length <- 1
      } else if (all(delta.time.mask == c(1,0))) {
        delta.time.length <- 2
      } else if (all(delta.time.mask == c(1,1))) {
        delta.time.length <- 4
      } else {
        cat("ERROR: delta time mask not recognized: ", delta.time.mask)
      }
      
      #mask 2
      mask2 <- read.as.binary.mask()
      if (VERBOSE) cat("   Mask 2:", mask1, "\n")
      
      # low mask within mask 2
      low.mask <- mask2[1:2]
      if (all(low.mask == c(0,0))) {
        low.length <- 0
      } else if (all(low.mask == c(0,1))) {
        low.length <- 1
      } else if (all(low.mask == c(1,0))) {
        low.length <- 2
      } else if (all(low.mask == c(1,1))) {
        low.length <- 4
      } else {
        cat("ERROR: low mask not recognized: ", low.mask)
      }
      
      # high mask within mask 2
      high.mask <- mask2[3:4]
      if (all(high.mask == c(0,0))) {
        high.length <- 0
      } else if (all(high.mask == c(0,1))) {
        high.length <- 1
      } else if (all(high.mask == c(1,0))) {
        high.length <- 2
      } else if (all(high.mask == c(1,1))) {
        high.length <- 4
      } else {
        cat("ERROR: high mask not recognized: ", high.mask)
      }
      
      # close mask within mask 2
      close.mask <- mask2[7:8]
      if (all(close.mask == c(0,0))) {
        close.length <- 0
      } else if (all(close.mask == c(0,1))) {
        close.length <- 1
      } else if (all(close.mask == c(1,0))) {
        close.length <- 2
      } else if (all(close.mask == c(1,1))) {
        close.length <- 4
      } else {
        cat("ERROR: close mask not recognized: ", close.mask)
      }
      
      # delta time and date
      delta.time <- ifelse(delta.time.length == 0, 1, read.as.unsigned.integer.BE(delta.time.length))
      if (VERBOSE) cat("   Delta time:", delta.time) 
      date <- date + delta.time;
      all.date[index] <- date
      if (VERBOSE) cat(" => Date:", format(date, format="%Y-%m-%d"), "\n")
      
      # open delta and open
      open.delta <- ifelse(open.length == 0, 0, read.as.unsigned.integer.BE(open.length)-open.floor)
      if (VERBOSE) cat("   Open delta:", open.delta)
      open <- open + open.delta * tick.size
      if (VERBOSE) cat(" => Open:", open, "\n")
      all.open[index] <- open
      
      # high delta and high
      high.delta <- ifelse(high.length == 0, 0, read.as.unsigned.integer.BE(high.length))
      if (VERBOSE) cat("   High delta:", high.delta)
      high <- open + high.delta * tick.size
      if (VERBOSE) cat(" => High:", high, "\n")
      all.high[index] <- high
      
      # low delta and low
      low.delta <- ifelse(low.length == 0, 0, read.as.unsigned.integer.BE(low.length))         
      if (VERBOSE) cat("   Low delta:", low.delta)
      low <- open - low.delta * tick.size
      if (VERBOSE) cat(" => Low:", low, "\n")
      all.low[index] <- low
      
      # close delta and close
      close.delta <- ifelse(close.length == 0, 0, read.as.unsigned.integer.BE(close.length))
      if (VERBOSE) cat("   Close delta:", close.delta)
      close <- low + close.delta * tick.size
      if (VERBOSE) cat(" => Close:", close, "\n")
      all.close[index] <- close
      
      # volume
      volume <- volume.multiplier * read.as.unsigned.integer.BE(volume.length)
      if (VERBOSE) cat("   Volume:", format(volume, big.mark=","), "\n") 
      all.volume[index] <- volume
    }
    
    close(file.con)
  }
  
  # 6. Creation of xts
  # ------------------
  
  quotes <- cbind(all.open, all.high, all.low, all.close, all.volume)
  colnames(quotes) <- c("open", "high", "low", "close", "volume")
  xts(x=quotes, order.by=all.date)
}


Hi Nicolas,

I am getting this error.

 
Code
Reading from .nft files within C:\Users\Anil\Documents\NinjaTrader 7\db\minute\SBINEQ\:
Error in as.Date.numeric(vector("integer", 1)) : 
  'origin' must be supplied

Please pardon me as I am not quite familiar with R.

I have also attached one of the file in my NT db.

EDIT: IT WORKED. Apparently the xts package was not installed. :-P

Attached Files
Register to download File Type: zip 20160609.Last.zip (2.4 KB, 6 views)

Last edited by testingnew; August 9th, 2016 at 04:15 AM. Reason: It Worked
Reply With Quote
The following user says Thank You to testingnew for this post:
 
  #19 (permalink)
Trading Apprentice
rome,italy,italy
 
Futures Experience: Advanced
Platform: NinjaTrader
Favorite Futures: forex
 
Posts: 3 since Aug 2010
Thanks: 1 given, 0 received

help for convert tick.ntd to XTS


Nicolas11 View Post
The below R code implements the above specification and allows reading .ntd daily files.

It provides a function, called ntdFileToXts, which accepts two arguments:
a) directory containing the daily .ntd files for a given security
b) a flag "VERBOSE" (true = all comments written on the console)
It returns an xts element containing the series of quotes for the given security.
I do not pretend that the code is fully optimized. But I hope that it is clear.

Let's take the example of AAPL daily quotes provided by (free) Kinetic End of Day.

On my computer, related .ntd files (containing data from 1990 to 2014) are stored on:
C:\Users\Nicolas\Documents\NinjaTrader 7\db\day\AAPL\ directory.

The 3 following lines use the above-mentioned function to read those files, and generate a xts file:
 
Code
directory <- "C:\\Users\\Nicolas\\Documents\\NinjaTrader 7\\db\\day\\AAPL\\"
ts.daily <- ntdFileToXts(directory, FALSE)
chartSeries(ts.daily, theme="white")
We obtain the following chart:
Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


Let's focus on 2014 :
 
Code
chartSeries(ts.daily["2014"], theme="white")
Chart:
Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


It really looks the same as NT chart:
Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


In order to be sure that there was no mistake in the reading of the data by the R code, I have compared the xts element to the file produced by NT's export function: 100% identical.

I have tested it also with MSFT, ^SP500, CL ##-## and ES ##-##

Nicolas

 
Code
library(xts)
library(BMS)
library(quantmod)

ntdFileToXts <- function(directory, VERBOSE) {
  # This function reads all .ntd files in a given directory, and returns the corresponding xts series
  
  cat("\nReading from .nft files within ",directory,":\n", sep="")
  
  # 1. Some auxiliary functions:
  # ----------------------------
  
  skip <- function(length) {
    readBin(file.con, raw(), n=length)
  }
  
  read.as.numeric.LE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    readBin(raw.vector, numeric(), n=1, endian="little")
  }
  
  read.as.numeric.BE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    readBin(raw.vector, numeric(), n=1, endian="big")
  }
  
  read.as.unsigned.integer.LE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    as.numeric(paste("0x", paste(rev(raw.vector), collapse=''), sep=""))
  }
  
  read.as.unsigned.integer.BE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    as.numeric(paste("0x", paste(raw.vector, collapse=''), sep=""))
  }
  
  read.as.binary.mask <- function() {
    raw.vector <- readBin(file.con, raw(), n=1)
    hex2bin(toString(raw.vector[1]))
  }
  
  # 2. Creation of the vectors which will temporarily store quotes,
  #    before the creation of the xts
  # ---------------------------------------------------------------
  
  all.date <- as.Date(vector("integer", 1))
  all.open <- vector("numeric", 1) 
  all.high <- vector("numeric", 1) 
  all.low <- vector("numeric", 1) 
  all.close <- vector("numeric", 1) 
  all.volume <- vector("integer", 1) 
  
  # 3. For each ntd file of the directory...
  # ---------------------------------------- 
  
  files <- list.files(path = directory, pattern = "*.ntd")
  files <- files[order(files)]
  index <- 0
  for (file.name0 in files) {
    
    full.file.name <- paste(directory, file.name0, sep="")
    cat("Reading", full.file.name, "...\n")
    file.con <- file(full.file.name, "rb")
    
    
    # 4. Reading of the first record
    # ------------------------------
    
    index = index + 1 
    
    if (VERBOSE) cat("\nRecord #1 (specific structure):\n")
    
    # tick size
    tick.size <--read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Tick size:", tick.size, "\n")
    
    # skip 4 bytes
    skip(4)
    
    # nb of records
    nb.of.records <- read.as.unsigned.integer.LE(4) 
    if (VERBOSE) cat("   Nb of records:", nb.of.records, "\n")
    
    # open
    open <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Open:", open, "\n")
    all.open[index] <- open
    
    # high
    high <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   High:", high, "\n")
    all.high[index] <- high
    
    #low
    low <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Low:", low, "\n")
    all.low[index] <- low
    
    # close
    close <- read.as.numeric.LE(8)  
    if (VERBOSE) cat("   Close:", close, "\n")
    all.close[index] <- close
    
    # date
    date <- as.Date(as.POSIXct(read.as.unsigned.integer.LE(8)/10000000, origin="1-01-01"))
    if (VERBOSE) cat("   Date:", format(date, format="%Y-%m-%d"), "\n")
    all.date[index] <- date
    
    # volume
    volume <- read.as.unsigned.integer.LE(8)
    if (VERBOSE) cat("   Volume:", format(volume, big.mark=","), "\n") 
    all.volume[index] <- volume
    
    # 5. Reading of record #2 and the subsequent ones
    # -----------------------------------------------
    
    for (record.number in 2:nb.of.records) { # 
      
      index <- index + 1
      
      if (VERBOSE) cat("\nRecord #", record.number, ":\n", sep="")
      
      # mask 1
      mask1 <- read.as.binary.mask()
      if (VERBOSE) cat("   Mask 1:", mask1, "\n")
      
      # volume mask within mask 1
      volume.mask <- mask1[2:4]
      if (all(volume.mask == c(0,0,1))) {
        volume.length <- 1
        volume.multiplier <- 1
      } else if (all(volume.mask == c(0,1,1))) {
        volume.length <- 1
        volume.multiplier <- 100
      } else if (all(volume.mask == c(1,0,0))) { 
        volume.length <- 1
        volume.multiplier <- 500    
      } else if (all(volume.mask == c(1,0,1))) { 
        volume.length <- 1
        volume.multiplier <- 1000    
      } else if (all(volume.mask == c(1,1,0))) {
        volume.length <- 2
        volume.multiplier <- 1
      } else if (all(volume.mask == c(1,1,1))) {
        volume.length <- 4
        volume.multiplier <- 1
      } else if (all(volume.mask == c(0,1,0))) {
        volume.length <- 8
        volume.multiplier <- 1
      } else {
        cat("ERROR: volume mask not recognized: ", volume.mask)
      }
      
      # open mask within mask 1
      open.mask <- mask1[5:6]
      if (all(open.mask == c(0,0))) {
        open.length <- 0
      } else if (all(open.mask == c(0,1))) {
        open.length <- 1
        open.floor <- 128
      } else if (all(open.mask == c(1,0))) {
        open.length <- 2
        open.floor <- 16384
      } else if (all(open.mask == c(1,1))) {
        open.length <- 4
        open.floor <- 1073741824
      } else {
        cat("ERROR: open mask not recognized: ", open.mask)
      }
      
      # delta time mask within mask 1
      delta.time.mask <- mask1[7:8]
      if (all(delta.time.mask == c(0,0))) {
        delta.time.length <- 0
      } else if (all(delta.time.mask == c(0,1))) {
        delta.time.length <- 1
      } else if (all(delta.time.mask == c(1,0))) {
        delta.time.length <- 2
      } else if (all(delta.time.mask == c(1,1))) {
        delta.time.length <- 4
      } else {
        cat("ERROR: delta time mask not recognized: ", delta.time.mask)
      }
      
      #mask 2
      mask2 <- read.as.binary.mask()
      if (VERBOSE) cat("   Mask 2:", mask1, "\n")
      
      # low mask within mask 2
      low.mask <- mask2[1:2]
      if (all(low.mask == c(0,0))) {
        low.length <- 0
      } else if (all(low.mask == c(0,1))) {
        low.length <- 1
      } else if (all(low.mask == c(1,0))) {
        low.length <- 2
      } else if (all(low.mask == c(1,1))) {
        low.length <- 4
      } else {
        cat("ERROR: low mask not recognized: ", low.mask)
      }
      
      # high mask within mask 2
      high.mask <- mask2[3:4]
      if (all(high.mask == c(0,0))) {
        high.length <- 0
      } else if (all(high.mask == c(0,1))) {
        high.length <- 1
      } else if (all(high.mask == c(1,0))) {
        high.length <- 2
      } else if (all(high.mask == c(1,1))) {
        high.length <- 4
      } else {
        cat("ERROR: high mask not recognized: ", high.mask)
      }
      
      # close mask within mask 2
      close.mask <- mask2[7:8]
      if (all(close.mask == c(0,0))) {
        close.length <- 0
      } else if (all(close.mask == c(0,1))) {
        close.length <- 1
      } else if (all(close.mask == c(1,0))) {
        close.length <- 2
      } else if (all(close.mask == c(1,1))) {
        close.length <- 4
      } else {
        cat("ERROR: close mask not recognized: ", close.mask)
      }
      
      # delta time and date
      delta.time <- ifelse(delta.time.length == 0, 1, read.as.unsigned.integer.BE(delta.time.length))
      if (VERBOSE) cat("   Delta time:", delta.time) 
      date <- date + delta.time;
      all.date[index] <- date
      if (VERBOSE) cat(" => Date:", format(date, format="%Y-%m-%d"), "\n")
      
      # open delta and open
      open.delta <- ifelse(open.length == 0, 0, read.as.unsigned.integer.BE(open.length)-open.floor)
      if (VERBOSE) cat("   Open delta:", open.delta)
      open <- open + open.delta * tick.size
      if (VERBOSE) cat(" => Open:", open, "\n")
      all.open[index] <- open
      
      # high delta and high
      high.delta <- ifelse(high.length == 0, 0, read.as.unsigned.integer.BE(high.length))
      if (VERBOSE) cat("   High delta:", high.delta)
      high <- open + high.delta * tick.size
      if (VERBOSE) cat(" => High:", high, "\n")
      all.high[index] <- high
      
      # low delta and low
      low.delta <- ifelse(low.length == 0, 0, read.as.unsigned.integer.BE(low.length))         
      if (VERBOSE) cat("   Low delta:", low.delta)
      low <- open - low.delta * tick.size
      if (VERBOSE) cat(" => Low:", low, "\n")
      all.low[index] <- low
      
      # close delta and close
      close.delta <- ifelse(close.length == 0, 0, read.as.unsigned.integer.BE(close.length))
      if (VERBOSE) cat("   Close delta:", close.delta)
      close <- low + close.delta * tick.size
      if (VERBOSE) cat(" => Close:", close, "\n")
      all.close[index] <- close
      
      # volume
      volume <- volume.multiplier * read.as.unsigned.integer.BE(volume.length)
      if (VERBOSE) cat("   Volume:", format(volume, big.mark=","), "\n") 
      all.volume[index] <- volume
    }
    
    close(file.con)
  }
  
  # 6. Creation of xts
  # ------------------
  
  quotes <- cbind(all.open, all.high, all.low, all.close, all.volume)
  colnames(quotes) <- c("open", "high", "low", "close", "volume")
  xts(x=quotes, order.by=all.date)
}


___________________________________________________

Hi , I have very appreciate and thank you for your superb R code to convert Ninja .ntd file to XTS .

But my ignorance is very "cool" :-) and becouse I need to convert tick Ninja .ntd file to XTS , appear many errors like:

volume mask not recognized: 0 0 0Error in read.as.unsigned.integer.BE(volume.length) converted from warning) NAs introduced by coercion


Any help...?

Many many thansk and hope in your answer

Kind regards by Giuseppe from Italy: stickyman:

Reply With Quote
 
  #20 (permalink)
Trading Apprentice
rome,italy,italy
 
Futures Experience: Advanced
Platform: NinjaTrader
Favorite Futures: forex
 
Posts: 3 since Aug 2010
Thanks: 1 given, 0 received



dalebru View Post
Thanks to MrJoe for the Tick file specification. Here is the additional information needed to read day or minute files, along with some code for reading them.

First, note that under "Address" MrJoe has:
0x10 : Price of the first record in the NTD file [8b IEEE754 LE].
0x18,0x20,0x28 : same data as 0x10 (don't know why)
For day and minute bars
the Open is at 0x10,
the High is at 0x18
the Low is at 0x20
the Close at 0x28

For subsequent records after the header:
The Open is calculated the same as for tick files, from the first 1-byte mask
But for Day and Minute files, there is a second mask byte (mask2) for reading DeltaPrices for High, Low, Close
Bits 01234567
0,1 bitmask for bytes to read for Low relative to Open (must be 0 or negative)
2,3 bitmask for bytes to read for High relative to Open (must be 0 or positive)
4,5 not used
6,7 bitmask for bytes to read for Close relative to Low (must be 0 or positive)
For each pair of bits:
00 no data
01 1B BE (Big Endian)
10 2B BE
11 4B BE

Rules, starting with Open
High is equal to Open or higher (data bits 2,3)
Low is equal to Open or lower (data bits 0,1)
Close is equal to Low or higher (data bits 6,7)

Delta Prices are in the following order: High, Low, Close
1. The first DeltaPrice is for the High, a positive number of ticks ABOVE the Open from the header record
2. The second DeltaPrice is for the Low, a positive number representing the number of ticks BELOW the Open
3. The third DeltaPrice is for the Close, a positive number representing the number of ticks ABOVE the CLOSE.

To use the attached file BruReadNtd.cs, make a test solution and make a test calls. Program.cs is also attached as an example for calling it.

_______________________________________________________

Hi dalebru, thanks fro your ninjascripts. I'm trying to convert .ntd tick file to json .

can you help me how to get tick data from your BruReadNtd script? The BruReadNtd.ReadAndCacheNtd return only a bool ...

exscuse my incompetence...

Thanks for your help

Reply With Quote

Reply



futures io > > > > > NTD File Specification

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

July Journal Challenge w/$1100 in prizes from TopstepTrader

July
 

An Afternoon with FIO member Softsoap

Elite only

Battlestations: Show us your trading desks and win $750 in prizes

August
 

Extended Ask Me Anything w/Brett Steenbarger

Elite only

Prototyping Python Strategies (part 3) w/Ran Aroussi

Elite only

Brannigan Barrett w/Axia Futures

Elite only

Introduction to Phillip Capital futures brokerage services

Aug TBD

How Monte Carlo Analysis Can Help Your Trading w/Kevin Davey

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Video file cabinet kbit Off-Topic 21 October 3rd, 2011 03:52 AM
CS file greatsky NinjaTrader 6 December 23rd, 2010 10:27 AM
zip file downloads ollie NinjaTrader Programming 5 July 30th, 2010 09:11 AM


All times are GMT -4. The time now is 04:48 AM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-07-26 in 0.16 seconds with 20 queries on phoenix via your IP 23.20.50.146