NexusFi: Find Your Edge


Home Menu

 





NTD File Specification


Discussion in NinjaTrader

Updated
      Top Posters
    1. looks_one gomi with 3 posts (10 thanks)
    2. looks_two mrjoe with 3 posts (18 thanks)
    3. looks_3 intermarkettrade with 3 posts (0 thanks)
    4. looks_4 Nicolas11 with 2 posts (22 thanks)
      Best Posters
    1. looks_one Nicolas11 with 11 thanks per post
    2. looks_two dalebru with 9 thanks per post
    3. looks_3 mrjoe with 6 thanks per post
    4. looks_4 gomi with 3.3 thanks per post
    1. trending_up 17,847 views
    2. thumb_up 77 thanks given
    3. group 16 followers
    1. forum 22 posts
    2. attach_file 8 attachments




 
Search this Thread

NTD File Specification

  #11 (permalink)
dom993
OTTAWA ON/CANADA
 
Posts: 11 since May 2012
Thanks Given: 0
Thanks Received: 12

Would you guys be able to generate NTM files from historical tick data (say, in the Ninja .txt format) ? I do not need L1 info, this is just to do tick-by-tick replay for real-time testing of trading strategies (CalculateBarOnClose=false)

Thanks in advance
D.

Reply With Quote

Can you help answer these questions
from other members on NexusFi?
The space time continuum and the dynamics of a financial …
Emini and Emicro Index
NexusFi Journal Challenge - April 2024
Feedback and Announcements
New Micros: Ultra 10-Year & Ultra T-Bond -- Live Now
Treasury Notes and Bonds
NT7 Indicator Script Troubleshooting - Camarilla Pivots
NinjaTrader
My NT8 Volume Profile Split by Asian/Euro/Open
NinjaTrader
 
Best Threads (Most Thanked)
in the last 7 days on NexusFi
Get funded firms 2023/2024 - Any recommendations or word …
61 thanks
Funded Trader platforms
39 thanks
Battlestations: Show us your trading desks!
24 thanks
NexusFi site changelog and issues/problem reporting
24 thanks
The Program
17 thanks
  #12 (permalink)
 dalebru 
Indianapolis/IN
 
Experience: Intermediate
Platform: SC, NT, ToS, my own
Broker: EdgeClear
Trading: ES
Posts: 67 since Jan 2013
Thanks Given: 1,035
Thanks Received: 110

Thanks to MrJoe for the Tick file specification. Here is the additional information needed to read day or minute files, along with some code for reading them.

First, note that under "Address" MrJoe has:
0x10 : Price of the first record in the NTD file [8b IEEE754 LE].
0x18,0x20,0x28 : same data as 0x10 (don't know why)
For day and minute bars
the Open is at 0x10,
the High is at 0x18
the Low is at 0x20
the Close at 0x28

For subsequent records after the header:
The Open is calculated the same as for tick files, from the first 1-byte mask
But for Day and Minute files, there is a second mask byte (mask2) for reading DeltaPrices for High, Low, Close
Bits 01234567
0,1 bitmask for bytes to read for Low relative to Open (must be 0 or negative)
2,3 bitmask for bytes to read for High relative to Open (must be 0 or positive)
4,5 not used
6,7 bitmask for bytes to read for Close relative to Low (must be 0 or positive)
For each pair of bits:
00 no data
01 1B BE (Big Endian)
10 2B BE
11 4B BE

Rules, starting with Open
High is equal to Open or higher (data bits 2,3)
Low is equal to Open or lower (data bits 0,1)
Close is equal to Low or higher (data bits 6,7)

Delta Prices are in the following order: High, Low, Close
1. The first DeltaPrice is for the High, a positive number of ticks ABOVE the Open from the header record
2. The second DeltaPrice is for the Low, a positive number representing the number of ticks BELOW the Open
3. The third DeltaPrice is for the Close, a positive number representing the number of ticks ABOVE the CLOSE.

To use the attached file BruReadNtd.cs, make a test solution and make a test calls. Program.cs is also attached as an example for calling it.

Attached Files
Elite Membership required to download: BruReadNtd.cs
Elite Membership required to download: Program.cs
Reply With Quote
  #13 (permalink)
 
Nicolas11's Avatar
 Nicolas11 
near Paris, France
 
Experience: Beginner
Platform: -
Trading: -
Posts: 1,071 since Aug 2011
Thanks Given: 2,232
Thanks Received: 1,769


For some purpose, I needed to be able to read .ntd daily files.
The above great work of mrjoe, gomi and dalebru allowed me understanding the structure of these files
In the enclosed document, I have illustrated their findings with AAPL quotes for 2014.
Thanks again to them,

Nicolas

Attached Thumbnails
NTD File Specification-2014-05-17-ntd-files-structure-illustration-content.pdf  
Visit my NexusFi Trade Journal Reply With Quote
  #14 (permalink)
 
Nicolas11's Avatar
 Nicolas11 
near Paris, France
 
Experience: Beginner
Platform: -
Trading: -
Posts: 1,071 since Aug 2011
Thanks Given: 2,232
Thanks Received: 1,769

The below R code implements the above specification and allows reading .ntd daily files.

It provides a function, called ntdFileToXts, which accepts two arguments:
a) directory containing the daily .ntd files for a given security
b) a flag "VERBOSE" (true = all comments written on the console)
It returns an xts element containing the series of quotes for the given security.
I do not pretend that the code is fully optimized. But I hope that it is clear.

Let's take the example of AAPL daily quotes provided by (free) Kinetic End of Day.

On my computer, related .ntd files (containing data from 1990 to 2014) are stored on:
C:\Users\Nicolas\Documents\NinjaTrader 7\db\day\AAPL\ directory.

The 3 following lines use the above-mentioned function to read those files, and generate a xts file:
 
Code
directory <- "C:\\Users\\Nicolas\\Documents\\NinjaTrader 7\\db\\day\\AAPL\\"
ts.daily <- ntdFileToXts(directory, FALSE)
chartSeries(ts.daily, theme="white")
We obtain the following chart:


Let's focus on 2014 :
 
Code
chartSeries(ts.daily["2014"], theme="white")
Chart:


It really looks the same as NT chart:


In order to be sure that there was no mistake in the reading of the data by the R code, I have compared the xts element to the file produced by NT's export function: 100% identical.

I have tested it also with MSFT, ^SP500, CL ##-## and ES ##-##

Nicolas

 
Code
library(xts)
library(BMS)
library(quantmod)

ntdFileToXts <- function(directory, VERBOSE) {
  # This function reads all .ntd files in a given directory, and returns the corresponding xts series
  
  cat("\nReading from .nft files within ",directory,":\n", sep="")
  
  # 1. Some auxiliary functions:
  # ----------------------------
  
  skip <- function(length) {
    readBin(file.con, raw(), n=length)
  }
  
  read.as.numeric.LE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    readBin(raw.vector, numeric(), n=1, endian="little")
  }
  
  read.as.numeric.BE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    readBin(raw.vector, numeric(), n=1, endian="big")
  }
  
  read.as.unsigned.integer.LE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    as.numeric(paste("0x", paste(rev(raw.vector), collapse=''), sep=""))
  }
  
  read.as.unsigned.integer.BE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    as.numeric(paste("0x", paste(raw.vector, collapse=''), sep=""))
  }
  
  read.as.binary.mask <- function() {
    raw.vector <- readBin(file.con, raw(), n=1)
    hex2bin(toString(raw.vector[1]))
  }
  
  # 2. Creation of the vectors which will temporarily store quotes,
  #    before the creation of the xts
  # ---------------------------------------------------------------
  
  all.date <- as.Date(vector("integer", 1))
  all.open <- vector("numeric", 1) 
  all.high <- vector("numeric", 1) 
  all.low <- vector("numeric", 1) 
  all.close <- vector("numeric", 1) 
  all.volume <- vector("integer", 1) 
  
  # 3. For each ntd file of the directory...
  # ---------------------------------------- 
  
  files <- list.files(path = directory, pattern = "*.ntd")
  files <- files[order(files)]
  index <- 0
  for (file.name0 in files) {
    
    full.file.name <- paste(directory, file.name0, sep="")
    cat("Reading", full.file.name, "...\n")
    file.con <- file(full.file.name, "rb")
    
    
    # 4. Reading of the first record
    # ------------------------------
    
    index = index + 1 
    
    if (VERBOSE) cat("\nRecord #1 (specific structure):\n")
    
    # tick size
    tick.size <--read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Tick size:", tick.size, "\n")
    
    # skip 4 bytes
    skip(4)
    
    # nb of records
    nb.of.records <- read.as.unsigned.integer.LE(4) 
    if (VERBOSE) cat("   Nb of records:", nb.of.records, "\n")
    
    # open
    open <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Open:", open, "\n")
    all.open[index] <- open
    
    # high
    high <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   High:", high, "\n")
    all.high[index] <- high
    
    #low
    low <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Low:", low, "\n")
    all.low[index] <- low
    
    # close
    close <- read.as.numeric.LE(8)  
    if (VERBOSE) cat("   Close:", close, "\n")
    all.close[index] <- close
    
    # date
    date <- as.Date(as.POSIXct(read.as.unsigned.integer.LE(8)/10000000, origin="1-01-01"))
    if (VERBOSE) cat("   Date:", format(date, format="%Y-%m-%d"), "\n")
    all.date[index] <- date
    
    # volume
    volume <- read.as.unsigned.integer.LE(8)
    if (VERBOSE) cat("   Volume:", format(volume, big.mark=","), "\n") 
    all.volume[index] <- volume
    
    # 5. Reading of record #2 and the subsequent ones
    # -----------------------------------------------
    
    for (record.number in 2:nb.of.records) { # 
      
      index <- index + 1
      
      if (VERBOSE) cat("\nRecord #", record.number, ":\n", sep="")
      
      # mask 1
      mask1 <- read.as.binary.mask()
      if (VERBOSE) cat("   Mask 1:", mask1, "\n")
      
      # volume mask within mask 1
      volume.mask <- mask1[2:4]
      if (all(volume.mask == c(0,0,1))) {
        volume.length <- 1
        volume.multiplier <- 1
      } else if (all(volume.mask == c(0,1,1))) {
        volume.length <- 1
        volume.multiplier <- 100
      } else if (all(volume.mask == c(1,0,0))) { 
        volume.length <- 1
        volume.multiplier <- 500    
      } else if (all(volume.mask == c(1,0,1))) { 
        volume.length <- 1
        volume.multiplier <- 1000    
      } else if (all(volume.mask == c(1,1,0))) {
        volume.length <- 2
        volume.multiplier <- 1
      } else if (all(volume.mask == c(1,1,1))) {
        volume.length <- 4
        volume.multiplier <- 1
      } else if (all(volume.mask == c(0,1,0))) {
        volume.length <- 8
        volume.multiplier <- 1
      } else {
        cat("ERROR: volume mask not recognized: ", volume.mask)
      }
      
      # open mask within mask 1
      open.mask <- mask1[5:6]
      if (all(open.mask == c(0,0))) {
        open.length <- 0
      } else if (all(open.mask == c(0,1))) {
        open.length <- 1
        open.floor <- 128
      } else if (all(open.mask == c(1,0))) {
        open.length <- 2
        open.floor <- 16384
      } else if (all(open.mask == c(1,1))) {
        open.length <- 4
        open.floor <- 1073741824
      } else {
        cat("ERROR: open mask not recognized: ", open.mask)
      }
      
      # delta time mask within mask 1
      delta.time.mask <- mask1[7:8]
      if (all(delta.time.mask == c(0,0))) {
        delta.time.length <- 0
      } else if (all(delta.time.mask == c(0,1))) {
        delta.time.length <- 1
      } else if (all(delta.time.mask == c(1,0))) {
        delta.time.length <- 2
      } else if (all(delta.time.mask == c(1,1))) {
        delta.time.length <- 4
      } else {
        cat("ERROR: delta time mask not recognized: ", delta.time.mask)
      }
      
      #mask 2
      mask2 <- read.as.binary.mask()
      if (VERBOSE) cat("   Mask 2:", mask1, "\n")
      
      # low mask within mask 2
      low.mask <- mask2[1:2]
      if (all(low.mask == c(0,0))) {
        low.length <- 0
      } else if (all(low.mask == c(0,1))) {
        low.length <- 1
      } else if (all(low.mask == c(1,0))) {
        low.length <- 2
      } else if (all(low.mask == c(1,1))) {
        low.length <- 4
      } else {
        cat("ERROR: low mask not recognized: ", low.mask)
      }
      
      # high mask within mask 2
      high.mask <- mask2[3:4]
      if (all(high.mask == c(0,0))) {
        high.length <- 0
      } else if (all(high.mask == c(0,1))) {
        high.length <- 1
      } else if (all(high.mask == c(1,0))) {
        high.length <- 2
      } else if (all(high.mask == c(1,1))) {
        high.length <- 4
      } else {
        cat("ERROR: high mask not recognized: ", high.mask)
      }
      
      # close mask within mask 2
      close.mask <- mask2[7:8]
      if (all(close.mask == c(0,0))) {
        close.length <- 0
      } else if (all(close.mask == c(0,1))) {
        close.length <- 1
      } else if (all(close.mask == c(1,0))) {
        close.length <- 2
      } else if (all(close.mask == c(1,1))) {
        close.length <- 4
      } else {
        cat("ERROR: close mask not recognized: ", close.mask)
      }
      
      # delta time and date
      delta.time <- ifelse(delta.time.length == 0, 1, read.as.unsigned.integer.BE(delta.time.length))
      if (VERBOSE) cat("   Delta time:", delta.time) 
      date <- date + delta.time;
      all.date[index] <- date
      if (VERBOSE) cat(" => Date:", format(date, format="%Y-%m-%d"), "\n")
      
      # open delta and open
      open.delta <- ifelse(open.length == 0, 0, read.as.unsigned.integer.BE(open.length)-open.floor)
      if (VERBOSE) cat("   Open delta:", open.delta)
      open <- open + open.delta * tick.size
      if (VERBOSE) cat(" => Open:", open, "\n")
      all.open[index] <- open
      
      # high delta and high
      high.delta <- ifelse(high.length == 0, 0, read.as.unsigned.integer.BE(high.length))
      if (VERBOSE) cat("   High delta:", high.delta)
      high <- open + high.delta * tick.size
      if (VERBOSE) cat(" => High:", high, "\n")
      all.high[index] <- high
      
      # low delta and low
      low.delta <- ifelse(low.length == 0, 0, read.as.unsigned.integer.BE(low.length))         
      if (VERBOSE) cat("   Low delta:", low.delta)
      low <- open - low.delta * tick.size
      if (VERBOSE) cat(" => Low:", low, "\n")
      all.low[index] <- low
      
      # close delta and close
      close.delta <- ifelse(close.length == 0, 0, read.as.unsigned.integer.BE(close.length))
      if (VERBOSE) cat("   Close delta:", close.delta)
      close <- low + close.delta * tick.size
      if (VERBOSE) cat(" => Close:", close, "\n")
      all.close[index] <- close
      
      # volume
      volume <- volume.multiplier * read.as.unsigned.integer.BE(volume.length)
      if (VERBOSE) cat("   Volume:", format(volume, big.mark=","), "\n") 
      all.volume[index] <- volume
    }
    
    close(file.con)
  }
  
  # 6. Creation of xts
  # ------------------
  
  quotes <- cbind(all.open, all.high, all.low, all.close, all.volume)
  colnames(quotes) <- c("open", "high", "low", "close", "volume")
  xts(x=quotes, order.by=all.date)
}

Visit my NexusFi Trade Journal Reply With Quote
  #15 (permalink)
 BMMA14 
Montreal,Quebec/Canada
 
Experience: Intermediate
Platform: NinjaTrader
Trading: Forex
Posts: 17 since Jun 2014
Thanks Given: 5
Thanks Received: 15

very good code. Thanks for sharing

Reply With Quote
Thanked by:
  #16 (permalink)
 BMMA14 
Montreal,Quebec/Canada
 
Experience: Intermediate
Platform: NinjaTrader
Trading: Forex
Posts: 17 since Jun 2014
Thanks Given: 5
Thanks Received: 15


Nicolas11 View Post
For some purpose, I needed to be able to read .ntd daily files.
The above great work of mrjoe, gomi and dalebru allowed me understanding the structure of these files
In the enclosed document, I have illustrated their findings with AAPL quotes for 2014.
Thanks again to them,

Nicolas

Good and well organized document for the work done by those wonderful members. Keep up the good work. Thanks.

Reply With Quote
Thanked by:
  #17 (permalink)
 bboyle1234 
Australia
 
Experience: Advanced
Platform: NinjaTrader
Trading: CL
Posts: 1 since Feb 2013
Thanks Given: 6
Thanks Received: 1

Thanks for sharing everybody, here's what I created for my own use based on your ideas.

https://github.com/bboyle1234/NTDFileReader

Reply With Quote
Thanked by:
  #18 (permalink)
testingnew
Delhi+India
 
Posts: 1 since Jul 2016
Thanks Given: 1
Thanks Received: 1


Nicolas11 View Post
The below R code implements the above specification and allows reading .ntd daily files.

It provides a function, called ntdFileToXts, which accepts two arguments:
a) directory containing the daily .ntd files for a given security
b) a flag "VERBOSE" (true = all comments written on the console)
It returns an xts element containing the series of quotes for the given security.
I do not pretend that the code is fully optimized. But I hope that it is clear.

Let's take the example of AAPL daily quotes provided by (free) Kinetic End of Day.

On my computer, related .ntd files (containing data from 1990 to 2014) are stored on:
C:\Users\Nicolas\Documents\NinjaTrader 7\db\day\AAPL\ directory.

The 3 following lines use the above-mentioned function to read those files, and generate a xts file:
 
Code
directory <- "C:\\Users\\Nicolas\\Documents\\NinjaTrader 7\\db\\day\\AAPL\\"
ts.daily <- ntdFileToXts(directory, FALSE)
chartSeries(ts.daily, theme="white")
We obtain the following chart:


Let's focus on 2014 :
 
Code
chartSeries(ts.daily["2014"], theme="white")
Chart:


It really looks the same as NT chart:


In order to be sure that there was no mistake in the reading of the data by the R code, I have compared the xts element to the file produced by NT's export function: 100% identical.

I have tested it also with MSFT, ^SP500, CL ##-## and ES ##-##

Nicolas

 
Code
library(xts)
library(BMS)
library(quantmod)

ntdFileToXts <- function(directory, VERBOSE) {
  # This function reads all .ntd files in a given directory, and returns the corresponding xts series
  
  cat("\nReading from .nft files within ",directory,":\n", sep="")
  
  # 1. Some auxiliary functions:
  # ----------------------------
  
  skip <- function(length) {
    readBin(file.con, raw(), n=length)
  }
  
  read.as.numeric.LE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    readBin(raw.vector, numeric(), n=1, endian="little")
  }
  
  read.as.numeric.BE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    readBin(raw.vector, numeric(), n=1, endian="big")
  }
  
  read.as.unsigned.integer.LE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    as.numeric(paste("0x", paste(rev(raw.vector), collapse=''), sep=""))
  }
  
  read.as.unsigned.integer.BE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    as.numeric(paste("0x", paste(raw.vector, collapse=''), sep=""))
  }
  
  read.as.binary.mask <- function() {
    raw.vector <- readBin(file.con, raw(), n=1)
    hex2bin(toString(raw.vector[1]))
  }
  
  # 2. Creation of the vectors which will temporarily store quotes,
  #    before the creation of the xts
  # ---------------------------------------------------------------
  
  all.date <- as.Date(vector("integer", 1))
  all.open <- vector("numeric", 1) 
  all.high <- vector("numeric", 1) 
  all.low <- vector("numeric", 1) 
  all.close <- vector("numeric", 1) 
  all.volume <- vector("integer", 1) 
  
  # 3. For each ntd file of the directory...
  # ---------------------------------------- 
  
  files <- list.files(path = directory, pattern = "*.ntd")
  files <- files[order(files)]
  index <- 0
  for (file.name0 in files) {
    
    full.file.name <- paste(directory, file.name0, sep="")
    cat("Reading", full.file.name, "...\n")
    file.con <- file(full.file.name, "rb")
    
    
    # 4. Reading of the first record
    # ------------------------------
    
    index = index + 1 
    
    if (VERBOSE) cat("\nRecord #1 (specific structure):\n")
    
    # tick size
    tick.size <--read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Tick size:", tick.size, "\n")
    
    # skip 4 bytes
    skip(4)
    
    # nb of records
    nb.of.records <- read.as.unsigned.integer.LE(4) 
    if (VERBOSE) cat("   Nb of records:", nb.of.records, "\n")
    
    # open
    open <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Open:", open, "\n")
    all.open[index] <- open
    
    # high
    high <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   High:", high, "\n")
    all.high[index] <- high
    
    #low
    low <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Low:", low, "\n")
    all.low[index] <- low
    
    # close
    close <- read.as.numeric.LE(8)  
    if (VERBOSE) cat("   Close:", close, "\n")
    all.close[index] <- close
    
    # date
    date <- as.Date(as.POSIXct(read.as.unsigned.integer.LE(8)/10000000, origin="1-01-01"))
    if (VERBOSE) cat("   Date:", format(date, format="%Y-%m-%d"), "\n")
    all.date[index] <- date
    
    # volume
    volume <- read.as.unsigned.integer.LE(8)
    if (VERBOSE) cat("   Volume:", format(volume, big.mark=","), "\n") 
    all.volume[index] <- volume
    
    # 5. Reading of record #2 and the subsequent ones
    # -----------------------------------------------
    
    for (record.number in 2:nb.of.records) { # 
      
      index <- index + 1
      
      if (VERBOSE) cat("\nRecord #", record.number, ":\n", sep="")
      
      # mask 1
      mask1 <- read.as.binary.mask()
      if (VERBOSE) cat("   Mask 1:", mask1, "\n")
      
      # volume mask within mask 1
      volume.mask <- mask1[2:4]
      if (all(volume.mask == c(0,0,1))) {
        volume.length <- 1
        volume.multiplier <- 1
      } else if (all(volume.mask == c(0,1,1))) {
        volume.length <- 1
        volume.multiplier <- 100
      } else if (all(volume.mask == c(1,0,0))) { 
        volume.length <- 1
        volume.multiplier <- 500    
      } else if (all(volume.mask == c(1,0,1))) { 
        volume.length <- 1
        volume.multiplier <- 1000    
      } else if (all(volume.mask == c(1,1,0))) {
        volume.length <- 2
        volume.multiplier <- 1
      } else if (all(volume.mask == c(1,1,1))) {
        volume.length <- 4
        volume.multiplier <- 1
      } else if (all(volume.mask == c(0,1,0))) {
        volume.length <- 8
        volume.multiplier <- 1
      } else {
        cat("ERROR: volume mask not recognized: ", volume.mask)
      }
      
      # open mask within mask 1
      open.mask <- mask1[5:6]
      if (all(open.mask == c(0,0))) {
        open.length <- 0
      } else if (all(open.mask == c(0,1))) {
        open.length <- 1
        open.floor <- 128
      } else if (all(open.mask == c(1,0))) {
        open.length <- 2
        open.floor <- 16384
      } else if (all(open.mask == c(1,1))) {
        open.length <- 4
        open.floor <- 1073741824
      } else {
        cat("ERROR: open mask not recognized: ", open.mask)
      }
      
      # delta time mask within mask 1
      delta.time.mask <- mask1[7:8]
      if (all(delta.time.mask == c(0,0))) {
        delta.time.length <- 0
      } else if (all(delta.time.mask == c(0,1))) {
        delta.time.length <- 1
      } else if (all(delta.time.mask == c(1,0))) {
        delta.time.length <- 2
      } else if (all(delta.time.mask == c(1,1))) {
        delta.time.length <- 4
      } else {
        cat("ERROR: delta time mask not recognized: ", delta.time.mask)
      }
      
      #mask 2
      mask2 <- read.as.binary.mask()
      if (VERBOSE) cat("   Mask 2:", mask1, "\n")
      
      # low mask within mask 2
      low.mask <- mask2[1:2]
      if (all(low.mask == c(0,0))) {
        low.length <- 0
      } else if (all(low.mask == c(0,1))) {
        low.length <- 1
      } else if (all(low.mask == c(1,0))) {
        low.length <- 2
      } else if (all(low.mask == c(1,1))) {
        low.length <- 4
      } else {
        cat("ERROR: low mask not recognized: ", low.mask)
      }
      
      # high mask within mask 2
      high.mask <- mask2[3:4]
      if (all(high.mask == c(0,0))) {
        high.length <- 0
      } else if (all(high.mask == c(0,1))) {
        high.length <- 1
      } else if (all(high.mask == c(1,0))) {
        high.length <- 2
      } else if (all(high.mask == c(1,1))) {
        high.length <- 4
      } else {
        cat("ERROR: high mask not recognized: ", high.mask)
      }
      
      # close mask within mask 2
      close.mask <- mask2[7:8]
      if (all(close.mask == c(0,0))) {
        close.length <- 0
      } else if (all(close.mask == c(0,1))) {
        close.length <- 1
      } else if (all(close.mask == c(1,0))) {
        close.length <- 2
      } else if (all(close.mask == c(1,1))) {
        close.length <- 4
      } else {
        cat("ERROR: close mask not recognized: ", close.mask)
      }
      
      # delta time and date
      delta.time <- ifelse(delta.time.length == 0, 1, read.as.unsigned.integer.BE(delta.time.length))
      if (VERBOSE) cat("   Delta time:", delta.time) 
      date <- date + delta.time;
      all.date[index] <- date
      if (VERBOSE) cat(" => Date:", format(date, format="%Y-%m-%d"), "\n")
      
      # open delta and open
      open.delta <- ifelse(open.length == 0, 0, read.as.unsigned.integer.BE(open.length)-open.floor)
      if (VERBOSE) cat("   Open delta:", open.delta)
      open <- open + open.delta * tick.size
      if (VERBOSE) cat(" => Open:", open, "\n")
      all.open[index] <- open
      
      # high delta and high
      high.delta <- ifelse(high.length == 0, 0, read.as.unsigned.integer.BE(high.length))
      if (VERBOSE) cat("   High delta:", high.delta)
      high <- open + high.delta * tick.size
      if (VERBOSE) cat(" => High:", high, "\n")
      all.high[index] <- high
      
      # low delta and low
      low.delta <- ifelse(low.length == 0, 0, read.as.unsigned.integer.BE(low.length))         
      if (VERBOSE) cat("   Low delta:", low.delta)
      low <- open - low.delta * tick.size
      if (VERBOSE) cat(" => Low:", low, "\n")
      all.low[index] <- low
      
      # close delta and close
      close.delta <- ifelse(close.length == 0, 0, read.as.unsigned.integer.BE(close.length))
      if (VERBOSE) cat("   Close delta:", close.delta)
      close <- low + close.delta * tick.size
      if (VERBOSE) cat(" => Close:", close, "\n")
      all.close[index] <- close
      
      # volume
      volume <- volume.multiplier * read.as.unsigned.integer.BE(volume.length)
      if (VERBOSE) cat("   Volume:", format(volume, big.mark=","), "\n") 
      all.volume[index] <- volume
    }
    
    close(file.con)
  }
  
  # 6. Creation of xts
  # ------------------
  
  quotes <- cbind(all.open, all.high, all.low, all.close, all.volume)
  colnames(quotes) <- c("open", "high", "low", "close", "volume")
  xts(x=quotes, order.by=all.date)
}


Hi Nicolas,

I am getting this error.

 
Code
Reading from .nft files within C:\Users\Anil\Documents\NinjaTrader 7\db\minute\SBINEQ\:
Error in as.Date.numeric(vector("integer", 1)) : 
  'origin' must be supplied

Please pardon me as I am not quite familiar with R.

I have also attached one of the file in my NT db.

EDIT: IT WORKED. Apparently the xts package was not installed. :-P

Attached Files
Elite Membership required to download: 20160609.Last.zip
Reply With Quote
Thanked by:
  #19 (permalink)
intermarkettrade
rome,italy,italy
 
Posts: 3 since Aug 2010
Thanks Given: 1
Thanks Received: 0


Nicolas11 View Post
The below R code implements the above specification and allows reading .ntd daily files.

It provides a function, called ntdFileToXts, which accepts two arguments:
a) directory containing the daily .ntd files for a given security
b) a flag "VERBOSE" (true = all comments written on the console)
It returns an xts element containing the series of quotes for the given security.
I do not pretend that the code is fully optimized. But I hope that it is clear.

Let's take the example of AAPL daily quotes provided by (free) Kinetic End of Day.

On my computer, related .ntd files (containing data from 1990 to 2014) are stored on:
C:\Users\Nicolas\Documents\NinjaTrader 7\db\day\AAPL\ directory.

The 3 following lines use the above-mentioned function to read those files, and generate a xts file:
 
Code
directory <- "C:\\Users\\Nicolas\\Documents\\NinjaTrader 7\\db\\day\\AAPL\\"
ts.daily <- ntdFileToXts(directory, FALSE)
chartSeries(ts.daily, theme="white")
We obtain the following chart:


Let's focus on 2014 :
 
Code
chartSeries(ts.daily["2014"], theme="white")
Chart:


It really looks the same as NT chart:


In order to be sure that there was no mistake in the reading of the data by the R code, I have compared the xts element to the file produced by NT's export function: 100% identical.

I have tested it also with MSFT, ^SP500, CL ##-## and ES ##-##

Nicolas

 
Code
library(xts)
library(BMS)
library(quantmod)

ntdFileToXts <- function(directory, VERBOSE) {
  # This function reads all .ntd files in a given directory, and returns the corresponding xts series
  
  cat("\nReading from .nft files within ",directory,":\n", sep="")
  
  # 1. Some auxiliary functions:
  # ----------------------------
  
  skip <- function(length) {
    readBin(file.con, raw(), n=length)
  }
  
  read.as.numeric.LE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    readBin(raw.vector, numeric(), n=1, endian="little")
  }
  
  read.as.numeric.BE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    readBin(raw.vector, numeric(), n=1, endian="big")
  }
  
  read.as.unsigned.integer.LE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    as.numeric(paste("0x", paste(rev(raw.vector), collapse=''), sep=""))
  }
  
  read.as.unsigned.integer.BE <- function(length) {
    raw.vector <- readBin(file.con, raw(), n=length)
    as.numeric(paste("0x", paste(raw.vector, collapse=''), sep=""))
  }
  
  read.as.binary.mask <- function() {
    raw.vector <- readBin(file.con, raw(), n=1)
    hex2bin(toString(raw.vector[1]))
  }
  
  # 2. Creation of the vectors which will temporarily store quotes,
  #    before the creation of the xts
  # ---------------------------------------------------------------
  
  all.date <- as.Date(vector("integer", 1))
  all.open <- vector("numeric", 1) 
  all.high <- vector("numeric", 1) 
  all.low <- vector("numeric", 1) 
  all.close <- vector("numeric", 1) 
  all.volume <- vector("integer", 1) 
  
  # 3. For each ntd file of the directory...
  # ---------------------------------------- 
  
  files <- list.files(path = directory, pattern = "*.ntd")
  files <- files[order(files)]
  index <- 0
  for (file.name0 in files) {
    
    full.file.name <- paste(directory, file.name0, sep="")
    cat("Reading", full.file.name, "...\n")
    file.con <- file(full.file.name, "rb")
    
    
    # 4. Reading of the first record
    # ------------------------------
    
    index = index + 1 
    
    if (VERBOSE) cat("\nRecord #1 (specific structure):\n")
    
    # tick size
    tick.size <--read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Tick size:", tick.size, "\n")
    
    # skip 4 bytes
    skip(4)
    
    # nb of records
    nb.of.records <- read.as.unsigned.integer.LE(4) 
    if (VERBOSE) cat("   Nb of records:", nb.of.records, "\n")
    
    # open
    open <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Open:", open, "\n")
    all.open[index] <- open
    
    # high
    high <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   High:", high, "\n")
    all.high[index] <- high
    
    #low
    low <- read.as.numeric.LE(8) 
    if (VERBOSE) cat("   Low:", low, "\n")
    all.low[index] <- low
    
    # close
    close <- read.as.numeric.LE(8)  
    if (VERBOSE) cat("   Close:", close, "\n")
    all.close[index] <- close
    
    # date
    date <- as.Date(as.POSIXct(read.as.unsigned.integer.LE(8)/10000000, origin="1-01-01"))
    if (VERBOSE) cat("   Date:", format(date, format="%Y-%m-%d"), "\n")
    all.date[index] <- date
    
    # volume
    volume <- read.as.unsigned.integer.LE(8)
    if (VERBOSE) cat("   Volume:", format(volume, big.mark=","), "\n") 
    all.volume[index] <- volume
    
    # 5. Reading of record #2 and the subsequent ones
    # -----------------------------------------------
    
    for (record.number in 2:nb.of.records) { # 
      
      index <- index + 1
      
      if (VERBOSE) cat("\nRecord #", record.number, ":\n", sep="")
      
      # mask 1
      mask1 <- read.as.binary.mask()
      if (VERBOSE) cat("   Mask 1:", mask1, "\n")
      
      # volume mask within mask 1
      volume.mask <- mask1[2:4]
      if (all(volume.mask == c(0,0,1))) {
        volume.length <- 1
        volume.multiplier <- 1
      } else if (all(volume.mask == c(0,1,1))) {
        volume.length <- 1
        volume.multiplier <- 100
      } else if (all(volume.mask == c(1,0,0))) { 
        volume.length <- 1
        volume.multiplier <- 500    
      } else if (all(volume.mask == c(1,0,1))) { 
        volume.length <- 1
        volume.multiplier <- 1000    
      } else if (all(volume.mask == c(1,1,0))) {
        volume.length <- 2
        volume.multiplier <- 1
      } else if (all(volume.mask == c(1,1,1))) {
        volume.length <- 4
        volume.multiplier <- 1
      } else if (all(volume.mask == c(0,1,0))) {
        volume.length <- 8
        volume.multiplier <- 1
      } else {
        cat("ERROR: volume mask not recognized: ", volume.mask)
      }
      
      # open mask within mask 1
      open.mask <- mask1[5:6]
      if (all(open.mask == c(0,0))) {
        open.length <- 0
      } else if (all(open.mask == c(0,1))) {
        open.length <- 1
        open.floor <- 128
      } else if (all(open.mask == c(1,0))) {
        open.length <- 2
        open.floor <- 16384
      } else if (all(open.mask == c(1,1))) {
        open.length <- 4
        open.floor <- 1073741824
      } else {
        cat("ERROR: open mask not recognized: ", open.mask)
      }
      
      # delta time mask within mask 1
      delta.time.mask <- mask1[7:8]
      if (all(delta.time.mask == c(0,0))) {
        delta.time.length <- 0
      } else if (all(delta.time.mask == c(0,1))) {
        delta.time.length <- 1
      } else if (all(delta.time.mask == c(1,0))) {
        delta.time.length <- 2
      } else if (all(delta.time.mask == c(1,1))) {
        delta.time.length <- 4
      } else {
        cat("ERROR: delta time mask not recognized: ", delta.time.mask)
      }
      
      #mask 2
      mask2 <- read.as.binary.mask()
      if (VERBOSE) cat("   Mask 2:", mask1, "\n")
      
      # low mask within mask 2
      low.mask <- mask2[1:2]
      if (all(low.mask == c(0,0))) {
        low.length <- 0
      } else if (all(low.mask == c(0,1))) {
        low.length <- 1
      } else if (all(low.mask == c(1,0))) {
        low.length <- 2
      } else if (all(low.mask == c(1,1))) {
        low.length <- 4
      } else {
        cat("ERROR: low mask not recognized: ", low.mask)
      }
      
      # high mask within mask 2
      high.mask <- mask2[3:4]
      if (all(high.mask == c(0,0))) {
        high.length <- 0
      } else if (all(high.mask == c(0,1))) {
        high.length <- 1
      } else if (all(high.mask == c(1,0))) {
        high.length <- 2
      } else if (all(high.mask == c(1,1))) {
        high.length <- 4
      } else {
        cat("ERROR: high mask not recognized: ", high.mask)
      }
      
      # close mask within mask 2
      close.mask <- mask2[7:8]
      if (all(close.mask == c(0,0))) {
        close.length <- 0
      } else if (all(close.mask == c(0,1))) {
        close.length <- 1
      } else if (all(close.mask == c(1,0))) {
        close.length <- 2
      } else if (all(close.mask == c(1,1))) {
        close.length <- 4
      } else {
        cat("ERROR: close mask not recognized: ", close.mask)
      }
      
      # delta time and date
      delta.time <- ifelse(delta.time.length == 0, 1, read.as.unsigned.integer.BE(delta.time.length))
      if (VERBOSE) cat("   Delta time:", delta.time) 
      date <- date + delta.time;
      all.date[index] <- date
      if (VERBOSE) cat(" => Date:", format(date, format="%Y-%m-%d"), "\n")
      
      # open delta and open
      open.delta <- ifelse(open.length == 0, 0, read.as.unsigned.integer.BE(open.length)-open.floor)
      if (VERBOSE) cat("   Open delta:", open.delta)
      open <- open + open.delta * tick.size
      if (VERBOSE) cat(" => Open:", open, "\n")
      all.open[index] <- open
      
      # high delta and high
      high.delta <- ifelse(high.length == 0, 0, read.as.unsigned.integer.BE(high.length))
      if (VERBOSE) cat("   High delta:", high.delta)
      high <- open + high.delta * tick.size
      if (VERBOSE) cat(" => High:", high, "\n")
      all.high[index] <- high
      
      # low delta and low
      low.delta <- ifelse(low.length == 0, 0, read.as.unsigned.integer.BE(low.length))         
      if (VERBOSE) cat("   Low delta:", low.delta)
      low <- open - low.delta * tick.size
      if (VERBOSE) cat(" => Low:", low, "\n")
      all.low[index] <- low
      
      # close delta and close
      close.delta <- ifelse(close.length == 0, 0, read.as.unsigned.integer.BE(close.length))
      if (VERBOSE) cat("   Close delta:", close.delta)
      close <- low + close.delta * tick.size
      if (VERBOSE) cat(" => Close:", close, "\n")
      all.close[index] <- close
      
      # volume
      volume <- volume.multiplier * read.as.unsigned.integer.BE(volume.length)
      if (VERBOSE) cat("   Volume:", format(volume, big.mark=","), "\n") 
      all.volume[index] <- volume
    }
    
    close(file.con)
  }
  
  # 6. Creation of xts
  # ------------------
  
  quotes <- cbind(all.open, all.high, all.low, all.close, all.volume)
  colnames(quotes) <- c("open", "high", "low", "close", "volume")
  xts(x=quotes, order.by=all.date)
}


___________________________________________________

Hi , I have very appreciate and thank you for your superb R code to convert Ninja .ntd file to XTS .

But my ignorance is very "cool" :-) and becouse I need to convert tick Ninja .ntd file to XTS , appear many errors like:

volume mask not recognized: 0 0 0Error in read.as.unsigned.integer.BE(volume.length) converted from warning) NAs introduced by coercion


Any help...?

Many many thansk and hope in your answer

Kind regards by Giuseppe from Italy: stickyman:

Reply With Quote
  #20 (permalink)
intermarkettrade
rome,italy,italy
 
Posts: 3 since Aug 2010
Thanks Given: 1
Thanks Received: 0



dalebru View Post
Thanks to MrJoe for the Tick file specification. Here is the additional information needed to read day or minute files, along with some code for reading them.

First, note that under "Address" MrJoe has:
0x10 : Price of the first record in the NTD file [8b IEEE754 LE].
0x18,0x20,0x28 : same data as 0x10 (don't know why)
For day and minute bars
the Open is at 0x10,
the High is at 0x18
the Low is at 0x20
the Close at 0x28

For subsequent records after the header:
The Open is calculated the same as for tick files, from the first 1-byte mask
But for Day and Minute files, there is a second mask byte (mask2) for reading DeltaPrices for High, Low, Close
Bits 01234567
0,1 bitmask for bytes to read for Low relative to Open (must be 0 or negative)
2,3 bitmask for bytes to read for High relative to Open (must be 0 or positive)
4,5 not used
6,7 bitmask for bytes to read for Close relative to Low (must be 0 or positive)
For each pair of bits:
00 no data
01 1B BE (Big Endian)
10 2B BE
11 4B BE

Rules, starting with Open
High is equal to Open or higher (data bits 2,3)
Low is equal to Open or lower (data bits 0,1)
Close is equal to Low or higher (data bits 6,7)

Delta Prices are in the following order: High, Low, Close
1. The first DeltaPrice is for the High, a positive number of ticks ABOVE the Open from the header record
2. The second DeltaPrice is for the Low, a positive number representing the number of ticks BELOW the Open
3. The third DeltaPrice is for the Close, a positive number representing the number of ticks ABOVE the CLOSE.

To use the attached file BruReadNtd.cs, make a test solution and make a test calls. Program.cs is also attached as an example for calling it.

_______________________________________________________

Hi dalebru, thanks fro your ninjascripts. I'm trying to convert .ntd tick file to json .

can you help me how to get tick data from your BruReadNtd script? The BruReadNtd.ReadAndCacheNtd return only a bool ...

exscuse my incompetence...

Thanks for your help

Reply With Quote




Last Updated on December 23, 2021


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts