Speeding up NinjaTrader with a RAM drive (ramdisk) - NinjaTrader | futures io social day trading
futures io futures trading


Speeding up NinjaTrader with a RAM drive (ramdisk)
Updated: Views / Replies:25,274 / 110
Created: by pawnbroker Attachments:6

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors – all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you don’t need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 6  
 
Thread Tools Search this Thread
 

Speeding up NinjaTrader with a RAM drive (ramdisk)

  #1 (permalink)
Elite Member
Cheltenham
 
Futures Experience: Advanced
Platform: InvestorRT, NinjaTrader
Favorite Futures: ES
 
Posts: 54 since Jan 2012
Thanks: 8 given, 100 received

Speeding up NinjaTrader with a RAM drive (ramdisk)

I have been able to make a large performance gain in NT 7 by storing the NT database files on a RAM drive and I thought that I would share what I have done.

The time taken by NT to change the time frame of a chart from 3 to 5 minutes was 46 seconds and that reduced to 11 seconds after moving the NT database to a RAM drive. That is a "fake" disk, which uses the computers memory instead of a hard drive. It's very fast, because memory is much faster than a hard drive, including solid state drives.

The only problem with a RAM drive is that the contents are lost when the machine shuts down, unless steps are taken to copy the RAM drive's contents or if you use a RAM drive that copies data to a traditional disk to preserve a copy.

Warnings

My set up is experimental and it is described below for those of you with the IT skill needed to try it yourself.

Since this is an experimental trial, I do not recommend trying this on a production machine. Also, backup your machine before making any changes. There is a risk of corruption in the NT database if your machine crashes or shuts down abnormally, but that risk can be handled with regular backups and by using an uninterruptable power supply (UPS).


Details (which will be most helpful to those of you with IT knowledge)

I have found it frustrating that it has taken about 46 seconds to change the time frame on a chart with NT 7, running eleven charts and a number of indicators, including GomiCD. NT locks up and all charts freeze until NT completes the change of settings. This is on a reasonably good, quad core machine with a solid state hard drive and 8 GB of memory, of which about 4 GB in use.

To test if NT can be sped up, I installed a freeware RAM drive [Dataram RAMDisk] to host the directory NT database directory [1] by creating a symbolic link to another directory on the RAM drive, using Junction [2]. As far as NT is concerned, there is no change to the file structure, but NT gains very fast read and write access to the database.

The rate of reading the database is important to performance and that was not good enough with a solid state hard drive.


The first test involved the same method to relocate the Gomi files to the RAM drive, but that had no obvious effect on NT's performance.


This experimental arrangement uses the load and save feature of Dataram's RAMDisk to preserve the data on shutdown. THere is also an option to save the contents on a schedule. For belt and brace protection, I also use robocopy.


I have since noticed Primo RAM disk, which is a pay for application that has an option to save the contents in real time [3], but I have not tested that.


[1] C:\Users\Pawnbroker\Documents\NinjaTrader 7\db
[2] Junction
[3] Primo Ramdisk Specifications and Features

Reply With Quote
The following 16 users say Thank You to pawnbroker for this post:
 
  #2 (permalink)
Elite Member
london uk
 
Futures Experience: Beginner
Platform: NinjaTrader
Broker/Data: Zen-Fire
Favorite Futures: Stocks
 
tellytub's Avatar
 
Posts: 409 since Jun 2009
Thanks: 333 given, 118 received


pawnbroker View Post
I have been able to make a large performance gain in NT 7 by storing the NT database files on a RAM drive and I thought that I would share what I have done.

The time taken by NT to change the time frame of a chart from 3 to 5 minutes was 46 seconds and that reduced to 11 seconds after moving the NT database to a RAM drive. That is a "fake" disk, which uses the computers memory instead of a hard drive. It's very fast, because memory is much faster than a hard drive, including solid state drives.

The only problem with a RAM drive is that the contents are lost when the machine shuts down, unless steps are taken to copy the RAM drive's contents or if you use a RAM drive that copies data to a traditional disk to preserve a copy.

Warnings

My set up is experimental and it is described below for those of you with the IT skill needed to try it yourself.

Since this is an experimental trial, I do not recommend trying this on a production machine. Also, backup your machine before making any changes. There is a risk of corruption in the NT database if your machine crashes or shuts down abnormally, but that risk can be handled with regular backups and by using an uninterruptable power supply (UPS).


Details (which will be most helpful to those of you with IT knowledge)

I have found it frustrating that it has taken about 46 seconds to change the time frame on a chart with NT 7, running eleven charts and a number of indicators, including GomiCD. NT locks up and all charts freeze until NT completes the change of settings. This is on a reasonably good, quad core machine with a solid state hard drive and 8 GB of memory, of which about 4 GB in use.

To test if NT can be sped up, I installed a freeware RAM drive [Dataram RAMDisk] to host the directory NT database directory [1] by creating a symbolic link to another directory on the RAM drive, using Junction [2]. As far as NT is concerned, there is no change to the file structure, but NT gains very fast read and write access to the database.

The rate of reading the database is important to performance and that was not good enough with a solid state hard drive.


The first test involved the same method to relocate the Gomi files to the RAM drive, but that had no obvious effect on NT's performance.


This experimental arrangement uses the load and save feature of Dataram's RAMDisk to preserve the data on shutdown. THere is also an option to save the contents on a schedule. For belt and brace protection, I also use robocopy.


I have since noticed Primo RAM disk, which is a pay for application that has an option to save the contents in real time [3], but I have not tested that.


[1] C:\Users\Pawnbroker\Documents\NinjaTrader 7\db
[2] Junction
[3] Primo Ramdisk Specifications and Features

funny you mention this, I was looking at this today for Visual Studio, as you said its fast, but you lose it all if the machine reboots for no reason at all. I know there are ways to save the content, but I've heard it tends to be slow.

Reply With Quote
The following user says Thank You to tellytub for this post:
 
  #3 (permalink)
Elite Member
@ Germany
 
Futures Experience: Beginner
Platform: NinjaTrader
Broker/Data: Mirus Futures/Zen-Fire
Favorite Futures: FDAX
 
Posts: 441 since Nov 2011
Thanks: 254 given, 369 received


Hi pawnbroker, i've made the same test with SSD vs. RamDisk and couldn't measure a performance boost. But i don't had many charts open. I think it relies more at the barstyle, timeframe and the days to load. I'm using SuperSpeed RamDisk which is a very reliable product and high end SSDs. It's fine if it works for you. For a production machine i don't recommend using any other software products other then ninjatrader or similar.

Reply With Quote
The following user says Thank You to Koepisch for this post:
 
  #4 (permalink)
Elite Member
Cheltenham
 
Futures Experience: Advanced
Platform: InvestorRT, NinjaTrader
Favorite Futures: ES
 
Posts: 54 since Jan 2012
Thanks: 8 given, 100 received


tellytub View Post
funny you mention this, I was looking at this today for Visual Studio, as you said its fast, but you lose it all if the machine reboots for no reason at all. I know there are ways to save the content, but I've heard it tends to be slow.


Robocopy is very fast if you run that frequently as a scheduled task, but it would be better to use real time synchronisation as that would reduce the loss of data to an absolute minimum. WIth real time sync, there would be no need for a slow "save on shutdown" option.

Primo Ramdisk has an option for real time replication of data to a hard drive, but I have not tested that yet. There are also products that do real time sync, such as BeyondSync and that could be used to keep a "safe" copy on a SSD. So you could create a RAM drive and use native sync with Primo or another application like BeyondSync with a freeware RAM drive.


A RAM drive can work well with NT, since that writes new data once and it read the same data many times. In addition, the amount of data written by NT and Gomi is low, so real time sync would not have an effect on performance.

If Visual Studio's performance has a bottle neck relating to the disk, you can try a RAM disk safely using Junction to point the existing directory to a replica on a RAM drive. If you don't gain a benefit, just delete the symbolic link on the C drive and restore the files from the RAM drive to the original location.

If it does give a large gain, try the Primo disk or BeyindSync or robocopy to keep a safe copy.

Reply With Quote
The following 2 users say Thank You to pawnbroker for this post:
 
  #5 (permalink)
Elite Member
@ Germany
 
Futures Experience: Beginner
Platform: NinjaTrader
Broker/Data: Mirus Futures/Zen-Fire
Favorite Futures: FDAX
 
Posts: 441 since Nov 2011
Thanks: 254 given, 369 received

Keep in mind, that win has an sophisticated caching system which means that if you load thinks twice it's mostly loading from win's file cache and not from the RamDisk/HD anymore.

Reply With Quote
The following user says Thank You to Koepisch for this post:
 
  #6 (permalink)
Elite Member
Cheltenham
 
Futures Experience: Advanced
Platform: InvestorRT, NinjaTrader
Favorite Futures: ES
 
Posts: 54 since Jan 2012
Thanks: 8 given, 100 received


tellytub View Post
funny you mention this, I was looking at this today for Visual Studio, as you said its fast, but you lose it all if the machine reboots for no reason at all. I know there are ways to save the content, but I've heard it tends to be slow.


Koepisch View Post
Keep in mind, that win has an sophisticated caching system which means that if you load thinks twice it's mostly loading from win's file cache and not from the RamDisk/HD anymore.

I considered this and I did not think that it was helping in my use of NT.

If a cache was helping NT then that would be seen by NT taking a long time to process changes (e.g. 3 min chart changed to 5 min chart) on the first trial, but it should be faster on the second trial done soon after, because the files would be cached in memory. However, my tests showed no speed gain on repeating the test.

Also, I don't change settings often, but I want them applied fast when I do make a change. I would expect a cache to flush files that have not been accessed recently, so the NT database may not be cached when I make a change.

For those reasons, I don't expect a general file cache to work in principle and my tests found that it does not help in practice on Windows 7 x64.

In contrast, a RAM disk ensures that the files are always fast to read and I can control the amount of memory used. The only problem is preserving the content on shut down, but that feature is provided by some RAM drive products.

Reply With Quote
The following 2 users say Thank You to pawnbroker for this post:
 
  #7 (permalink)
Elite Member
Portland Oregon, United States
 
Futures Experience: Beginner
Platform: Ninjatrader®
Broker/Data: CQG, Kinetick
Favorite Futures: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,327 since Jul 2009
Thanks: 1,246 given, 2,635 received

Don't forget about code quality


Quoting 
The time taken by NT to change the time frame of a chart from 3 to 5 minutes was 46 seconds and that reduced to 11 seconds after moving the NT database to a RAM drive. That is a "fake" disk, which uses the computers memory instead of a hard drive. It's very fast, because memory is much faster than a hard drive, including solid state drives.

This is interesting and worthwhile, but there is something else to consider.

What indicators do you have on these charts? It's at least as important to reduce resources wasted by inefficient, sloppily coded software, as to look for a way to increase the speed at which that software travels along a 500 mile path to go across the street.

I've found that improving the way an indicator is coded will greatly reduce its loading time and real time CPU load, especially with COBC false. Applied to many indicators and charts, the responsiveness of Ninjatrader is greatly improved.

Only then will memory access improvements deliver to their full potential.

.
https://futures.io/ninjatrader-programming/3565-anyone-have-any-hints-optimizing-c-code-9.html#post207297


Last edited by Zondor; October 27th, 2012 at 01:53 AM.
Reply With Quote
 
  #8 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,240 since Jun 2009
Thanks: 29,352 given, 83,234 received

Very interesting.

But I don't understand why Windows is not doing its job of caching the database files in RAM to begin with. If it is being frequently used, and you have sufficient memory, then it should already be in memory. It could be that 8GB is simply not enough for your workstation in order for Windows to do this.

I am not sure if Windows has support for some type of O_DIRECT flag like exists on Linux, and that perhaps for some unknown reason the developers of NinjaTrader are purposely instructing Windows to do direct reads instead of cached reads?

How big is your database folder?

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #9 (permalink)
Elite Member
Amsterdam, The Netherlands
 
Futures Experience: Intermediate
Platform: Sierra, IRT, ToS, Ninja
Broker/Data: Stage 5
Favorite Futures: NQ
 
bob7123's Avatar
 
Posts: 588 since Oct 2011
Thanks: 109 given, 1,909 received


pawnbroker View Post
I have been able to make a large performance gain in NT 7 by storing the NT database files on a RAM drive and I thought that I would share what I have done.

The time taken by NT to change the time frame of a chart from 3 to 5 minutes was 46 seconds and that reduced to 11 seconds after moving the NT database to a RAM drive.

What Luck! A rainy Saturday here, and a new geeky thing to try. Cool, thanks!

I got everything working as you suggested. WOW charts load faster. Previously I thought delays were due to the IQFeed servers being slow, but I guess not. Great thread!

I found the freeware Dataram RAMDisk worked fine for me, available here. You don't have to pay anything unless you want to make a Ramdrive that is more than 4 GB. I made my RamDrive 1.5GB and it has tons of free space. It is set to backup to my hard drive every 5 minutes, so I am not worried about losing data.

Of course, I couldn't stop there though... I moved my Firefox cache to the ramdrive too. (I'm one of those people that keep a zillion tabs open.) Old but still correct instructions to do that are here: How to Change Cache Size and Location in Firefox

Also, the junction tool you mentioned is not needed after XP, so on my win7 machine I got this GUI tool to create the links:
LinkShellExtension
. I suggest that anyone who gets the LSE might want to read all of the documentation. Not only are junctions supported, but a WHOLE bunch of other options too.

Finally, in regards to SSDs, I have a SSD C drive, and a regular D drive. I manually moved most of my USERS folder to D. I did this so that files that are written to more often are not stored on the SSD, as writing to the SSD drive can degrade it over time. It looks like there is a better way to do that when doing a clean install, using something called "Audit mode" that I had never heard of.

Link to that info is here: Microsoft Answers Page

I've got a new bigger SSD on order, when it gets here I plan to use it to reinstall a fresh copy of windows to get rid of a lot of unused apps. I'll also use the link above to keep more files on the D drive.

And for laughs, I am including a screenshot of the results of HD Tune for my SSD, regular drive and RamDrive.

Attached Thumbnails
Speeding up NinjaTrader with a RAM drive (ramdisk)-diskperformance.jpg  
Reply With Quote
The following 10 users say Thank You to bob7123 for this post:
 
  #10 (permalink)
Elite Member
Portland Oregon, United States
 
Futures Experience: Beginner
Platform: Ninjatrader®
Broker/Data: CQG, Kinetick
Favorite Futures: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,327 since Jul 2009
Thanks: 1,246 given, 2,635 received

RAMDisk big enough?


Question for @bob7123 and @pawnbroker:

The size of my Documents\NinjaTrader7\db folder is 28 GB, most of that being in the data folder.

Will this work with a 1.5 GB RAMDisk partition?

If so could you give a more detailed description of how to create the links using LinkShellExtension?

Reply With Quote
The following 2 users say Thank You to Zondor for this post:

Reply



futures io > > > > Speeding up NinjaTrader with a RAM drive (ramdisk)

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Jigsaw Trading: TBA

Elite only

FuturesTrader71: TBA

Elite only

NinjaTrader: TBA

Jan 18

RandBots: TBA

Jan 23

GFF Brokers & CME Group: Futures & Bitcoin

Elite only

Adam Grimes: TBA

Elite only

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
.....SEC May Ticket Speeding Traders kbit News and Current Events 0 February 23rd, 2012 04:08 PM
NT7.4 - RAM hog kashter NinjaTrader 40 April 14th, 2011 02:10 PM
Drive cory Off-Topic 3 October 1st, 2010 04:17 PM
ram nt7 vps MyTraderAdvisor NinjaTrader 5 August 18th, 2010 04:49 PM
Toyota Blames Driver Error for Unwanted Speeding Quick Summary News and Current Events 0 July 14th, 2010 03:20 AM


All times are GMT -4. The time now is 10:56 PM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-14 in 0.17 seconds with 20 queries on phoenix via your IP 54.90.207.75