NexusFi: Find Your Edge


Home Menu

 





Speeding up NinjaTrader with a RAM drive (ramdisk)


Discussion in NinjaTrader

Updated
      Top Posters
    1. looks_one Big Mike with 20 posts (13 thanks)
    2. looks_two Zondor with 18 posts (24 thanks)
    3. looks_3 kronie with 14 posts (2 thanks)
    4. looks_4 bob7123 with 12 posts (15 thanks)
      Best Posters
    1. looks_one pawnbroker with 3.4 thanks per post
    2. looks_two Zondor with 1.3 thanks per post
    3. looks_3 bob7123 with 1.3 thanks per post
    4. looks_4 Big Mike with 0.7 thanks per post
    1. trending_up 59,279 views
    2. thumb_up 94 thanks given
    3. group 41 followers
    1. forum 111 posts
    2. attach_file 6 attachments




 
Search this Thread

Speeding up NinjaTrader with a RAM drive (ramdisk)

  #1 (permalink)
 pawnbroker 
Cheltenham
 
Experience: Advanced
Platform: InvestorRT, NinjaTrader
Trading: ES
Posts: 54 since Jan 2012
Thanks Given: 8
Thanks Received: 107

I have been able to make a large performance gain in NT 7 by storing the NT database files on a RAM drive and I thought that I would share what I have done.

The time taken by NT to change the time frame of a chart from 3 to 5 minutes was 46 seconds and that reduced to 11 seconds after moving the NT database to a RAM drive. That is a "fake" disk, which uses the computers memory instead of a hard drive. It's very fast, because memory is much faster than a hard drive, including solid state drives.

The only problem with a RAM drive is that the contents are lost when the machine shuts down, unless steps are taken to copy the RAM drive's contents or if you use a RAM drive that copies data to a traditional disk to preserve a copy.

Warnings

My set up is experimental and it is described below for those of you with the IT skill needed to try it yourself.

Since this is an experimental trial, I do not recommend trying this on a production machine. Also, backup your machine before making any changes. There is a risk of corruption in the NT database if your machine crashes or shuts down abnormally, but that risk can be handled with regular backups and by using an uninterruptable power supply (UPS).


Details (which will be most helpful to those of you with IT knowledge)

I have found it frustrating that it has taken about 46 seconds to change the time frame on a chart with NT 7, running eleven charts and a number of indicators, including GomiCD. NT locks up and all charts freeze until NT completes the change of settings. This is on a reasonably good, quad core machine with a solid state hard drive and 8 GB of memory, of which about 4 GB in use.

To test if NT can be sped up, I installed a freeware RAM drive [Dataram RAMDisk] to host the directory NT database directory [1] by creating a symbolic link to another directory on the RAM drive, using Junction [2]. As far as NT is concerned, there is no change to the file structure, but NT gains very fast read and write access to the database.

The rate of reading the database is important to performance and that was not good enough with a solid state hard drive.


The first test involved the same method to relocate the Gomi files to the RAM drive, but that had no obvious effect on NT's performance.


This experimental arrangement uses the load and save feature of Dataram's RAMDisk to preserve the data on shutdown. THere is also an option to save the contents on a schedule. For belt and brace protection, I also use robocopy.


I have since noticed Primo RAM disk, which is a pay for application that has an option to save the contents in real time [3], but I have not tested that.


[1] C:\Users\Pawnbroker\Documents\NinjaTrader 7\db
[2] Junction
[3] Primo Ramdisk Specifications and Features

Started this thread Reply With Quote

Can you help answer these questions
from other members on NexusFi?
NT7 Indicator Script Troubleshooting - Camarilla Pivots
NinjaTrader
My NT8 Volume Profile Split by Asian/Euro/Open
NinjaTrader
ZombieSqueeze
Platforms and Indicators
Exit Strategy
NinjaTrader
Better Renko Gaps
The Elite Circle
 
Best Threads (Most Thanked)
in the last 7 days on NexusFi
Get funded firms 2023/2024 - Any recommendations or word …
61 thanks
Funded Trader platforms
39 thanks
NexusFi site changelog and issues/problem reporting
26 thanks
Battlestations: Show us your trading desks!
26 thanks
The Program
18 thanks
  #2 (permalink)
 
tellytub's Avatar
 tellytub 
london uk
 
Experience: Beginner
Platform: NinjaTrader
Broker: Zen-Fire
Trading: Stocks
Posts: 410 since Jun 2009
Thanks Given: 333
Thanks Received: 121


pawnbroker View Post
I have been able to make a large performance gain in NT 7 by storing the NT database files on a RAM drive and I thought that I would share what I have done.

The time taken by NT to change the time frame of a chart from 3 to 5 minutes was 46 seconds and that reduced to 11 seconds after moving the NT database to a RAM drive. That is a "fake" disk, which uses the computers memory instead of a hard drive. It's very fast, because memory is much faster than a hard drive, including solid state drives.

The only problem with a RAM drive is that the contents are lost when the machine shuts down, unless steps are taken to copy the RAM drive's contents or if you use a RAM drive that copies data to a traditional disk to preserve a copy.

Warnings

My set up is experimental and it is described below for those of you with the IT skill needed to try it yourself.

Since this is an experimental trial, I do not recommend trying this on a production machine. Also, backup your machine before making any changes. There is a risk of corruption in the NT database if your machine crashes or shuts down abnormally, but that risk can be handled with regular backups and by using an uninterruptable power supply (UPS).


Details (which will be most helpful to those of you with IT knowledge)

I have found it frustrating that it has taken about 46 seconds to change the time frame on a chart with NT 7, running eleven charts and a number of indicators, including GomiCD. NT locks up and all charts freeze until NT completes the change of settings. This is on a reasonably good, quad core machine with a solid state hard drive and 8 GB of memory, of which about 4 GB in use.

To test if NT can be sped up, I installed a freeware RAM drive [Dataram RAMDisk] to host the directory NT database directory [1] by creating a symbolic link to another directory on the RAM drive, using Junction [2]. As far as NT is concerned, there is no change to the file structure, but NT gains very fast read and write access to the database.

The rate of reading the database is important to performance and that was not good enough with a solid state hard drive.


The first test involved the same method to relocate the Gomi files to the RAM drive, but that had no obvious effect on NT's performance.


This experimental arrangement uses the load and save feature of Dataram's RAMDisk to preserve the data on shutdown. THere is also an option to save the contents on a schedule. For belt and brace protection, I also use robocopy.


I have since noticed Primo RAM disk, which is a pay for application that has an option to save the contents in real time [3], but I have not tested that.


[1] C:\Users\Pawnbroker\Documents\NinjaTrader 7\db
[2] Junction
[3] Primo Ramdisk Specifications and Features

funny you mention this, I was looking at this today for Visual Studio, as you said its fast, but you lose it all if the machine reboots for no reason at all. I know there are ways to save the content, but I've heard it tends to be slow.

Reply With Quote
Thanked by:
  #3 (permalink)
 Koepisch 
@ Germany
 
Experience: Beginner
Platform: NinjaTrader
Broker: Mirus Futures/Zen-Fire
Trading: FDAX
Posts: 569 since Nov 2011
Thanks Given: 440
Thanks Received: 518


Hi pawnbroker, i've made the same test with SSD vs. RamDisk and couldn't measure a performance boost. But i don't had many charts open. I think it relies more at the barstyle, timeframe and the days to load. I'm using SuperSpeed RamDisk which is a very reliable product and high end SSDs. It's fine if it works for you. For a production machine i don't recommend using any other software products other then ninjatrader or similar.

Reply With Quote
Thanked by:
  #4 (permalink)
 pawnbroker 
Cheltenham
 
Experience: Advanced
Platform: InvestorRT, NinjaTrader
Trading: ES
Posts: 54 since Jan 2012
Thanks Given: 8
Thanks Received: 107


tellytub View Post
funny you mention this, I was looking at this today for Visual Studio, as you said its fast, but you lose it all if the machine reboots for no reason at all. I know there are ways to save the content, but I've heard it tends to be slow.


Robocopy is very fast if you run that frequently as a scheduled task, but it would be better to use real time synchronisation as that would reduce the loss of data to an absolute minimum. WIth real time sync, there would be no need for a slow "save on shutdown" option.

Primo Ramdisk has an option for real time replication of data to a hard drive, but I have not tested that yet. There are also products that do real time sync, such as BeyondSync and that could be used to keep a "safe" copy on a SSD. So you could create a RAM drive and use native sync with Primo or another application like BeyondSync with a freeware RAM drive.


A RAM drive can work well with NT, since that writes new data once and it read the same data many times. In addition, the amount of data written by NT and Gomi is low, so real time sync would not have an effect on performance.

If Visual Studio's performance has a bottle neck relating to the disk, you can try a RAM disk safely using Junction to point the existing directory to a replica on a RAM drive. If you don't gain a benefit, just delete the symbolic link on the C drive and restore the files from the RAM drive to the original location.

If it does give a large gain, try the Primo disk or BeyindSync or robocopy to keep a safe copy.

Started this thread Reply With Quote
Thanked by:
  #5 (permalink)
 Koepisch 
@ Germany
 
Experience: Beginner
Platform: NinjaTrader
Broker: Mirus Futures/Zen-Fire
Trading: FDAX
Posts: 569 since Nov 2011
Thanks Given: 440
Thanks Received: 518

Keep in mind, that win has an sophisticated caching system which means that if you load thinks twice it's mostly loading from win's file cache and not from the RamDisk/HD anymore.

Reply With Quote
Thanked by:
  #6 (permalink)
 pawnbroker 
Cheltenham
 
Experience: Advanced
Platform: InvestorRT, NinjaTrader
Trading: ES
Posts: 54 since Jan 2012
Thanks Given: 8
Thanks Received: 107


tellytub View Post
funny you mention this, I was looking at this today for Visual Studio, as you said its fast, but you lose it all if the machine reboots for no reason at all. I know there are ways to save the content, but I've heard it tends to be slow.


Koepisch View Post
Keep in mind, that win has an sophisticated caching system which means that if you load thinks twice it's mostly loading from win's file cache and not from the RamDisk/HD anymore.

I considered this and I did not think that it was helping in my use of NT.

If a cache was helping NT then that would be seen by NT taking a long time to process changes (e.g. 3 min chart changed to 5 min chart) on the first trial, but it should be faster on the second trial done soon after, because the files would be cached in memory. However, my tests showed no speed gain on repeating the test.

Also, I don't change settings often, but I want them applied fast when I do make a change. I would expect a cache to flush files that have not been accessed recently, so the NT database may not be cached when I make a change.

For those reasons, I don't expect a general file cache to work in principle and my tests found that it does not help in practice on Windows 7 x64.

In contrast, a RAM disk ensures that the files are always fast to read and I can control the amount of memory used. The only problem is preserving the content on shut down, but that feature is provided by some RAM drive products.

Started this thread Reply With Quote
Thanked by:
  #7 (permalink)
 
Zondor's Avatar
 Zondor 
Portland Oregon, United States
 
Experience: Beginner
Platform: NinjatraderŽ
Broker: CQG, Kinetick
Trading: Gameplay KlownbineŽ Trading of Globex
Posts: 1,333 since Jul 2009
Thanks Given: 1,246
Thanks Received: 2,731


Quoting 
The time taken by NT to change the time frame of a chart from 3 to 5 minutes was 46 seconds and that reduced to 11 seconds after moving the NT database to a RAM drive. That is a "fake" disk, which uses the computers memory instead of a hard drive. It's very fast, because memory is much faster than a hard drive, including solid state drives.

This is interesting and worthwhile, but there is something else to consider.

What indicators do you have on these charts? It's at least as important to reduce resources wasted by inefficient, sloppily coded software, as to look for a way to increase the speed at which that software travels along a 500 mile path to go across the street.

I've found that improving the way an indicator is coded will greatly reduce its loading time and real time CPU load, especially with COBC false. Applied to many indicators and charts, the responsiveness of Ninjatrader is greatly improved.

Only then will memory access improvements deliver to their full potential.

.

Follow me on Twitter Visit my NexusFi Trade Journal Reply With Quote
  #8 (permalink)
 
Big Mike's Avatar
 Big Mike 
Manta, Ecuador
Site Administrator
Developer
Swing Trader
 
Experience: Advanced
Platform: Custom solution
Broker: IBKR
Trading: Stocks & Futures
Frequency: Every few days
Duration: Weeks
Posts: 50,396 since Jun 2009
Thanks Given: 33,172
Thanks Received: 101,534

Very interesting.

But I don't understand why Windows is not doing its job of caching the database files in RAM to begin with. If it is being frequently used, and you have sufficient memory, then it should already be in memory. It could be that 8GB is simply not enough for your workstation in order for Windows to do this.

I am not sure if Windows has support for some type of O_DIRECT flag like exists on Linux, and that perhaps for some unknown reason the developers of NinjaTrader are purposely instructing Windows to do direct reads instead of cached reads?

How big is your database folder?

Mike

We're here to help: just ask the community or contact our Help Desk

Quick Links: Change your Username or Register as a Vendor
Searching for trading reviews? Review this list
Lifetime Elite Membership: Sign-up for only $149 USD
Exclusive money saving offers from our Site Sponsors: Browse Offers
Report problems with the site: Using the NexusFi changelog thread
Follow me on Twitter Visit my NexusFi Trade Journal Reply With Quote
  #9 (permalink)
 
bob7123's Avatar
 bob7123 
Amsterdam, The Netherlands
Legendary Market Wizard
 
Experience: Intermediate
Platform: Ninja, IRT, ToS
Broker: EdgeClear, LMAX
Trading: NQ
Posts: 668 since Oct 2011
Thanks Given: 134
Thanks Received: 2,682


pawnbroker View Post
I have been able to make a large performance gain in NT 7 by storing the NT database files on a RAM drive and I thought that I would share what I have done.

The time taken by NT to change the time frame of a chart from 3 to 5 minutes was 46 seconds and that reduced to 11 seconds after moving the NT database to a RAM drive.

What Luck! A rainy Saturday here, and a new geeky thing to try. Cool, thanks!

I got everything working as you suggested. WOW charts load faster. Previously I thought delays were due to the IQFeed servers being slow, but I guess not. Great thread!

I found the freeware Dataram RAMDisk worked fine for me, available here. You don't have to pay anything unless you want to make a Ramdrive that is more than 4 GB. I made my RamDrive 1.5GB and it has tons of free space. It is set to backup to my hard drive every 5 minutes, so I am not worried about losing data.

Of course, I couldn't stop there though... I moved my Firefox cache to the ramdrive too. (I'm one of those people that keep a zillion tabs open.) Old but still correct instructions to do that are here: How to Change Cache Size and Location in Firefox

Also, the junction tool you mentioned is not needed after XP, so on my win7 machine I got this GUI tool to create the links:
LinkShellExtension
. I suggest that anyone who gets the LSE might want to read all of the documentation. Not only are junctions supported, but a WHOLE bunch of other options too.

Finally, in regards to SSDs, I have a SSD C drive, and a regular D drive. I manually moved most of my USERS folder to D. I did this so that files that are written to more often are not stored on the SSD, as writing to the SSD drive can degrade it over time. It looks like there is a better way to do that when doing a clean install, using something called "Audit mode" that I had never heard of.

Link to that info is here: Microsoft Answers Page

I've got a new bigger SSD on order, when it gets here I plan to use it to reinstall a fresh copy of windows to get rid of a lot of unused apps. I'll also use the link above to keep more files on the D drive.

And for laughs, I am including a screenshot of the results of HD Tune for my SSD, regular drive and RamDrive.

Attached Thumbnails
Click image for larger version

Name:	DiskPerformance.jpg
Views:	621
Size:	381.6 KB
ID:	93230  
Follow me on Twitter Reply With Quote
  #10 (permalink)
 
Zondor's Avatar
 Zondor 
Portland Oregon, United States
 
Experience: Beginner
Platform: NinjatraderŽ
Broker: CQG, Kinetick
Trading: Gameplay KlownbineŽ Trading of Globex
Posts: 1,333 since Jul 2009
Thanks Given: 1,246
Thanks Received: 2,731


Question for @bob7123 and @pawnbroker:

The size of my Documents\NinjaTrader7\db folder is 28 GB, most of that being in the data folder.

Will this work with a 1.5 GB RAMDisk partition?

If so could you give a more detailed description of how to create the links using LinkShellExtension?

Follow me on Twitter Visit my NexusFi Trade Journal Reply With Quote
Thanked by:




Last Updated on August 6, 2020


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts