Speeding up NinjaTrader with a RAM drive (ramdisk)
I have been able to make a large performance gain in NT 7 by storing the NT database files on a RAM drive and I thought that I would share what I have done.
The time taken by NT to change the time frame of a chart from 3 to 5 minutes was 46 seconds and that reduced to 11 seconds after moving the NT database to a RAM drive. That is a "fake" disk, which uses the computers memory instead of a hard drive. It's very fast, because memory is much faster than a hard drive, including solid state drives.
The only problem with a RAM drive is that the contents are lost when the machine shuts down, unless steps are taken to copy the RAM drive's contents or if you use a RAM drive that copies data to a traditional disk to preserve a copy.
My set up is experimental and it is described below for those of you with the IT skill needed to try it yourself.
Since this is an experimental trial, I do not recommend trying this on a production machine. Also, backup your machine before making any changes. There is a risk of corruption in the NT database if your machine crashes or shuts down abnormally, but that risk can be handled with regular backups and by using an uninterruptable power supply (UPS).
Details (which will be most helpful to those of you with IT knowledge)
I have found it frustrating that it has taken about 46 seconds to change the time frame on a chart with NT 7, running eleven charts and a number of indicators, including GomiCD. NT locks up and all charts freeze until NT completes the change of settings. This is on a reasonably good, quad core machine with a solid state hard drive and 8 GB of memory, of which about 4 GB in use.
To test if NT can be sped up, I installed a freeware RAM drive [Dataram RAMDisk] to host the directory NT database directory  by creating a symbolic link to another directory on the RAM drive, using Junction . As far as NT is concerned, there is no change to the file structure, but NT gains very fast read and write access to the database.
The rate of reading the database is important to performance and that was not good enough with a solid state hard drive.
The first test involved the same method to relocate the Gomi files to the RAM drive, but that had no obvious effect on NT's performance.
This experimental arrangement uses the load and save feature of Dataram's RAMDisk to preserve the data on shutdown. THere is also an option to save the contents on a schedule. For belt and brace protection, I also use robocopy.
I have since noticed Primo RAM disk, which is a pay for application that has an option to save the contents in real time , but I have not tested that.
funny you mention this, I was looking at this today for Visual Studio, as you said its fast, but you lose it all if the machine reboots for no reason at all. I know there are ways to save the content, but I've heard it tends to be slow.
The following user says Thank You to tellytub for this post:
Hi pawnbroker, i've made the same test with SSD vs. RamDisk and couldn't measure a performance boost. But i don't had many charts open. I think it relies more at the barstyle, timeframe and the days to load. I'm using SuperSpeed RamDisk which is a very reliable product and high end SSDs. It's fine if it works for you. For a production machine i don't recommend using any other software products other then ninjatrader or similar.
The following user says Thank You to Koepisch for this post:
Robocopy is very fast if you run that frequently as a scheduled task, but it would be better to use real time synchronisation as that would reduce the loss of data to an absolute minimum. WIth real time sync, there would be no need for a slow "save on shutdown" option.
Primo Ramdisk has an option for real time replication of data to a hard drive, but I have not tested that yet. There are also products that do real time sync, such as BeyondSync and that could be used to keep a "safe" copy on a SSD. So you could create a RAM drive and use native sync with Primo or another application like BeyondSync with a freeware RAM drive.
A RAM drive can work well with NT, since that writes new data once and it read the same data many times. In addition, the amount of data written by NT and Gomi is low, so real time sync would not have an effect on performance.
If Visual Studio's performance has a bottle neck relating to the disk, you can try a RAM disk safely using Junction to point the existing directory to a replica on a RAM drive. If you don't gain a benefit, just delete the symbolic link on the C drive and restore the files from the RAM drive to the original location.
If it does give a large gain, try the Primo disk or BeyindSync or robocopy to keep a safe copy.
The following 2 users say Thank You to pawnbroker for this post:
I considered this and I did not think that it was helping in my use of NT.
If a cache was helping NT then that would be seen by NT taking a long time to process changes (e.g. 3 min chart changed to 5 min chart) on the first trial, but it should be faster on the second trial done soon after, because the files would be cached in memory. However, my tests showed no speed gain on repeating the test.
Also, I don't change settings often, but I want them applied fast when I do make a change. I would expect a cache to flush files that have not been accessed recently, so the NT database may not be cached when I make a change.
For those reasons, I don't expect a general file cache to work in principle and my tests found that it does not help in practice on Windows 7 x64.
In contrast, a RAM disk ensures that the files are always fast to read and I can control the amount of memory used. The only problem is preserving the content on shut down, but that feature is provided by some RAM drive products.
The following 2 users say Thank You to pawnbroker for this post:
Favorite Futures: Gameplay Klownbine® Trading of Globex
Posts: 1,276 since Jul 2009
Thanks: 1,227 given,
Don't forget about code quality
This is interesting and worthwhile, but there is something else to consider.
What indicators do you have on these charts? It's at least as important to reduce resources wasted by inefficient, sloppily coded software, as to look for a way to increase the speed at which that software travels along a 500 mile path to go across the street.
I've found that improving the way an indicator is coded will greatly reduce its loading time and real time CPU load, especially with COBC false. Applied to many indicators and charts, the responsiveness of Ninjatrader is greatly improved.
Only then will memory access improvements deliver to their full potential.
But I don't understand why Windows is not doing its job of caching the database files in RAM to begin with. If it is being frequently used, and you have sufficient memory, then it should already be in memory. It could be that 8GB is simply not enough for your workstation in order for Windows to do this.
I am not sure if Windows has support for some type of O_DIRECT flag like exists on Linux, and that perhaps for some unknown reason the developers of NinjaTrader are purposely instructing Windows to do direct reads instead of cached reads?
How big is your database folder?
Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.
Need help? 1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first. 2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses. 3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make. 4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance. 5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers. 6) Help using the forum? Watch this video to learn general tips on using the site.
If you want to support our community, become an Elite Member.
What Luck! A rainy Saturday here, and a new geeky thing to try. Cool, thanks!
I got everything working as you suggested. WOW charts load faster. Previously I thought delays were due to the IQFeed servers being slow, but I guess not. Great thread!
I found the freeware Dataram RAMDisk worked fine for me, available here. You don't have to pay anything unless you want to make a Ramdrive that is more than 4 GB. I made my RamDrive 1.5GB and it has tons of free space. It is set to backup to my hard drive every 5 minutes, so I am not worried about losing data.
Of course, I couldn't stop there though... I moved my Firefox cache to the ramdrive too. (I'm one of those people that keep a zillion tabs open.) Old but still correct instructions to do that are here: How to Change Cache Size and Location in Firefox
Also, the junction tool you mentioned is not needed after XP, so on my win7 machine I got this GUI tool to create the links:
LinkShellExtension. I suggest that anyone who gets the LSE might want to read all of the documentation. Not only are junctions supported, but a WHOLE bunch of other options too.
Finally, in regards to SSDs, I have a SSD C drive, and a regular D drive. I manually moved most of my USERS folder to D. I did this so that files that are written to more often are not stored on the SSD, as writing to the SSD drive can degrade it over time. It looks like there is a better way to do that when doing a clean install, using something called "Audit mode" that I had never heard of.
I've got a new bigger SSD on order, when it gets here I plan to use it to reinstall a fresh copy of windows to get rid of a lot of unused apps. I'll also use the link above to keep more files on the D drive.
And for laughs, I am including a screenshot of the results of HD Tune for my SSD, regular drive and RamDrive.
The following 10 users say Thank You to bob7123 for this post: