NexusFi: Find Your Edge


Home Menu

 





Speeding up NinjaTrader with a RAM drive (ramdisk)


Discussion in NinjaTrader

Updated
      Top Posters
    1. looks_one Big Mike with 20 posts (13 thanks)
    2. looks_two Zondor with 18 posts (24 thanks)
    3. looks_3 kronie with 14 posts (2 thanks)
    4. looks_4 bob7123 with 12 posts (15 thanks)
      Best Posters
    1. looks_one pawnbroker with 3.4 thanks per post
    2. looks_two Zondor with 1.3 thanks per post
    3. looks_3 bob7123 with 1.3 thanks per post
    4. looks_4 Big Mike with 0.7 thanks per post
    1. trending_up 60,629 views
    2. thumb_up 94 thanks given
    3. group 41 followers
    1. forum 111 posts
    2. attach_file 6 attachments




 
Search this Thread

Speeding up NinjaTrader with a RAM drive (ramdisk)

  #71 (permalink)
 
Big Mike's Avatar
 Big Mike 
Manta, Ecuador
Site Administrator
Developer
Swing Trader
 
Experience: Advanced
Platform: Custom solution
Broker: IBKR
Trading: Stocks & Futures
Frequency: Every few days
Duration: Weeks
Posts: 50,440 since Jun 2009
Thanks Given: 33,215
Thanks Received: 101,599

I've been trying out PrimoCache. Under most circumstances it makes more sense than a dedicated ramdisk, and will offer the same performance benefits.

I am using a delayed write setting of 60 seconds. I see NinjaTrader writing 50,000 blocks per minute during large backtests, so this can really help both in terms of performance as well as lifespan (SSD wear level).

I am using an 8GB cache and with my normal day to day stuff, my cache hit ratio is only around 25%. If I increased my cache size to 16GB I would have much higher ratio, as I've only read 16.4GB from disk total since my last reboot. The problem is, I need around 20GB of memory to start Sierra Chart when it is loading all my charts (once it loads them, usage is only 1GB), so the max I can safely use for the cache disk is around 8GB. I have 32GB total.

Anyway, you might try it. No need for symlink and it will work with any program, not just ones you've configured to use the ramdrive.

Naturally you should always have a backup of any important data before using something like this.

Mike

We're here to help: just ask the community or contact our Help Desk

Quick Links: Change your Username or Register as a Vendor
Searching for trading reviews? Review this list
Lifetime Elite Membership: Sign-up for only $149 USD
Exclusive money saving offers from our Site Sponsors: Browse Offers
Report problems with the site: Using the NexusFi changelog thread
Follow me on Twitter Visit my NexusFi Trade Journal Reply With Quote
Thanked by:

Can you help answer these questions
from other members on NexusFi?
How to apply profiles
Traders Hideout
NT7 Indicator Script Troubleshooting - Camarilla Pivots
NinjaTrader
REcommedations for programming help
Sierra Chart
Increase in trading performance by 75%
The Elite Circle
Pivot Indicator like the old SwingTemp by Big Mike
NinjaTrader
 
Best Threads (Most Thanked)
in the last 7 days on NexusFi
Just another trading journal: PA, Wyckoff & Trends
36 thanks
Tao te Trade: way of the WLD
24 thanks
Spoo-nalysis ES e-mini futures S&P 500
20 thanks
Bigger Wins or Fewer Losses?
19 thanks
GFIs1 1 DAX trade per day journal
16 thanks
  #72 (permalink)
 
Big Mike's Avatar
 Big Mike 
Manta, Ecuador
Site Administrator
Developer
Swing Trader
 
Experience: Advanced
Platform: Custom solution
Broker: IBKR
Trading: Stocks & Futures
Frequency: Every few days
Duration: Weeks
Posts: 50,440 since Jun 2009
Thanks Given: 33,215
Thanks Received: 101,599


Big Mike View Post
I've been trying out PrimoCache. Under most circumstances it makes more sense than a dedicated ramdisk, and will offer the same performance benefits.

I am using a delayed write setting of 60 seconds. I see NinjaTrader writing 50,000 blocks per minute during large backtests, so this can really help both in terms of performance as well as lifespan (SSD wear level).

I am using an 8GB cache and with my normal day to day stuff, my cache hit ratio is only around 25%. If I increased my cache size to 16GB I would have much higher ratio, as I've only read 16.4GB from disk total since my last reboot. The problem is, I need around 20GB of memory to start Sierra Chart when it is loading all my charts (once it loads them, usage is only 1GB), so the max I can safely use for the cache disk is around 8GB. I have 32GB total.

Anyway, you might try it. No need for symlink and it will work with any program, not just ones you've configured to use the ramdrive.

Naturally you should always have a backup of any important data before using something like this.

Mike

Hmm oops, math a bit wrong. Total read 16.5GB but cached read just under 4GB, so it means since boot I've 16GB from disk, but after the initial read was performed, only 4GB has been re-read and thus was delivered from the cache. So 8GB is plenty for my daily needs looks like, could even get by with less.

I think this is mainly attributed to Sierra Chart, because the data is roughly 50GB but they store it compressed (NTFS compressed) on disk, at about 18GB total. This is why on startup it balloons to 20GB of memory usage, due to Windows decompressing the data stream as it builds my charts. And since I usually only start Sierra Chart once every month or two (about as often as I reboot) then there is no real "caching", because it has to read it from disk once anyway to feed it to the cache, and I don't re-read it because I don't ever restart Sierra.

Your particular applications/daily use could be different.

Mike

We're here to help: just ask the community or contact our Help Desk

Quick Links: Change your Username or Register as a Vendor
Searching for trading reviews? Review this list
Lifetime Elite Membership: Sign-up for only $149 USD
Exclusive money saving offers from our Site Sponsors: Browse Offers
Report problems with the site: Using the NexusFi changelog thread
Follow me on Twitter Visit my NexusFi Trade Journal Reply With Quote
  #73 (permalink)
 Jlab 
Brooklyn, New York/USA
 
Experience: Intermediate
Platform: NinjaTrader
Trading: Stocks
Posts: 1 since Sep 2013
Thanks Given: 0
Thanks Received: 0


Hi,

I have installed dataram drive but do not know what settings to use speed up NT7. Where do I move the db file? Can someone list the steps?

Example

Basic Settings

*Disk Size 1.5MB*
*FAT16 Partition*

Format Options

*Create Temp Dir*

Load/Save

*??*

ETC....?

Any help would be appreciated.

Reply With Quote
  #74 (permalink)
 cdk1212 
Calgary, Canada
 
Experience: Intermediate
Platform: Ninja Trader
Broker: AMP - CQG
Trading: 6E, 6J, ES & TF
Posts: 5 since May 2011
Thanks Given: 3
Thanks Received: 1

Thanks for the thread! I've been wondering what I could do to help speed things up with NT-7. I am running Win-7 Pro & have 32gb of RAM. Thinking of buying the latest extreme chip from Intel & up the RAM to 64.
I run about 140 charts imbedded into one chart for two different markets.

thanks,
CK









Reply With Quote
  #75 (permalink)
 Guss   is a Vendor
 
Posts: 138 since Aug 2011
Thanks Given: 60
Thanks Received: 86

I have been experiencing freezing issues with NY using zenfire data in fast market environment.
I was told by NT support, that it is the 3rdparty indicators that are causing this issue.The developer is not totally convinced about that.
After reading the posts in this thread about using Ram disk, does it resolve those freezing issues in fast markets using zenfire?
I used to have 6 workspaces opened with all indicators when I was using IB filtered data without any issue, now with zenfire I am using a single workspace, removed most of my 3rdparty indicators, and still having issues

I am running windows 7 64 BT on 8 GB ram with Intel i7-3820 CPU @ 3.6 GHz

Any feedback and help on this is highly appreciated.

Thanks in advance
Guss

Follow me on Twitter Visit my NexusFi Trade Journal Reply With Quote
  #76 (permalink)
 
Beljevina's Avatar
 Beljevina 
Toronto, Canada
 
Experience: Advanced
Platform: NinjaTrader, SierraCharts
Broker: Amp/CQG, IB
Trading: CL, GC, 6E, ES, RTY, EURUSD, GBPJPY, AUDUSD
Posts: 352 since Sep 2010


Guss View Post
I have been experiencing freezing issues with NY using zenfire data in fast market environment.
I was told by NT support, that it is the 3rdparty indicators that are causing this issue.The developer is not totally convinced about that.
After reading the posts in this thread about using Ram disk, does it resolve those freezing issues in fast markets using zenfire?
I used to have 6 workspaces opened with all indicators when I was using IB filtered data without any issue, now with zenfire I am using a single workspace, removed most of my 3rdparty indicators, and still having issues

I am running windows 7 64 BT on 8 GB ram with Intel i7-3820 CPU @ 3.6 GHz

It definitely sounds like indicators clogging things up. I've been here, and still kind of am today. The fact that you had no issues with IB 'condensed' or 'averaged' data is the telltale sign to me. Once you introduced a feed that is completely unfiltered, sending in all ticks - which increase significantly for often many instruments at the same time - as surges in volume (both buying and selling) come through in ES, CL, other indexes, etc - is the tip off to me.

I have not yet implemented this RAM disk thingy myself for other reasons, but I do not see it helping with the issue you have, as it is to speed loading of data from the SSD/harddisk (and writing), and I don't see the rush of market ticks or bid/ask data as a write throughput issue. When that data rushes in, it's NT doing something with it, that is causing the freezes, white screens/redraws, round blue "I'm busy" Windows cursor, etc.

I'm assuming most if not all indicators are set to 'Calculate on bar close = true'. Given the indicator removals you've mentioned and the drastic reduction in workspaces - on a highly capable CPU - I wonder if recreating & replacing each chart with a new chart that contains the exact same indicators would help, and then saving that as a completely new workspace. I mention this because I have had suspicions in the past where I feel there is something else 'clogging up' the chart in it's associated template XML file. I have seen too many quirky things with NT7 charts that are part of my workspace where, for example, I'm humming along day after day for weeks or months, and then all of a sudden, some indicators disappear off my chart. I have seen this occur on multiple machines and different installs. NT7 forums have reports like this, with the answer coming back 'pack up your log and template files' and send them to us. That is of course a time killing approach that realistically few undertake since, recreating or fixing the displayed chart is the quickest action. It's possible since the XML files are getting updated, parsed, etc by NT7, that it makes a mistake. And, if it can drop indicators, it can maybe be running others we're not seeing, or, having something else sucking up valuable CPU cycles. I have over the years undertaken this, and, it has cleared things up, for a time. Often, the 'clogging' returns over time, or, other unexplained issues start occurring.

I'm not looking to gripe, but it's a huge shortcoming that @NinjaTrader does not have some sort of utility to show CPU cycles used by each chart, or, say a breakdown by indicators. Hell, it is such a robust program, and if a much smaller memory footprint program like Chrome can offer it (via SHIFT-ESC), it's really upsetting that NT cant provide us an add on utility or build it into a release update. It would offer unbelievably untold benefit to us, as, we all have vastly different indicator needs & configurations.

One thing I am quite certain of, is that based on your described circumstances, there is no way you should be seeing that type of performance hit, which, ultimately points to a NT7 flaw, or inability to coexist with something, somewhere.

Follow me on Twitter Reply With Quote
  #77 (permalink)
 Guss   is a Vendor
 
Posts: 138 since Aug 2011
Thanks Given: 60
Thanks Received: 86

Thank you for your informative reply, much appreciated
will build a new workspace over the weekend, and test the performance again
Regards
Guss



Beljevina View Post
It definitely sounds like indicators clogging things up. I've been here, and still kind of am today. The fact that you had no issues with IB 'condensed' or 'averaged' data is the telltale sign to me. Once you introduced a feed that is completely unfiltered, sending in all ticks - which increase significantly for often many instruments at the same time - as surges in volume (both buying and selling) come through in ES, CL, other indexes, etc - is the tip off to me.

I have not yet implemented this RAM disk thingy myself for other reasons, but I do not see it helping with the issue you have, as it is to speed loading of data from the SSD/harddisk (and writing), and I don't see the rush of market ticks or bid/ask data as a write throughput issue. When that data rushes in, it's NT doing something with it, that is causing the freezes, white screens/redraws, round blue "I'm busy" Windows cursor, etc.

I'm assuming most if not all indicators are set to 'Calculate on bar close = true'. Given the indicator removals you've mentioned and the drastic reduction in workspaces - on a highly capable CPU - I wonder if recreating & replacing each chart with a new chart that contains the exact same indicators would help, and then saving that as a completely new workspace. I mention this because I have had suspicions in the past where I feel there is something else 'clogging up' the chart in it's associated template XML file. I have seen too many quirky things with NT7 charts that are part of my workspace where, for example, I'm humming along day after day for weeks or months, and then all of a sudden, some indicators disappear off my chart. I have seen this occur on multiple machines and different installs. NT7 forums have reports like this, with the answer coming back 'pack up your log and template files' and send them to us. That is of course a time killing approach that realistically few undertake since, recreating or fixing the displayed chart is the quickest action. It's possible since the XML files are getting updated, parsed, etc by NT7, that it makes a mistake. And, if it can drop indicators, it can maybe be running others we're not seeing, or, having something else sucking up valuable CPU cycles. I have over the years undertaken this, and, it has cleared things up, for a time. Often, the 'clogging' returns over time, or, other unexplained issues start occurring.

I'm not looking to gripe, but it's a huge shortcoming that @NinjaTrader does not have some sort of utility to show CPU cycles used by each chart, or, say a breakdown by indicators. Hell, it is such a robust program, and if a much smaller memory footprint program like Chrome can offer it (via SHIFT-ESC), it's really upsetting that NT cant provide us an add on utility or build it into a release update. It would offer unbelievably untold benefit to us, as, we all have vastly different indicator needs & configurations.

One thing I am quite certain of, is that based on your described circumstances, there is no way you should be seeing that type of performance hit, which, ultimately points to a NT7 flaw, or inability to coexist with something, somewhere.


Follow me on Twitter Visit my NexusFi Trade Journal Reply With Quote
  #78 (permalink)
 
kickmic's Avatar
 kickmic 
Melbourne, Victoria, Australia
 
Experience: Advanced
Platform: NinjaTrader + Gomicators
Broker: InteractiveBrokers, CQG
Trading: 6A
Posts: 281 since May 2011
Thanks Given: 57
Thanks Received: 343


Big Mike View Post
I've been trying out PrimoCache. Under most circumstances it makes more sense than a dedicated ramdisk, and will offer the same performance benefits.


Mike

Hi Mike,

Thanks for mentioning this option. I'll have a crack at setting this up later, if I get stuck I may have a couple of questions

Follow me on Twitter Reply With Quote
  #79 (permalink)
 
kronie's Avatar
 kronie 
NYC + NY / USA
 
Experience: Advanced
Platform: "I trade, therefore, I AM!"; Theme Song: "Atomic Dog!"
Trading: EMD, 6J, ZB
Posts: 796 since Oct 2009


pawnbroker View Post
Robocopy is very fast if you run that frequently as a scheduled task, but it would be better to use real time synchronisation as that would reduce the loss of data to an absolute minimum. WIth real time sync, there would be no need for a slow "save on shutdown" option.

Primo Ramdisk has an option for real time replication of data to a hard drive, but I have not tested that yet. There are also products that do real time sync, such as BeyondSync and that could be used to keep a "safe" copy on a SSD. So you could create a RAM drive and use native sync with Primo or another application like BeyondSync with a freeware RAM drive.


A RAM drive can work well with NT, since that writes new data once and it read the same data many times. In addition, the amount of data written by NT and Gomi is low, so real time sync would not have an effect on performance.

If Visual Studio's performance has a bottle neck relating to the disk, you can try a RAM disk safely using Junction to point the existing directory to a replica on a RAM drive. If you don't gain a benefit, just delete the symbolic link on the C drive and restore the files from the RAM drive to the original location.

If it does give a large gain, try the Primo disk or BeyindSync or robocopy to keep a safe copy.

Full and Free Software Downloads » >Primo Ramdisk Professional Edition 5.1.0

just on this description page alone, this one, might actually be worth paying for, what with all the additional features that accompany the actual allocation and maintenance of the virtual ram disk

perhaps I'll report back on my findings

Reply With Quote
  #80 (permalink)
 futuretrader 
Como Italy
 
Experience: Intermediate
Platform: Ninjatrader, customized
Trading: ES
Posts: 525 since Feb 2010
Thanks Given: 471
Thanks Received: 643


Hi all,

I have a notebook with a SSD as the primary disk and a SD for some extra storage. Would it make sense to move the GOMFOLDER and NT db to the SD in order to save wear and tear on the SSD? And would it possibly make some difference to NT getting a bit sluggish in high volume bursts?

Thanks

Reply With Quote




Last Updated on August 6, 2020


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts