Speeding up NinjaTrader with a RAM drive (ramdisk) - NinjaTrader | futures io social day trading
futures io futures trading


Speeding up NinjaTrader with a RAM drive (ramdisk)
Updated: Views / Replies:25,260 / 110
Created: by pawnbroker Attachments:6

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors – all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you don’t need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 6  
 
Thread Tools Search this Thread
 

Speeding up NinjaTrader with a RAM drive (ramdisk)

  #11 (permalink)
Elite Member
Amsterdam, The Netherlands
 
Futures Experience: Intermediate
Platform: Sierra, IRT, ToS, Ninja
Broker/Data: Stage 5
Favorite Futures: NQ
 
bob7123's Avatar
 
Posts: 588 since Oct 2011
Thanks: 109 given, 1,908 received


Big Mike View Post
But I don't understand why Windows is not doing its job of caching the database files in RAM to begin with.

Well, that is a good question, but unfortunately somewhat rhetorical, as it is hard to know what NT is doing under the hood, or what windows is doing for that matter. I just know the RAMDrive helps.

Interestingly, Pawnbroker noticed that putting the Gomfolder on the ramdrive did not help. Purely speculation, but I think it is possible that our favorite Parisian puts a little more time into thinking about performance than other coders.

And to further speculate, I think one problem may be the popular yet dangerous idea that "abstracter is better." I know I found myself pissing up a rope trying implement a SQL back end for gomi data. ...But it seemed so beautiful on paper...

Coincidentally, I ran in a a Yogi Berra quote yesterday. "In theory, there is no difference between theory and practice, but in practice, there is."


Zondor View Post
Question for @bob7123 and @pawnbroker:

The size of my Documents\NinjaTrader7\db folder is 28 GB, most of that being in the data folder.

Will this work with a 1.5 GB RAMDisk partition?

If so could you give a more detailed description of how to create the links using LinkShellExtension?

WHOA! Mine is twenty-someodd MEGA bytes. I just did a recent reinstall of NT, but even my old Ninja folder was only about 150 MB.

My understanding is the DB folder is analogous to a browser cache, in that if something is missing then it tries to get it from the datafeed. I would check with Ninja, or at least do a good backup, but you may be able to clear a lot of that out.

But to answer your question, no it will not fit. There is no compression on the RAMDrive or the image backup it does. (Maybe with Primo but not the one I used)

Tying up a ton of memory would seem like an expensive last resort to me. I'd try emptying the folders (after backup of course) and see if everything still works. If so, then you should be fine with a smaller RAMDrive.

And please share your results. I'm curious to know what my options will be when my RAMDrive fills up too.

-Bob

Reply With Quote
The following user says Thank You to bob7123 for this post:
 
  #12 (permalink)
Elite Member
Cheltenham
 
Futures Experience: Advanced
Platform: InvestorRT, NinjaTrader
Favorite Futures: ES
 
Posts: 54 since Jan 2012
Thanks: 8 given, 100 received

The Slow Indicator

I have found the resource hog that is slowing down NT; it's the emini-watch.com indicators _Better_Momentum and _Better_Momemntum2 (latest versions- Better_Indicators_Ninja_25Aug12_jsy).

Here are some times, starting with just a candle chart and changing between 3 to 5 minutes charts.

Candles only = instant.
with Better_Momentum = 9s
with Better_Momentum2 = 9s
with both indicators = 16s

These tests were done with and without a RAM cache and the times were the same. NT was not recording data at the time, because the markets were closed.

Note that NT and my PC run OK whith these indicators, but changing settings takes a long time. There is not much I can do about this directly as the code is closed source, but I can contact the emini-watch and notify the author that he needs a "better" coder.

Size of my NT database

My NT db directory is 218 MB, so it easy for me to cache all of that in a RAM drive.


Notes on Caching SSD's

There are a number of sites on the WWW having recommendations on settings to optimise SSD's. It's common to state that you should disable prefetch and superfetch [e.g. 1 & 2].

"The purpose of these is to pre-load the programs you load from slow hard drive to fast memory (cache) in case you want to run them. With your SSD, there is no need. Disable them and free up some memory and resources and stop a lot of writes to the SSD. However, this may hurt performance in older first gen SSDs and HDDs".


Disabling prefetch makes sense to increase the life of your SSD, but there is debate about disabling superfetch. Read the article on that URL and the comments. If you do turn off superfetch you will disable caching and that may slow down NT (if you don't use a RAM drive).


RAM Disk benefits

NT was no faster with the NT database on a RAM drive when I tested today, with the markets closed. However, it was much faster when I tested in the week with the markets open. More testing is needed to check if my fists tests were accurate, but the activity from processing new data would slow things down in principle.

RAM Disk alternatives

I installed SuperCache 5 today, which can be used to add a memory cache for a disk or to mirror an entire disk in memory. No improvement was seen in NT's speed, but note the comments in the RAM disk benefits section, as the tests were done with the markets closed.

There is a competing comment, FancyCache, which is in beta testing and free to use for 180 days. The product is stable as far as I can tell from forum posts.

I will retest in the week, when NT has market data is coming in.

Both supercache and Fancy cache can optionally cache writes to disk. I strongly recommend not using that option, because you will loose data and risk system corruption if your machine crashes or shuts down before data is written from memory to disk. Also, NT will benefit much more by having ultra fast read speeds and I would expect very little benefit to NT from caching writes.


References

[1] Sean's Windows 7 Install & Optimization Guide for SSDs & HDDs
[2] How to: Disable Prefetch and Superfetch in Windows 7 - YouTube

Reply With Quote
The following 4 users say Thank You to pawnbroker for this post:
 
  #13 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,240 since Jun 2009
Thanks: 29,352 given, 83,223 received



bob7123 View Post
WHOA! Mine is twenty-someodd MEGA bytes. I just did a recent reinstall of NT, but even my old Ninja folder was only about 150 MB.

My understanding is the DB folder is analogous to a browser cache, in that if something is missing then it tries to get it from the datafeed. I would check with Ninja, or at least do a good backup, but you may be able to clear a lot of that out.

It stores all chart data. Mine is many GB in size because I have years of tick data.

After you reach your conclusions in the thread, I encourage you to post in the NT8 request thread. This is something that NT should be doing under the hood in order to improve speed, caching these files in memory as needed.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
The following 2 users say Thank You to Big Mike for this post:
 
  #14 (permalink)
Elite Member
Portland Oregon, United States
 
Futures Experience: Beginner
Platform: Ninjatrader®
Broker/Data: CQG, Kinetick
Favorite Futures: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,327 since Jul 2009
Thanks: 1,246 given, 2,635 received

NT Database Size, Better Momentum Indicator

Maybe instead of the entire db folder, only recent vintage datastore folders need to placed on the RAMDisk.

By moving old data files to an archive outside of the db folder I reduced the folder size to 500 MB so now would be able to use the RAMDisk to contain the entire db folder, or just the data files. I would assume that the database engine gets loaded into RAM anyway and does not need to be on the RAMDisk.

I am already using Windows ReadyBoost, but have not tried to measure any improvement that it might be giving.

Meanwhile I don't think you cant go wrong by putting in as much RAM as possible, as Mike suggests in next post, and putting any spare memory sticks to work as Ready Boost caches.

The Better Momentum Indicator is said to be a momentum oscillator applied to buy sell volume. There are a few different ways to do this and the slow loading time of tbe emini-watch version could probably be improved:
  • Take the Momentum of GomCD. Fast loading. You can use BidAsk buy sell volume if you have Gom data, but even if you don't have Gom data, you can use UpDownTick data from the Ninja Tick files.
  • For real time data use momentum of bid ask volume from OnMarketDepth and for historical data use Momentum of On Balance Volume or Momentum of UpDownTick Volume of a secondary data series. If the secondary data series has a high resolution, such as 1 tick, the indicator will be very slow to load, especially if days back is more than one.

For many days back the most practical way would be to either use OBV or to use a moderately coarse secondary data series, such as around 2000 tick for the ES.

After plotting the oscillator curve you would have to add the features that show the signals.

If anyone is interested in pursuing this we will have to create an Elite thread since Gom indicators fall under that category,


Last edited by Zondor; October 28th, 2012 at 07:03 PM.
Reply With Quote
The following user says Thank You to Zondor for this post:
 
  #15 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,240 since Jun 2009
Thanks: 29,352 given, 83,223 received

Memory is dirt cheap right now. I just paid like $120 for 32GB for a new server. 16GB would be roughly half that.

You can check Windows 7 memory limits here
Memory Limits for Windows Releases

@Zondor, please keep in mind this is a non-Elite thread so do not post Elite indicators (Gom*)

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
The following 2 users say Thank You to Big Mike for this post:
 
  #16 (permalink)
Elite Member
Cheltenham
 
Futures Experience: Advanced
Platform: InvestorRT, NinjaTrader
Favorite Futures: ES
 
Posts: 54 since Jan 2012
Thanks: 8 given, 100 received


Zondor View Post
If anyone is interested in pursuing this we will have to create an Elite thread since Gom indicators fall under that category,

A comparison of the peak heights of GomMomentum swings gives a very similar assessment of the market conditions and the strength of the buying and selling (or lack thereof). That is not a clone of his work, so it does not cause me any moral problems to mention that!

Reply With Quote
 
  #17 (permalink)
Elite Member
Portland Oregon, United States
 
Futures Experience: Beginner
Platform: Ninjatrader®
Broker/Data: CQG, Kinetick
Favorite Futures: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,327 since Jul 2009
Thanks: 1,246 given, 2,635 received

DB Data Folders Moved to RAMDisk, working fine so far

My PC is now running a 1.5 GB RAMDisk containing the cache, data, day, tick and minute folders moved from the db folder, where there are now junctions linking to them. My workspaces do seem to be loading quite a bit faster, although I have not taken any measurements.

I don't think it's a serious problem if the data in these folders gets lost. They would then be automatically rebuilt from server supplied historical data. I am only backing up the RAMDisk about once per hour, and on computer shutdown.

I am assuming that the folder called NinjaTrader that contains the SQL Server CE Database does not include any data files, so there would be nothing gained by moving it to the RAMDisk.

Reply With Quote
 
  #18 (permalink)
Elite Member
Amsterdam, The Netherlands
 
Futures Experience: Intermediate
Platform: Sierra, IRT, ToS, Ninja
Broker/Data: Stage 5
Favorite Futures: NQ
 
bob7123's Avatar
 
Posts: 588 since Oct 2011
Thanks: 109 given, 1,908 received


pawnbroker View Post
The Slow Indicator

I have found the resource hog that is slowing down NT; it's the emini-watch.com indicators

ROFL! I guess it is "Better" to pay for a proprietary indicator that breaks your system than pay to become an elite member and get indicators that are open and have a community of people who will give you their unbiased opinions of their real value.


pawnbroker View Post
Notes on Caching SSD's

There are a number of sites on the WWW having recommendations on settings to optimise SSD's. It's common to state that you should disable prefetch and superfetch [e.g. 1 & 2].

"The purpose of these is to pre-load the programs you load from slow hard drive to fast memory (cache) in case you want to run them. With your SSD, there is no need. Disable them and free up some memory and resources and stop a lot of writes to the SSD. However, this may hurt performance in older first gen SSDs and HDDs".

Disabling prefetch makes sense to increase the life of your SSD, but there is debate about disabling superfetch. Read the article on that URL and the comments. If you do turn off superfetch you will disable caching and that may slow down NT (if you don't use a RAM drive).

My NinjaTrader folder is on D, which is not SSD.

I use this tool to set up my SSD: SSD Tweaker by Elpamsoft.com
I'll stay out of which parameter settings are best, but this tool makes it easy to set them from one convenient panel.

It also allows me to initiate a TRIM cleanup and ensure I the OS is using AHCI drivers. (Which are a lot faster than older ones.)

Reply With Quote
The following user says Thank You to bob7123 for this post:
 
  #19 (permalink)
Elite Member
Amsterdam, The Netherlands
 
Futures Experience: Intermediate
Platform: Sierra, IRT, ToS, Ninja
Broker/Data: Stage 5
Favorite Futures: NQ
 
bob7123's Avatar
 
Posts: 588 since Oct 2011
Thanks: 109 given, 1,908 received


Big Mike View Post
It stores all chart data. Mine is many GB in size because I have years of tick data.

How did you come by this data? Or more to the point, what is the quality? Did you just keep NT running on a server someplace, or did you pony up for tick data? Can any of it be converted to binary Gom files?

In thinking about how the backfill application could be extended, I think it would be nice to have a cataloging database to know what I have and if there are any gaps in it.


Big Mike View Post
After you reach your conclusions in the thread, I encourage you to post in the NT8 request thread. This is something that NT should be doing under the hood in order to improve speed, caching these files in memory as needed.


Big Mike View Post
Memory is dirt cheap right now. I just paid like $120 for 32GB for a new server. 16GB would be roughly half that.

Well, that's the thing. I cut out a comment I thought was a bit snarky in my previous post, but I guess remains relevant.

I was thinking that all the nice new transistors we get courtesy of Moore's law are wasted by increasingly bloated software. I'll be happy to tell the NT folks to write cleaner code, but I doubt there will be much beyond lip service; a lot of developers these days just figure you can always throw more hardware at the problem.

Reply With Quote
 
  #20 (permalink)
Elite Member
Amsterdam, The Netherlands
 
Futures Experience: Intermediate
Platform: Sierra, IRT, ToS, Ninja
Broker/Data: Stage 5
Favorite Futures: NQ
 
bob7123's Avatar
 
Posts: 588 since Oct 2011
Thanks: 109 given, 1,908 received



Zondor View Post
I am assuming that the folder called NinjaTrader that contains the SQL Server CE Database does not include any data files, so there would be nothing gained by moving it to the RAMDisk.

I think you are right, but one tool I ran across if anyone ever wants to open it up and have a look inside:

SQL Compact Query Analyzer

Reply With Quote

Reply



futures io > > > > Speeding up NinjaTrader with a RAM drive (ramdisk)

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Jigsaw Trading: TBA

Elite only

FuturesTrader71: TBA

Elite only

NinjaTrader: TBA

Jan 18

RandBots: TBA

Jan 23

GFF Brokers & CME Group: Futures & Bitcoin

Elite only

Adam Grimes: TBA

Elite only

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
.....SEC May Ticket Speeding Traders kbit News and Current Events 0 February 23rd, 2012 04:08 PM
NT7.4 - RAM hog kashter NinjaTrader 40 April 14th, 2011 02:10 PM
Drive cory Off-Topic 3 October 1st, 2010 04:17 PM
ram nt7 vps MyTraderAdvisor NinjaTrader 5 August 18th, 2010 04:49 PM
Toyota Blames Driver Error for Unwanted Speeding Quick Summary News and Current Events 0 July 14th, 2010 03:20 AM


All times are GMT -4. The time now is 06:55 PM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-13 in 0.18 seconds with 20 queries on phoenix via your IP 54.90.92.204