Speeding up NinjaTrader with a RAM drive (ramdisk) - NinjaTrader | futures io social day trading
futures io futures trading


Speeding up NinjaTrader with a RAM drive (ramdisk)
Updated: Views / Replies:25,330 / 110
Created: by pawnbroker Attachments:6

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors – all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you don’t need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 6  
 
Thread Tools Search this Thread
 

Speeding up NinjaTrader with a RAM drive (ramdisk)

  #21 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,240 since Jun 2009
Thanks: 29,354 given, 83,237 received


bob7123 View Post
How did you come by this data? Or more to the point, what is the quality?

https://futures.io/elite-circle/7279-official-gomrecorder-qcollector-data-sharing-thread.html

https://futures.io/elite-circle/719-historical-es-s-p-500-futures-contract-tick-data-2003-up.html

https://futures.io/elite-circle/21664-using-mysql-storing-tick-data.html

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #22 (permalink)
Elite Member
Amsterdam, The Netherlands
 
Futures Experience: Intermediate
Platform: Sierra, IRT, ToS, Ninja
Broker/Data: Stage 5
Favorite Futures: NQ
 
bob7123's Avatar
 
Posts: 588 since Oct 2011
Thanks: 109 given, 1,909 received



...but how do you get it all into the NT DB folder?

---
DUMB QUESTION I NOW KNOW! But at least I found the answer myself...
Importing Historical data

Still, I wonder if storing all that data in NTD files is a good idea, as it is (essentially) a one way street. I guess it depends on your application. If you are doing a lot of backtesting, then of course, but if charting active month contracts, I guess not. YMMV as they say.


Last edited by bob7123; October 29th, 2012 at 10:31 AM.
Reply With Quote
 
  #23 (permalink)
Elite Member
las vegas
 
Futures Experience: Intermediate
Platform: Sierra Chart
Broker/Data: Velocity/IB
Favorite Futures: 6E
 
Posts: 1,147 since Feb 2010
Thanks: 304 given, 836 received


Can you specify certain files to load into ram drive or does it have to be a folder? I want to put SSD in trading box (yes, I tried this previously, long story), but supposedly intense reading and writing will shorten the life, so thinking ram drive but I am limited to 16 gigs memory and my SC data folder is 40 gigs. I don't really want to put historical data in a separate folder and have to move it over every time I want to access it.

Any ideas would be helpful, thanks.

Reply With Quote
 
  #24 (permalink)
Elite Member
Portland Oregon, United States
 
Futures Experience: Beginner
Platform: Ninjatrader®
Broker/Data: CQG, Kinetick
Favorite Futures: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,327 since Jul 2009
Thanks: 1,246 given, 2,635 received

Using the new setup with historical data on RAMDisk, I just shut down and restarted Ninjatrader with five workspaces. The restart and reload were much faster than I have ever seen before. This really does seem to work.

Re preceding post, using Link Shell Extension it seems to me that you could create a junction leading to a file just as well as to a folder. I just did this for a png image but the only option given by the right click menu in LSE was to drop a symbolic link. It did not give me a choice of what type of symbolic link.

I don't think that the historical data in the ntd files has anything to do with the SQLServer CE database other than being in the same folder. I don't think it's a good idea to put the SQLServer CE database on the RAMDisk. Ninjatrader has said that in NT7, this database is NOT used to store historical data.

To deal with the problem of large amounts of data stored in the ntd files, it is not necessary to put the top level folders data, tick, minute, day, data and cache into the RAMDisk. You could judiciously choose lower level folders from within those folders, for example just the ones for the front month contracts.

The method that I use to move a folder from the Ninjatrader db folder to the RAMDisk is:
  • Ninjatrader is shut down.
  • Verified that I have an NTFS formatted RAMDisk Partition set to start up automatically each time the computer starts. Only needs to e set up once.
  • Save a copy of the folder to an archive on the hard disk if I don't want to risk losing it.
  • Use Windows cut and paste to move the folder to the RAMDisk drive.
  • Using LinkShellExtension, right click on the folder in the RAMDisk to designate it as the source of the link and then right click on the db folder to create the junction, which will have the same name as the folder in the RAMDisk partition.


Last edited by Zondor; October 29th, 2012 at 03:21 PM.
Reply With Quote
The following 5 users say Thank You to Zondor for this post:
 
  #25 (permalink)
Elite Member
Portland Oregon, United States
 
Futures Experience: Beginner
Platform: Ninjatrader®
Broker/Data: CQG, Kinetick
Favorite Futures: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,327 since Jul 2009
Thanks: 1,246 given, 2,635 received

Huge improvement with RAMDisk

Just did a shutdown and restart with seven workspaces. Most of those workspaces have charts with multiple indicators that use one tick secondary dataseries, and Gom data. Restart only took a couple of minutes.

This is the best startup performance I have ever seen, by far. Before I set up the RAMdisk a seven workspace instance would usually not restart at all.

Will need to wait until Wednesday to see if improvement continues when markets are active.

Reply With Quote
The following user says Thank You to Zondor for this post:
 
  #26 (permalink)
Elite Member
Portland Oregon, United States
 
Futures Experience: Beginner
Platform: Ninjatrader®
Broker/Data: CQG, Kinetick
Favorite Futures: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,327 since Jul 2009
Thanks: 1,246 given, 2,635 received

Progress Report on RAMDisk experiment

Thanks @pawnbroker, this is one of the best Ninjatrader performance improvements ever.

I tried putting the Templates folder and the SQLServer CE database on the RAMDisk but did not see any performance improvement, so moved them back to their default locations. Will try that with the Indicator folder too, mainly out of curiosity.

For now the RAMDisk contains only the five data storage folders (data, day, tick, minute, cache) from the db folder, and the db folder has junctions that link to them. I used Link Shell Extension to create the links.

Technically, the database files are inside the SQLServer CE folders called Ninjatrader and Ninjatrader.Old. The files in the five folders are not part of the SQLServer database. They are data storage files, but strictly speaking are not database files. When Ninjatrader refers to the database they are referring to the SQLServer database, which does NOT store historical data.

This seems like a stable setup. The RAMDisk software is a production version so I don't consider this to be a white knuckle "experiment". Ninjatrader loads faster and with less drama, especially with multiple workspaces. There is none of the flashing and freezing that I was used to seeing as workspaces and charts loaded from data stored on the hard disk. The RAMDisk backup image is only being updated once per hour and on shutdown, but the loss of some of that data due to an unexpected event would not be a serious problem anyway.

The main problem is that the datastore folders will eventually outgrow the RAMDisk partition. There are various ways to handle this, which may be a little awkward but are worth the trouble based on the improvement in performance.


Putting the datastore folders on their own dedicated physical disk, preferably an SSD and linking to them from junctions in the db folder would simplify dealing with large amounts of historical data. This would not perform as well as the RAMDisk but would certainly be an improvement on the base configuration due to reduction of I/O conflicts - even if using a hard drive.


Last edited by Zondor; October 30th, 2012 at 07:31 PM.
Reply With Quote
The following 3 users say Thank You to Zondor for this post:
 
  #27 (permalink)
Elite Member
Cheltenham
 
Futures Experience: Advanced
Platform: InvestorRT, NinjaTrader
Favorite Futures: ES
 
Posts: 54 since Jan 2012
Thanks: 8 given, 100 received


Zondor View Post
Putting the datastore folders on their own dedicated physical disk, preferably an SSD and linking to them from junctions in the db folder would simplify dealing with large amounts of historical data. This would not perform as well as the RAMDisk but would certainly be an improvement on the base configuration due to reduction of I/O conflicts - even if using a hard drive.

You may like to try SuperCache 5 to create a RAM cache for a drive. I tried that while the markets were closed and saw no improvement, but I saw the same with a RAM drive too. Perhaps my set up needs the stress of live data coming in to get an improvement.

SuperCache sounds good, because you can dedicate say 2 GB of memory for a disk of any size. That would avoid the disk space issue, but it is only worthwhile if there is a performance gain.

I have not tried FancyCache yet, which is a similar product in beta and free while in beta.

With both of those apps, you should not use the option to cache writes. That is very dangerous as lost writes can cause corruption and NT will be improved most by fast reads.

Anyway, I am glad to see that the RAM disk idea is having some benefits!

Reply With Quote
The following 2 users say Thank You to pawnbroker for this post:
 
  #28 (permalink)
Elite Member
Milan Italy
 
Futures Experience: Intermediate
Platform: Ninjatrader, Sierra Chart
Favorite Futures: ES
 
Posts: 523 since Feb 2010
Thanks: 470 given, 629 received

I'd been getting increasingly frustrated with NT's slow startup. After reading this thread, and particularly Zondor's posts, I decided to try reducing the size of the db folder by archiving what I wasn't using. By itself, this cut startup time by about two thirds. Perhaps NT runs a check on the entire db on initial startup.

Reply With Quote
The following user says Thank You to futuretrader for this post:
 
  #29 (permalink)
Identify Evaluate Execute
Mercer Island WA
 
Futures Experience: Advanced
Platform: Ninjatrader/Strategy Desk
Broker/Data: Various
Favorite Futures: TF/NQ/ES/Stocks
 
djkiwi's Avatar
 
Posts: 561 since May 2010
Thanks: 980 given, 1,524 received


futuretrader View Post
I'd been getting increasingly frustrated with NT's slow startup. After reading this thread, and particularly Zondor's posts, I decided to try reducing the size of the db folder by archiving what I wasn't using. By itself, this cut startup time by about two thirds. Perhaps NT runs a check on the entire db on initial startup.

I must try that one. I've been deleting the database cache which seemed to help.

This has been a bone of contention over the years. I'm hoping NT8 overhauls the entire database structure. I suspect Ninjatrader doesn't have a hardcore database expert on staff who has the skills to redesign the architecture. It does seem extremely slow particularly compared to other packages I see people using in demos. I'm seeing FT71 loading his volume profiles back a few years in seconds....

Cheers
DJ

Reply With Quote
 
  #30 (permalink)
Elite Member
Portland Oregon, United States
 
Futures Experience: Beginner
Platform: Ninjatrader®
Broker/Data: CQG, Kinetick
Favorite Futures: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,327 since Jul 2009
Thanks: 1,246 given, 2,635 received

Big improvement in replay mode


Have not given much thought to disk caching software since I am already dedicating two USB memory sticks to Windows Ready Boost®.

Noticing that with the historical data archives moved to the RAM disk I can run multiple complex workspaces simultaneously in replay mode, reliably. Up until now this would cause Ninja to freeze and crash. However with this setup, during day session hours the CPU load is very high and with the contoller set for 500x, the replay is only at around 3x actual speed.

If using inefficiently coded indicators the results would not be as good. Maybe we should call the revision of indicators to reduce resource load "economization" rather than optimizaion, to avoid confusion with strategy optimization.


Last edited by Zondor; November 3rd, 2012 at 03:58 PM.
Reply With Quote
The following user says Thank You to Zondor for this post:

Reply



futures io > > > > Speeding up NinjaTrader with a RAM drive (ramdisk)

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Jigsaw Trading: TBA

Elite only

FuturesTrader71: TBA

Elite only

NinjaTrader: TBA

Jan 18

RandBots: TBA

Jan 23

GFF Brokers & CME Group: Futures & Bitcoin

Elite only

Adam Grimes: TBA

Elite only

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
.....SEC May Ticket Speeding Traders kbit News and Current Events 0 February 23rd, 2012 04:08 PM
NT7.4 - RAM hog kashter NinjaTrader 40 April 14th, 2011 02:10 PM
Drive cory Off-Topic 3 October 1st, 2010 04:17 PM
ram nt7 vps MyTraderAdvisor NinjaTrader 5 August 18th, 2010 04:49 PM
Toyota Blames Driver Error for Unwanted Speeding Quick Summary News and Current Events 0 July 14th, 2010 03:20 AM


All times are GMT -4. The time now is 11:55 AM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-17 in 0.15 seconds with 20 queries on phoenix via your IP 54.90.92.204