NinjaTrader 8 (NT8) Performance Improvements and Tweaks - NinjaTrader | futures io social day trading
futures io futures trading


NinjaTrader 8 (NT8) Performance Improvements and Tweaks
Updated: Views / Replies:533 / 6
Created: by quantera Attachments:0

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 100,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 
Thread Tools Search this Thread
 

NinjaTrader 8 (NT8) Performance Improvements and Tweaks

  #1 (permalink)
Elite Member
Palo Alto, California, USA
 
Futures Experience: Intermediate
Platform: NT8,MC,Quantiacs,Python
Favorite Futures: Metals
 
Posts: 4 since Apr 2018
Thanks: 2 given, 9 received

NinjaTrader 8 (NT8) Performance Improvements and Tweaks

I'd like to propose a single thread (maybe even a Sticky) to pull together tips, tricks, settings, Windows OS mods, hardware and any general tweaks to improve performance of NT8. I've been surprised how little (useful) research is out there on more advanced tweaks to improve NT8. If another thread already exists along these lines, please let me know but I couldn't find anything.

Since I started using NT8 a couple years ago, I've tested dozens of tweaks, most of which offered modest gains at best but a few have made a big difference and I'll share some of those below. Some personal suggestions mostly regarding Backtest performance:


* PC hardware
Hardware specs are obviously a primary factor. I've experimented with 5 or 6 rigs and learned some expensive lessons along the way.
1: IMO CPU/Ram Speed is more important than additional cores: a 4.2ghz 4Core outperforms a 2.9ghz 8Core rig.
2: HD speed makes a huge difference: HD => SSD is obvious but moving from SSD => M.2 Nvme was huge as well
3: Ram Speed seems to also make a huge difference. 3000DDR4 significantly outperforms 1600DDR3
4: Clearly Amt of Ram is also key running > 1000 iterations. 32gb was a big help over 16gb*.

Note: All Backtest Data below is avg for tests over 12mos of 1-minute date (my standard to make comparisons easier.)
Rig1: (4.6ghz I7-7820x 16-Thread / 32gb 3000DDR4 / m.2 Nvme) : Best Performance @ ~ 300 /Min
Rig2: (3.6 ghz Xeon E51620v3 8-Thread / 12gb 1600DDR3 / Sata SSD) : Good Value @ 120 / Min
Rig3: (2.9ghz Dual Xeon 24-thread / 56gb 1600DDR3/ Sata SSD) Disappointing @ ~ 100 / Min
Rig4: (4.1ghz 8-Thread Intel i5 / 8gb DDR3 / Sata SSD) : ok Value @ 100/min
Rig5: (3.2ghz 4-Thread 8gb / SSD Dell Laptop) : ~ @ 50-60 / min
AWS EC2 R4.large Instance 8-thread /8 ) : Not sure why but R4 instances did not perform as well as similar T2 instances
AWS EC2 T2.xlarge 4-Thread 16gb): Suprisingly Seems better than R4 : ~100/Min


* BIG Tip Regarding Virtual Memory:
I was constantly maxing out 32gb of ram on Rig1 (during backtests the CPU and the backtest would idle for 8-10 seconds every 40 seconds while I presume cached data was moved to virtual memory) - while Rig2 seemed to manage ok with only 12gb. So I started looking into why and I realized Rig2 had a much larger Windows page file. I immediately doubled the page file size on Rig 1 from 4gb to 8gb and noticed a modest improvement (Ram would still max out but less frequently) ... so I plugged in an unused 120gb SataSSD in Rig1 and moved the page file from the OS disk to the SSD and removed the original page file. I dedicated the entire SSD to host the windows page file (set to 64gb ) and noticed a huge improvement... so I did some more research and thought maybe i'll use a lower-latency M.2 drive for the virtual memory... and then I found intel's optane m.2 drives. The consumer versions are only 32gb (~$60 ) but with much (much) lower random 4k latency than any other SSD or M.2 Nvme. I dedicated the entire 32gb to the page file and am very happy with the results! No more issues maxing out ram! I think part of this benefit is simply because the primary OS disk no longer has to serve cached ram files so if you have an extra sata ssd drive or m.2 slot... (as an aside, i plan to try moving the NT8 DB folder to this drive to see if there are any improvements and will report results)

* Windows and NT8 Tweaks

FYI, Backtest caches seem to be retained for as long as the respective Strategy analyzer window is open (This really should be fixed by NT as it seems ridiculous) so if you're having memory issues, be sure to Shutdown and reopen the strategy analyzer window in-between memory-intensive tests.

Running NT8 as an Administrator seems to improve responsiveness in NT8 a bit (and causes an annoying security prompt)

Turn off Windows Search and Indexing for NT8 folders (or all together)

If you have a decent graphics card: open NVIDIA control panel 2. select manage 3D settings 3. select program settings
4. select or add Ninjatrader 8 64bit 5. set features to use your GPU and you should notice much better graphics


Hopefully others have found some useful performance tweaks too so please share!


Some other threads I've found useful:
https://ninjatrader.com/support/forum/showthread.php?t=106465
https://ninjatrader.com/support/forum/showthread.php?t=101568
https://ninjatrader.com/support/helpGuides/nt7/?performance_tips2.htm (old / NT7 but might still help)


Last edited by quantera; May 13th, 2018 at 01:17 AM.
Reply With Quote
The following 8 users say Thank You to quantera for this post:
 
  #2 (permalink)
Elite Member
charlotte nc
 
Futures Experience: Master
Platform: ninjatrader
Broker/Data: NinjaTrader
Favorite Futures: Emini (ES, YM, NQ, ect.)
 
Posts: 304 since Jan 2015
Thanks: 56 given, 635 received

When it comes to performance in market replay...

The single best hack to improve the over speed, or put another way, the item that bogs the system down the most.... Is the database. After every use I just delete the DB and the speed improvement is significant. The starting DB size is around 5mb or so. After a nice long market replay run, this will grow to 10,20 mb. After 10 to 20 long market replay runs you will hit >50 mb and be lucky to even get it to load. All you need to do is delete it. NT rebuilds it for you so it's not an issue. I do this trick on my test machine and my live trading machine. For live I do this after every trading day just to squeeze every bit if performance I can...

Cool thread. Thanks for starting it!

Ian

In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
Reply With Quote
The following 2 users say Thank You to iantg for this post:
 
  #3 (permalink)
Elite Member
Palo Alto, California, USA
 
Futures Experience: Intermediate
Platform: NT8,MC,Quantiacs,Python
Favorite Futures: Metals
 
Posts: 4 since Apr 2018
Thanks: 2 given, 9 received

@ Ian : Regarding DB


Thanks for the suggestion Ian. I'd be curious to hear other people's experience... and what benefits you've observed by wiping the DB directory. I've amassed a significant DB of historical data (over 15gb/ 10 years of tick and 1 min data for all commodities and dow-30 equities). I use this historical DB pretty often so don't want to just delete and re-download historical data every time i need to run a large backtest. I have several backups but I consistently use at least a few years of data.

Reply With Quote
 
  #4 (permalink)
Elite Member
charlotte nc
 
Futures Experience: Master
Platform: ninjatrader
Broker/Data: NinjaTrader
Favorite Futures: Emini (ES, YM, NQ, ect.)
 
Posts: 304 since Jan 2015
Thanks: 56 given, 635 received

Sorry, I wasn't clear before. You should delete the actual database. It's a single database file. The database folder with all the various downloaded market replay files are not what I was referring to. Though you may see some improvements there as well, the big gain will be keeping the actual DB file small.

Ian


quantera View Post
Thanks for the suggestion Ian. I'd be curious to hear other people's experience... and what benefits you've observed by wiping the DB directory. I've amassed a significant DB of historical data (over 15gb/ 10 years of tick and 1 min data for all commodities and dow-30 equities). I use this historical DB pretty often so don't want to just delete and re-download historical data every time i need to run a large backtest. I have several backups but I consistently use at least a few years of data.


In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
Reply With Quote
 
  #5 (permalink)
Trading for Fun
Shanghai
 
Futures Experience: Intermediate
Platform: NinjaTrader
Favorite Futures: liquid
 
Posts: 81 since Sep 2013
Thanks: 8 given, 10 received


iantg View Post
Sorry, I wasn't clear before. You should delete the actual database. It's a single database file. The database folder with all the various downloaded market replay files are not what I was referring to. Though you may see some improvements there as well, the big gain will be keeping the actual DB file small.

Ian

do you mean NinjaTrader.sdf?

If it is deleted, would it take longer for charts to load ? I mean, is the chart tick data stored in that file?

Reply With Quote
 
  #6 (permalink)
Elite Member
charlotte nc
 
Futures Experience: Master
Platform: ninjatrader
Broker/Data: NinjaTrader
Favorite Futures: Emini (ES, YM, NQ, ect.)
 
Posts: 304 since Jan 2015
Thanks: 56 given, 635 received

Yes the NinjaTrader.sdf file is the one you want to delete. The playback data that runs your market replay (I.E is loaded into your chart) is not from the NinjaTrader.sdf file. That data lives in the following directory: NinjaTrader8 > DB > Replay > Instrument / Period (ES 03-18 for example) and inside of the instrument specific period specific folder lives the individual replay files.

The NinjaTrader.sdf is where all your trades live from market replay / SIM. Your live trades are going to be preserved via the internet because they are held by your broker and when you point to your live account and connect you can always see these. So the only thing you are deleting when you delete the NinjaTrader.sdf file are the historical market replay / SIM trades and the associated metadata around these. This is essentially garbage data that you likely won't want to keep for more than the current trading session if you are an algorythmic trader. If you are a chart trader, then you likely don't take enough trades for the DB to fill up and grow to 25MB to 50MB.

You might be wondering, "what happens when I just delete this, won't it break something." No, NT just recreates a new one for you that is back to the original specs and nice and tiny around 5MB.

For any algo traders here is the test to run to see the performance gain.... Run a strategy in market replay that takes 100 trades a day, and run this for 3 months.... Then close NT and then try to restart it. It will literally take a few minutes just to load and allow you to even open stuff... Now try to run another market replay simulation.... It will go extremely slow, if it even runs at all. Now check your DB size. It will likely be between 25 MB and 100 MB. Now close NT, delete the DB and open NT a second time. No issues! Everything runs as fast as possible. Run a market replay strategy... Back to 100% speed. Problem solved.

Hope this helps others. It sure helped me a ton.

Ian

In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
Reply With Quote
The following user says Thank You to iantg for this post:
 
  #7 (permalink)
Trading for Fun
Shanghai
 
Futures Experience: Intermediate
Platform: NinjaTrader
Favorite Futures: liquid
 
Posts: 81 since Sep 2013
Thanks: 8 given, 10 received

I wasn't really wondering if I'd break something because it seemed back in the day (older versions of NT) - that was the way you fixed NinjaTrader...

Thanks for the explanation

Reply With Quote

Reply



futures io > > > > NinjaTrader 8 (NT8) Performance Improvements and Tweaks




Upcoming Webinars and Events (4:30PM ET unless noted)

Battlestations: $750 in prizes - Show us your trading desk!

August
 

The Business of Trading w/Darren Reed & Jigsaw Trading

Aug 14
 

Trading Preparedness w/Vinny Nizian @ GFF Brokers

Aug 16
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
NinjaTrader 8 strategy improvements Big Mike Elite Automated NinjaTrader Trading 14 October 24th, 2017 01:14 PM
NT8 Performance Analysis Thoughts jflaggs NinjaTrader Programming 20 August 22nd, 2017 05:35 PM
Webinar: Preparing for NinjaTrader 8 (NT8) with Raymond Deux, CEO of NinjaTrader Big Mike NinjaTrader 21 October 28th, 2016 07:11 PM
Why would Strategy Performance differ in NT7 vs NT8? Aufidius NinjaTrader Programming 3 August 4th, 2015 03:31 AM
Webinar: NinjaTrader Brokerage and NinjaTrader 8 (NT8) discussion Big Mike NinjaTrader 8 July 2nd, 2014 08:17 PM


All times are GMT -4. The time now is 08:55 PM. (this page content is cached, log in for real-time version)

Copyright © 2018 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2018-08-14 in 0.14 seconds with 14 queries on phoenix