NexusFi: Find Your Edge


Home Menu

 





NinjaTrader 8 (NT8) Performance Improvements and Tweaks


Discussion in NinjaTrader

Updated
      Top Posters
    1. looks_one iantg with 3 posts (9 thanks)
    2. looks_two quantera with 2 posts (26 thanks)
    3. looks_3 ycomp with 2 posts (0 thanks)
    4. looks_4 stoicbuddha with 1 posts (1 thanks)
    1. trending_up 7,771 views
    2. thumb_up 36 thanks given
    3. group 9 followers
    1. forum 8 posts
    2. attach_file 0 attachments




 
 

NinjaTrader 8 (NT8) Performance Improvements and Tweaks

 
 quantera 
Palo Alto, California, USA
 
Experience: Intermediate
Platform: NT8,MC,Quantiacs,Python
Trading: Metals
Posts: 5 since Apr 2018
Thanks Given: 2
Thanks Received: 27

I'd like to propose a single thread (maybe even a Sticky) to pull together tips, tricks, settings, Windows OS mods, hardware and any general tweaks to improve performance of NT8. I've been surprised how little (useful) research is out there on more advanced tweaks to improve NT8. If another thread already exists along these lines, please let me know but I couldn't find anything.

Since I started using NT8 a couple years ago, I've tested dozens of tweaks, most of which offered modest gains at best but a few have made a big difference and I'll share some of those below. Some personal suggestions mostly regarding Backtest performance:


* PC hardware
Hardware specs are obviously a primary factor. I've experimented with 5 or 6 rigs and learned some expensive lessons along the way.
1: IMO CPU/Ram Speed is more important than additional cores: a 4.2ghz 4Core outperforms a 2.9ghz 8Core rig.
2: HD speed makes a huge difference: HD => SSD is obvious but moving from SSD => M.2 Nvme was huge as well
3: Ram Speed seems to also make a huge difference. 3000DDR4 significantly outperforms 1600DDR3
4: Clearly Amt of Ram is also key running > 1000 iterations. 32gb was a big help over 16gb*.

Note: All Backtest Data below is avg for tests over 12mos of 1-minute date (my standard to make comparisons easier.)
Rig1: (4.6ghz I7-7820x 16-Thread / 32gb 3000DDR4 / m.2 Nvme) : Best Performance @ ~ 300 /Min
Rig2: (3.6 ghz Xeon E51620v3 8-Thread / 12gb 1600DDR3 / Sata SSD) : Good Value @ 120 / Min
Rig3: (2.9ghz Dual Xeon 24-thread / 56gb 1600DDR3/ Sata SSD) Disappointing @ ~ 100 / Min
Rig4: (4.1ghz 8-Thread Intel i5 / 8gb DDR3 / Sata SSD) : ok Value @ 100/min
Rig5: (3.2ghz 4-Thread 8gb / SSD Dell Laptop) : ~ @ 50-60 / min
AWS EC2 R4.large Instance 8-thread /8 ) : Not sure why but R4 instances did not perform as well as similar T2 instances
AWS EC2 T2.xlarge 4-Thread 16gb): Suprisingly Seems better than R4 : ~100/Min


* BIG Tip Regarding Virtual Memory:
I was constantly maxing out 32gb of ram on Rig1 (during backtests the CPU and the backtest would idle for 8-10 seconds every 40 seconds while I presume cached data was moved to virtual memory) - while Rig2 seemed to manage ok with only 12gb. So I started looking into why and I realized Rig2 had a much larger Windows page file. I immediately doubled the page file size on Rig 1 from 4gb to 8gb and noticed a modest improvement (Ram would still max out but less frequently) ... so I plugged in an unused 120gb SataSSD in Rig1 and moved the page file from the OS disk to the SSD and removed the original page file. I dedicated the entire SSD to host the windows page file (set to 64gb ) and noticed a huge improvement... so I did some more research and thought maybe i'll use a lower-latency M.2 drive for the virtual memory... and then I found intel's optane m.2 drives. The consumer versions are only 32gb (~$60 ) but with much (much) lower random 4k latency than any other SSD or M.2 Nvme. I dedicated the entire 32gb to the page file and am very happy with the results! No more issues maxing out ram! I think part of this benefit is simply because the primary OS disk no longer has to serve cached ram files so if you have an extra sata ssd drive or m.2 slot... (as an aside, i plan to try moving the NT8 DB folder to this drive to see if there are any improvements and will report results)

* Windows and NT8 Tweaks

FYI, Backtest caches seem to be retained for as long as the respective Strategy analyzer window is open (This really should be fixed by NT as it seems ridiculous) so if you're having memory issues, be sure to Shutdown and reopen the strategy analyzer window in-between memory-intensive tests.

Running NT8 as an Administrator seems to improve responsiveness in NT8 a bit (and causes an annoying security prompt)

Turn off Windows Search and Indexing for NT8 folders (or all together)

If you have a decent graphics card: open NVIDIA control panel 2. select manage 3D settings 3. select program settings
4. select or add Ninjatrader 8 64bit 5. set features to use your GPU and you should notice much better graphics


Hopefully others have found some useful performance tweaks too so please share!


Some other threads I've found useful:
https://forum.ninjatrader.com/showthread.php?t=106465
https://forum.ninjatrader.com/showthread.php?t=101568
https://ninjatrader.com/support/helpGuides/nt7/?performance_tips2.htm (old / NT7 but might still help)

Started this thread

Can you help answer these questions
from other members on NexusFi?
New Micros: Ultra 10-Year & Ultra T-Bond -- Live Now
Treasury Notes and Bonds
Exit Strategy
NinjaTrader
Are there any eval firms that allow you to sink to your …
Traders Hideout
Deepmoney LLM
Elite Quantitative GenAI/LLM
Better Renko Gaps
The Elite Circle
 
Best Threads (Most Thanked)
in the last 7 days on NexusFi
Get funded firms 2023/2024 - Any recommendations or word …
61 thanks
Funded Trader platforms
39 thanks
NexusFi site changelog and issues/problem reporting
26 thanks
GFIs1 1 DAX trade per day journal
18 thanks
The Program
18 thanks
 
 iantg 
charlotte nc
 
Experience: Advanced
Platform: My Own System
Broker: Optimus
Trading: Emini (ES, YM, NQ, ect.)
Posts: 408 since Jan 2015
Thanks Given: 90
Thanks Received: 1,147

The single best hack to improve the over speed, or put another way, the item that bogs the system down the most.... Is the database. After every use I just delete the DB and the speed improvement is significant. The starting DB size is around 5mb or so. After a nice long market replay run, this will grow to 10,20 mb. After 10 to 20 long market replay runs you will hit >50 mb and be lucky to even get it to load. All you need to do is delete it. NT rebuilds it for you so it's not an issue. I do this trick on my test machine and my live trading machine. For live I do this after every trading day just to squeeze every bit if performance I can...

Cool thread. Thanks for starting it!

Ian

In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
Visit my NexusFi Trade Journal
 
 quantera 
Palo Alto, California, USA
 
Experience: Intermediate
Platform: NT8,MC,Quantiacs,Python
Trading: Metals
Posts: 5 since Apr 2018
Thanks Given: 2
Thanks Received: 27


Thanks for the suggestion Ian. I'd be curious to hear other people's experience... and what benefits you've observed by wiping the DB directory. I've amassed a significant DB of historical data (over 15gb/ 10 years of tick and 1 min data for all commodities and dow-30 equities). I use this historical DB pretty often so don't want to just delete and re-download historical data every time i need to run a large backtest. I have several backups but I consistently use at least a few years of data.

Started this thread
 
 iantg 
charlotte nc
 
Experience: Advanced
Platform: My Own System
Broker: Optimus
Trading: Emini (ES, YM, NQ, ect.)
Posts: 408 since Jan 2015
Thanks Given: 90
Thanks Received: 1,147

Sorry, I wasn't clear before. You should delete the actual database. It's a single database file. The database folder with all the various downloaded market replay files are not what I was referring to. Though you may see some improvements there as well, the big gain will be keeping the actual DB file small.

Ian


quantera View Post
Thanks for the suggestion Ian. I'd be curious to hear other people's experience... and what benefits you've observed by wiping the DB directory. I've amassed a significant DB of historical data (over 15gb/ 10 years of tick and 1 min data for all commodities and dow-30 equities). I use this historical DB pretty often so don't want to just delete and re-download historical data every time i need to run a large backtest. I have several backups but I consistently use at least a few years of data.


In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
Visit my NexusFi Trade Journal
 
ycomp
Europe-ish
 
Posts: 177 since Sep 2013
Thanks Given: 12
Thanks Received: 58


iantg View Post
Sorry, I wasn't clear before. You should delete the actual database. It's a single database file. The database folder with all the various downloaded market replay files are not what I was referring to. Though you may see some improvements there as well, the big gain will be keeping the actual DB file small.

Ian

do you mean NinjaTrader.sdf?

If it is deleted, would it take longer for charts to load ? I mean, is the chart tick data stored in that file?

 
 iantg 
charlotte nc
 
Experience: Advanced
Platform: My Own System
Broker: Optimus
Trading: Emini (ES, YM, NQ, ect.)
Posts: 408 since Jan 2015
Thanks Given: 90
Thanks Received: 1,147

Yes the NinjaTrader.sdf file is the one you want to delete. The playback data that runs your market replay (I.E is loaded into your chart) is not from the NinjaTrader.sdf file. That data lives in the following directory: NinjaTrader8 > DB > Replay > Instrument / Period (ES 03-18 for example) and inside of the instrument specific period specific folder lives the individual replay files.

The NinjaTrader.sdf is where all your trades live from market replay / SIM. Your live trades are going to be preserved via the internet because they are held by your broker and when you point to your live account and connect you can always see these. So the only thing you are deleting when you delete the NinjaTrader.sdf file are the historical market replay / SIM trades and the associated metadata around these. This is essentially garbage data that you likely won't want to keep for more than the current trading session if you are an algorythmic trader. If you are a chart trader, then you likely don't take enough trades for the DB to fill up and grow to 25MB to 50MB.

You might be wondering, "what happens when I just delete this, won't it break something." No, NT just recreates a new one for you that is back to the original specs and nice and tiny around 5MB.

For any algo traders here is the test to run to see the performance gain.... Run a strategy in market replay that takes 100 trades a day, and run this for 3 months.... Then close NT and then try to restart it. It will literally take a few minutes just to load and allow you to even open stuff... Now try to run another market replay simulation.... It will go extremely slow, if it even runs at all. Now check your DB size. It will likely be between 25 MB and 100 MB. Now close NT, delete the DB and open NT a second time. No issues! Everything runs as fast as possible. Run a market replay strategy... Back to 100% speed. Problem solved.

Hope this helps others. It sure helped me a ton.

Ian

In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
Visit my NexusFi Trade Journal
 
ycomp
Europe-ish
 
Posts: 177 since Sep 2013
Thanks Given: 12
Thanks Received: 58

I wasn't really wondering if I'd break something because it seemed back in the day (older versions of NT) - that was the way you fixed NinjaTrader...

Thanks for the explanation

 
 ValLiant 
redding, CA
 
Experience: None
Platform: ninja/timingsolutions(Wave59 jurys out)
Broker: iB
Trading: CL/GC/NQ
Posts: 67 since May 2010
Thanks Given: 78
Thanks Received: 22

has anyone noticed if going from 32 to 64 GB or ram will improve performance noticeably with nt8?

 
 
stoicbuddha's Avatar
 stoicbuddha 
Seattle, WA
 
Experience: Intermediate
Platform: NinjaTrader 8
Broker: AMP/CQG
Trading: Indices
Frequency: Every few days
Duration: Minutes
Posts: 96 since Feb 2012
Thanks Given: 1,034
Thanks Received: 96

I start NinjaTrader 8 from a batch file and delete the links created by the Ninjatrader install. My desktop shortcut link to the batch file is set to start in the default c:\users\{username}\Documents\Ninjatrader 8 directory and clears out what slowed down Ninjatrader for me previously. The code's below, hope it helps -

@echo off

rd /s /q "cache"
md "cache"

rd /s /q "db\cache"
md "db/cache"

rd /s /q "db\day"
md "db\day"

rd /s /q "db\minute"
md "db\minute"

rd /s /q "db\tick"
md "db\tick"

rd /s /q "tmp"
md "tmp"

rd /s /q "trace"
md "trace"

rd /s /q "log"
md "log"

start "Ninjatrader 8" /high "C:\Program Files (x86)\NinjaTrader 8\bin64\NinjaTrader.exe"

Our life is our own today. Tomorrow you will be dust, a shade, a tale that is told. Live mindful of death, the hour flies.
Visit my NexusFi Trade Journal
Thanked by:

 



Last Updated on April 16, 2020


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts