Anyone have any hints for optimizing C# code? - futures io
futures io futures trading



Anyone have any hints for optimizing C# code?


Discussion in NinjaTrader

Updated
      Top Posters
    1. looks_one Zondor with 20 posts (21 thanks)
    2. looks_two ZTR with 13 posts (0 thanks)
    3. looks_3 Richard with 11 posts (20 thanks)
    4. looks_4 gomi with 7 posts (15 thanks)
      Best Posters
    1. looks_one Fat Tails with 2.6 thanks per post
    2. looks_two gomi with 2.1 thanks per post
    3. looks_3 Richard with 1.8 thanks per post
    4. looks_4 Zondor with 1.1 thanks per post
    1. trending_up 33,400 views
    2. thumb_up 108 thanks given
    3. group 40 followers
    1. forum 111 posts
    2. attach_file 13 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 125,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

Anyone have any hints for optimizing C# code?

(login for full post details)
  #1 (permalink)
 
Experience: Advanced
Platform: NinjaTrader7
Broker: Mirus RCG/Zen-Fire
Trading: CL & 6e, looking at ES, ZB and AU again.
 
ZTR's Avatar
 
Posts: 2,099 since Nov 2009
Thanks: 1,099 given, 1,392 received

Anyone have any hints for optimizing C# code? Currently using Visual Studio for C#. Want to run some indicators through the process to see if there are any excessive computational issues.

Visit my futures io Trade Journal Started this thread Reply With Quote

Can you help answer these questions
from other members on futures io?
Short Interest Ratio indicator?
ThinkOrSwim
SimplerOptions
Trading Reviews and Vendors
Second Entry Indicator (Looking for testers/feedback)
NinjaTrader
MACD line not plotting after change of plot order
NinjaTrader
Reading A set of Trading Signals
EasyLanguage Programming
 
 
(login for full post details)
  #3 (permalink)
USA
 
Experience: Beginner
Platform: NinjaTrader
 
Posts: 26 since Jul 2009
Thanks: 4 given, 17 received


Do you see a performance issue anywhere in your indicators?
Basically you can use .NET profiler, I don't know any freewares, but JetBrains dotTrace is very good (can use 30 days free trial), or look here:
Any Good Free .NET Profiler? - Stack Overflow

Reply With Quote
 
(login for full post details)
  #4 (permalink)
 
Experience: Advanced
Platform: NinjaTrader7
Broker: Mirus RCG/Zen-Fire
Trading: CL & 6e, looking at ES, ZB and AU again.
 
ZTR's Avatar
 
Posts: 2,099 since Nov 2009
Thanks: 1,099 given, 1,392 received

Thanks for the answer. Will be looking at the site.

Some indicators take longer to load than others, especially the recording ones, which read data from local storage, rather than my Zen-Fire feed.

R.I.P. Andy Zektzer (ZTR), 1960-2010.
Please visit this thread for more information.
Visit my futures io Trade Journal Started this thread Reply With Quote
 
(login for full post details)
  #5 (permalink)
Frankfurt
 
Experience: Intermediate
Platform: NinjaTrader
Trading: FGBL 6E B4
 
max-td's Avatar
 
Posts: 1,754 since Jun 2009
Thanks: 2,309 given, 924 received


ZTrade101 View Post
Some indicators take longer to load than others.

thats very true ZT - not only the recording stuff.
i would also be interessted in the reasons of this.

last indicator i realized this heavyly was the BBand2 posted some days ago here.
i am shure this has to do wtih the plotting of the drawRegions in BBand2 and i would like to know if this eating up recources is only when loading / reloading it in the chart or also when running the chart live. and for shure also if it can be optimized some how.

so if you get deeper into this - let me know - i `d like to take part here.

thanks max

max-td
Reply With Quote
 
(login for full post details)
  #6 (permalink)
 
Experience: Advanced
Platform: NinjaTrader7
Broker: Mirus RCG/Zen-Fire
Trading: CL & 6e, looking at ES, ZB and AU again.
 
ZTR's Avatar
 
Posts: 2,099 since Nov 2009
Thanks: 1,099 given, 1,392 received

Thank you Max,

I will look into the .NET more soon, I have a friend visiting next month who is very skilled on optimizing windows applications and he has contacts who he believes can provide him with tools to see what process are using the most computational power. Seems obvious once I thought about, but it is especially important to remove duplicate efforts.

He said that the really skilled C# developers check these things at each step of building a large application. Indicators become part of NT when initiated, so there may be much more efficient was to utilize computation power.

Your investigation into mult-cpu usage, opened my eyes to one of the short comings of NT. He wondered if there was a way to get indicators to use less busy processors. That is way beyond my understanding and hope someone on the forum can enlighten me & possibly find a work around to this limitation. Or just hope NT does this for 8, the consensus on the forum is 2015.

R.I.P. Andy Zektzer (ZTR), 1960-2010.
Please visit this thread for more information.
Visit my futures io Trade Journal Started this thread Reply With Quote
 
(login for full post details)
  #7 (permalink)
Szczecin
 
Experience: Intermediate
Platform: Ninja, writing own now
 
Posts: 212 since Mar 2010

Your friend, sadly, says a lot of bad things. MOst developers - skillfull or not - do NOT check performance on each step of development. It simply makes no sense - as most program parts normally are not relevant for performance at all. THis is a 90/10 issue - your program spends like 90% of the time in 10% of the code. No sense to optimize the other 90%

His "contacts" also should not have to provide him with the tools needed if he indeed is that skillfull. THis is like saying "I know a great cook, and his contacts will provide him with a cookbook". A good developer will know and have the tools of his trade, ready and available to him. I have Ants here, as well as the (currently preferred) VS 2010 Profilers. Do I run them often? No - hell no. I Have better things to do. Writing code. Occasionally I fire them up to see whether something sticks out - and otherwise I use them when, well, I do performance hunting Which is about once every other month. Pretty often you handle one or two items and you are over any performance bottleneck Naturally trading frameworks are a lot different than 90% of the applications out there - where the performance is simply not really dictated by the application itself. And like always, architecture is a lot more important. With all the optimization in the world the crappy NT architecture would not start using multiple cores - and that is where it would start shining.

THAT SAID - especially in something like Ninja or a similar platform, the visualization is the most likely culprit anyway. Drawing things takes time - more than calculating And on a platform / system using as primitive technology as Ninja (WinForms - I long moved to WPF, which is a lot nicer to work with) the optimization you CAN do may simply not exist - at least not on the drawing side. Ninja, as far as I can say, likes being cheap - and uses he Windows integrated user interface elements for many things. Sadly, this blows on simple items like time and sales when too many updates happen. Because it basically does a lot of redraw. On my own T&S implemntation I actually manage to offload a lot of stuff onto the GPU and avoid redrawing the text - without any written code. WPF just is SO much more powerfull

That said, carefull:


Quoting 
to see what process are using the most computational power.

Open task manager, open process list, sort by CPU usage. No other tools needed.

Here sadly the problem starts: If you talk about something like Ninja, knowing Ninja uses all the CPU is not the solution. You need to look INTO the process (in this case a .NET profiler) and locate the code block blowing CPU - which in most cases will lead you to a class that is an indicator/strategy. Process level is sadly no use here - because all indicators / strategies run within one process, the NinjaTrader process.

My only concern would be that all Ninja code is heavily crypted to make decompiling impossible (most likely to stop people from posting emabarassing bad code samples?). I am not sure how wel lthat will run together with a profiler. It SHOULD - as the resulting classes etc. still have to be legal in the .NET framework - but they may be hard to read, confusing and possibly crash something in the profiler.

Reply With Quote
 
(login for full post details)
  #8 (permalink)
 
Experience: Advanced
Platform: NinjaTrader7
Broker: Mirus RCG/Zen-Fire
Trading: CL & 6e, looking at ES, ZB and AU again.
 
ZTR's Avatar
 
Posts: 2,099 since Nov 2009
Thanks: 1,099 given, 1,392 received

That is what I was afraid of. Thank you.

I checked, he told me my interpretations of his comments were overly optimistic. He was basically telling me there was little I could do other than avoid duplicate efforts when adding indicators. Until the underlying architecture was upgraded.

R.I.P. Andy Zektzer (ZTR), 1960-2010.
Please visit this thread for more information.
Visit my futures io Trade Journal Started this thread Reply With Quote
 
(login for full post details)
  #9 (permalink)
London
 
 
Posts: 16 since Mar 2010
Thanks: 0 given, 13 received

NetTecture knows what he is talking about. Wise words there.

Premature optimisation is the root of all evil. Having said that, coding with performance in mind saves a lot of optimisation in the future.

It you want to squeeze every last piece of performance out of the .NET framework the 2 most important things you need to understand are:

(a) Threading (how and when to use it)
(b) Unsafe Code (again, how and when to use it)

Get yourself a box with 4 or 8 cores and offload any market data analysis work that can safely be done in parallel onto background threads for asynchronous processing. If you code it right all your cores should be sharing the work equally. For short running tasks (< 1 sec) you can make things easier by using the .NET thread pool.

Unsafe code is a whole different ball game though and you should only use it if you really need the extra performance. However, if you are iterating through large collections in your analytics code it can speed things up significantly.

Best of luck

P.S. If you looking for a good free profiler try this one from Eqatec. I've been using it for a while and it's great for a freebie.

Reply With Quote
 
(login for full post details)
  #10 (permalink)
Szczecin
 
Experience: Intermediate
Platform: Ninja, writing own now
 
Posts: 212 since Mar 2010


Actually I work on something at the moment that HOPEFULLY is soon good enough to show that is among similar lines than NInja (not as powerfull, though - charting is a seconrdary tought at the moment).

Anyhow, I use ONLY the ThreadPool. Seriously. I have tons of what I call ActionQueue items - a queue that, when items are in, queues an item in the ThreadPool that then extracts them and processes them. I use a different (well, that part is still in the programming) UI approach than Ninja - I hate having tons of windows, so I use "Workspaces" that are windows that have controls on them, and a control can be a chart, T&S etc. - this allows me to basically run one window per physical screen.

Here is the trick: EVERY window gets it's own thread for message processing. That simple. 5 screens? 6 threads (one for the conrol window and some small stuff one may open separately). All connected via ActionQueues - LOTS of work (which makes it understandable Ninja never went that route) but the performance is hard. On top, I use WPF so far, and I am surprised how easy it is to beat Ninja hands down with WPF. Especially smart GPU offloading.

Example: Time and sales. Ninja uses as it loks a standard list there - which regularly redraws. BAD for performance, but extept using a more powerfull control (too cheap for that?) there is not a lot they can do. I use a standard WPF list. So far so bad - I ALSO redraw. THanks to WPF in .NET 4.0 though I can tell the control to keep the lines as bitmaps ON THE GRAPHICS CARD, so when it redraws after a scroll - only the new line is "drawn", the others come from cache. Result? Let's just say I fly compared to Ninja, with me showing every tick. STILL not working perfect - I dont have to show every tick if they come by the hundreds, so I need optimization here. But basically, WPF allows me WITHOUT a lot more programming to really really really shine

Reply With Quote
 
(login for full post details)
  #11 (permalink)
 
Experience: Advanced
Platform: NinjaTrader7
Broker: Mirus RCG/Zen-Fire
Trading: CL & 6e, looking at ES, ZB and AU again.
 
ZTR's Avatar
 
Posts: 2,099 since Nov 2009
Thanks: 1,099 given, 1,392 received

NetTecture,

When is you package coming out?

I'm going to look at Equtec, ScoobyStoo,

Downloaded it last night, but it might be a while before I actually get to optimizing Zondor's & I's indy's

R.I.P. Andy Zektzer (ZTR), 1960-2010.
Please visit this thread for more information.
Visit my futures io Trade Journal Started this thread Reply With Quote
 
(login for full post details)
  #12 (permalink)
Szczecin
 
Experience: Intermediate
Platform: Ninja, writing own now
 
Posts: 212 since Mar 2010

No idea. Some months maybe. SO far the goal is to have something that I can use for trade control and some visualization. It was on a backburner for some months, not it goes in full speed again. Working fulltime on it. I hope to put out a usable internal version of WHATEVER FUNCTIONALITY I CAN GET mid of next week. Working on the client/server part now (news: this is not ninja - it will be a client server tool running against a real sql server as data store, allowing multiple users to connect - the idea is something that has the power of ninja and is also usable in a prop shop style of environment).

Reply With Quote
 
(login for full post details)
  #13 (permalink)
Site Administrator
Swing Trader
Data Scientist & DevOps
Manta, Ecuador
 
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
 
Big Mike's Avatar
 
Posts: 49,785 since Jun 2009
Thanks: 32,314 given, 97,577 received

Good advice here.

[soap box on]

But let me also say, you are likely to see a better improvement in overall performance by just eliminating all the crap people fall for on Windows installs.

Don't install all the registry cleaning crap, all the fancy program uninstallers, all the extra special haxx0r stuff that promises to increase speed. All this stuff is loaded down bloatware, worse even a lot of it is spyware.

Latency is a big issue for trading systems. So if you are really looking for performance, you need to run a wired connection (not wifi), make sure you aren't playing any internet music or downloading torrents, etc.

Personally, I don't install the crap people fall for, but I don't worry so much about the latency side on the internet connection. If I can only be successful trader by entering a trade 250ms faster, then I am not a good trader.

Also, let me state the obvious --- it seems many people have really old computers. If you don't have a quad core CPU, it is old. You can't get blood from a turnip, and you can't make an out of date computer fast. You should buy a new system. If you don't build it yourself, then the first thing you should do when you get the system is blow away everything and reformat the partition, reinstall your OS from scratch. Most tier 1/2/3 system manufacturers load up their installs with crap, crap and more crap because people that don't know any better just can't get enough crap!



[/soap box off]

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to Big Mike for this post:
 
(login for full post details)
  #14 (permalink)
Szczecin
 
Experience: Intermediate
Platform: Ninja, writing own now
 
Posts: 212 since Mar 2010

Amen to that. Seriously. It is called "Crapware" for a reason. I love our company internal Widows installs - yes, it is not bare bone (office etc.), but no - no damn crapware on it

And I TRADE from a pure windows virtual machine sitting in a data center I Dont trust neither my office power supply nore my internet connection

Reply With Quote
 
(login for full post details)
  #15 (permalink)
 
Experience: Advanced
Platform: NinjaTrader7
Broker: Mirus RCG/Zen-Fire
Trading: CL & 6e, looking at ES, ZB and AU again.
 
ZTR's Avatar
 
Posts: 2,099 since Nov 2009
Thanks: 1,099 given, 1,392 received

Recently purchased a system built around an Intel i7 Quad Core Processor 860, 2.80GHz, 1333 MHz clock speed memory with RAID 1 & Nvidia Quadpro 450

Guaranteed to come with nothing but Windows 7 pro.

Will be using my old system in Global Simulation mode for testing/
development of indicators and strategies, and as a backup Gom & Zondor recorder.


R.I.P. Andy Zektzer (ZTR), 1960-2010.
Please visit this thread for more information.
Visit my futures io Trade Journal Started this thread Reply With Quote
 
(login for full post details)
  #16 (permalink)
Site Administrator
Swing Trader
Data Scientist & DevOps
Manta, Ecuador
 
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
 
Big Mike's Avatar
 
Posts: 49,785 since Jun 2009
Thanks: 32,314 given, 97,577 received

Very nice, enjoy the new system

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to Big Mike for this post:
 
(login for full post details)
  #17 (permalink)
USA
 
Experience: Beginner
Platform: NinjaTrader
 
Posts: 26 since Jul 2009
Thanks: 4 given, 17 received


NetTecture View Post
... My only concern would be that all Ninja code is heavily crypted to make decompiling impossible (most likely to stop people from posting emabarassing bad code samples?). I am not sure how wel lthat will run together with a profiler. It SHOULD - as the resulting classes etc. still have to be legal in the .NET framework - but they may be hard to read, confusing and possibly crash something in the profiler.

Ninja's assemblies are not encrypted just don't have the IL in them, RemoteSoft protector takes the IL out but LEAVES the metadata, so basically he should have no problem seeing how much time each method in ninja took.

Also, I wouldn't bash WinForms so much, it is true the msft put their efforts on WPF but still WinForms is a good thing. Ninja, for saving development time, uses Infragistics UI controls which are cool, but with "cool" may come overkill and performance issues (not saying anything bad about Infragistics, they have great controls, but might be misused).

If I understood correctly ZTrade is talking about indicators that has heavy(?) I/O to disk, I guess that it might take some time to read a file full of ticks, not talking about the recording.

Reply With Quote
 
(login for full post details)
  #18 (permalink)
 
Experience: Advanced
Platform: NinjaTrader7
Broker: Mirus RCG/Zen-Fire
Trading: CL & 6e, looking at ES, ZB and AU again.
 
ZTR's Avatar
 
Posts: 2,099 since Nov 2009
Thanks: 1,099 given, 1,392 received

It was suggested by one the the Ninja_Traders that code optimizations was necessary, I asked how? This is what I was told by another:

If you want someone to professionally evaluate your code and make recommendations you could try one of the 3rd party NinjaScript Consultants here: Online Trading. NINJATRADER. NinjaScript Consultants. They make a business out of programming scripts for people and if you ask some of them they can provide you with code evaluations too.


The recording indicators can cause long load times - Especially earlier versions of BuySellVolume.

Wanted to find some way to understand what was happening and avoid duplication of effort. Any time something is changed another long wait is required, as the data must be reloaded from disk.

Zondor has added some clean up code but load times are still very, very long. I'll let him explain that part, but it is basically lifted from the indicators that Ninja Trader has built in.

R.I.P. Andy Zektzer (ZTR), 1960-2010.
Please visit this thread for more information.
Visit my futures io Trade Journal Started this thread Reply With Quote
 
(login for full post details)
  #19 (permalink)
Szczecin
 
Experience: Intermediate
Platform: Ninja, writing own now
 
Posts: 212 since Mar 2010


YuvalW View Post
Ninja's assemblies are not encrypted just don't have the IL in them, RemoteSoft protector takes the IL out but LEAVES the metadata, so basically he should have no problem seeing how much time each method in ninja took.

Also, I wouldn't bash WinForms so much, it is true the msft put their efforts on WPF but still WinForms is a good thing. Ninja, for saving development time, uses Infragistics UI controls which are cool, but with "cool" may come overkill and performance issues (not saying anything bad about Infragistics, they have great controls, but might be misused).

If I understood correctly ZTrade is talking about indicators that has heavy(?) I/O to disk, I guess that it might take some time to read a file full of ticks, not talking about the recording.

Well...
* RemoteSoft protector. Interesting. This, though, seriously sucks - with all respect for the attempts of RemoteSoft, I dont see them coming up with efficient assembly level optimizations. Ouch.
* NinjaTrader6.5 does NOT use Infragistics, and the 7.0 uses such an outdated version I dont wonder they have performance problems - a lot of performance optimizations was done recently by infragistics. Their version is ancient (makes me wonder whether they care at all about bugfixes etc.). Add that to the usual "program quality focus" we all know from Ninja and "crapware" gets a ninja meaning Seriously - never seen a program with so many brain dead decisions. Just to make things true- the version they use is like 2 years old (8.3, now we are at 10.1). Thhe 9 version had a lot of UI optimizations, especially in grids etc.
* Winforms - sorry, winforms DESERVES A bashing for charting etc. It simply is not modern enough. MS has put up a tremendous amount of groundwork towards being able to offload higher functions to the GPU - which really will shine when you have lot of interactive updates (I.e. real time charts). Even text rendering is now part of a DirectX method call - instead of the CPU working. The performane gains are pretty fantastic. NO way to get that out of Winforms. Espeicalyl the new 4.0 version (why the heck will NinjaTrader 7 not be .NET 4.0? I moved my own code over within days) with the cache hinting that allows arbitrary elements to be GPU-cached is REALLY a godsent. I could se a ton of really nice stuff one could do with charts to allow more control

IO indicators are always an issue - file IO is not that simple. If you do a lot of reading and writing, try using BufferedStreams around the files - with a buffer good enough to actually be relevant (2-4mb maybe). After that, a better disc subsystem is needed

Reply With Quote
 
(login for full post details)
  #20 (permalink)
Market Wizard
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,247 since Oct 2009
Thanks: 272 given, 4,415 received


NetTecture View Post
On my own T&S implemntation I actually manage to offload a lot of stuff onto the GPU and avoid redrawing the text - without any written code. WPF just is SO much more powerfull

I started to develop a visualization app for trading with WPF but quickly went back to GDI because it was impossible to draw a 1 pixel width line (the whole "dpi independant" paradigm...", and fonts were blurry. I think now you can draw 1 pixel lines, but aren't fonts still blurry ?

Reply With Quote
 
(login for full post details)
  #21 (permalink)
Market Wizard
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,247 since Oct 2009
Thanks: 272 given, 4,415 received

I'm answering my own question : as a matter of fact you can now develop non-blurry WPF apps with .NET version 4.0
cf Scott Hanselman - WPF and Text Blurriness, now with complete Clarity

This version was released on april 12th 2010, so let's say it's been 11 days that's it's possible to develop a visualization app with WPF.. ;-)

Reply With Quote
The following 2 users say Thank You to gomi for this post:
 
(login for full post details)
  #22 (permalink)
Szczecin
 
Experience: Intermediate
Platform: Ninja, writing own now
 
Posts: 212 since Mar 2010

Small font rendering was significantly improoved with 4.0 - you can not tell draw operations to align with physical pixels (good for small things) and a lot was done for making fonts more readable. I assume it helped that they made VIsual Studio in WPF now, so a lot of the problems start appearing in their own applications and THUS GET FIXED

Reply With Quote
 
(login for full post details)
  #23 (permalink)
British Columbia
 
Experience: Advanced
Platform: NINJA
Broker: ZEN
Trading: Crude
 
Posts: 1,091 since May 2010

Hey Mike,

I was wondering if you would be interested in starting a thread on PC tweaking for traders. I want to learn this stuff and run my PC like a car freak mechanic would run his muscle car..

I'd also like to start on for learning how to tweak NT7, maintain it, and how best to learn C# and what other things you need to know to tweak indicators. I noticed one book on C# in the books thread. Is that a good place to start do you think?

Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #24 (permalink)
NSW Australia
 
Experience: Advanced
Platform: MT4/MT5/ Multicharts/Ninja Trader
Broker: gain
Trading: forex
 
waza1960's Avatar
 
Posts: 33 since May 2010
Thanks: 38 given, 39 received

Hoping somebody can help me here.I'm running N6.5 using a VPS (windows server2003 750mb Ram single core.I run a simulation with 12 strategies for a month went perfectly.
I've started trading live 3 days ago and I suddenly get these messages saying that ninja was unable to submit an order due to not enough free threads in the thread Pool.
I contacted Ninja support forum and they say its a VPS problem I contact the VPS people and they say that its not their problem. I can't find any relevant info on the net.
I was wondering if the RAM being less than recommended was an issue (I don't use the VPS for any thing other than simulation and live automated trading).Maybe I need to upgrade to windows server2008 or multiple cores. opinion or advise welcome

Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #25 (permalink)
Site Administrator
Swing Trader
Data Scientist & DevOps
Manta, Ecuador
 
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
 
Big Mike's Avatar
 
Posts: 49,785 since Jun 2009
Thanks: 32,314 given, 97,577 received


waza1960 View Post
Hoping somebody can help me here.I'm running N6.5 using a VPS (windows server2003 750mb Ram single core.I run a simulation with 12 strategies for a month went perfectly.
I've started trading live 3 days ago and I suddenly get these messages saying that ninja was unable to submit an order due to not enough free threads in the thread Pool.
I contacted Ninja support forum and they say its a VPS problem I contact the VPS people and they say that its not their problem. I can't find any relevant info on the net.
I was wondering if the RAM being less than recommended was an issue (I don't use the VPS for any thing other than simulation and live automated trading).Maybe I need to upgrade to windows server2008 or multiple cores. opinion or advise welcome

750mb ram is not enough to run NinjaTrader. I assume your home machine has 2GB or more, hopefully 4GB or more, just run the strategy live from home for a day and see if you have the same problems.

Server 2008 isn't going to do anything except add more services and slow things down. Your best bet is XP or Windows Seven, and 2GB of ram if you are using NT 6.5.

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #26 (permalink)
London
 
Experience: None
Platform: MC & Ninja mainly
 
Posts: 132 since Jan 2010
Thanks: 3 given, 49 received

Try to use online algorithms wherever possible. Online algorithm - Wikipedia, the free encyclopedia
For example put any of the VWAP algorithms on a 1 tick chart of the ES to see what I mean!

Essentially you want to avoid looping through the data as much as possible. A good example is a simple moving average. The way most new to programming would tackle that is some sort of loop to iterate over the last N bars and sum the total. Really you should

add close[0]/period
subtract close[period]/period

That is about the simplest type of optimisation but wherever you see a loop you want to try and look at eliminating it if possible. There are always places where you are iterating through ticks or bars or something or other.

I know very little about .NET but have started to try and get up to speed on it. Having said that I hear one area to be wary of is garbage collection (as with any managed framework), it is widely written about.

Reply With Quote
 
(login for full post details)
  #27 (permalink)
London
 
 
Posts: 16 since Mar 2010
Thanks: 0 given, 13 received


waza1960 View Post
I've started trading live 3 days ago and I suddenly get these messages saying that ninja was unable to submit an order due to not enough free threads in the thread Pool.

This is called thread starvation. The .NET runtime allocates one thread pool per process (i.e. NT). In .NET2.0 the thread pool has a default size of 25 worker threads per available processor. In .NET3.5 Microsoft increased this to 250 worker threads. However, NT6.5 runs on .NET2.0 so you are only getting 25 worker threads.

Just found your post on the NT support forum. From the screenshot I can see you are getting an InvalidOperationException when NT invokes HttpWebRequest.BeginGetRequestStream. This is a common problem which NT should be aware of.

See this link...

Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications

Try adding the following line of code to the initialisation method of your NT strategy.

 
Code
System.Threading.ThreadPool.SetMaxThreads(250, 1000);
See if that helps. It'll bump up the number of threads in the pool.


waza1960 View Post
I contacted Ninja support forum and they say its a VPS problem I contact the VPS people and they say that its not their problem.

This is a software issue, not a VPS issue. Send that link to NT and tell them to pull their finger out and look at their code rather than being so quick to point their finger at the VPS. They should be providing support for their product, not me. I suspect they are making numerous HttpWebRequest calls without understanding the ramifications for the thread pool. On a powerful machine you won't notice a problem because the worker threads will be freed up quickly but on a less powerful machine with limited CPU resources (such as your VPS) you've got a problem.


waza1960 View Post
I was wondering if the RAM being less than recommended was an issue (I don't use the VPS for any thing other than simulation and live automated trading).

This problem is completely unconnected to the RAM. But yes, you need a couple of Gig really to run NT comfortably. It's a memory hungry beast.


waza1960 View Post
Maybe I need to upgrade to windows server2008 or multiple cores. opinion or advise welcome

Stick with Win2003. There's no point in you upgrading to Win2008. Any increased CPU resources (such as additional cores) will definitely help. It sounds like you have a fairly low powered VPS instance and that's going to mean you don't get much CPU time allocated to you. Either get yourself a more powerful VPS or ditch the VPS entirely and run NT on a dedicated box instead (my preference).

Reply With Quote
The following 2 users say Thank You to ScoobyStoo for this post:
 
(login for full post details)
  #28 (permalink)
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

RWT-004 :: Move the Markets


Very interesting post at MoveTheMarkets. The way we usually do things is terribly inefficient and there is a better way:

"In all of the coding tutorials and examples I've seen, Ninjatrader indicators are used in-line. So, if you want to know when the SMA is going up, you'd check:
SMA(Close, 5)[0] > SMA(Close, 5)[1]
I find this practice repulsive for two reasons: redundancy and inefficiency. This paper chronicles the path I took to fifinding a better way. If you just want to know what the better way is, skip to the 'Solution' section at the end......"

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #29 (permalink)
London
 
Experience: None
Platform: MC & Ninja mainly
 
Posts: 132 since Jan 2010
Thanks: 3 given, 49 received

"The problem is in the caching scheme that Ninjatrader uses for momoization. First off, it's a simple linear search!"

Hmm that is precisely the kind of thing I was eluding too above. You really don't want to be looping through 'stuff'. Loops are bad. Of course if ninja is doing it internally you have problems (as Richard discovered)

Reply With Quote
 
(login for full post details)
  #30 (permalink)
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

Anyone writing Ninjascript code should look at that post at MoveTheMarkets, and implement that idea in their code.

A good test case is the BigMike CollectiveMA. It makes six calls to external MA's.

I already changed it as Richard suggested, and I can easily see that it loads much faster now.

These changes are very easy to make, only taking a few minutes. When you think about it, doing that in an indicator that uses six calls to external MA's will cut the resources used by the indicator to a small fraction of the original demands.

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #31 (permalink)
Dallas TX/USA
 
 
Posts: 154 since Jun 2009
Thanks: 33 given, 283 received


Zondor View Post
Anyone writing Ninjascript code should look at that post at MoveTheMarkets, and implement that idea in their code.

Yeah, it's faster, and it keeps you from repeating the arguments every time. Just all-around better. It's too bad ninja encourages the style that they do.

Another big source of inefficiency are indicators using MAX and MIN either directly or indirectly. I also have an article with an online algorithm for maintaining a sorted sliding window, which is much faster than doing an Array.Sort() every time OnBarUpdate is called (think about it... last bar you had a sorted window for everything but the newest value... make use of it rather than throw the almost correct answer away!). Much better for median filters and interquartile ranges, etc. There are other easy things to do, but I haven't had time to write them up. I'm kinda lazy about writing.

Reply With Quote
The following user says Thank You to Richard for this post:
 
(login for full post details)
  #32 (permalink)
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

Richard, this is great! I optimized several custom indicators using this approach and the performance improvement is tremendous. Posted in the Elite Section is the optimized version of the Big Mike CMA Envelope Bands.. it loads in a split second. Thanks so much!




Sometimes the definitions of the external indicators include parameters that are updated dynamically. Whenever that happens, the definition of the external indicator must also be updated. Here is an example:


For this reason, and also to avoid problems with NT6.5, I prefer to define the external indicators in a procedure module such as OnBarUpdate() rather than in OnStartUp().

Is there anything we can do about MAX and MIN? One particular custom indicator that uses them a lot is painfully slow to load. (BetterBuySellVolume, part of double Z trading package)

Do Math.Max and Math.Min also have problems?

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to Zondor for this post:
 
(login for full post details)
  #33 (permalink)
East Rochester, NY
 
Experience: Intermediate
Platform: NT
 
eDanny's Avatar
 
Posts: 329 since Jul 2009
Thanks: 17 given, 423 received

But what if the called indicator does not have just one output? Which is more efficient?

This?

in Variables:
private DataSeries myMacdAvg;

in Initialize():
myMacdAvg = new DataSeries(this);

in OnBarUpdate();
myMacdAvg.Set(MACD(12,26,9).Avg[0]);

if(myMacdAvg[0] > 0)
DoSomething

Or this?

in Variables:
private MACD myMacd:

in OnBarUpdate();
if(!init)
{
myMacd = MACD(12,26,9); //myMacd = MACD(12,26,9).Avg will not work here
init = true
}
if(myMacd.Avg[0] > 0)
DoSomething

Reply With Quote
 
(login for full post details)
  #34 (permalink)
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

If you want a dataseries to capture the SMA of the MACD: (Note... MACD12 should have been MyNewMACD)

in Variables:

private DataSeries myMacdAvg;
private bool init = false;
private MACD MyNewMACD
private SMA SMAofMACD


in Initialize():

myMacdAvg = new DataSeries(this);


in OnBarUpdate();

if(!init)
{
MyNewMACD= MACD(12,26,9);
SMAofMACD=SMA(MyNewMACD, 7); // 7 period SMA of MACD(12,26,9)
init = true;
}

MyMacdAvg.Set(SMAofMACD[0]);
if(MyMacdAvg[0] > 0) // if current value of 7 period SMA of MACD(12,26,9)>0
(
Print("Hello Vorld");
FabianCancellera.WinWorldTimeTrialChampionship;
)

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #35 (permalink)
East Rochester, NY
 
Experience: Intermediate
Platform: NT
 
eDanny's Avatar
 
Posts: 329 since Jul 2009
Thanks: 17 given, 423 received

Very nice (except I don't see where MACD12 comes in or is actually used). In my previous example, which would be better in accomplishing the same result, using a DataSeries or not using a DataSeries? That is my question which might be better left to Richard.

Dan

Reply With Quote
 
(login for full post details)
  #36 (permalink)
Dallas TX/USA
 
 
Posts: 154 since Jun 2009
Thanks: 33 given, 283 received


eDanny View Post
But what if the called indicator does not have just one output? Which is more efficient?

The main thing is to remember a reference to the indicator, so that you don't have to ask ninja to look it up again. The number of DataSeries you need to look at isn't a factor.

So, your second example--just referencing myMacd.Avg[0]--is the one I'd use. Putting the information in an additional DataSeries isn't a necessary step, and wastes time and memory.

If you really just want a reference to the Avg part of the MACD indicator, you should be able to do that like this, though:

 
Code
in Variables:
private DataSeries macdAvg;

in OnBarUpdate():
if(!init)
{
macdAvg= MACD(12,26,9).Avg; 
init = true;
}

if(macdAvg[0] > 0)
DoSomething
.. but you lose easy access to the rest of the MACD that way, and the performance increase would be negligible.

Reply With Quote
The following 2 users say Thank You to Richard for this post:
 
(login for full post details)
  #37 (permalink)
East Rochester, NY
 
Experience: Intermediate
Platform: NT
 
eDanny's Avatar
 
Posts: 329 since Jul 2009
Thanks: 17 given, 423 received

Thanks Richard. I was trying to avoid the memory consumption of an additional DataSeries and that is why I asked. Also I could not get this to work (I was using something other than the MACD but same idea).

Dan


macdAvg= MACD(12,26,9).Avg;

Reply With Quote
The following user says Thank You to eDanny for this post:
 
(login for full post details)
  #38 (permalink)
Dallas TX/USA
 
 
Posts: 154 since Jun 2009
Thanks: 33 given, 283 received


Zondor View Post
Sometimes the definitions of the external indicators include parameters that are updated dynamically.

Yeah, I can't see the code you are posting, but just be aware that when you change the inputs you are actually creating an additional indicator that runs in parallel with the original one. So, that can easily get out of hand if you don't do some quantizing... like: instead of changing the SMA length each bar, only change it in length multiples of 5, or whatever.



Quoting 
Is there anything we can do about MAX and MIN?

There are fast algorithms for maintaining the max/min of the last n samples. As with the sliding sorted window I posted about, the key is to realize that on the previous bar you had almost the right answer. If you can just incorporate the current bar's information without just searching all over again for the maximum, you are much better off. I just haven't had time to write those up yet. Maybe soon, but I'm sure someone motivated to speed them up can work out the details. Hint: if the maximum was 3 bars ago on the last bar, then it was 4 bars ago on this bar. So if the current bar's not larger than the previous maximum, and your length is >= 4, then there's no need to do anything. The maximum hasn't changed.

Reply With Quote
The following user says Thank You to Richard for this post:
 
(login for full post details)
  #39 (permalink)
Dallas TX/USA
 
 
Posts: 154 since Jun 2009
Thanks: 33 given, 283 received


eDanny View Post
Thanks Richard. I was trying to avoid the memory consumption of an additional DataSeries and that is why I asked. Also I could not get this to work (I was using something other than the MACD but same idea).

Dan


macdAvg= MACD(12,26,9).Avg;

Avg will be a Dataseries or IDataSeries... so macdAvg must be defined as one, as well (not a MACD). Does that help? So...

 
Code
DataSeries macdAvg;
macdAvg = MACD(12,26,9).Avg;
... compiles for me.

Reply With Quote
The following user says Thank You to Richard for this post:
 
(login for full post details)
  #40 (permalink)
East Rochester, NY
 
Experience: Intermediate
Platform: NT
 
eDanny's Avatar
 
Posts: 329 since Jul 2009
Thanks: 17 given, 423 received

Thanks Richard. I would be right back to using a DataSeries in that example so that is NOT the way to go.

Dan

Reply With Quote
The following user says Thank You to eDanny for this post:
 
(login for full post details)
  #41 (permalink)
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

"Yeah, I can't see the code you are posting, but just be aware that when you change the inputs you are actually creating an additional indicator that runs in parallel with the original one. So, that can easily get out of hand if you don't do some quantizing... like: instead of changing the SMA length each bar, only change it in length multiples of 5, or whatever."

Richard,

Is there a way to destroy the original one when I create the new one, so that only one instance will exist? Can I use Dispose() to do that?

Thanks to you we are making some great progress in this thread now. It's making a real difference.

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #42 (permalink)
Dallas TX/USA
 
 
Posts: 154 since Jun 2009
Thanks: 33 given, 283 received


eDanny View Post
Thanks Richard. I would be right back to using a DataSeries in that example so that is NOT the way to go.

I just want to make sure we're on the same page. Storing a reference to an existing DataSeries doesn't cost anything. Creating a new dataseries and filling it with values from another one is wasteful.

So, putting this in the init code:
 
Code
DataSeries macdavg; 
macdavg = MACD(12,26,9).Avg;
... costs nothing, and leaves you with access to the Dataseries that the MACD is filling. On the other hand:

 
Code
// Init Code
macdavg = new DataSeries(this);
...
// OnBarUpdate():
macdavg.Set(MACD(12,26,9).Avg[0]);
... stores copies of data that you already had access to, which is wasteful. In both cases, you end up with a DataSeries called macdavg that has the correct values in it, but the first version is practically free and the second one does a bunch of unnecessary work.

Reply With Quote
The following 4 users say Thank You to Richard for this post:
 
(login for full post details)
  #43 (permalink)
Dallas TX/USA
 
 
Posts: 154 since Jun 2009
Thanks: 33 given, 283 received


Zondor View Post
Is there a way to destroy the original one when I create the new one, so that only one instance will exist? Can I use Dispose() to do that?

I don't think so, because ninja is caching them and you don't know who else is looking at that same indicator reference. In the same way, you can't just change an indicator's inputs on the fly, because someone else may be depending on the object continuing to compute the 5 SMA rather than the 10, or whatever.

The three easiest workarounds I know of are:
  • Quantize so that you only ever create a few indicators to cover the input space and you don't have to worry about it.
  • Make an indicator that knows how to vary itself in the right way, rather than having the main indicator in control of adjusting the inputs (for bonus points, market it to people as an "adaptive indicator" for tons of cash to fund further development)
  • Suck the indicator code into a private function of the main indicator, so that you can call it with different inputs at will

Reply With Quote
The following user says Thank You to Richard for this post:
 
(login for full post details)
  #44 (permalink)
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

Based on the suggestion by Richard in Post 38, I changed the DoubleStochasticsOptimized to check within the indicator, when a new bar starts, whether the [Period +1] value is equal to the minimum or maximum value for each of four different dataseries.

It is necessary to call the external MIN or MAX function ONLY when that is true.

A 30 tick, 30 days back chart of the YM with this indicator refreshes in less than one second.

(Printing the value of CurrentBar to the Output Window shows that this chart has 99,798 bars. ) That seems pretty good.....

Attached Files
Register to download File Type: cs DoubleStochasticsOptimized.cs (10.2 KB, 28 views)
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to Zondor for this post:
 
(login for full post details)
  #45 (permalink)
Site Administrator
Swing Trader
Data Scientist & DevOps
Manta, Ecuador
 
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
 
Big Mike's Avatar
 
Posts: 49,785 since Jun 2009
Thanks: 32,314 given, 97,577 received

In linux cli world, we can easily time stuff to find which method is most CPU efficient, just:

time <what you want to time>

It would be nice if a small block of code could be written to time these new optimized functions, so we can compare old vs. new and see what we are really accomplishing.

I'm all for writing efficient code and am just curious how much CPU time we're saving here. I think with .NET framework our only hope is to measure CPU time, measuring resource consumption like memory is not going to happen I don't think.

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #46 (permalink)
 
Experience: Advanced
Platform: NinjaTrader7
Broker: Mirus RCG/Zen-Fire
Trading: CL & 6e, looking at ES, ZB and AU again.
 
ZTR's Avatar
 
Posts: 2,099 since Nov 2009
Thanks: 1,099 given, 1,392 received

Anecdotally, there is a definite difference in load time and workspace switching times since Zondor started his optimization project.

I am running 3 workspaces of 3 GOM Volume Ladders & 5 copies of BSV39 (both notoriously slow loaders). Load times went from 30 + seconds for the attached chart when hitting F5 to less than 8 seconds with 4 worksapces running. The fourth workspace has 12 GOM recorders -2 each of YM, CL, 6e flat and binary and 2 contract months each. Load time is less then 3 seconds if this is the only chart loaded in an empty workspace & that includes data.

R.I.P. Andy Zektzer (ZTR), 1960-2010.
Please visit this thread for more information.
Attached Thumbnails
Click image for larger version

Name:	YM 06-10 (1024 Volume)  5_26_2010_2034.jpg
Views:	106
Size:	183.4 KB
ID:	14365   Click image for larger version

Name:	Indys.PNG
Views:	69
Size:	26.7 KB
ID:	14366  
Visit my futures io Trade Journal Started this thread Reply With Quote
 
(login for full post details)
  #47 (permalink)
Dallas TX/USA
 
 
Posts: 154 since Jun 2009
Thanks: 33 given, 283 received


Zondor View Post
A 30 tick, 30 days back chart of the YM with this indicator refreshes in less than one second.

I had in mind more of a FastMAX and FastMIN indicator that you can use as replacements for the original MAX and MIN... so other indicators wouldn't need to know the difference. Like the attached.

I can run the new 89-max and 89-min on a chart with 2.8 million bars in 6 seconds, versus more like 19 seconds with the built-in max and min. It definitely makes a difference to avoid redundant computations.

Attached Files
Register to download File Type: cs Z20100527FastMAX.cs (7.1 KB, 56 views)
Register to download File Type: cs Z20100527FastMIN.cs (7.5 KB, 48 views)
Reply With Quote
The following 5 users say Thank You to Richard for this post:
 
(login for full post details)
  #48 (permalink)
Dallas TX/USA
 
 
Posts: 154 since Jun 2009
Thanks: 33 given, 283 received

While we're on the topic of optimization, DoubleStochasticsOptimized has a line like this in it:

 
Code
			p2.Set(pEMA[0]);
Whenever you see a line like this, it is a sign that you probably don't need p2 at all. Just use pEMA.Value instead. Also, by using fastMAX and fastMIN you can put those into OnStartUp() and avoid the cache lookup.

Reply With Quote
The following user says Thank You to Richard for this post:
 
(login for full post details)
  #49 (permalink)
san ramon
 
Experience: Beginner
Platform: ninja
Broker: zen
Trading: es, 6e
 
Posts: 136 since Oct 2009
Thanks: 98 given, 76 received

I propose a new platform by Gomi and Richard ........ "Be Rich GoRich" .....

Reply With Quote
The following user says Thank You to zikonc for this post:
 
(login for full post details)
  #50 (permalink)
London
 
 
Posts: 16 since Mar 2010
Thanks: 0 given, 13 received


Big Mike View Post
It would be nice if a small block of code could be written to time these new optimized functions, so we can compare old vs. new and see what we are really accomplishing.

I'm all for writing efficient code and am just curious how much CPU time we're saving here. I think with .NET framework our only hope is to measure CPU time, measuring resource consumption like memory is not going to happen I don't think.

Try these puppies...

CPU profiling: EQATEC Profiler &mdash; EQATEC

Memory profiling: Download details: CLR Profiler for the .NET Framework 2.0

No need to part with any cash money.

Reply With Quote
The following user says Thank You to ScoobyStoo for this post:
 
(login for full post details)
  #51 (permalink)
Market Wizard
Berlin, Europe
 
Experience: Advanced
Platform: NinjaTrader, MultiCharts
Broker: Interactive Brokers
Trading: Keyboard
 
Fat Tails's Avatar
 
Posts: 9,855 since Mar 2010
Thanks: 4,238 given, 26,731 received

I am impressed with all the contributions in this thread. I will read this again and try to make use of it...

However, the real problems I had with NT 7 so far, came from another source. My NT sometimes used more than 50% of the CPU, when I had added some custom indicators to various charts of my workspace, although all the indicators were in CalculateOnBarClose = "True" mode.

The reason for the high CPU usage was the customized plot. The custom plots are calculated for every incoming tick, and this can have a huge impact on performance.

The solution here is to define an external method, which writes the new input values to a DataSeries or arrays object, and then let the Plot Override check for changed values, before performing the calculations.

Reply With Quote
The following 2 users say Thank You to Fat Tails for this post:
 
(login for full post details)
  #52 (permalink)
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

Great participation and help from everyone who has been visiting here lately!

I attempted to incorporate Richard's latest suggestions into the DoubleStochasticOptimized. I posted it briefly until I realized it had problems. Will re-post after it is fixed.

Regarding previous post by FatTails, if the PlotGraphics method is used, the update frequency can be throttled back by having a minimum millisecond value between screen refreshes, as is done in the RealStats indicator.


if (lastRefresh.AddMilliseconds(LevelIIRefreshDelay) < DateTime.Now){

lastRefresh = DateTime.Now;
}

It looks like the two biggest drains on performance are excessive graphics demand, and excessive and redundant use of calls to external indicators. Addressing both as applicable is very productive, especially combined with the efficiency improvements of NT7.

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to Zondor for this post:
 
(login for full post details)
  #53 (permalink)
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

This version of the Double Stochastics has every coding trick I can think of.

All calls to external indicators are pre-instantiated.

It uses Richard's FastMAX and FastMin indicators instead of MIN and MAX.

Redundant p2 DataSeries from earlier version has been removed.

Calls to the FastMIN and FastMAX indicators can only happen during the first tick of a bar. And they are only called if the internal logic can no longer be sure of the actual minimum and maximum values. Usually it WILL be sure, and will not need to call these external indicators at all.

I consider this to be a coding exercise. Is it worth it? Not sure....



.... I am Ming the Merciless. I will RULE THE UNIVERSE!

Attached Files
Register to download File Type: cs DoubleStochasticsOptimized.cs (10.5 KB, 81 views)
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following 2 users say Thank You to Zondor for this post:
 
(login for full post details)
  #54 (permalink)
Market Wizard
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,247 since Oct 2009
Thanks: 272 given, 4,415 received

Another hint : if you use collections in your code, you have to beware of the complexity of access/insert the data.

Here are the objects I use the most


Most simple object is List
  • It's basically a checked, autogrowing ,and typed, array.
  • Access/Read is very quick (if you access by index of course)
  • You can sort the List, which is o(n log n) complexity

Then you have the Dictionary
  • It's a generic and typed Hashtable.
  • Access/Read is o(1) complexity, meaning access time doesn't depend on the size of the Dictionary, and is very quick (obviously less than an array as there is the hashing stuff to compute)

Then you have the sorted objects : SortedList and SortedDictionary
  • SortedList and SortedDictionary have o(log n) access
  • SortedList has o(n) insert, SortedDictionary has o(log n) insert
  • You can access SortedList by index (not possible with Dictionary)


So you have to remember if you're accessing tick data that you will be manipulating large structures, so choose the object with the less complexity that fits your need.

In special cases, if you do lots of random inserts and modifcations, and very rarely you need to access the sorted values, it can be quicker to use a List and sort it when needed than using a SortedList

Reply With Quote
The following 2 users say Thank You to gomi for this post:
 
(login for full post details)
  #55 (permalink)
 
Experience: Advanced
Platform: NinjaTrader7
Broker: Mirus RCG/Zen-Fire
Trading: CL & 6e, looking at ES, ZB and AU again.
 
ZTR's Avatar
 
Posts: 2,099 since Nov 2009
Thanks: 1,099 given, 1,392 received

Since this is an answer box & can just click thanks.

Thank you Gomi.

R.I.P. Andy Zektzer (ZTR), 1960-2010.
Please visit this thread for more information.
Visit my futures io Trade Journal Started this thread Reply With Quote
 
(login for full post details)
  #56 (permalink)
Dallas TX/USA
 
 
Posts: 154 since Jun 2009
Thanks: 33 given, 283 received


Zondor View Post
This version of the Double Stochastics has every coding trick I can think of.

Since MAX and MIN maintain the max and min, I'm not sure why the indicator code keeps checking to see if the low is lower than the minLow[0]... if it's working correctly it never should be... so without running the code, I imagine OnBarUpdate() should look more like this:

 
Code
double r = maxHigh[0] - minLow[0];
p1.Set(((Close[0] - minLow[0]) / (r == 0 ? 1 : r)) * 100);
			
double s = maxpEMA[0] - minpEMA[0];
p3.Set(((pEMA[0] - minpEMA[0]) / (s == 0 ? 1 : s)) * 100);
K.Set(kEMA[0]);
			
if(Rising(K))PlotColors[0][0]=upColor;
if(Falling(K))PlotColors[0][0]=downColor;
much simpler. Maybe I misunderstood something, but glancing at the code that's what it looks like to me.

Reply With Quote
The following 2 users say Thank You to Richard for this post:
 
(login for full post details)
  #57 (permalink)
 
Experience: Advanced
Platform: NinjaTrader7
Broker: Mirus RCG/Zen-Fire
Trading: CL & 6e, looking at ES, ZB and AU again.
 
ZTR's Avatar
 
Posts: 2,099 since Nov 2009
Thanks: 1,099 given, 1,392 received

Richard are you willing to do a Webinar?

R.I.P. Andy Zektzer (ZTR), 1960-2010.
Please visit this thread for more information.
Visit my futures io Trade Journal Started this thread Reply With Quote
 
(login for full post details)
  #58 (permalink)
Asia
 
Experience: Beginner
Platform: NinjaTrader, TOS
 
Posts: 798 since Jun 2009
Thanks: 109 given, 799 received

Hi guys,

I'm still trying to get my head around this, so can you guys help me with something simple? An example of something that I find to be painfully slow is a Market Analyzer grid I use every week or so that looks at the 14, 21, 50, and 200 day ROC of a basket of roughly 70 ETFs. The default ROC code is this:

 
Code
                            
protected override void OnBarUpdate()
{
int barsAgo Math.Min(CurrentBarPeriod);
Value.Set(((Input[0] - Input[barsAgo]) / Input[barsAgo]) * 100);

Is there an example of how this code could be optimized? I'm still not getting what types of calculations should be taken out of OnBarUpdate... or am I totally missing something? Thanks for the enlightenment.

Attached Thumbnails
Click image for larger version

Name:	NinjaTrader Grid 2010-05-28 10_38 AM.jpg
Views:	101
Size:	239.8 KB
ID:	14436  
Reply With Quote
 
(login for full post details)
  #59 (permalink)
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

protected override void OnBarUpdate()
{
int barsAgo = Math.Min(CurrentBar, Period);
Value.Set(((Input[0] - Input[barsAgo]) / Input[barsAgo]) * 100);
}


The above will execute every time a BarUpate is triggered. However, during the lifetime of a bar, the value of CurrentBar never changes. So after executing the first time, on successive updates to a bar, this statement is calculating the same thing over and over and over and over.... and over again.

It should only execute ONCE per bar, as soon as a new bar starts.... ie on FirstTickOfBar.

So try this, (and before doing the division on the Value.Set line, be sure that the value of the denominator cannot be zero)


protected override void OnBarUpdate()
{
if(FirstTickOfBar)
{
int barsAgo = Math.Min(CurrentBar, Period);
Value.Set(((Input[0] - Input[barsAgo]) / Input[barsAgo]) * 100);
}
}





I am MING THE MERCILESS. I will RULE THE UNIVERSE!

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to Zondor for this post:
 
(login for full post details)
  #60 (permalink)
Asia
 
Experience: Beginner
Platform: NinjaTrader, TOS
 
Posts: 798 since Jun 2009
Thanks: 109 given, 799 received

Thanks Zondor,

I get the FirstTickOfBar thing if CalculateOnBarClose == false, but in this particular case its historical EOD data.

In retrospect, my performance issue is probably because I'm calling up four indicators per instrument (one each for the 14, 21, 50, and 200 ROC), on roughly 90 instruments.

So perhaps that was a poor choice on my part to use an example.

Let me look at some of the code posted here and I'll post back if I'm still confused.

Reply With Quote
 
(login for full post details)
  #61 (permalink)
Dallas TX/USA
 
 
Posts: 154 since Jun 2009
Thanks: 33 given, 283 received


MXASJ View Post
Is there an example of how this code could be optimized? I'm still not getting what types of calculations should be taken out of OnBarUpdate... or am I totally missing something? Thanks for the enlightenment.

There's no way to meaningfully speed up that indicator. It basically does the minimum that it needs to do. Make sure the market analyzer only uses the minimum number of bars back that you need. And if you only need the realtime numbers, you could try adding:

 
Code
if(!Historical) { ... }
... around the whole thing, to avoid doing any calculation at all, except for the most recent bar. (you'd do this in a copy of ROC, and not ROC itself!)

Since ROC does so little, I don't know how much that would help, but it's the first thing I'd try.

Reply With Quote
The following user says Thank You to Richard for this post:
 
(login for full post details)
  #62 (permalink)
East Rochester, NY
 
Experience: Intermediate
Platform: NT
 
eDanny's Avatar
 
Posts: 329 since Jul 2009
Thanks: 17 given, 423 received

Richard, what is the difference here?

private EMA emathingy;
emathingy = EMA(CCI(CCIPeriod),10);

and

private Indicator emathingy;
emathingy = EMA(CCI(CCIPeriod),10);

Which is better here?

private MACD macdThingamajig;
macdThingamajig = MACD(12,26,9);

then use: macdThingamajig.Avg[0]

or

DataSeries macdThingamajig;
macdThingamajig = MACD(12,26,9).Avg;

then use: macdThingamajig[0]

Reply With Quote
 
(login for full post details)
  #63 (permalink)
Dallas TX/USA
 
 
Posts: 154 since Jun 2009
Thanks: 33 given, 283 received


eDanny View Post
Richard, what is the difference here?

private EMA emathingy;
emathingy = EMA(CCI(CCIPeriod),10);

and

private Indicator emathingy;
emathingy = EMA(CCI(CCIPeriod),10);

It's the same thing, except in the second example you can hold a reference to any indicator, versus just the EMA. The main difference you'll see in practice is when indicators define properties for their plots. So, for instance, MACD has the "Avg" property, which gets you access to the dataseries for that particular plot. If you have "Indicator macd", then the best you can do is look up macd.Values[1]... because "Indicator" does not have a property named "Avg."

Performance-wise, it's a wash, though. It's the same objects. Just a matter of what you need access to through the variables you use.



eDanny View Post
private MACD macdThingamajig;
macdThingamajig = MACD(12,26,9);

then use: macdThingamajig.Avg[0]

or

DataSeries macdThingamajig;
macdThingamajig = MACD(12,26,9).Avg;

then use: macdThingamajig[0]

Again, it's all the same objects doing the same computations... the only difference is what you can access through the variables you are holding. In this case, you are storing off a reference to a particular dataseries in the MACD. So, from that point forward, you have direct access to it, but not to the MACD. You didn't store off a reference to the MACD.

But, if all you were needing was the 'Avg' data, then it's all the same, and in fact a few cycles faster. Nothing you'd notice, so I tend to store references to the indicators for maximum flexibility.

Reply With Quote
The following user says Thank You to Richard for this post:
 
(login for full post details)
  #64 (permalink)
East Rochester, NY
 
Experience: Intermediate
Platform: NT
 
eDanny's Avatar
 
Posts: 329 since Jul 2009
Thanks: 17 given, 423 received

Thanks Richard, those answers are exactly what I was thinking but wanted confirmation.

Dan

Reply With Quote
 
(login for full post details)
  #65 (permalink)
Toronto, Canada
 
Experience: Beginner
Platform: Ninja Trader
Broker: Amp Futures/ Zen-Fire
Trading: Still not sure what is my favourite instrument.
 
SteveM's Avatar
 
Posts: 155 since Jul 2009
Thanks: 464 given, 111 received

I am unable to download the indicator Double Stochastic Optimized, LATEST VERSION V1.4 on NT6.5. I am getting a message: Import failed. Ninja Trader could not decompress the Ninja Script Archive File: Version required to extract this entry not supported (45).

Can some one help me with this. Please.

Thanks

Steve

Thanks
Steve
Be like a JAGUAR. Crouching, perfectly still, waiting for the set up to come to you, then pouncing without hesitation or thought. You only need one to feed the family
Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #66 (permalink)
 
Experience: Advanced
Platform: NinjaTrader7
Broker: Mirus RCG/Zen-Fire
Trading: CL & 6e, looking at ES, ZB and AU again.
 
ZTR's Avatar
 
Posts: 2,099 since Nov 2009
Thanks: 1,099 given, 1,392 received


SteveM View Post
I am unable to download the indicator Double Stochastic Optimized, LATEST VERSION V1.4 on NT6.5. I am getting a message: Import failed. Ninja Trader could not decompress the Ninja Script Archive File: Version required to extract this entry not supported (45).

Can some one help me with this. Please.

Thanks

Steve

Try deleting the info.xml file from the installation folder and paste in the .cs file into an empty shell. I have posted about this if you have questions.

But you may have to upgrade to NT7 to get the really good stuff on the forum, Zondor and I no longer even have 6.5 on our computers.

R.I.P. Andy Zektzer (ZTR), 1960-2010.
Please visit this thread for more information.
Visit my futures io Trade Journal Started this thread Reply With Quote
 
(login for full post details)
  #67 (permalink)
Nederland
 
Experience: Intermediate
Platform: NinjaTrader
Broker: Zen-Fire
 
Posts: 18 since Jun 2009
Thanks: 42 given, 4 received

Can someone please make this indicator working for NT 6.5
Thanks

Reply With Quote
 
(login for full post details)
  #68 (permalink)
Site Administrator
Swing Trader
Data Scientist & DevOps
Manta, Ecuador
 
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
 
Big Mike's Avatar
 
Posts: 49,785 since Jun 2009
Thanks: 32,314 given, 97,577 received


kees View Post
Can someone please make this indicator working for NT 6.5
Thanks

I think you are in the wrong place. You should post requests for indicators in this thread:



Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #69 (permalink)
 
Experience: Advanced
Platform: NinjaTrader7
Broker: Mirus RCG/Zen-Fire
Trading: CL & 6e, looking at ES, ZB and AU again.
 
ZTR's Avatar
 
Posts: 2,099 since Nov 2009
Thanks: 1,099 given, 1,392 received

Great ideas to optimize NT indy's From Richad Todd: http://www.movethemarkets.com/downloads/rwt/rwt004/

I truely believe this person runderstands computer processing, especially when applied to NinjaTrader 7 & hope this is not considered an advertisement for this site as I have no idea if he is selling anything directly on this page.

R.I.P. Andy Zektzer (ZTR), 1960-2010.
Please visit this thread for more information.
Visit my futures io Trade Journal Started this thread Reply With Quote
 
(login for full post details)
  #70 (permalink)
San Francisco Bay Area
 
Experience: Intermediate
Platform: NT,TOS,IB
Trading: ES,CL,TF
 
Posts: 280 since Jun 2010
Thanks: 155 given, 271 received

I originally wrote code for solving NP-type problems and performance was the one benchmark we were measured on. When I originally started writing my trading system, I was surprised to see that most commercial libraries do not use data-persistence and loop through the entire data-series. To me that was like driving forward while looking backwards.

It took me some time to realize that most of the world, including most programmers, are not measured on performance and are focussed on getting the job done quickly instead of performance. However when you start running this kind of code on tick data, you very quickly will start maxing out.

This brings me to the question of NT. Though NT7 uses multi-threading the main charting loop is still a single thread. This means that your super-duper quad-core CPU will basically be no better than a single/dual core CPU with the same core and memory system (assuming you are not doing anything else). Back-testing and optimization does use the multiple threads and will run faster on more cores.

I have often faced performance problems with NT with the CPU maxing out at 25% (One core of a quad core). I now try to use CalculateOnBarClose or write my own sampled version for larger time-frame bars i.e. the indicator will update 10 times, during the bar but not on every tick. However this is still inefficient since the OnBarUpdate() method is called on every tick and the check for sampling does take some time.

I am attaching the basic sampled indicator from which other indicators can derive from. It is very much like the gomi indicators. There is also an example which converts the TSI to the SampledTSI.

Attached Files
Register to download File Type: cs AviatSampledIndicator.cs (7.9 KB, 43 views)
Register to download File Type: cs BlauTSISampled.cs (11.4 KB, 40 views)
Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #71 (permalink)
Market Wizard
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,247 since Oct 2009
Thanks: 272 given, 4,415 received

Did you try using the "Display update interval" in NT ?
It's 0.5 sec by default but you can increase it.

Reply With Quote
 
(login for full post details)
  #72 (permalink)
Israel
 
Experience: Intermediate
Platform: NinjaTrader
Broker: pfg
Trading: eminis
 
Posts: 323 since Jun 2009
Thanks: 6 given, 207 received

"Display update interval" has nothing to do with strategies.

Baruch

Reply With Quote
 
(login for full post details)
  #73 (permalink)
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

Here is an indicator that uses optimization techniques including predefined instances of external indicators and minimization of the number of calls to the MIN and MAX indicators.

As a result, this indicator loads and refreshes super fast.

Thanks again to Richard Todd for explaining the how and why of these techniques.

I hope that some of you will take the time to look at this code and use these techniques in your own work.





Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #74 (permalink)
Site Sponsor

Web: NinjaTrader
AMA: Ask Me Anything
Webinars: NinjaTrader Webinars
Elite offer: Click here
 
 
NinjaTrader's Avatar
 
Posts: 1,576 since May 2010
Thanks: 185 given, 2,441 received


Zondor View Post
Here is an indicator that uses optimization techniques including predefined instances of external indicators and minimization of the number of calls to the MIN and MAX indicators.

As a result, this indicator loads and refreshes super fast.

Thanks again to Richard Todd for explaining the how and why of these techniques.

I hope that some of you will take the time to look at this code and use these techniques in your own work.





FYI

Our MIN/MAX indicators have been re-written in NT7 Beta 17. There is a known bug with this that is resolved for Beta 18. Point being, no need to roll your own MIN/MAX functions. We are also going through all our system indicators to see if we can make some further improvements where it would make a difference.

Follow me on Twitter Reply With Quote
The following 4 users say Thank You to NinjaTrader for this post:
 
(login for full post details)
  #75 (permalink)
 
Experience: Advanced
Platform: NinjaTrader7
Broker: Mirus RCG/Zen-Fire
Trading: CL & 6e, looking at ES, ZB and AU again.
 
ZTR's Avatar
 
Posts: 2,099 since Nov 2009
Thanks: 1,099 given, 1,392 received

Thank you. As I just can't click a button to say this in this form.

R.I.P. Andy Zektzer (ZTR), 1960-2010.
Please visit this thread for more information.
Visit my futures io Trade Journal Started this thread Reply With Quote
 
(login for full post details)
  #76 (permalink)
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

In Double Stochastics Optimized (DSCO), I did two things to reduce the number of times that the MIN and MAX functions would be called in the first place. Reason: these functions do loop processing, so are expensive to run.
  • Made sure that each one could be called no more than once per bar, on FirstTickOfBar.
  • Made sure that even then, each one would be called only when the item last removed from the series, or the item most recently added to the series, could have an effect on what the MIN or MAX indicator would return.
Then, I moved the code from the MIN and MAX indicators into the DSCO indicator itself. I used the same code Ninja used in the original MIN and MAX indicators.

I did this based on the theory that calling a private function might be more efficient than calling an external indicator, however I am not sure about that.

There is a tremendous decrease in loading and refresh time vs. the @DoubleStochastics indicator. I think it's mostly because the number of executions of the MIN and MAX loops has been reduced by a couple of orders of magnitude.

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #77 (permalink)
Madison, WI
 
Experience: Advanced
Platform: ALT
Trading: ES
 
aslan's Avatar
 
Posts: 616 since Jan 2010
Thanks: 351 given, 1,109 received


Zondor View Post
I moved the code from the MIN and MAX indicators into the DSCO indicator itself.

Exactly! All of the nested stuff makes it easy to write code, but you pay a price. The built in stuff makes it easy to get up and running and try things out. Once you know it is working though, you can go back and optimize it greatly, if it is really needed.

I don't really use indicators anymore, but when I did, I had coded most of my indicators to remove all of the nested stuff (especially the trivial stuff like MIN/MAX).

Reply With Quote
 
(login for full post details)
  #78 (permalink)
Santa Maria
 
Experience: Advanced
Platform: NinjaTrader, ThinkOrSwim
Broker: Mirus/Zen-Fire
Trading: ES
 
Posts: 295 since May 2010
Thanks: 97 given, 321 received


Zondor View Post
... and minimization of the number of calls to the MIN and MAX indicators.

As 'Ninjatrader' stated, MIN and MAX iterating when not required is fixed in b18. As I see it the two biggest efficiency problems are iteration and memory consumption. I just posted jhlMIN and jhlMAX which when used in building other indicators and not displayed have memory overhead of 8 * n + 16, vs. I believe over 2K for the standard implementations.



Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #79 (permalink)
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

Post #70 in this thread is a perfect example of what not to do. The writer, xxxx72, talks about his programming background, his expert observations regarding NT7, and his scheme to reduce CPU load by skipping OnBarUpdate cycles.

However he failed to correct some extremely obvious and totally ridiculous features of the original code that were the real causes of the poor performance. Adding sampling of intrabar ticks while ignoring the deficiencies of the bar update code is a wonderful example of putting lipstick on a pig.

Here are the major problems:
  • Variables that only need to be calculated once (ONCE, not once per bar) are being calculated on every tick. These variables retain constant values over the entire life span of each instance of the indicator.
  • Variables that only need to be calculated once per bar are also being calculated on every tick. Avoiding that situation is why God gave us the if(FirstTickOfBar) condition.

These problems were a cinch to fix. I guess you have to be a non-programmer like me to notice things like that. C'mon guys, this is inexcusable.

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following 2 users say Thank You to Zondor for this post:
 
(login for full post details)
  #80 (permalink)
Market Wizard
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,247 since Oct 2009
Thanks: 272 given, 4,415 received

Just a remark if lots of divisions are involved : remember on a pure performance perspective that division takes much longer than multiplication.

So if you have lost of divisions by a rarely changing x, it's much faster to compute invx=1/x once, and perform multiplications by invx rather than divisions by x.

Reply With Quote
The following 2 users say Thank You to gomi for this post:
 
(login for full post details)
  #81 (permalink)
Market Wizard
Berlin, Europe
 
Experience: Advanced
Platform: NinjaTrader, MultiCharts
Broker: Interactive Brokers
Trading: Keyboard
 
Fat Tails's Avatar
 
Posts: 9,855 since Mar 2010
Thanks: 4,238 given, 26,731 received


Zondor View Post
Post #70 in this thread is a perfect example of what not to do. The writer, xxxx72, talks about his programming background, his expert observations regarding NT7, and his scheme to reduce CPU load by skipping OnBarUpdate cycles.

However he failed to correct some extremely obvious and totally ridiculous features of the original code that were the real causes of the poor performance. Adding sampling of intrabar ticks while ignoring the deficiencies of the bar update code is a wonderful example of putting lipstick on a pig.

Here are the major problems:
  • Variables that only need to be calculated once (ONCE, not once per bar) are being calculated on every tick. These variables retain constant values over the entire life span of each instance of the indicator.
  • Variables that only need to be calculated once per bar are also being calculated on every tick. Avoiding that situation is why God gave us the if(FirstTickOfBar) condition.

These problems were a cinch to fix. I guess you have to be a non-programmer like me to notice things like that. C'mon guys, this is inexcusable.

The TSI is a NinjaTrader default indicator, itis not the worst indicator that I have seen. At least it calculates correctly.

I have tried to use the FirstTickOfBar() condition for many indicators, and it does not always improve performance. The problem is that I do not know what NinjaTrader does if I check for FirstTickOfBar ...

In this case the FirstTickOfBar condition would just save me a few multiplications, so I would not consider it worth using FirstTickOfBar. This does not invalidate the concept. For one of my MTF VWAPs, I was certainly able to improve performance over 1,000 times during news releases, by using both FirstTickOfBar and a recursive formula. It all depends what is in the bracket.

For the TSI I just would take out the divisions into OnStartUp(). This gives me a piglet without lipstick. The gain in performance was not measurable.



Attached Files
Register to download File Type: zip TSI_Piglet_No_Lipstick.zip (3.9 KB, 28 views)
Reply With Quote
The following 3 users say Thank You to Fat Tails for this post:
 
(login for full post details)
  #82 (permalink)
los angeles
 
Experience: Intermediate
Platform: nt
Trading: cl
 
Sunil P's Avatar
 
Posts: 292 since Jan 2012

Pig reminds me of my ex wife....biatch

A King Cobra is a more docile creature and trading CL is more predictable.

Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #83 (permalink)
Market Wizard
Berlin, Europe
 
Experience: Advanced
Platform: NinjaTrader, MultiCharts
Broker: Interactive Brokers
Trading: Keyboard
 
Fat Tails's Avatar
 
Posts: 9,855 since Mar 2010
Thanks: 4,238 given, 26,731 received


Sunil P View Post
Pig reminds me of my ex wife....biatch

A King Cobra is a more docile creature and trading CL is more predictable.

C# does neither stand for Cheating on Your Spouse, Cobra or CL. It is just a coding language.

Reply With Quote
The following user says Thank You to Fat Tails for this post:
 
(login for full post details)
  #84 (permalink)
los angeles
 
Experience: Intermediate
Platform: nt
Trading: cl
 
Sunil P's Avatar
 
Posts: 292 since Jan 2012

Sorry Fats ...I got confused with another forum and its more colorful vernicular that C and CL reminded me of...

As you noticed,I usually just crack jokes....nothing of hard substance in my comments...usually.

Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #85 (permalink)
los angeles
 
Experience: Intermediate
Platform: nt
Trading: cl
 
Sunil P's Avatar
 
Posts: 292 since Jan 2012

MIT OpenCourseWare | Electrical Engineering and Computer Science | 6.096 Introduction to C++, January IAP 2011 | Home


FYI

Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #86 (permalink)
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

During the lifetime of any bar, the output values of any DataSeries that have an index value other than zero do not change. Therefore the calculation of any algebraic expressions involving terms such as Close[1] only needs to be done OnFirstTickOfBar, NOT on every tick. There are about half a dozen of these expressions that can be isolated in Blau TSI. Some have arithmetic divisions in them, so not doing them on every tick is a big help since division is costly per Gomi.

I was surprised to see so much interest in this post... good!

@gomi, doesn't the calculation of 1/x entail division? Is multiplying by 1/x less costly than dividing by x?

FatTails, the piglet is so cute, what is his name? Does he have a Facebook® page?

Three things about FirstTickOfBar that we need to be aware of:
  • For historical data it is actually LAST Tick of Bar.
  • In a multiseries indicator you will get crazy results unless you are specific to a barsarray series For example If(BarsInPRogress==0 && FirstTickOfBar)
  • By the time FirstTickOfBar is true, the values of all the variables have ALREADY been set to the values appropriate to the new bar.

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to Zondor for this post:
 
(login for full post details)
  #87 (permalink)
Market Wizard
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,247 since Oct 2009
Thanks: 272 given, 4,415 received


Zondor View Post
@ gomi, doesn't the calculation of 1/x entail division? Is multiplying by 1/x less costly than dividing by x.

Of course calculation of 1/x implies a division, that's why I mentioned "if you have lost of divisions by a rarely changing x", so you can only make the division once , and use the result afterwards.

Also remember that with NT , you're likely to see the improvement only on very heavy numerical computations...

And a last word, only try to outsmart the compiler when you know what you're doing. For instance if you're only dividing by constant numbers (available at compile time), the compiler will optimize the division.
example : optimization - What's the fastest way to divide an integer by 3? - Stack Overflow

Reply With Quote
The following 2 users say Thank You to gomi for this post:
 
(login for full post details)
  #88 (permalink)
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

There are now two complementary approaches that, together, will give a big performance boost to Ninjatrader. And there is only one place on the web where you will find this information... Big Mike's. The two approaches are:

  1. Reducing resource usage of indicators

  2. Moving the historical chart and replay data from the System Disk to a faster or less busy device, ideally a RAM disk but alternately a non system SSD or HDD, as discussed in


Regarding the first item:

What is being called "Optimizing" of code should not be confused with the process of optimizing a strategy by adjusting parameters. Maybe code improvement should be referred to as "Economizing".

I think that the CPU consumption of some indicators can easily be reduced by one or two orders of magnitude (90 to 99%) however, I have never tried to measure the change in performance. I think this could be done by inserting code into a control version and an economized version of an indicator that would measure, and print to the output window, things like the total time in ms to backfill historical data, and the total time per bar spent processing OnBarUpdate events.

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following 3 users say Thank You to Zondor for this post:
 
(login for full post details)
  #89 (permalink)
Toledo, OH
 
 
Posts: 41 since Nov 2012
Thanks: 16 given, 46 received

Hello,
I agree with the previous post. There is nothing better than outputting/printing time to the output window before and after critical chunks of code to figure out the bottleneck. Pay attention to variables, array size for memory, switch statements instead of if/else where applicable and use of Ninja's multi threading. ( using OnMarketUpdate with OnBarUpdate to divvy up the functions as they will be called at different events but may accomplish the task faster together)

Thx

Reply With Quote
 
(login for full post details)
  #90 (permalink)
Cheltenham
 
Experience: Advanced
Platform: InvestorRT, NinjaTrader
Trading: ES
 
Posts: 54 since Jan 2012
Thanks: 8 given, 106 received


eDanny View Post

private EMA emathingy;
emathingy = EMA(CCI(CCIPeriod),10);

In the example above, there is a EMA and a CCI indicator, but a reference is created only for the EMA.

Should there be a reference for the CCI too, such as in the example below?


private EMA emathingy;
private CCI cmithingy;

cmithingy = CCI(CCIPeriod);
emathingy = EMA(cmithingy,10);

Reply With Quote
 
(login for full post details)
  #91 (permalink)
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received


Quoting 
Should there be a reference for the CCI too, such as in the example below?


private EMA emathingy;
private CCI cmithingy;

cmithingy = CCI(CCIPeriod);
emathingy = EMA(cmithingy,10);

Yes, as long as the value of CCIPeriod does not change within the indicator.

Once in a while, as in "Adaptive" indicators, the parameters of external classes vary under programmatic control.

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #92 (permalink)
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

The LinReg indicator does not have much of a performance impact but this example shows how it can be made more efficient.
  • Calls to the external SUM class eliminated. SUM algorithm moved into this indicator. Beats using a reusable instance of the external SUM class.
  • Variables that do not change intrabar are calculated only on FirstTickOfBar.
  • OnBarUpdate does not process redundant intrabar ticks.
  • Single precision "Float" variables reduce amount of arithmetic and memory load since a Float has half the bytes of a Double.

If you want to use this, rename the file to @LinReg and REPLACE the existing file that has that name.

"If we don't loosen up some money, this sucker is going down." -GW Bush, 2008
“Lack of proof that something is true does not prove that it is not true - when you want to believe.” -Humpty Dumpty, 2014
“The greatest shortcoming of the human race is our inability to understand the exponential function.”
Prof. Albert Bartlett
Attached Files
Register to download File Type: cs LinReg.cs (6.8 KB, 41 views)
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following 3 users say Thank You to Zondor for this post:
 
(login for full post details)
  #93 (permalink)
Legendary Bought TSLA at 880
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
 
Posts: 1,127 since Jul 2012
Thanks: 765 given, 2,536 received

@Zondor

You actually lose performance by downcasting those doubles. C# double addition takes 0.11 ns more than float addition on a 2 GHz machine, all else is the same. So downcasting is more expensive for your algorithm. I would only do it if the memory bandwidth was comparable to the L2 cache.

Reply With Quote
The following 2 users say Thank You to artemiso for this post:
 
(login for full post details)
  #94 (permalink)
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

Thanks @artemiso, I was wondering about that. The float downcasting is a sledgehammer approach.

I was brought up in the days of slide rules, so am very aware that doing arithmetic with excessive non signficant precision is very wasteful. But I am not sure that there is any good way to curtail it. Math.Round, Trunc, and string formatting don't sound like good options either.

Noticed that lots of people have looked at this post. You are probably admiring my necktie and wondering where you can get your own.

"If we don't loosen up some money, this sucker is going down." -GW Bush, 2008
“Lack of proof that something is true does not prove that it is not true - when you want to believe.” -Humpty Dumpty, 2014
“The greatest shortcoming of the human race is our inability to understand the exponential function.”
Prof. Albert Bartlett
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following 2 users say Thank You to Zondor for this post:
 
(login for full post details)
  #95 (permalink)
Legendary Bought TSLA at 880
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
 
Posts: 1,127 since Jul 2012
Thanks: 765 given, 2,536 received


Zondor View Post
Thanks @artemiso, I was wondering about that.

Doing arithmetic with excessive non signficant precision is very wasteful, but I am not sure that there is any good way to curtail it. Math.Round, Trunc, and string formatting don't sound like good options either.

You're welcome. Unfortunately, modern processors are so fast that the primitives nearly all take 1 clock cycle. The extra precision is indeed wasteful, but it's here to stay.

Reply With Quote
The following user says Thank You to artemiso for this post:
 
(login for full post details)
  #96 (permalink)
Lubbock TX
 
Experience: Intermediate
Platform: NinjaTrader
Broker: Stage 5 Trading
Trading: CL
 
MWinfrey's Avatar
 
Posts: 1,880 since Jul 2009
Thanks: 1,450 given, 3,325 received


Zondor View Post
RWT-004 :: Move the Markets


Very interesting post at MoveTheMarkets. The way we usually do things is terribly inefficient and there is a better way:

"In all of the coding tutorials and examples I've seen, Ninjatrader indicators are used in-line. So, if you want to know when the SMA is going up, you'd check:
SMA(Close, 5)[0] > SMA(Close, 5)[1]
I find this practice repulsive for two reasons: redundancy and inefficiency. This paper chronicles the path I took to fifinding a better way. If you just want to know what the better way is, skip to the 'Solution' section at the end......"

@Zondor, @Richard

That movethemarkets link does not work any longer. Any ideas?

Reply With Quote
 
(login for full post details)
  #97 (permalink)
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

@Richard closed it down to pursue other interests, but hopefully is still around on futures.io (formerly BMT).

Too bad, there was some great stuff posted there.

"If we don't loosen up some money, this sucker is going down." -GW Bush, 2008
“Lack of proof that something is true does not prove that it is not true - when you want to believe.” -Humpty Dumpty, 2014
“The greatest shortcoming of the human race is our inability to understand the exponential function.”
Prof. Albert Bartlett
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to Zondor for this post:
 
(login for full post details)
  #98 (permalink)
Market Wizard
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,247 since Oct 2009
Thanks: 272 given, 4,415 received


MWinfrey View Post
@Zondor, @Richard

That movethemarkets link does not work any longer. Any ideas?

RWT-004 :: Move the Markets ?

Found it on NT support site
Best practice for programming efficient indicators for the NT Strategy Analyzer? - NinjaTrader Support Forum

Reply With Quote
The following 7 users say Thank You to gomi for this post:
 
(login for full post details)
  #99 (permalink)
Orange County, CA / United States
 
Experience: Advanced
Platform: NinjaTrader, Excel/VBA
Broker: IB / Kinetick / eSignal
Trading: Equities
 
Posts: 17 since Nov 2012
Thanks: 12 given, 16 received

I'm learning more about this topic and found this thread really helpful. Thank you!

Here is the movethemarkets PDF "Calling Upon Indicators Efficiently in NinjaScript" (attached)

Also, found this thread which has some more good info on this topic:

Best practice for programming efficient indicators for the NT Strategy Analyzer
Best practice for programming efficient indicators for the NT Strategy Analyzer? - NinjaTrader Support Forum

Attached Thumbnails
Anyone have any hints for optimizing C# code?-calling-upon-indicators-efficiently-ninjascript.pdf  
Reply With Quote
The following 2 users say Thank You to ntvola for this post:
 
(login for full post details)
  #100 (permalink)
Calgary AB
 
Experience: Intermediate
Platform: AmiBroker, IB
Broker: Interactive Brokers
Trading: Stocks
 
Posts: 10 since Mar 2013
Thanks: 5 given, 2 received

How would you go about this "economization" of indicators if you are using a multi-instrument strategy?

ie, if you declared

 
Code
class MyIndicator : Indicator {
private SMA sma1;
private SMA sma2;
...
protected void override OnStartup() {
sma1 = SMA(Close,5);
sma2 = SMA(Close, 13);
...
}
How could you then call sma1 or sma2 for a BarsArray[index] item?

Reply With Quote


futures io Trading Community Platforms and Indicators NinjaTrader > Anyone have any hints for optimizing C# code?


Last Updated on December 24, 2015


Upcoming Webinars and Events
 

NinjaTrader Indicator Challenge!

Ongoing
 

Battlestations! Show us your trading desk - $1,500 in prizes!

March
 

Importance of Finding Your Own Way w/Adam Grimes

Elite only
 

Journal Challenge w/Jigsaw

April
     



Copyright © 2021 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts