Indicator performance bottleneck - NinjaTrader Programming | futures io social day trading
futures io futures trading


Indicator performance bottleneck
Updated: Views / Replies:2,116 / 24
Created: by Adamus Attachments:1

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 1  
 
Thread Tools Search this Thread
 

Indicator performance bottleneck

  #1 (permalink)
Elite Member
London, UK
 
Futures Experience: Beginner
Platform: NinjaTrader, home-grown Java
Broker/Data: IB/IQFeed
Favorite Futures: EUR/USD
 
Adamus's Avatar
 
Posts: 1,085 since Dec 2010
Thanks: 470 given, 778 received

Indicator performance bottleneck

I wrote an indicator to display support and resistance levels and a few other types of horizontal lines on the chart. It's got pretty complex but that's not to say that it's necessarily a pile of spaghetti - I've started from scratch on a couple of occasions when I've found that the design I had was no longer working.

At this point the indicator that does everything required of it finally, after months of work, but it's got a bad performance problem in that the time to process each historical bar creeps up steadily, making it a pain to load more than about 1500 bars (less 3 days on a 3-min chart), which is stifling my ability to use the indicator to do historical research on the setups I use it for.

I log the time each OnBarUpdate takes. It starts off at 20 millisecs for each OBU and varies around 20 ms, but every 10 or 20 bars it increases by roughly 10 ms. After about 1000 bars it gets up to over 100 ms per OBU. That's like a second for every ten bars, so another day (480 bars) takes almost a minute and then longer and longer.

Despite knowing the indi inside out, I don't really know where to look to see what is causing the problem. I'm not a C# expert so I don't know what the expensive operations are in the code. What foxes me is that I can see from my logging that the indicator can have nothing more to do from one bar to the next over perhaps 100 bars, but the performance degrades anyway.

A few of the things I'm doing that might be causing problems are:
  • using several lists and searching them with delegate/predicate methods
  • using dictionaries in the same way
  • instantiating new objects a lot
  • these objects are based on a class with 2 subclasses, so lots of inheritance going on
  • using multiple time-frames
  • pre-loading data myself with code using GetBars() (this is quick but I thought it might have unknown side-effects)

That list represents my ignorance of C# rather than my insightful experience. Here's what it's doing:

Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


I can post more info, data, logs, whatever - if any C# coders out there have come across this sort of unidentifiable bottle-neck before, I'd love to hear about it and perhaps get some tips on where to investigate.

You can discover what your enemy fears most by observing the means he uses to frighten you.
Reply With Quote
 
  #2 (permalink)
Quick Summary
Quick Summary Post

Quick Summary is created and edited by users like you... Add FAQ's, Links and other Relevant Information by clicking the edit button in the lower right hand corner of this message.

 
  #3 (permalink)
Elite Member
Birmingham UK
 
Futures Experience: Intermediate
Platform: NinjaTrader
Broker/Data: IG/eSignal
Favorite Futures: Dax
 
ratfink's Avatar
 
Posts: 3,340 since Dec 2012
Thanks: 11,282 given, 7,092 received


Just sounds like you need to work out a lot more clearly what it needs to do, rather than what it is doing. I can offer speed-up tips, but to be honest you want to be in the micro-second ranges at worst anyway, so I'm guessing at this stage it's orders of magnitude too slow and clearly running geometrically or exponentially so needs a major redesign. All the bullet points you mention are found in many unwieldy OO implementations, I suggest you do some pondering and look for some KISS approaches instead.

And check for obvious no-no's like redrawing every Ninja rectangle from the beginning of time on every tick and such like....

Travel Well

Last edited by ratfink; November 28th, 2013 at 10:46 AM.
Reply With Quote
The following user says Thank You to ratfink for this post:
 
  #4 (permalink)
Elite Member
London, UK
 
Futures Experience: Beginner
Platform: NinjaTrader, home-grown Java
Broker/Data: IB/IQFeed
Favorite Futures: EUR/USD
 
Adamus's Avatar
 
Posts: 1,085 since Dec 2010
Thanks: 470 given, 778 received


ratfink View Post
Just sounds like you need to work out a lot more clearly what it needs to do, rather than what it is doing. I can offer speed-up tips, but to be honest you want to be in the micro-second ranges at worst anyway, so I'm guessing at this stage it's orders of magnitude too slow and clearly running geometrically or exponentially so needs a major redesign. All the bullet points you mention are found in many unwieldy OO implementations, I suggest you do some pondering and look for some KISS approaches instead.

And check for obvious no-no's like redrawing every Ninja rectangle from the beginning of time on every tick and such like....

OK I can see you come from the kill it fast and kill it hard school of software development

However the Plot is fine. I've got good durations for each call to Plot(). It's just historical bar processing.

In my defence though, the complexity of the indicator is necessary to keep the visual display on the chart simple, e.g. if I've got 5 lines across 10 points, it'll merge them so you see one zone on the chart, not 5 clustered lines. But I will put my mind to it on how to KISS it. I don't want to be guilty of trying to justify a pile of horseshit.

Another thing I just discovered: the bar update duration drops back down to sub-millisecond levels once it starts processing live bars.

Why? What is the difference between historical and live bars OnBarUpdates?

You can discover what your enemy fears most by observing the means he uses to frighten you.
Reply With Quote
 
  #5 (permalink)
Elite Member
@ Germany
 
Futures Experience: Beginner
Platform: NinjaTrader
Broker/Data: Mirus Futures/Zen-Fire
Favorite Futures: FDAX
 
Posts: 441 since Nov 2011
Thanks: 254 given, 369 received

@Adamus It's impossible to help you out without looking at the source codes. You can check if your code iterates thru the whole set of historical bars at new bar arrival (include some function loop counter). Have you used the debugger to resolve your problems?

Reply With Quote
The following user says Thank You to Koepisch for this post:
 
  #6 (permalink)
Elite Member
Birmingham UK
 
Futures Experience: Intermediate
Platform: NinjaTrader
Broker/Data: IG/eSignal
Favorite Futures: Dax
 
ratfink's Avatar
 
Posts: 3,340 since Dec 2012
Thanks: 11,282 given, 7,092 received


Adamus View Post
OK I can see you come from the kill it fast and kill it hard school of software development

You bet. In 40 years I have met many writers of correct software but, since OO and Ghz CPUs, a lot less writers of (correct && efficient) software.

On the other hand you may just be unlucky at this stage and have a simple gotcha in there that just needs hunting down. It sounds like you know what you are doing, maybe just need to dig deeper. My first thought was that you are exponentially processing all bars on every tick or something, but I'm only guessing at a distance so can't really help that much.

Travel Well
Reply With Quote
 
  #7 (permalink)
Elite Member
Birmingham UK
 
Futures Experience: Intermediate
Platform: NinjaTrader
Broker/Data: IG/eSignal
Favorite Futures: Dax
 
ratfink's Avatar
 
Posts: 3,340 since Dec 2012
Thanks: 11,282 given, 7,092 received


Adamus View Post
Why? What is the difference between historical and live bars OnBarUpdates?

There are several, but one that comes to mind is the drawing of off-screen graphical artefacts, Ninja doesn't seem too good at clipping, e.g. particularly of rectangles/triangles, which is why I mentioned them. You can speed up a lot of things by not drawing unnecessarily, but it is always worth proving what is taking the time first just by commenting out calls, etc. Lots of red-herrings to find and eliminate when you really start to run a speed-up exercise! It can always be our own code or just unrealistic expectations of somebody else's.

Travel Well
Reply With Quote
The following user says Thank You to ratfink for this post:
 
  #8 (permalink)
Elite Member
Birmingham UK
 
Futures Experience: Intermediate
Platform: NinjaTrader
Broker/Data: IG/eSignal
Favorite Futures: Dax
 
ratfink's Avatar
 
Posts: 3,340 since Dec 2012
Thanks: 11,282 given, 7,092 received

Just another quick point, getting things up into the milli-second rather than micro-second levels usually takes memory or even hard page faults to become an issue, have you eliminated that side of things? (i.e. the TM is showing (100/number-of-cores)% flat-out for Ninja and no hard faults so it is all cpu). Would be unusual but not unknown, just depends what you are doing.

Travel Well
Reply With Quote
 
  #9 (permalink)
Market Wizard
virginia
 
Futures Experience: Intermediate
Platform: ninja
Favorite Futures: ES
 
cory's Avatar
 
Posts: 5,220 since Jun 2009
Thanks: 628 given, 6,318 received
Forum Reputation: Legendary

set a calculation kill switch on after calculation on history data is done.

Reply With Quote
The following user says Thank You to cory for this post:
 
  #10 (permalink)
Elite Member
London, UK
 
Futures Experience: Beginner
Platform: NinjaTrader, home-grown Java
Broker/Data: IB/IQFeed
Favorite Futures: EUR/USD
 
Adamus's Avatar
 
Posts: 1,085 since Dec 2010
Thanks: 470 given, 778 received



Koepisch View Post
@Adamus It's impossible to help you out without looking at the source codes. You can check if your code iterates thru the whole set of historical bars at new bar arrival (include some function loop counter). Have you used the debugger to resolve your problems?

Hi Koepisch, good comment, thanks. I shall check while I am going through adding more detailed performance monitoring.

The code runs to 2800 lines. I assume nobody would want to look at it - I can't imagine I'd want to look at somebody else's dogfood. If I'm wrong, I'd gladly attach it - just say the word.

You can discover what your enemy fears most by observing the means he uses to frighten you.
Reply With Quote

Reply



futures io > > > > > Indicator performance bottleneck

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Jigsaw Trading: TBA

Elite only

FuturesTrader71: TBA

Elite only

NinjaTrader: TBA

Jan 18

RandBots: TBA

Jan 23

GFF Brokers & CME Group: Futures & Bitcoin

Elite only

Adam Grimes: TBA

Elite only

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Performance impact of calling multiple instances of an external indicator Big Mike NinjaTrader Programming 12 June 27th, 2015 04:47 AM
Strategy Performance JamesCBO NinjaTrader 11 October 20th, 2013 05:48 AM
How do you see the overnight performance? arnie Traders Hideout 3 January 28th, 2013 11:26 PM
Relative Performance Indicator soulartist NinjaTrader 4 November 22nd, 2010 10:21 PM
Performance Indicator MXASJ NinjaTrader Programming 2 April 16th, 2010 01:13 AM


All times are GMT -4. The time now is 06:30 AM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-17 in 0.14 seconds with 20 queries on phoenix via your IP 54.82.81.154