Indicator performance bottleneck

November 28th, 2013, 05:29 PM

ratfink

Just another quick point, getting things up into the milli-second rather than micro-second levels usually takes memory or even hard page faults to become an issue, have you eliminated that side of things? (i.e. the TM is showing (100/number-of-cores)% flat-out for Ninja and no hard faults so it is all cpu). Would be unusual but not unknown, just depends what you are doing.

I had to look up what hard faults are and then I discovered that my resource monitor doesn't display properly, according to Microsoft the work-around for an incompatibility with IE10 is that I reduce my display font size to 100%, but Windows now wants me to log out and log back in so this will have to wait.

I see a 25% CPU utilisation which is 100% on one core when loading the indi. As I said can't tell yet about hard faults (what should I see?)

November 29th, 2013, 01:44 AM

Adamus

I see a 25% CPU utilisation which is 100% on one core when loading the indi. As I said can't tell yet about hard faults (what should I see?)

Then it is just a total cpu hog so the hard faults will likely be showing zero anyway and can be ignored. Definitely check for redundant drawing, calculation and object allocation operations, poor data fetch/create methods, excessive routine calling, unexpected looping, etcetera. Tests including dumb prints on small sets, eliminations and and binary chop style commenting out, your mission, should you choose to accept it, starts now.

November 29th, 2013, 04:05 AM

cory

set a calculation kill switch on after calculation on history data is done.

hi cory, at first I didn't understand and just thought 'oh well' but then I saw you have an awesome 3,777 thank-yous and a higher thank-you to post ratio so I figure what you said is probably well worth heeding and I better find out what you mean. What confuses me is that all the calculations I do on history data need to be on every live bar as well, and in fact I don't think I've ever written anything for Ninja where that wasn't so. I do occasionally use the 'Historical' flag to switch when there is something hacky going on with historical vs live data or processing, but usually not. Could you expand a little on what you meant? Thanks.

November 29th, 2013, 01:49 PM

Adamus

hi cory, at first I didn't understand and just thought 'oh well' but then I saw you have an awesome 3,777 thank-yous and a higher thank-you to post ratio so I figure what you said is probably well worth heeding and I better find out what you mean. What confuses me is that all the calculations I do on history data need to be on every live bar as well, and in fact I don't think I've ever written anything for Ninja where that wasn't so. I do occasionally use the 'Historical' flag to switch when there is something hacky going on with historical vs live data or processing, but usually not. Could you expand a little on what you meant? Thanks.

see if this help

Quoting

.....

public class NoGapBarbyBar : Indicator
{
#region Variables
// Wizard generated variables
// User defined variables (add any user defined variables below)
private DateTime currentDate = Cbi.Globals.MinDate;
private DateTime now;
private DateTime prevDay;
private DateTime prev2Day;
private DateTime prev3Day;

.....
#endregion

/// <summary>
/// This method is used to configure the indicator and is called once before any bar data is loaded.
/// </summary>
protected override void Initialize()
{
CalculateOnBarClose = true;
Overlay = true;

now = DateTime.Today; // today date
prevDay = now.AddDays(-1); // yesterday date
prev2Day = now.AddDays(-2);
prev3Day = now.AddDays(-3);
....

}

/// <summary>
/// Called on each bar update event (incoming tick)
/// </summary>
protected override void OnBarUpdate()
{

// do one (firsttickofbar) for each live bar (bar date = today date) only
//
if (FirstTickOfBar && Time[0].Date == now)
{

// Print("open 0 "+Open[0]+ " close 1 " +Close[1] + " open 1 "+Open[1] + "Close 2 "+Close[2] + " open 2 "+Open[2]);
// do calculation
.....

} // end first tick

} // end on bar update

...

November 29th, 2013, 02:02 PM

Thanks Cory for that. I see what you mean but I am doing it the way it is for the very reason that I can do that. I need to seriously think about whether I want to carry on that way. I also found the main culprit so if I can't fix it, I'll be into major re-design territory.

Now got a smaller screen resolution to make the resource manager work and no page faults. I might have to get more powerful reading glasses though.

ratfink

Tests including dumb prints on small sets, eliminations and and binary chop style commenting out, your mission, should you choose to accept it, starts now.

I should employ Dilbert Jayavarman via freelancer.com to help.

Dumb prints are my forte - no worries there - although I suspect the eliminations I've managed so far owe more to the curry I had last night

Binary chop style commenting out sounds like an art form.

My experiments so far revealed some fairly embarrassing dog food but nothing that would go further than a peer code review if I was gainfully employed

plus I've learnt too much about C#

It seems that .NET linq allows you to do all sorts of cool stuff but it runs like a snail. In all my C# Ninjascripting naivety I just wanted to find out if 2 lists of objects are the same, so I did this:

Code

IEnumerable<SRLevel> firstNotSecond = mergedLevelsList.Except(
	otherMergedLevels);
IEnumerable<SRLevel> secondNotFirst = 
	otherMergedLevels.Except(mergedLevelsList);
if (firstNotSecond.Count() > 0) return false;
if (secondNotFirst.Count() > 0) return false;
return true;

That alone would have been alright but every now and again the routine needs to go over info from all the bars - which is exactly what everyone here was warning about.

I might be able to send back my laptop cooler when this is sorted.

December 5th, 2013, 09:16 AM

I have achieved great performance with the OnBarUpdate after re-coding some of the loops I was doing, but I am still not satisfied with the performance of the Plot()

Can anybody who's been there give me a faster way of doing this:

'level' is my simple data class containing the prices to draw at and the bars to display on. I keep them in a list for each bar in a dictionary indexed by bar.

Code

for (int i = 0; i < BarsArray[0].Count; i++) //loop over all bars
{
    if (i < ChartControl.FirstBarPainted) continue;
    if (i > this.LastBarIndexPainted) continue;
    List<SRLevel> levels = srLevels[i];
    for (int x = 0; x < levels.Count; x++)
    {
        SRLevel level = levels[x];
        int leftX = ChartControl.GetXByBarIdx(BarsArray[0], i);
        int rightX = ChartControl.GetXByBarIdx(BarsArray[0], i + 1);
        int width = rightX - leftX; 
        // higher price is lower Y - Y origin is top
        int startY = ChartControl.GetYByValue(this, level.topY) - displayMargin; 
        int stopY;
        if (level.topY.Compare(level.bottomY, PRECISION) == 0)
            stopY = startY + displayMargin;
        else
            stopY = ChartControl.GetYByValue(this, level.bottomY) + displayMargin; 
        int height = stopY - startY;
        if (height < 1) height = 1;
        int labelY = 0; 
        //LastBarPainted only references primary dataseries
        if (level.behaviour == SRLevelBehaviour.SUPPORT)
        {
            brush.Color = supportColor;
            labelY = stopY + textFont.Height / 2; // labelY is for text
        }
        else if (level.behaviour == SRLevelBehaviour.RESISTANCE)
        {
            brush.Color = resistanceColor;
            labelY = startY - textFont.Height;
        }  
        graphics.FillRectangle(brush, leftX, startY, width, height);
    }
}

It seems ChartControl.GetXByBarIdx() and GetYByValue() are costly operations. Is there any other way of doing this?

December 5th, 2013, 10:19 AM

ninja performance tip
Last Historical Bar? - [AUTOLINK]NinjaTrader[/AUTOLINK] Support Forum

December 5th, 2013, 10:37 AM

Adamus

...
It seems ChartControl.GetXByBarIdx() and GetYByValue() are costly operations. Is there any other way of doing this?

@Adamus: Nice coding so far. You can test that you don't call the Get.. functions every time. If you call it 2 times for 2 consecutive IDXs then you can interpolate the other values. You can also check if "List<SRLevel> levels = srLevels[i];" is copying your list (worse) - there is no need for this, work with srlevels[i] directly.

December 5th, 2013, 10:38 AM

cory

see if this help

Hey Cory,

Your example references FirstTickOfBar and appears to be CalcOnbarclose=true. According to the ninja reference this would not work.

"10.4.3.26 FirstTickOfBar
Definition
Indicates if the incoming tick is the first tick of a new bar. This property is only of value in
scripts that run tick by tick which is when the CalculateOnBarClose property is set to false."

Did I miss something?

Regards,

December 5th, 2013, 11:07 AM

Tasker_182

Hey Cory,

Your example references FirstTickOfBar and appears to be CalcOnbarclose=true. According to the ninja reference this would not work.

"10.4.3.26 FirstTickOfBar
Definition
Indicates if the incoming tick is the first tick of a new bar. This property is only of value in
scripts that run tick by tick which is when the CalculateOnBarClose property is set to false."

Did I miss something?

Regards,

use FirstTickOfBar so you don't have to set CalcOnbarclose=true.

Indicator performance bottleneck

Discussion in NinjaTrader

Indicator performance bottleneck