Slicing ISeries objects

March 3rd, 2022, 11:44 AM

This post is about my creating an alternative to array slicing for NinjaScript, since NT8 is still under the thumb of C#5. Please let loose your suggestions and criticism, or maybe you've already done this in a better way. I'd be glad to hear what y'all have to say. Ultimately, my hope is that it can help other NinjaScript coders.

Onward.

As a recovering MATLAB user...well...who am I kidding. I actually do like MATLAB. One of the things I especially like, in fact, is its array-slicing operator, the ":":

Code

apple = 1:1e6; % Creates the array [1 2 3 4 ... 999999 1000000].
sliceOfApple = apple(1:3:end); % Selecting every third value.

I'd like to do this for creating slices out of ISeries objects. C# 8(?) gives us a reasonable alternative, the ".." operator, but again, NinjaTrader does not yet give us such treasures. As far as I understand it, the coder has a couple options for creating this functionality, namely operator overloading and method extensions.

And if ISeries objects were LINQable, that would also present an easy alternative. I did experiment with this in a roundabout way but the code is...how do we say...ugly?

Code

// _x is a double[] of ordinal numbers; the independent variable (0, 1, 2, 3...).
// _dataArray will be the slice out of the ISeries object.

_x.ToList().ForEach(index => _dataArray[(int)(_x.Count() - index)] = Input[(int)index]); // And of course you can change this depending on how you want things ordered.

What I dislike about this approach, though, is its requirement to pass in the independent variable object, and I'd much rather have a dynamic way of doing it such that I wouldn't have to be constantly recreating that vector.

As a somewhat new C# coder, operator overloading seems hairy, to me. A method extension seemed simple enough, so I chose ISeries<double>.Range. After all that preamble, here is what I wanted to share in this post and get suggestions about:

Code

using NinjaTrader.Cbi;

namespace NinjaTrader.NinjaScript
{public static class MyUtilities
{public static double[] Range(this ISeries<double> series, int startBarsAgo, int endBarsAgo)
{if (startBarsAgo < 0 || startBarsAgo <= endBarsAgo)NinjaTrader.NinjaScript.NinjaScript.Log("Some error message", LogLevel.Error);int numValues = startBarsAgo - endBarsAgo + 1;
double[] values = new double[numValues];

for (var ii = startBarsAgo; ii >= endbarsAgo; ii--)
{if (series.IsValidDataPoint(ii))values[startBarsAgo - ii] = series[ii]; // values vector will be built in the same order as they're indexed in the ISeries object.}

return values;
}

To use it in an indicator, now, I do the following:

Code

_dataArray = Input.Range(Period, EndBarsAgo);
// or...
_dataArray = Input.Range(12, 1)

Well, that's it. I hope it helps. And again, please let me know your thoughts. I did approach this from the perspective of needing to create the independent-variable object for an indicator I was writing, so there's the possibility I pidgeon-holed myself to a specific approach.

--Caleb

March 3rd, 2022, 01:21 PM

I'm unfamiliar with matlab and "slicing" but if you mean taking an array and breaking it up into 2 or 3 smaller arrays, doesn't Array.Copy work with iSeries? I haven't tried that, but I would think that would be much faster than iterating through your array.

March 3rd, 2022, 01:45 PM

Update:
Return the independent and dependent variable vectors. I used a jagged array so I could do vector assignment in one line, outside of the loop.
Method summary added.

Code

/// <summary>
/// Slice ISeries object.
/// </summary>
/// <param name="series">ISeries object</param>
/// <param name="startBarsAgo>Start index</param>
/// <param name="endBarsAgo>End index</param>
/// <returns>Jagged array: [0] is independent variable vector; [1] is dependent variable vector.</returns>
public static double[][] Range(this ISeries<double> series, int startBarsAgo, int endBarsAgo)
{if (startBarsAgo < 0 || startBarsAgo <= endBarsAgo)NinjaTrader.NinjaScript.NinjaScript.Log("Indexing error", LogLevel.Error);int numValues = startBarsAgo - endBarsAgo + 1;
double[][] values = new double[2][];
values[0] = Enumerable.Range(endBarsAgo, numValues).Select(Convert.ToDouble).ToArray();
values[1] = new double[numValues];

for (var ii = startBarsAgo; ii >= endBarsAgo; ii--)
{if (series.IsValidDataPoint(ii))values[1][startBarsAgo - ii] =  series[ii];}

return values;
}

March 3rd, 2022, 01:58 PM

chipwitch

I'm unfamiliar with matlab and "slicing" but if you mean taking an array and breaking it up into 2 or 3 smaller arrays, doesn't Array.Copy work with iSeries? I haven't tried that, but I would think that would be much faster than iterating through your array.

By "slicing" I mean cutting out elements of an array and assigning the subset to a new object. For example, let's say you want the last ten elements of Input. Input.Range(10, 1), as I have it, would be equivalent to the following:

Code

double[] vector = new double[10];

for (var ii = 10; ii >= 1; ii--) vector[10 - ii] = Input[ii];

Slicing simply hides the for loop.

That would be interesting, if I could use Array, somehow. So far, I've not been able to successfully compile anything in which I pass an ISeries<double> as an argument to an Array method. E.g. _dataArray = Array.Copy(Input, 60, _dataArray, 0, 60); I may not have the remaining arguments correct, but I can't compile to troubleshoot. Error regards conversion of NinjaScript.ISeries<double> to System.Array.

March 3rd, 2022, 06:25 PM

I'm not sure I totally understand exactly what the issue is, but I think I may have a use-case, albeit different?

Here I have a big list... but I only do operations on a small segment of that list. In this case, it is simply calculating an average.

So, at the class level... I define the List that will hold my vals:

Code

List<double> ADRs3 = new List<double>();

In onBarUpdate(), I push values into this list either by:

Code

ADRs3.Add(someValue); //Which pushes it to the end of the list

or I can change the LATEST value IN PLACE by:

Code

ADRs3[ADRs3.Count-1] = someValue;

If I ever need to iterate over that list to see what's in it, I can do something like, on the last bar on the chart, iterate over the list once an show me the values:

Code

if (CurrentBar==Bars.Count-2) //Call for last bar on chart( ==1532
{
  for (int i=0; i < ADRs3.Count; i++)
    {
      Print( i+ "  " +   ADRs3[i]  );					
    }
 }

To the part about taking a slice of the list.... So, the INDICATOR simply calculates an average of a daily r@nge. So the list contans ALL values on the chart, but I only want to use a SLICE of that data...

So I have a function that takes the list... makes a COPY(at least thats what I think its doing?), and I only calculate and average based on the last n-values, using that GetRange extension:

Code

public double GetAverage( List<double> myList, int LookBackPeriod )	//Gets pipsize given a string value
{
  myList = myList.ToList().GetRange(myList.Count() - LookBackPeriod, LookBackPeriod); 	//Reduce list to on last n-samples
  return Math.Round( myList.Average(),2);
}

When I need to calculate that average, I pass my main list to the function, along with how many items to calculate the average for... in a function call via:

Code

s3Avg = GetAverage( ADRs3.ToList(), Period); //Get ADR

I'm not sure if this is helpful, but when I need to store data, instead of using arrays, I use Lists... and this is an example of only using a slice of that list to perform calculations on.

The key point, is that I am not modifying the original list, I'm am essentially copying it, and modifying the copy.

March 3rd, 2022, 07:16 PM

if you can use forrestang's suggestion, there is a copy method for List. I have never used it though. I couldn't find a way to copy an ISeries.

March 3rd, 2022, 11:21 PM

It's not really an issue, just a desire. I just wanted an easier way to take chunks out of the ISeries objects. @forrestang, your one block of code where you run a for-loop to inspect the values in your List is similar to what is required to extract values from ISeries objects, but only because NT is still limited by C# 5. With the latest version of C#, whenever NT gets updated to it, this method extension's usefulness will likely expire.

Let me describe one use case for why using a List or something similar is inconsequential. I have a numeric-derivative indicator that uses the Math.NET library. Let's say I want to do a three-point numeric derivative of, say, NT's built-in linear regression indicator with a period of 14. (There's no need for this since LinRegSlope already exists, but it makes for a good example since people actually use it.)

Prereqs:

Code

private double[] _threePointVector = double[3];
private NumericalDerivative _D = new NumericalDerivative(3, 1);

Ideally, if array slicing was possible, we would do the following to get the values from the last three bars of LinReg(14):

Code

 _threePointVector = LinReg(14)[3:1];

Since this is not possible, the following is required (as far as I know, anyway):

Code

for (var ii = 3; ii > 0; ii++)
{
    _threePointVector[3 - ii] = LinReg(14)[ii];
    // The [3 -ii] indexing insures the values are put in the same order into _threePointVector as they are in LinReg(14).  You could just as easily call Reverse() for the array after you build it.  It's all the same.
}

// And then the difference calculation:
double diff = _D.EvaluateDerivative(_threePointVector, 1, 1);

So _threePointVector had to be made no matter what, and whether it's a double[] or List or Queue or whatever is irrelevant (although better as double[], in this case), but it had to be made from values stored in the ISeries object, LinReg(14). It's the ISeries object I want to slice, so this method extension would work for PriceSeries objects and the rest, also. Granted, the for loop is short, but I'd still like to hide it and simplify the expressions. Notably, this feature must have been a popular C# request such that slicing is now a part of the latest version.

So with, say, the Range method extension, I can cut out the for loop to the following:

Code

_threePointVector = LinReg(14).Range(3, 1);
double diff = _D.EvaluateDerivative(_threePointVector, 1, 1);

Does that clarify things, at all? I'd still like to learn about operator overloading, which could have an even simpler appearance, but I'm not there, yet.

June 6th, 2022, 02:18 PM

Caleb,
Your static class was exactly what I was looking for. Thanks very much. I am so pleased to forgo the for loops since we know there is a better way.
Thanks,
Warren

TigerStripes · June 14th, 2022, 12:35 PM

benJephunneh

By "slicing" I mean cutting out elements of an array and assigning the subset to a new object. For example, let's say you want the last ten elements of Input. Input.Range(10, 1), as I have it, would be equivalent to the following:

Code

double[] vector = new double[10];

for (var ii = 10; ii >= 1; ii--) vector[10 - ii] = Input[ii];

Slicing simply hides the for loop.

That would be interesting, if I could use Array, somehow. So far, I've not been able to successfully compile anything in which I pass an ISeries<double> as an argument to an Array method. E.g. _dataArray = Array.Copy(Input, 60, _dataArray, 0, 60); I may not have the remaining arguments correct, but I can't compile to troubleshoot. Error regards conversion of NinjaScript.ISeries<double> to System.Array.

So part of your code I may be new but not sure why "StartBarsAgo < 0" when to start you want bars to begin, or if it is to instantiate a variable you could do that pre if conditional and then assign a more literal instatiation when the time is right.

Also it is possible you could use the items itterate inside a list, give conditions for when outside the list a conditions is met and inside the list itterate based on a set of conditions to allow usage to conditions only for ceratin int funtions or values.

EDIT: what mentioned here is exactly as forrstang had mentioned above, again newbie and not versed in classes as well but sure that there are important features, and think .Range is a useful function as well hope to see more improvments possible in updates of NT, as well thnking ab when NT becomes obselete or there is a "9" rather useful imporvments in the .NET and C# current code base.

October 8th, 2023, 09:08 PM

This came up in a separate conversation, so I'm adding a reply merely to ensure the code is up to date. If anybody has modified it, I'd be glad to hear what improvements you've made.
One thing I may have forgotten to mention is that I wanted the returned array to be indexable in the same order as NinjaTrader's array ordering. Close[0], for example, gets the last (latest) value, whereas C# uses the 0-index to get the first array value. So if I create a slice such as

Code

double[] vals = Close.Range(2, 0)[1];

, I build the C# array such that vals[0] returns what Close[0] would return.

I removed the error checking (e.g. checking for negative values of endBarsAgo) because I do that separately, but it's easy enough to combine them as you like. Combining them would likely save some clock cycles, but I don't personally have anything plugged into NT that would make that important.

Code

        /// <summary>
        /// Slice ISeries object.
        /// </summary>
        /// <param name="series">ISeries object</param>
        /// <param name="startBarsAgo">Earliest bar of range.</param>
        /// <param name="endBarsAgo">Latest bar of range.</param>
        /// <returns>Jagged array: [0] is independent variable vector; [1] is dependent variable vector.</returns>
        public static double[][] Range(this ISeries<double> series, int startBarsAgo, int endBarsAgo = 0)
        {
            double[][] slice = new double[2][];
            int numValues = startBarsAgo - endBarsAgo + 1;

            slice[0] = Enumerable.Range(endBarsAgo, numValues).Select(Convert.ToDouble).ToArray();
            slice[1] = new double[numValues];

            foreach (int point in slice[0])
                slice[1][point - endBarsAgo] = series[point];

            return slice;
        }

Here is an error checking method:

Code

        /// <summary>
        /// Check the series for valid data points in a given range.
        /// </summary>
        /// <param name="series">ISeries object</param>
        /// <param name="startBarsAgo">Earliest bar of range to verify.</param>
        /// <param name="endBarsAgo">Latest bar of range to verify.</param>
        /// <returns>True if all data points exist, otherwise false.</returns>
        public static bool VerifyRange(this ISeries<double> series, int startBarsAgo, int endBarsAgo = 0)
        {
            if (endBarsAgo < 0)
            {
                NinjaScript.Log(string.Format("VerifyRange({0}, {1}) indexing error: endsAgo must be non-negative.", startBarsAgo, endBarsAgo), LogLevel.Error);
                return false;
            }
            if (startBarsAgo <= endBarsAgo)
            {
                NinjaScript.Log(string.Format("VerifyRange({0}, {1}) indexing error: startBarsAgo must be greater than endBarsAgo.", startBarsAgo, endBarsAgo), LogLevel.Error);
                return false;
            }
            
            int index = 0;

            double temp;
            for (var ii = endBarsAgo; ii <= startBarsAgo; ii++)
            {
                try
                {
                    if (series is PriceSeries || series is Bars)
                    {
                        temp = series[ii];
                        index = ii;
                    }
                    else if (series.IsValidDataPoint(ii))
                        index = ii;
                    else
                    {
                        // NinjaScript.Log(string.Format("{0}.Range({1}, {2}) out of range.", series.ToString(), startBarsAgo, endBarsAgo), LogLevel.Error);
                        return false;
                    }
                }
                catch (ArgumentOutOfRangeException e)
                {
                    // NinjaScript.Log(string.Format("{0}.Range({1}, {2}) out of range.", series.ToString(), startBarsAgo, endBarsAgo), LogLevel.Error);
                    return false;
                }
            }

            return index == startBarsAgo;
        }

Usage:

Code

double[] vals;
if (SMA(Weighted, 14).VerifyRange(2, 0))
    vals = SMA(Weighted, 14).Range(2, 0)[1]; // Latest three values of the SMA.

The array of independent values can be useful:

Code

double[][] vals = SMA(Weighted, 14).Range(5, 2); // vals[0] = [2, 3, 4, 5]. vals[1] = the SMA values at SMA[2], [3], [4], [5].
for (int ii = 0; ii < vals.Length; ii++)
{
    // Compare to "longer" SMA values for whatever reason:
    int t = vals[0][ii]; // Again, t = 2, 3, 4, 5 through the for-loop.
    double maDifference = SMA(Weighted, 39)[t] - vals[1][ii];
}

By the way, what's up with code indentation? Is there a way to force it?

Slicing ISeries objects

Discussion in NinjaTrader

Slicing ISeries objects