Rounding issues using MIN, MAX, and TickSize

September 13th, 2012, 08:15 PM

I am working on a new project, and am getting inconsistent results using MIN, MAX, and TickSize. I believe it is a rounding issue, but can't pinpoint it.

Code

//Down-Trend
			else if(Phase == "DT")
			{

				dtMin = MIN(Close,TrendPeriods)[0];
				for(int i=0;i<TrendPeriods;i++)
				{
					if(Close[i] == dtMin)
					{
						dtMinIndex = CurrentBar - i;
						break;
					}
				}
				
				if(Close[0] - dtMin >= PullBackTicks*TickSize)
				{
					Phase = "DTPB";
					TrendPlot.Set(-1);
					PlotColors[0][0] = Color.Fuchsia;
					DrawTriangleDown("MyTriangleDown"+CurrentBar, true, 0, High[0] + 2 * TickSize, Color.Fuchsia);
				}
				else
				{
					TrendPlot.Set(-1);
					PlotColors[0][0] = Color.Red;					
				}
			}

PullBackTicks is a double and set to 30 in the following picture:

Am I missing something? (I have tried drawing the dtMin on the chart, and they seem to be correct)

September 14th, 2012, 12:49 AM

If it is a rounding issue, it is related to the use of ">=". Comparing two variables of type double for equity does not make sense. I would try to replace

Code

if(Close[0] - dtMin >= PullBackTicks*TickSize)

with

Code

if(Close[0] - dtMin > (PullBackTicks-1)*TickSize)

In case that does not help, you would need to find out what is the bar high and bar low of a BetterRenko bar. The bars are backtestable and use real highs and lows, but those highs and lows are not displayed with the brick style.

September 14th, 2012, 03:58 AM

I don't know with which datatype ninjatrader works internally, but in my own tools i use only "decimal". With this type you can define an exact decimal value and all comparisions works like a charm. With double or float you can't describe a decimal value exactly - it's only a approximation. "Decimal" should be used in financial related software products (usually).

September 14th, 2012, 06:10 AM

Fat Tails

If it is a rounding issue, it is related to the use of ">=". Comparing two variables of type double for equity does not make sense. I would try to replace

Code

if(Close[0] - dtMin >= PullBackTicks*TickSize)

with

Code

if(Close[0] - dtMin > (PullBackTicks-1)*TickSize)

In case that does not help, you would need to find out what is the bar high and bar low of a BetterRenko bar. The bars are backtestable and use real highs and lows, but those highs and lows are not displayed with the brick style.

Worked like a charm.

September 14th, 2012, 07:43 AM

Koepisch

I don't know with which datatype ninjatrader works internally, but in my own tools i use only "decimal". With this type you can define an exact decimal value and all comparisions works like a charm. With double or float you can't describe a decimal value exactly - it's only a approximation. "Decimal" should be used in financial related software products (usually).

@Koepisch: This is an interesting argument. However, a DataSeries of type decimal requires the double amount of RAM compared to a DataSeries of type double. And I am not sure that there are any advantages.

Let us examine the problem above:

1st case: You want to compare today's high to yesterday's high

In case that yesterday's high was 7403.50 and today's high is also 7403.50, you can check for equality and the
expression

(high_yesterday == high_today) = true

becomes true. The expression returns equality, because the two values have been directly written to the variable that holds them. The internal representation of the values is identical, which explains the equality.

2nd case: You want to compare today's high to yesterday's close and yesterday's range (floor pivot R2)

Let us assume that yesterday's close was 7303.50, yesterday's range 200 ticks (Ticksize 0.50) and today's high is 7403.50. Now you are very astonished to find out that

(close_yesterday + 200 * TickSize == high_today) = false

The reason here is that the numerical expression on the left side produces a tiny rounding error. The rounding error is due to the internal representation of the value. Not every decimal number has a finite binary representation. An example is

0.1 (decimal) = 0.00011001100110011..... (binary)

Would using decimals have prevented the rounding error?

No.

Due to the higher precision, you would have got a smaller rounding error, but not the exact value. For variables of type double and float, never compare two values for equality. This is FORBIDDEN. If you want to check for equality you can use a small epsilon value and check whether

Math.Abs (value1 - value2) < epsilon

In the case above any epsilon < 0.5 * TickSize would have done.

September 14th, 2012, 08:03 AM

Thats right @Fat Tails. Here are the truth: "decimal" is internal based on Base 10, which isn't native supported to processor. This slows down the execution (10 times slower then double). Since real literals are in base 10, "decimal" can precisely represent numbers such as 0.1. BUT NO real number value type can precisely represent a fractional number whose base 10 representation is recurring. You have to handle comparisions in a special manner. For instance with overloaded comparision operators. Because the performance impact is small, you should use "decimal" in your own tools. If Ninjatrader goes a other way you should do it also in your indicator/strategy development. But you should know the differences.

Because 7303.50 + 200 * 0.5 = 7403.5 (not recurring) the "decimal" appoach should give the precise value and the comparision should be TRUE.

September 14th, 2012, 08:28 AM

Koepisch

Thats right @Fat Tails. Here are the truth: "decimal" is internal based on Base 10, which isn't native supported to processor. This slows down the execution (10 times slower then double). Since real literals are in base 10, "decimal" can precisely represent numbers such as 0.1. BUT NO real number value type can precisely represent a fractional number whose base 10 representation is recurring. You have to handle comparisions in a special manner. For instance with overloaded comparision operators. Because the performance impact is small, you should use "decimal" in your own tools. If Ninjatrader goes a other way you should do it also in your indicator/strategy development. But you should know the differences.

Because 7303.50 + 200 * 0.5 = 7403.5 (not recurring) the "decimal" appoach should give the precise value and the comparision should be TRUE.

@Koepisch: Thank you again for your answer, but ....

... probably a simple division by 3 would turn your wonderfully exact decimal number into something that has no finite representation. And you are back to case 1. Why should NinjaTrader use a data type, which slows down calculations by a factor 10 and has higher RAM requirements, if in the end it does not cure the problem?

Clean code should not check for equality of two variables, whether they are of type double or decimal. You need to define an epsilon an check the absolute amount of the difference of two values for being smaller than epsilon.

I only check for equality when the values are integers or booleans.

Further the use of decimals goes against the architecture of NinjaTrader. All the DataSeries and PlotSeries objects are based on the data type double.

September 14th, 2012, 09:10 AM

Thats right, the important issues (with ninja) are, that you know

1. a double is only a approximate representation of a real value
2. if you do math with real value types, the rounding differences increases at every step (mostly)
3. you can't use the "==" comparision operator with real type values at all (@Fat Tails laid down a proper way)
4. Ninja uses "double" so you use "double" too and you have to be cautiously due to rounding deviations

If you are a C# developer you should additionally know that
1. there is a real value type "decimal" which is based at base 10 and is precise at all numbers except if the number represent a fractional number whose base 10 representation is recurring
2. you can't use the "==" operator with decimal too
3. you can overload the "==" operator in custom types, to keep the code clean
4. "decimal" was build for financial calculations
5. the MATH functions don't cover "decimal" types at full extent
6. it is your task to decide when to use "double" and when to use "decimal"

Koepisch

February 20th, 2014, 06:42 AM

Sorry to resurrect an old thread, but I wanted to question this "decimal value type makes your code 10 times slower than when using real data type" claim.

I have a complicated indicator in development that is more than 2300 lines of code (just my own code; not including the "NinjaScript generated code." I have used the decimal datatype throughout because it makes the accuracy of the results much easier to guarantee. When grabbing prices or other data from the chart, I always convert from double to decimal, and all internal calculations in the indicator use decimals. There are hundreds of these decimal datatypes; many of them are arrays. A typical test of the indicator on several years of minute data will involve many millions of calculations.

I created a test indicator that is identical to the original except that doubles were used throughout instead of decimals. I used the Stopwatch datatype to test the execution speeds of the two versions.

On an extremely long backtest run on 12 years of minute data, which generated over a thousand trades over this period, the double version completed the run in about .145 seconds. The decimal version completed it in 0.587 seconds, about 4 times slower. (I averaged 10 test runs to get an accurate timing; the standard deviations in the timings were tiny.)

Normally I do not have such large backtests with so many trades. A typical backtest will generate about 100 trades. On a test like this, the difference was much smaller. The double version completed the test in around 0.1 seconds, while the decimal version completed it in 0.18 seconds, about 1.8 times slower.

I would say from these tests that using decimals will slow everything down by 10x is a big exaggeration. Note also that I had a big difference in the accuracy of my results. The number of trades and the performance of the backtest differed by between 1% and 5% when comparing the double version to the decimal version, and the difference there is all due to the inaccuracy of the double datatype. Now of course I could recode everything to try to minimise the effects of rounding errors when using doubles, but it seems much easier, for a relatively small performance hit, to use decimals. The decimal version will of course use more memory; how much more is not something I measured. I have no shortage of memory (32GB) so this is not an issue for me.

I think whether you use doubles or decimals is something that will vary according to your needs; a "one size fits all" policy here is not a good one. In my opinion Ninjatrader should support both using doubles and decimals internally (in the Dataseries datatype and the like) and allow the individual programmer to make the decision as to which tradeoff is the better one for his or her particular project.

Rounding issues using MIN, MAX, and TickSize

Discussion in NinjaTrader

Rounding issues using MIN, MAX, and TickSize