Ninjatrader Threading issue with Custom columns, can anyone help?

KhaosTrader · (login for full post details)

Hi,

I am writing a custom column, where use my own indicator.

When I grab data from my indicator, I get a ThreadingLockRecursionException.

I get the data as follows...

Suggestions Please...

Thank you in advance...

Code

protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)
		{

			
				if (!IsFirstTickOfBar)
				{
					return;
				}
			
			
			enuColumnAlertAction ColumnAlertAction = enuColumnAlertAction.None;
			enuTradeDirection TradeDirection = enuTradeDirection.Unknown;
			

		
			double InfoVal = Khaos_QuantumCalc.Market_Analyzer_Setup_Info[1]; // <-- problem

Popsicle · (login for full post details)

Hi @KhaosTrader,

Try providing a lock for the indicator call you are making. At the top of the class (or anywhere outside a method) declare a variable that you can use as a lock:

Code

private static object _syncLock = new object();

Then change this line of code:

Code

double InfoVal = Khaos_QuantumCalc.Market_Analyzer_Setup_Info[1];

to:

Code

double InfoVal = 0.0d;
lock(_syncLock)
{
      InfoVal = Khaos_QuantumCalc.Market_Analyzer_Setup_Info[1];
}
//Use the value in the InfoVal variable here....

This will ensure that only one thread at a time can call that piece of code. Please just note that this may have a bit of a performance impact on your code as it effectively creates a "one call at a time queue" basically into that call. It does correctly ensure that only one thread at a time calls the code though.

If this does not fix the issue, you might have code in your indicator that recursively calls itself which is a totally different beast to solve.

Hope this helps,
Popsicle

rleplae · (login for full post details)

KhaosTrader

Hi,

I am writing a custom column, where use my own indicator.

When I grab data from my indicator, I get a ThreadingLockRecursionException.

I get the data as follows...

Suggestions Please...

Thank you in advance...

Code

protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)
		{

			
				if (!IsFirstTickOfBar)
				{
					return;
				}
			
			
			enuColumnAlertAction ColumnAlertAction = enuColumnAlertAction.None;
			enuTradeDirection TradeDirection = enuTradeDirection.Unknown;
			

		
			double InfoVal = Khaos_QuantumCalc.Market_Analyzer_Setup_Info[1]; // <-- problem

Probably there is (already) a lock in that code ?

Can you dig deeper ?

what is behind Khaos_QuantumCalc.Market_Analyzer_Setup_Info[1] ?

KhaosTrader · (login for full post details)

Hi,

Thank you guys for your help..

I didnt put any locks in any of my code prior to this point...

I added the locks, and I still got errors, but not as many errors...

Here is my modified code...

I put the following line outside the scope of the method...

Code

private static object _syncLock = new object();

And here is the modified code in the method...

Code

		protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)

		{

	if (!IsFirstTickOfBar)
			{
				return;
			}
			
			if (CurrentBar < 300)
			{
				return;
			}
			
				
			int InfoValue = 0;
			int SqueezeState = 0;
		lock(_syncLock)
		{
			 InfoValue =  (int) Khaos_QuantumCalc.Market_Analyzer_Setup_Info[0];
			SqueezeState = (int)Khaos_QuantumCalc.Market_Analyzer_Squeeze_Info[0];
		}

rleplae · (login for full post details)

I don't think it's a good idea to put a lock and serialize the 'on market data' event
this will create a single bottle neck in your system

If you read your indicator, that does not come from OnMarketData anyway

It would be good to know, what is happening when you are reading the value
of your indicator... and why there is a conflict

Normaly (according to my understanding)
you need a lock, when two threads are accessing the same data
and more precisely, if at least one of the two threads is changing
the data. If two thread are 'reading' the data, that is not a concurrency
issue.

Here i think you are doing something in the code for 'reading' your indicator
i would be interested to see your 'getter' piece of code, what happens there..

KhaosTrader · (login for full post details)

Hi rleplae,

Thank you for your helping.. I really appreciate it.

The code calls my indicator that is quite complex, it datamines various bar patterns and uses all sorts of intelligent filtering to come up with setups. I like to use it to trade forex, and I basically have about 17 or so pairs I trade. I have the custom column on the market analyzer such that it will act as a scanner and tell me how many bars ago the setup was, what direction the setup is, and a rating of the setup.

The cool thing is I can put in several columns with different time frames, thereby having my scanner show me my setups on dozens of charts, and its really quite handy.

rleplae · (login for full post details)

KhaosTrader

Hi rleplae,

Thank you for your helping.. I really appreciate it.

The code calls my indicator that is quite complex, it datamines various bar patterns and uses all sorts of intelligent filtering to come up with setups. I like to use it to trade forex, and I basically have about 17 or so pairs I trade. I have the custom column on the market analyzer such that it will act as a scanner and tell me how many bars ago the setup was, what direction the setup is, and a rating of the setup.

The cool thing is I can put in several columns with different time frames, thereby having my scanner show me my setups on dozens of charts, and its really quite handy.

Something must be there in that code...

Just on another direction, if you only have one single column for one single instrument, are you also running into the same issue ? or is it only, when multiple columns for multiple instruments are updating simultaneously ?

KhaosTrader · (login for full post details)

i checked it with one column with 18 instruments... I didnt check with 1 column with 1 instrument, should I try that? it might take a while for the problem to come up , if it will come up with just 1 instrument...

rleplae · (login for full post details)

KhaosTrader

i checked it with one column with 18 instruments... I didnt check with 1 column with 1 instrument, should I try that? it might take a while for the problem to come up , if it will come up with just 1 instrument...

if it never comes up with 1 column 1 instrument
then it's the indicator that is not thread safe

it's worth doing the test to eliminate that possibility

KhaosTrader · (login for full post details)

Ok I will test this overnight on Forex 5 minute interval and get back to you.. thank you

Sent from my iPad using futures.io futures trading

Zondor · (login for full post details)

Try this, if it works it should eliminate the need for the lock.

Anyone have any hints for optimizing C# code?

But what if the called indicator does not have just one output? Which is more efficient?

The main thing is to remember a reference to the indicator, so that you don't have to ask ninja to look it up again. The number of DataSeries you need to look …

Code

        ....   variable declarations...

        private Khaos_QuantumCalc KQC;


        ...     OnStateChange....

        if(State==State.Configure)
               {   KQC=Khaos_QuantumCalc();

               }


        protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)

		{

	if (!IsFirstTickOfBar || CurrentBar < 300) return;			{
				
			
	lock(_syncLock)
        	{   InfoValue =    (int) KQC.Market_Analyzer_Setup_Info[0];
		    SqueezeState = (int) KQC.Market_Analyzer_Squeeze_Info[0];
		}

KhaosTrader · (login for full post details)

Hi Zondor,

I have it this way... and still the problem, I am not sure if I am doing it where i am using the "init" concept in your referenced post.. but here is my code..

please if you have another idea or if i am not employing your suggestion correctly, let me know how to do it.

Thank you..

Code Below:

Declare

Code

public class KhaosQuantum : MarketAnalyzerColumn
	{
		
		private Khaos_Quantum_Calculator Khaos_QuantumCalc;
...
...

Reference

Code

		else if (State == State.Configure)
			{
		Khaos_QuantumCalc = Khaos_Quantum_Calculator
				(
					9, // BarsLength
					20, // MA Short Length
					50, // MA Medium Length
					200, // MA Long Length
....
....

Method Logic

Code

		protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)

		{
			

				
			
			if (!IsFirstTickOfBar || CurrentBar < 300)
			{
				return;
			}
			
				
			int InfoValue = 0;
			int SqueezeState = 0;
		
		lock(_syncLock)
		{
			 InfoValue =  (int) Khaos_QuantumCalc.Market_Analyzer_Setup_Info[0];
			SqueezeState = (int)Khaos_QuantumCalc.Market_Analyzer_Squeeze_Info[0];
		}

...
...

*** Also note, it seems that when I click reload ninjascript, I really get errors, I just put into testing on a 10 second interval with about 18 or so instruments, when I hit reload ninjascript, and while it is already reloadingi again, many errors occur... Here is the log info:

More info....

It seems if I press the "refresh" on the Market Analyzer it triggers these errors...

Here is the log file...
2016-10-18 14:02:56:851|2|4|Session Break (Version 8.0.0.14)
2016-10-18 14:02:59:038|1|4|Verifying license at primary server...
2016-10-18 14:02:59:308|1|4|Type=Simulation State=Verified Start Date=12/31/2004 End Date=11/30/2099
2016-10-18 14:02:59:418|1|4|Global simulation mode enabled
2016-10-18 14:03:03:511|1|4|Vendor assembly 'System.Speech' version='4.0.0.0' loaded.
2016-10-18 14:03:04:319|1|2|Using IS (is-us-003.ninjatrader.com/31658)
2016-10-18 14:03:04:357|1|4|Automated trading disabled
2016-10-18 14:03:04:358|1|4|Auto connecting 'My FXCM'...
2016-10-18 14:03:04:401|1|2|My FXCM: Primary connection=Connecting, Price feed=Connecting
2016-10-18 14:03:04:651|1|32|Order entry hot keys disabled
2016-10-18 14:03:04:653|1|4|Auto close enabled=False
2016-10-18 14:03:12:147|1|2|My FXCM: Primary connection=Connected, Price feed=Connected
2016-10-18 14:03:12:227|1|2|Time to auto close position='00:00:00', Enabled=False
2016-10-18 14:03:12:228|1|2|Using HDS (hds-us-003.ninjatrader.com/31655)
2016-10-18 14:05:54:630|3|4|Failed to call 'Add' method: System.Threading.LockRecursionException: Write lock may not be acquired with read lock held. This pattern is prone to deadlocks. Please ensure that read locks are released before taking a write lock. If an upgrade is necessary, use an upgrade lock in place of the read lock.
at System.Threading.ReaderWriterLockSlim.TryEnterWriteLockCore(TimeoutTracker timeout)
at System.Threading.ReaderWriterLockSlim.TryEnterWriteLock(TimeoutTracker timeout)
at NinjaTrader.Data.BarsSeries.Add(Bars bars, Double open, Double high, Double low, Double close, DateTime time, Int64 volume, Double tickSize, Boolean isBar, Double bid, Double ask)
2016-10-18 14:31:57:361|3|4|Market analyzer column 'KhaosQuantum': Error on calling 'OnMarketData' method on bar -1: Object reference not set to an instance of an object.
2016-10-18 14:31:57:421|3|4|Market analyzer column 'KhaosQuantum': Error on calling 'OnMarketData' method on bar -1: Object reference not set to an instance of an object.
2016-10-18 14:31:57:472|3|4|Market analyzer column 'KhaosQuantum': Error on calling 'OnMarketData' method on bar -1: Object reference not set to an instance of an object.
2016-10-18 14:31:57:537|3|4|Market analyzer column 'KhaosQuantum': Error on calling 'OnMarketData' method on bar -1: Object reference not set to an instance of an object.
2016-10-18 14:31:57:622|3|4|Market analyzer column 'KhaosQuantum': Error on calling 'OnMarketData' method on bar -1: Object reference not set to an instance of an object.
2016-10-18 14:31:57:628|3|4|Market analyzer column 'KhaosQuantum': Error on calling 'OnMarketData' method on bar -1: You are accessing an index with a value that is invalid since it is out-of-range. I.E. accessing a series [barsAgo] with a value of 5 when there are only 4 bars on the chart.
2016-10-18 14:31:57:698|3|4|Market analyzer column 'KhaosQuantum': Error on calling 'OnMarketData' method on bar -1: Object reference not set to an instance of an object.
2016-10-18 14:31:57:723|3|4|Market analyzer column 'KhaosQuantum': Error on calling 'OnMarketData' method on bar -1: Object reference not set to an instance of an object.
2016-10-18 14:31:57:773|3|4|Market analyzer column 'KhaosQuantum': Error on calling 'OnMarketData' method on bar -1: Object reference not set to an instance of an object.
2016-10-18 14:31:57:816|3|4|Market analyzer column 'KhaosQuantum': Error on calling 'OnMarketData' method on bar -1: Object reference not set to an instance of an object.
2016-10-18 14:31:57:844|3|4|Market analyzer column 'KhaosQuantum': Error on calling 'OnMarketData' method on bar -1: Object reference not set to an instance of an object.
2016-10-18 14:31:57:921|3|4|Market analyzer column 'KhaosQuantum': Error on calling 'OnMarketData' method on bar -1: Object reference not set to an instance of an object.
2016-10-18 14:31:57:977|3|4|Market analyzer column 'KhaosQuantum': Error on calling 'OnMarketData' method on bar -1: Object reference not set to an instance of an object.
2016-10-18 14:31:58:010|3|4|Market analyzer column 'KhaosQuantum': Error on calling 'OnMarketData' method on bar -1: Object reference not set to an instance of an object.
2016-10-18 14:31:58:095|3|4|Market analyzer column 'KhaosQuantum': Error on calling 'OnMarketData' method on bar -1: Object reference not set to an instance of an object.
2016-10-18 14:31:58:095|3|4|Market analyzer column 'KhaosQuantum': Error on calling 'OnMarketData' method on bar -1: You are accessing an index with a value that is invalid since it is out-of-range. I.E. accessing a series [barsAgo] with a value of 5 when there are only 4 bars on the chart.
2016-10-18 14:31:58:095|3|4|Market analyzer column 'KhaosQuantum': Error on calling 'OnMarketData' method on bar -1: You are accessing an index with a value that is invalid since it is out-of-range. I.E. accessing a series [barsAgo] with a value of 5 when there are only 4 bars on the chart.

KhaosTrader · (login for full post details)

I tried with 1 instrument, error comes up.

What I am going to do is put in an indicator column and just have that work for me.

Zondor · (login for full post details)

You did not make the change I suggested, and don't seem to understand what I said.

I don't think that this is a threading issue.

The indicator crashes because it does not have error handling code. If you trapped the errors, it would probably work.

Refer to a C# tutorial to learn about error handling.

To eliminate the Object Reference exceptions, as a first step add this condition to beginning of OnMarketData and OnBarUpdate methods:

if(CurrentBar == -1) return;

If that doesn't get rid of them, something else is wrong - just like it says, failure to have set an object reference to an instance of an object..

That's about all I can offer based on the information you posted since the problems are in code not revealed..

KhaosTrader · (login for full post details)

Hi Zandor,

---> I did do a try catch block and posted the results follow up on ninjatrader forum. Eventually ninjatrader suggested I make a new indicator that calls the main indicator, and just use an indicator column.

With the suggested approach, it works fine.

In the ninjatrader form thread, the support person said this

" KhaosTrader,

Thank you for your patience.

To sum it up, even if the custom column has the thread of the instrument and indicator locked we still don't have the bars loaded. We could make a bars request but those bars would not be associated with the indicator and thus we would really be re-writing the entire indicator into the column (which may be an approach you wish to take).

However, I am going to make a feature request to allow for Series<T> types to be the value that is used from the Indicator Column Plot option as well. This would resolve your entire scenario as you could just have an indicator that looks for the conditions and sets a Series<string> to the desired string for the column.

"

The thread is located here:

Custom Column Question - NinjaTrader Support Forum

As you can see Ninjatrader is thinking about allowing the return of a string value for a custom indicator column. That would be excellent if they do that.

Thank you for your patience and help Zandor.

Ninjatrader Threading issue with Custom columns, can anyone help?

Discussion in NinjaTrader

Ninjatrader Threading issue with Custom columns, can anyone help?

NinjaTrader Indicator Challenge!

Battlestations! Show us your trading desk - $1,500 in prizes!

Importance of Finding Your Own Way w/Adam Grimes

Journal Challenge w/Jigsaw