Design a DayTrader Scalping Order Flow Indicator

September 28th, 2021, 07:14 PM

hyperscalper

Does the Cumulative Delta you are using, allow for "Big Lot" filtering ?
I've never used it...

With the TradeFlowTutorial, the parameter BIGLOT_MINIMUM needs to be set to filter,
say, contracts at a minimum lot size, e.g. 10 in NQ for example... Then you should
see a significant difference; and also the RETENTION_SECONDS will affect
how much data the "moving window" operates upon..... Stuff like that?...

hyperscalper

I am surprised you don't know this indicator maybe that would spare you the trouble of reinventing the wheel ;-)

Look at the section Parameters > Size filter:

https://ninjatrader.com/support/helpGuides/nt8/NT%20HelpGuide%20English.html?order_flow_cumulative_delta2.htm

September 28th, 2021, 07:18 PM

TWDsje

It's delta with a rolling window.

Thanks. So the parameter RETENTION_SECONDS = rolling window but what is the meaning of the TAPER_SIZE parameter?

September 28th, 2021, 07:29 PM

trendisyourfriend

Thanks. So the parameter RETENTION_SECONDS = rolling window but what is the meaning of the TAPER_SIZE parameter?

The Integrator has optional "tapering" which is the oldest 25% of the moving retention window.

So if retention is 10 minutes, then anything 7.5 - 10 minutes old, will experience a LINEAR Taper from
100% down to ZERO as it moves through that last 25% of the window toward expiration.

This is a bool (literally true or false) which determines whether tapering is done as data moves into
that last 25% of the window toward EXPIRATION.

Assuming the tapering code is correct, it will do a LINEAR TAPER from the 75% age point to the
100% maximum retention age. Otherwise, data will just remain "as is" and perhaps a very BIG
LOT could cause the Indicator Net Inventory to "jump abruptly" simply because it is TOO OLD
and is suddenly no longer within the integration time window.

REMINDER: The data is NOT "raw" but is Compressed by a factor before being placed
into the Integrator, Be Aware !!!

IF THERE ARE BUGS, IT'S BECAUSE I WAS IN A RUSH

Hope this helps !!

hyperxtrading

September 28th, 2021, 07:38 PM

trendisyourfriend

I am surprised you don't know this indicator maybe that would spare you the trouble of reinventing the wheel ;-)

Look at the section Parameters > Size filter:

https://ninjatrader.com/support/helpGuides/nt8/NT%20HelpGuide%20English.html?order_flow_cumulative_delta2.htm

Hey, of course I don't want NEEDLESSLY to re-invent any wheel; but I do want to know exactly
what I'm dealing with; so I code my own most of the time.

The Integration Time Window is as significant a design feature as is the "Big Lot Filter", and
I'm glad that better minds may already have done what I've done, but I like to be sure...

I don't use Indicators this way anyway; but they are embedded within my much more complex
Order Processing and Analytics Custom Strategy, which outstrips anything that can be done
within the standard Indicator structure anyway..... just sayin' I do very advanced stuff, and
"the Devil is Always in the Details", as I'm sure you all know !!!

My custom Strategy is heavily multi-threaded; does Unmanaged Order Processing,
and can manage dozens of simultaneous positions, targets and stops; with built-in
multi-Triggering criteria; and embedded Indicators running in their own threads,
contributing data to triggering criteria.... so... and it has a WinForms user interface
so complicated that several weeks of training would be required to 'fly it" LOL

Just sayin' ... I do things differently

I literally respond to up to 100,000 events per minute which is incoming
Market Depth data from a full Rithmic unaggregated DOM. To operate that
fast, data has to be Queued and processed later, so the OnMarketDepth
can return and get the next callback... Not so much trying to impress you,
but just to say I do know a little bit whereof I speak, and thought that this
little Trade Flow Integration exercise would help, or inspire someone ? LOL

hyperxtrading

September 29th, 2021, 07:46 AM

hyperscalper

... Not so much trying to impress you,
but just to say I do know a little bit whereof I speak, and thought that this
little Trade Flow Integration exercise would help, or inspire someone ? LOL

hyperxtrading

But i am impressed :-) I must commend you for your efforts in sharing your thoughts and scripts. It's rare on a public forum.

September 29th, 2021, 11:32 AM

Thank you for sharing your ideas and providing samples.
To get rid of your textfile config-Interface you could have a look at this indicator on how to place stuff like buttons on top of a chart to make it configurable through NT.

regarding the dynamic adjustements you mentioned - have you thought about tracking all "bigger" let's say 5+ for a given period (like 10x the retention time) and automatically adjust the indicator threshold to e.g. the lower quartile?

cheers and don't stop posting

askerix

September 29th, 2021, 11:41 AM

TWDsje

I've been looking at some similar metrics, and found them to have incredible power in predicting turns. However, the signals are terrible for trend days, and I haven't found a good way to filter for that yet.

I'd suggest simplifying the code a little. At the moment it's probably too hard for less experienced coders to follow even though what it's doing right now is quite simple.

JUST RESPONDING TO THE PERCEIVED NEED FOR A VERSION THAT WOULD
WORK IN BACK-TESTING...

I'd suggest, Seth, that you might be the person to deliver a backtestable version here, but as
you know, I wouldn't under-estimate the conversion to usage of a virtual 'playback time".
No doubt you could easily do it !!

I do only Real Time analysis, but it MIGHT be the case that the replay of Time and Sales
is accurate enough. The reason I don't do historical, is that I use a "shyte storm" of
incoming Market Depth events. I've said elsewhere that can approach 100k events
per minuts. So it's just about IMPOSSIBLE for me to use replay, unless I have my own
replay-able file, etc.......

But for Time and Sales analysis Trade Flow replay, it's probably entirely feasible; so that code
could be run in a Strategy Analyzer (which I've never used).

Obviously, instead of "real time" you'd have to have a "time provider" which would capture
the "virtual historical time" from what the Strategy Manager could deliver from the
historical data source.

Creation of a "virtual time provider" could be fairly straightforward, or it could be very
difficult. Will the Integrator implementation remain correct using a "virtual clock"?
Are categorizations of Buy versus Sell also accurate during playback?

If the answers to these questions (including a bunch of changes to the code) are YES,
then I'd say please go ahead and convert.

In that case, my Real Time parameters adjustment code is no longer useful for playback
in a Strategy Analyzer, and the parameters would be delivered in the usual way through
the Indicator startup, for each "run".

Time would certainly be less accurate, as a "virtual time" can be sampled only when some
timestamped data comes in from the historical source, obviously. However, these variations
are probably insignificant, so why not?

I'm just not a back tester; so I'd suggest that you, Seth, might spearhead some code that
would work in the Strategy Tester, as you have suggested elsewhere... I'm sure that would
interest a lot of forum Traders, and it would be interesting, no doubt to see how the various
parameterizations affect predictability.

WHO'S INTERESTED IN A STRATEGY TESTER VERSION OF THE CODE? I think my
work here is done, but it's a version that I think appears quite feasible. Seth?

I realize it's not "polite" to try and force another Trader to do something, but
PRETTY PLEASE, on behalf of those who do want to Back Test? Maybe you'd want to
do backtesting yourself ?

hyperxtrading

September 29th, 2021, 11:54 AM

askerix

Thank you for sharing your ideas and providing samples.
To get rid of your textfile config-Interface you could have a look at this indicator on how to place stuff like buttons on top of a chart to make it configurable through NT.

regarding the dynamic adjustements you mentioned - have you thought about tracking all "bigger" let's say 5+ for a given period (like 10x the retention time) and automatically adjust the indicator threshold to e.g. the lower quartile?

cheers and don't stop posting

askerix

[EDIT] To answer your query; YES, it's often a great way to make something
"dynamic" by comparing incoming events with some "slower" moving average;
and when you see a 3x or 5x or 10x event, then you'll flag it !!! Yes, I do a lot
of that stuff, maybe too much of it !! What I'm trying to measure is often so
difficult; for example, "instantaneous" Ratios of Inside BID size against inside
ASK/OFFER size. And I mean "instantaneous"... This is when Market Makers
approach the market; and those are almost always "significant" events. So the
Ratios of those sizes, relative to "average ratios" is a great way to try and "see"
those events...

How much "Fun" are we having ??? LOL

Placing a per-instance User Interface on Indicators is certainly NOT SUPPORTED by
NinjaScript Platform support; but they do always suggest such things. However,
there are a lot of "gotchas" when you try to do that, due to the "multiple cloning",
allocated resource deallocations requirements, which that imposes...? Made no sense
to you???? Good ! Never try to do it... LOL

So completely bypassing those significant issues, is the Best way forward. A simple
parameters file, with an automatic reader; indeed, just the code I provided... is guaranteed
to work, no complications, etc..... YOU HAVE BEEN WARNED. Just kidding...

NinjaTrader has decided to use the WPF format, since that's where Microsoft is moving
for user interfaces. But when you need a Real "dense" Highly Functional User Interface,
you're on your own.

I'll post my WinForms User Interface, which runs a Custom Strategy combining
Analytics with highly Configurable Order Triggering, so you can see what a "really
complex" User Interface means.

BY THE WAY, so Horrified were Microsoft users that they were "ditching" the WinForms
in favor of WPF, that WPF was forced to have a "WinForms Host Control" within which
WinForms can be implemented within an "outer shell" of WPF.

When you have several Dozens of Interactive Controls you need to EASILY implement,
there is no substitute for the tried and true WinForms which is still "supported" via
this WPF "hosting" control.

But, believe me, it's a nightmare, and that's why they DO NOT Support such code.

I'll attach a picture of my User Interface, just to reinforce my point here... You're
looking only at 1 of 7 Tabbed control panels.... So, yeah..... crazy. But YOU CAN'T
DO STUFF LIKE THAT with WPF; Just Sayin'.... LOL

[EDIT2] But I'd like to emphasize that this kind of stuff is so DIFFICULT, due to
what I call the "weird" lifecycle of an Indicator or Strategy instance. Often it's best
to start it running and NEVER try to adjust it, since anything really complex is
likely to fail; due to the "cloning" and "reflection" method NT8 uses for parameters
and NinjaScript like instances..... don't argue with them; just try to bypass the issues.

[EDIT3] THERE IS NOTHING, repeat NOTHING as powerful as NinjaTrader 8, and they
have a monopoly on that platform; I need it; and there's nowhere else for me to go,
so I have to live with the way they've engineered it. It's amazing; but nothing's
absolutely perfect

hyperxtrading

SpeculatorSeth · September 29th, 2021, 12:17 PM

hyperscalper

JUST RESPONDING TO THE PERCEIVED NEED FOR A VERSION THAT WOULD
WORK IN BACK-TESTING...

I'd suggest, Seth, that you might be the person to deliver a backtestable version here, but as
you know, I wouldn't under-estimate the conversion to usage of a virtual 'playback time".
No doubt you could easily do it !!

I do only Real Time analysis, but it MIGHT be the case that the replay of Time and Sales
is accurate enough. The reason I don't do historical, is that I use a "shyte storm" of
incoming Market Depth events. I've said elsewhere that can approach 100k events
per minuts. So it's just about IMPOSSIBLE for me to use replay, unless I have my own
replay-able file, etc.......

But for Time and Sales analysis Trade Flow replay, it's probably entirely feasible; so that code
could be run in a Strategy Analyzer (which I've never used).

Obviously, instead of "real time" you'd have to have a "time provider" which would capture
the "virtual historical time" from what the Strategy Manager could deliver from the
historical data source.

Creation of a "virtual time provider" could be fairly straightforward, or it could be very
difficult. Will the Integrator implementation remain correct using a "virtual clock"?
Are categorizations of Buy versus Sell also accurate during playback?

If the answers to these questions (including a bunch of changes to the code) are YES,
then I'd say please go ahead and convert.

In that case, my Real Time parameters adjustment code is no longer useful for playback
in a Strategy Analyzer, and the parameters would be delivered in the usual way through
the Indicator startup, for each "run".

Time would certainly be less accurate, as a "virtual time" can be sampled only when some
timestamped data comes in from the historical source, obviously. However, these variations
are probably insignificant, so why not?

I'm just not a back tester; so I'd suggest that you, Seth, might spearhead some code that
would work in the Strategy Tester, as you have suggested elsewhere... I'm sure that would
interest a lot of forum Traders, and it would be interesting, no doubt to see how the various
parameterizations affect predictability.

WHO'S INTERESTED IN A STRATEGY TESTER VERSION OF THE CODE? I think my
work here is done, but it's a version that I think appears quite feasible. Seth?

I realize it's not "polite" to try and force another Trader to do something, but
PRETTY PLEASE, on behalf of those who do want to Back Test? Maybe you'd want to
do backtesting yourself ?

hyperxtrading

First to the accuracy of tick replay data there are definitely inaccuracies, but it depends on what you are doing as to whether it matters. For instance just looking at the imbalance between market buys and market sells in a 5 minute rolling window appears to be relatively accurate. When you start looking at 1 second intervals or you're in a quiet market the errors are enough to start screwing up the entries.

I have a system of storing the market depth data, but I have not yet been using it in backtests. A key is that I'm only storing the end result. Otherwise I wouldn't have enough resources to save it. So if you're developing a new filter it's not going to help you. Once you have your filter though you can start saving the live data and look at it historically to test theories about how to use it. So far I use my tick replay backtests to develop the system, and then add on the level2 enhancements as an extra little improvement at the forward testing stage.

One of the issues with sharing code is some parts rely on my framework that I provide for paid YouTube subscribers. Namely the parts that save level2 data. NinjaTrader also doesn't want people distributing things that recreate full license features. However, we can just use the built in OrderFlowDelta. So you get something like this where you add deltas for the new bar and remove deltas for the bar that is leaving the frame.

Code

#region Using declarations
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.ComponentModel.DataAnnotations;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Input;
using System.Windows.Media;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Gui;
using NinjaTrader.Gui.Chart;
using NinjaTrader.Gui.SuperDom;
using NinjaTrader.Gui.Tools;
using NinjaTrader.Data;
using NinjaTrader.NinjaScript;
using NinjaTrader.Core.FloatingPoint;
using NinjaTrader.NinjaScript.DrawingTools;
#endregion

//This namespace holds Indicators in this folder and is required. Do not change it. 
namespace NinjaTrader.NinjaScript.Indicators.SpeculatorSethFree
{
	public class spsMarketOrderImbalance : Indicator
	{
		OrderFlowDelta MyDelta;
		
		double buys = 0;
		double sells = 0;
		
		protected override void OnStateChange()
		{
			if (State == State.SetDefaults)
			{
				Description									= @"Enter the description for your new custom Indicator here.";
				Name										= "spsMarketOrderImbalance";
				Calculate									= Calculate.OnBarClose;
				IsOverlay									= false;
				DisplayInDataBox							= true;
				DrawOnPricePanel							= true;
				DrawHorizontalGridLines						= true;
				DrawVerticalGridLines						= true;
				PaintPriceMarkers							= true;
				ScaleJustification							= NinjaTrader.Gui.Chart.ScaleJustification.Right;
				//Disable this property if your indicator requires custom values that cumulate with each new market data event. 
				//See Help Guide for additional information.
				IsSuspendedWhileInactive					= false;
				
				Period = 10;
				
				AddPlot(Brushes.Orange, "Imbalance");
			}
			else if (State == State.Configure)
			{
			}
			else if (State == State.DataLoaded)
			{
				MyDelta = OrderFlowDelta();
			}
		}

		protected override void OnBarUpdate()
		{
			buys = buys + MyDelta.BuyVolume[0];
			sells = sells + MyDelta.SellVolume[0];
			//Add your custom indicator logic here.
			
			if(CurrentBar > Period-1)
			{
				buys = buys - MyDelta.BuyVolume[Period-1];
				sells = sells - MyDelta.SellVolume[Period-1];
			}
			
			Values[0][0] = buys / (buys - sells);
		}
		
		[NinjaScriptProperty]
		public int Period
		{ get; set; }
		
		
	}
}

September 29th, 2021, 01:25 PM

TWDsje

First to the accuracy of tick replay data there are definitely inaccuracies, but it depends on what you are doing as to whether it matters. For instance just looking at the imbalance between market buys and market sells in a 5 minute rolling window appears to be relatively accurate. When you start looking at 1 second intervals or you're in a quiet market the errors are enough to start screwing up the entries.

I have a system of storing the market depth data, but I have not yet been using it in backtests. A key is that I'm only storing the end result. Otherwise I wouldn't have enough resources to save it. So if you're developing a new filter it's not going to help you. Once you have your filter though you can start saving the live data and look at it historically to test theories about how to use it. So far I use my tick replay backtests to develop the system, and then add on the level2 enhancements as an extra little improvement at the forward testing stage.

One of the issues with sharing code is some parts rely on my framework that I provide for paid YouTube subscribers. Namely the parts that save level2 data. NinjaTrader also doesn't want people distributing things that recreate full license features. However, we can just use the built in OrderFlowDelta. So you get something like this where you add deltas for the new bar and remove deltas for the bar that is leaving the frame.

Thanks for taking an interest in it, and thinking about it...

As for Level2; well that doesn't play into this strictly Trade Flow Indicator obviously.
And, by the way, what CQG calls Level 2 is nothing compared to what I use
from Rithmic, which extends maybe at least 60+ levels either side of the market,
and it is UN-aggregated as well... so the QUALITY of Level 2 data ranges drastically,
and if you're not in a Co-Located Server situation, latencies will factor in...
BUT NONE OF THAT MATTERS, fortunately...

The difficulty in retaining Market Depth, which for me means 100k events per minute,
is one of the main reasons I don't try to back-test it at all...

Oddly enough, one of my main Triggering conditions is a Net Inventory MAX or
MIN; however, the dynamism of NQ/MNQ makes it a difficult "Beast" to handle...

I REALLY WISH you would be willing to do a Strategy Tester compatible version of
the Trade Flow, and make it available.... PLEASE... I'd run the Strategy Tester
against it, out of curiosity !

[EDIT] Or, I could just modify the code to do all that with a static switch to enable
one or the other modes..... Hmmm.

hyperxtrading

Design a DayTrader Scalping Order Flow Indicator

Discussion in Traders Hideout

Design a DayTrader Scalping Order Flow Indicator