NexusFi: Find Your Edge


Home Menu

 





Missing data from OnMarketData?


Discussion in NinjaTrader

Updated
    1. trending_up 2,288 views
    2. thumb_up 8 thanks given
    3. group 2 followers
    1. forum 12 posts
    2. attach_file 0 attachments




 
Search this Thread

Missing data from OnMarketData?

  #1 (permalink)
Cilla
Oxford UK
 
Posts: 10 since Apr 2019
Thanks Given: 4
Thanks Received: 5

I think I'm beginning to understand this issue now, but thought I’d post it here to help me be sure.

To summarise, I use OnMarketData (OMD) and believe I have demonstrated missing or incorrect data from the Kinetick feed. I posted a request on the NinjaTrader Forum:

“Hi

I'm trying to understand some of the output of OnMarketData in which I think I see more traded volume at the Ask than available volume at the Ask. I ran the following code against YM 06-19 future:

protected override void OnMarketData(MarketDataEventArgs e)
{
if (e.MarketDataType == MarketDataType.Last)
{
Print("Last: " + e.Volume + " @ " + e.Price);
}
else if (e.MarketDataType == MarketDataType.Ask) {
var diff = e.Volume - lastVolAtAsk;
Print("Ask: " + diff + " to " + e.Volume + " @ " + e.Price);
lastVolAtAsk=e.Volume;
}
else if (e.MarketDataType == MarketDataType.Bid) {
var diff = e.Volume - lastVolAtBid;
Print("Bid: " + diff + " to " + e.Volume + " @ " + e.Price);
lastVolAtBid=e.Volume;
}
}

The output I don't understand:

Ask: -1 to 15 @ 25989
Bid: 1 to 5 @ 25988
Bid: 1 to 6 @ 25988
Bid: 1 to 7 @ 25988
Ask: -1 to 14 @ 25989
Bid: 1 to 8 @ 25988
Bid: -1 to 7 @ 25988
Ask: -1 to 13 @ 25989
Bid: -1 to 6 @ 25988
Last: 1 @ 25989
Last: 1 @ 25989
Last: 1 @ 25989
Last: 14 @ 25989

Ask: -7 to 6 @ 25990
Bid: 1 to 7 @ 25988
Bid: 1 to 8 @ 25988
Ask: 1 to 7 @ 25990
Bid: 1 to 9 @ 25988
Bid: -1 to 8 @ 25988

So, it looks to me like there were 17 contracts traded at the Ask (25989) when there was only an ask volume of 13 available.

Please could someone help me make sense of this.”

NinjaTrader Customer Support were helpful and I think I eventually convinced them there was a problem of some kind. I sent them more examples of the issue and they tested various data providers and found a difference in the synchronicity of OnMarketData between all of them - “The OnMarketData ask/bid stream is subject to the data feed you are using and should not be expected to perfectly sync with the last update events.”
I responded that the NinjaTrader documentation states that OnMarketData is “guaranteed to be in the correct sequence for every change in level one market data for the underlying instrument", so I would expect the ask/bid stream to perfectly sync with last update events.
They accepted this and their final post was that they would make a change to the help guide to make it more clear that OnMarketData will give you every change in level one data that is given through your data provider. I took this to mean that NinjaTrader were saying that the problem lay with my data provider, Kinetick.

(hopefully, people are still with me on this…!)

Forgetting that NinjaTrader and Kinetick are the same company, I took this to Kinetick support, who were surprisingly unhelpful, twice not reading my (brief, polite and to the point) emails, until I told them I expected better support as a paying customer. I again sent them an example of the missing data with timestamp details as well and was told “I do not see any data missing on our end” and “the behaviour is expected and is not an issue with the Kinetick data feed.”

Throughout both support requests (NT and Kinetick), I did not blame any party, I was just requesting an explanation for the missing data.

My thinking about this since is that the data is being sent out as a UDP multicast and packets are being dropped occasionally, somewhere in the Internet. Would anybody have more information on this, or have another explanation?

When posting this thread, the forum found a similar thread (no. 6025) (I'm not yet allowed to post links):

There was a response (post 69691) from Ray at NT that appears to contradict my thinking - it is stated that "no data is ever dropped".

So, I'm more confused again...

Thanks

Reply With Quote

Can you help answer these questions
from other members on NexusFi?
New Micros: Ultra 10-Year & Ultra T-Bond -- Live Now
Treasury Notes and Bonds
Are there any eval firms that allow you to sink to your …
Traders Hideout
Online prop firm The Funded Trader (TFT) going under?
Traders Hideout
Deepmoney LLM
Elite Quantitative GenAI/LLM
Build trailing stop for micro index(s)
Psychology and Money Management
 
Best Threads (Most Thanked)
in the last 7 days on NexusFi
Get funded firms 2023/2024 - Any recommendations or word …
60 thanks
Funded Trader platforms
43 thanks
NexusFi site changelog and issues/problem reporting
24 thanks
GFIs1 1 DAX trade per day journal
22 thanks
The Program
19 thanks
  #2 (permalink)
 
RJay's Avatar
 RJay 
Hartford, CT. USA
 
Experience: Intermediate
Platform: NinjaTrader
Broker: AMP/CQG, Kinetick
Trading: RTY
Posts: 682 since Jun 2009
Thanks Given: 757
Thanks Received: 787


Cilla View Post
I think I'm beginning to understand this issue now, but thought I’d post it here to help me be sure.

To summarise, I use OnMarketData (OMD) and believe I have demonstrated missing or incorrect data from the Kinetick feed. I posted a request on the NinjaTrader Forum:

“Hi

I'm trying to understand some of the output of OnMarketData in which I think I see more traded volume at the Ask than available volume at the Ask. I ran the following code against YM 06-19 future:

protected override void OnMarketData(MarketDataEventArgs e)
{
if (e.MarketDataType == MarketDataType.Last)
{
Print("Last: " + e.Volume + " @ " + e.Price);
}
else if (e.MarketDataType == MarketDataType.Ask) {
var diff = e.Volume - lastVolAtAsk;
Print("Ask: " + diff + " to " + e.Volume + " @ " + e.Price);
lastVolAtAsk=e.Volume;
}
else if (e.MarketDataType == MarketDataType.Bid) {
var diff = e.Volume - lastVolAtBid;
Print("Bid: " + diff + " to " + e.Volume + " @ " + e.Price);
lastVolAtBid=e.Volume;
}
}

The output I don't understand:

Ask: -1 to 15 @ 25989
Bid: 1 to 5 @ 25988
Bid: 1 to 6 @ 25988
Bid: 1 to 7 @ 25988
Ask: -1 to 14 @ 25989
Bid: 1 to 8 @ 25988
Bid: -1 to 7 @ 25988
Ask: -1 to 13 @ 25989
Bid: -1 to 6 @ 25988
Last: 1 @ 25989
Last: 1 @ 25989
Last: 1 @ 25989
Last: 14 @ 25989

Ask: -7 to 6 @ 25990
Bid: 1 to 7 @ 25988
Bid: 1 to 8 @ 25988
Ask: 1 to 7 @ 25990
Bid: 1 to 9 @ 25988
Bid: -1 to 8 @ 25988

So, it looks to me like there were 17 contracts traded at the Ask (25989) when there was only an ask volume of 13 available.

Please could someone help me make sense of this.”

NinjaTrader Customer Support were helpful and I think I eventually convinced them there was a problem of some kind. I sent them more examples of the issue and they tested various data providers and found a difference in the synchronicity of OnMarketData between all of them - “The OnMarketData ask/bid stream is subject to the data feed you are using and should not be expected to perfectly sync with the last update events.”
I responded that the NinjaTrader documentation states that OnMarketData is “guaranteed to be in the correct sequence for every change in level one market data for the underlying instrument", so I would expect the ask/bid stream to perfectly sync with last update events.
They accepted this and their final post was that they would make a change to the help guide to make it more clear that OnMarketData will give you every change in level one data that is given through your data provider. I took this to mean that NinjaTrader were saying that the problem lay with my data provider, Kinetick.

(hopefully, people are still with me on this…!)

Forgetting that NinjaTrader and Kinetick are the same company, I took this to Kinetick support, who were surprisingly unhelpful, twice not reading my (brief, polite and to the point) emails, until I told them I expected better support as a paying customer. I again sent them an example of the missing data with timestamp details as well and was told “I do not see any data missing on our end” and “the behaviour is expected and is not an issue with the Kinetick data feed.”

Throughout both support requests (NT and Kinetick), I did not blame any party, I was just requesting an explanation for the missing data.

My thinking about this since is that the data is being sent out as a UDP multicast and packets are being dropped occasionally, somewhere in the Internet. Would anybody have more information on this, or have another explanation?

When posting this thread, the forum found a similar thread (no. 6025) (I'm not yet allowed to post links):

There was a response (post 69691) from Ray at NT that appears to contradict my thinking - it is stated that "no data is ever dropped".

So, I'm more confused again...

Thanks

Hi Cilla,

I believe the reason why 17 contracts got filled is that it used all the available contracts at the current price then jumped to the next price tick up to complete the order. Note the Ask price after the transaction.

Ask: -1 to 13 @ 25989 <----- Ask price level before trade.
Bid: -1 to 6 @ 25988
Last: 1 @ 25989
Last: 1 @ 25989
Last: 1 @ 25989
Last: 14 @ 25989
Ask: -7 to 6 @ 25990 <----- Ask price level after trade.


RJay

Reply With Quote
  #3 (permalink)
Cilla
Oxford UK
 
Posts: 10 since Apr 2019
Thanks Given: 4
Thanks Received: 5



RJay View Post
Hi Cilla,

I believe the reason why 17 contracts got filled is that it used all the available contracts at the current price then jumped to the next price tick up to complete the order. Note the Ask price after the transaction.

Ask: -1 to 13 @ 25989 <----- Ask price level before trade.
Bid: -1 to 6 @ 25988
Last: 1 @ 25989
Last: 1 @ 25989
Last: 1 @ 25989
Last: 14 @ 25989
Ask: -7 to 6 @ 25990 <----- Ask price level after trade.


RJay

Hi RJay - thanks for responding.

I agree that the Ask price changes after the trades, but my point was that it took 17 contracts (when there was only an Ask volume of 13) before the Ask price changed.

In any case, I sent other examples, including this one:

Bid: 2 @ 25639
Ask: 4 @ 25640
Last: 1 @ 25639
Bid: 1 @ 25639
Last: 20 @ 25639
Last: 1 @ 25639
Bid: 2 @ 25639

Last: 1 @ 25639
Bid: 1 @ 25639
Last: 2 @ 25639
Bid: 3 @ 25638

In this one, it looks like 21 contracts are traded at the Bid, but the Bid volume actually increases by 1.

Reply With Quote
  #4 (permalink)
 
ratfink's Avatar
 ratfink 
Birmingham UK
Market Wizard
 
Experience: Intermediate
Platform: NinjaTrader
Broker: TST/Rithmic
Trading: YM/Gold
Posts: 3,633 since Dec 2012
Thanks Given: 17,423
Thanks Received: 8,425

You should only count transacted volume on a Last event, not on the Ask or Bid events, hence the need to store current (last seen) Ask and Bid values in local variables.

Ask and/or Bid can change hundreds of times and mean nothing until the next Last event is seen.

edit: NT8 does pass current e.Bid and e.Ask prices on each event, different to NT7 so local storage not the same requirement, assuming they're updated properly. Transaction volume still only counts on the 'Last' event.

Travel Well
Visit my NexusFi Trade Journal Reply With Quote
Thanked by:
  #5 (permalink)
Cilla
Oxford UK
 
Posts: 10 since Apr 2019
Thanks Given: 4
Thanks Received: 5


ratfink View Post
You should only count transacted volume on a Last event, not on the Ask or Bid events, hence the need to store current (last seen) Ask and Bid values in local variables.

Ask and/or Bid can change hundreds of times and mean nothing until the next Last event is seen.

edit: NT8 does pass current e.Bid and e.Ask prices on each event, different to NT7 so local storage not the same requirement, assuming they're updated properly. Transaction volume still only counts on the 'Last' event.

Hi ratfink

Thanks for the explanation. Unfortunately, I think I'm still missing something here!

I thought OnMarketData was supposed to give a complete picture of the level one changes. If this is the case, when there is a Last event (a trade), I would expect something to also happen to the Bid or Ask volume, matching the traded volume. This is what is seen in most of the data I have looked at, there is even an instance of this at the beginning of the second example I gave:

Bid: 2 @ 25639
Ask: 4 @ 25640
Last: 1 @ 25639
Bid: 1 @ 25639


Here, there was originally a Bid volume of 2 at a Bid price of 25639, followed by a trade (Last) volume of 1 (also at 25639, presumably the Bid price) and then the Bid volume fell by 1 and the Bid price remained at 25639. This is what I would expect to see normally. So I'm not sure what you mean by "Ask and/or Bid can change hundreds of times and mean nothing until the next Last event is seen." - what is the point of Ask or Bid (volume) then? I understand that the Ask or Bid volume can change as new orders come in or are cancelled, but I want to distinguish these from the traded volume.

Please keep explaining, I might get it eventually!

Cheers

Reply With Quote
Thanked by:
  #6 (permalink)
 
ratfink's Avatar
 ratfink 
Birmingham UK
Market Wizard
 
Experience: Intermediate
Platform: NinjaTrader
Broker: TST/Rithmic
Trading: YM/Gold
Posts: 3,633 since Dec 2012
Thanks Given: 17,423
Thanks Received: 8,425


Cilla View Post
So I'm not sure what you mean by "Ask and/or Bid can change hundreds of times and mean nothing until the next Last event is seen." - what is the point of Ask or Bid (volume) then? I understand that the Ask or Bid volume can change as new orders come in or are cancelled, but I want to distinguish these from the traded volume.

Hi @Cilla The only point of Ask and Bid volume in OnMarketData is as a snapshot of the inside orderbook levels at any given time. For or a fuller picture you can look at OnMarketDepth.

The orderbook (including the inside levels) can and does change independently or concurrently with transacted volume, so the inside level changes can't in themselves be used to distinguish much of use.

Picture a hundred traders adding and removing best bids and offers every second, then once in a while a market order comes in and hits a limit order causing one unit of transaction volume on one side, but at the same time (in our limited retail timestamp space) the latter may or may not be simultaneously replaced by another limit order on the same side in amongst all the other ongoing changes, so your wished for indication may not be seen.

The only information we have is where in Bid-Ask price space the transaction occurred and, as we have now seen, that is still flawed in NT8 though not as much as it was in NT7.

[

Cheers

Look here if you want to go further down the hole...


Travel Well
Visit my NexusFi Trade Journal Reply With Quote
Thanked by:
  #7 (permalink)
Cilla
Oxford UK
 
Posts: 10 since Apr 2019
Thanks Given: 4
Thanks Received: 5


ratfink View Post
Hi @Cilla The only point of Ask and Bid volume in OnMarketData is as a snapshot of the inside orderbook levels at any given time. For or a fuller picture you can look at OnMarketDepth.

The orderbook (including the inside levels) can and does change independently or concurrently with transacted volume, so the inside level changes can't in themselves be used to distinguish much of use.

Picture a hundred traders adding and removing best bids and offers every second, then once in a while a market order comes in and hits a limit order causing one unit of transaction volume on one side, but at the same time (in our limited retail timestamp space) the latter may or may not be simultaneously replaced by another limit order on the same side in amongst all the other ongoing changes, so your wished for indication may not be seen.

The only information we have is where in Bid-Ask price space the transaction occurred and, as we have now seen, that is still flawed in NT8 though not as much as it was in NT7.

[

Cheers

Look here if you want to go further down the hole...


Thanks again ratfink. In your "hundred traders" picture, I was thinking that would mean that each time a best bid and offer was added or removed, there would be an individual update of the bid or ask volume. Similarly, I was thinking that each trade should be followed immediately by an individual change to the bid or ask volume. But I wasn't catering for "simultaneous" updates (I take it that this can happen on the futures exchange?) being reported as a single update to the bid or ask volume...

Reply With Quote
  #8 (permalink)
 
ratfink's Avatar
 ratfink 
Birmingham UK
Market Wizard
 
Experience: Intermediate
Platform: NinjaTrader
Broker: TST/Rithmic
Trading: YM/Gold
Posts: 3,633 since Dec 2012
Thanks Given: 17,423
Thanks Received: 8,425


Cilla View Post
Thanks again ratfink. In your "hundred traders" picture, I was thinking that would mean that each time a best bid and offer was added or removed, there would be an individual update of the bid or ask volume. Similarly, I was thinking that each trade should be followed immediately by an individual change to the bid or ask volume. But I wasn't catering for "simultaneous" updates (I take it that this can happen on the futures exchange?) being reported as a single update to the bid or ask volume...

At some level in the matching engine(s) they will be individual updates, but all we are ever seeing in NT is a summary of the order book. Only transacted volume is consecutive for us.

Always remember to think of the massive server farms in New Jersey and the microwave and optical data links and micro/nano second speeds rather than the second and millisecond levels that we are seeing.

Cheers

Travel Well
Visit my NexusFi Trade Journal Reply With Quote
Thanked by:
  #9 (permalink)
Cilla
Oxford UK
 
Posts: 10 since Apr 2019
Thanks Given: 4
Thanks Received: 5


ratfink View Post
At some level in the matching engine(s) they will be individual updates, but all we are ever seeing in NT is a summary of the order book. Only transacted volume is consecutive for us.

Always remember to think of the massive server farms in New Jersey and the microwave and optical data links and micro/nano second speeds rather than the second and millisecond levels that we are seeing.

Cheers

I guess we get what we pay for. I will try feeling smug that there really is "missing data" somewhere in the system!
Cheers!

Reply With Quote
Thanked by:
  #10 (permalink)
 
ratfink's Avatar
 ratfink 
Birmingham UK
Market Wizard
 
Experience: Intermediate
Platform: NinjaTrader
Broker: TST/Rithmic
Trading: YM/Gold
Posts: 3,633 since Dec 2012
Thanks Given: 17,423
Thanks Received: 8,425



Cilla View Post
I guess we get what we pay for. I will try feeling smug that there really is "missing data" somewhere in the system!
Cheers!

I think better considered as missing information rather than missing data.

Travel Well
Visit my NexusFi Trade Journal Reply With Quote




Last Updated on April 11, 2019


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts