Missing data from OnMarketData? - NinjaTrader | futures io social day trading
futures io futures trading


Missing data from OnMarketData?
Updated: Views / Replies:290 / 12
Created: by Cilla Attachments:0

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 100,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors – all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you don’t need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 
Thread Tools Search this Thread
 

Missing data from OnMarketData?

  #1 (permalink)
Oxford UK
 
Trading Experience: Intermediate
Platform: NinjaTrader
Favorite Futures: YM
 
Posts: 10 since Apr 2019
Thanks: 4 given, 5 received

Missing data from OnMarketData?

I think I'm beginning to understand this issue now, but thought I’d post it here to help me be sure.

To summarise, I use OnMarketData (OMD) and believe I have demonstrated missing or incorrect data from the Kinetick feed. I posted a request on the NinjaTrader Forum:

“Hi

I'm trying to understand some of the output of OnMarketData in which I think I see more traded volume at the Ask than available volume at the Ask. I ran the following code against YM 06-19 future:

protected override void OnMarketData(MarketDataEventArgs e)
{
if (e.MarketDataType == MarketDataType.Last)
{
Print("Last: " + e.Volume + " @ " + e.Price);
}
else if (e.MarketDataType == MarketDataType.Ask) {
var diff = e.Volume - lastVolAtAsk;
Print("Ask: " + diff + " to " + e.Volume + " @ " + e.Price);
lastVolAtAsk=e.Volume;
}
else if (e.MarketDataType == MarketDataType.Bid) {
var diff = e.Volume - lastVolAtBid;
Print("Bid: " + diff + " to " + e.Volume + " @ " + e.Price);
lastVolAtBid=e.Volume;
}
}

The output I don't understand:

Ask: -1 to 15 @ 25989
Bid: 1 to 5 @ 25988
Bid: 1 to 6 @ 25988
Bid: 1 to 7 @ 25988
Ask: -1 to 14 @ 25989
Bid: 1 to 8 @ 25988
Bid: -1 to 7 @ 25988
Ask: -1 to 13 @ 25989
Bid: -1 to 6 @ 25988
Last: 1 @ 25989
Last: 1 @ 25989
Last: 1 @ 25989
Last: 14 @ 25989

Ask: -7 to 6 @ 25990
Bid: 1 to 7 @ 25988
Bid: 1 to 8 @ 25988
Ask: 1 to 7 @ 25990
Bid: 1 to 9 @ 25988
Bid: -1 to 8 @ 25988

So, it looks to me like there were 17 contracts traded at the Ask (25989) when there was only an ask volume of 13 available.

Please could someone help me make sense of this.”

NinjaTrader Customer Support were helpful and I think I eventually convinced them there was a problem of some kind. I sent them more examples of the issue and they tested various data providers and found a difference in the synchronicity of OnMarketData between all of them - “The OnMarketData ask/bid stream is subject to the data feed you are using and should not be expected to perfectly sync with the last update events.”
I responded that the NinjaTrader documentation states that OnMarketData is “guaranteed to be in the correct sequence for every change in level one market data for the underlying instrument", so I would expect the ask/bid stream to perfectly sync with last update events.
They accepted this and their final post was that they would make a change to the help guide to make it more clear that OnMarketData will give you every change in level one data that is given through your data provider. I took this to mean that NinjaTrader were saying that the problem lay with my data provider, Kinetick.

(hopefully, people are still with me on this…!)

Forgetting that NinjaTrader and Kinetick are the same company, I took this to Kinetick support, who were surprisingly unhelpful, twice not reading my (brief, polite and to the point) emails, until I told them I expected better support as a paying customer. I again sent them an example of the missing data with timestamp details as well and was told “I do not see any data missing on our end” and “the behaviour is expected and is not an issue with the Kinetick data feed.”

Throughout both support requests (NT and Kinetick), I did not blame any party, I was just requesting an explanation for the missing data.

My thinking about this since is that the data is being sent out as a UDP multicast and packets are being dropped occasionally, somewhere in the Internet. Would anybody have more information on this, or have another explanation?

When posting this thread, the forum found a similar thread (no. 6025) (I'm not yet allowed to post links):

There was a response (post 69691) from Ray at NT that appears to contradict my thinking - it is stated that "no data is ever dropped".

So, I'm more confused again...

Thanks

Reply With Quote
 
  #2 (permalink)
Hartford, CT. USA
 
Trading Experience: Intermediate
Platform: NinjaTrader
Broker/Data: AMP/CQG, Kinetick
Favorite Futures: RTY
 
RJay's Avatar
 
Posts: 684 since Jun 2009
Thanks: 739 given, 760 received


Cilla View Post
I think I'm beginning to understand this issue now, but thought I’d post it here to help me be sure.

To summarise, I use OnMarketData (OMD) and believe I have demonstrated missing or incorrect data from the Kinetick feed. I posted a request on the NinjaTrader Forum:

“Hi

I'm trying to understand some of the output of OnMarketData in which I think I see more traded volume at the Ask than available volume at the Ask. I ran the following code against YM 06-19 future:

protected override void OnMarketData(MarketDataEventArgs e)
{
if (e.MarketDataType == MarketDataType.Last)
{
Print("Last: " + e.Volume + " @ " + e.Price);
}
else if (e.MarketDataType == MarketDataType.Ask) {
var diff = e.Volume - lastVolAtAsk;
Print("Ask: " + diff + " to " + e.Volume + " @ " + e.Price);
lastVolAtAsk=e.Volume;
}
else if (e.MarketDataType == MarketDataType.Bid) {
var diff = e.Volume - lastVolAtBid;
Print("Bid: " + diff + " to " + e.Volume + " @ " + e.Price);
lastVolAtBid=e.Volume;
}
}

The output I don't understand:

Ask: -1 to 15 @ 25989
Bid: 1 to 5 @ 25988
Bid: 1 to 6 @ 25988
Bid: 1 to 7 @ 25988
Ask: -1 to 14 @ 25989
Bid: 1 to 8 @ 25988
Bid: -1 to 7 @ 25988
Ask: -1 to 13 @ 25989
Bid: -1 to 6 @ 25988
Last: 1 @ 25989
Last: 1 @ 25989
Last: 1 @ 25989
Last: 14 @ 25989

Ask: -7 to 6 @ 25990
Bid: 1 to 7 @ 25988
Bid: 1 to 8 @ 25988
Ask: 1 to 7 @ 25990
Bid: 1 to 9 @ 25988
Bid: -1 to 8 @ 25988

So, it looks to me like there were 17 contracts traded at the Ask (25989) when there was only an ask volume of 13 available.

Please could someone help me make sense of this.”

NinjaTrader Customer Support were helpful and I think I eventually convinced them there was a problem of some kind. I sent them more examples of the issue and they tested various data providers and found a difference in the synchronicity of OnMarketData between all of them - “The OnMarketData ask/bid stream is subject to the data feed you are using and should not be expected to perfectly sync with the last update events.”
I responded that the NinjaTrader documentation states that OnMarketData is “guaranteed to be in the correct sequence for every change in level one market data for the underlying instrument", so I would expect the ask/bid stream to perfectly sync with last update events.
They accepted this and their final post was that they would make a change to the help guide to make it more clear that OnMarketData will give you every change in level one data that is given through your data provider. I took this to mean that NinjaTrader were saying that the problem lay with my data provider, Kinetick.

(hopefully, people are still with me on this…!)

Forgetting that NinjaTrader and Kinetick are the same company, I took this to Kinetick support, who were surprisingly unhelpful, twice not reading my (brief, polite and to the point) emails, until I told them I expected better support as a paying customer. I again sent them an example of the missing data with timestamp details as well and was told “I do not see any data missing on our end” and “the behaviour is expected and is not an issue with the Kinetick data feed.”

Throughout both support requests (NT and Kinetick), I did not blame any party, I was just requesting an explanation for the missing data.

My thinking about this since is that the data is being sent out as a UDP multicast and packets are being dropped occasionally, somewhere in the Internet. Would anybody have more information on this, or have another explanation?

When posting this thread, the forum found a similar thread (no. 6025) (I'm not yet allowed to post links):

There was a response (post 69691) from Ray at NT that appears to contradict my thinking - it is stated that "no data is ever dropped".

So, I'm more confused again...

Thanks

Hi Cilla,

I believe the reason why 17 contracts got filled is that it used all the available contracts at the current price then jumped to the next price tick up to complete the order. Note the Ask price after the transaction.

Ask: -1 to 13 @ 25989 <----- Ask price level before trade.
Bid: -1 to 6 @ 25988
Last: 1 @ 25989
Last: 1 @ 25989
Last: 1 @ 25989
Last: 14 @ 25989
Ask: -7 to 6 @ 25990 <----- Ask price level after trade.


RJay

Reply With Quote
 
  #3 (permalink)
Oxford UK
 
Trading Experience: Intermediate
Platform: NinjaTrader
Favorite Futures: YM
 
Posts: 10 since Apr 2019
Thanks: 4 given, 5 received



RJay View Post
Hi Cilla,

I believe the reason why 17 contracts got filled is that it used all the available contracts at the current price then jumped to the next price tick up to complete the order. Note the Ask price after the transaction.

Ask: -1 to 13 @ 25989 <----- Ask price level before trade.
Bid: -1 to 6 @ 25988
Last: 1 @ 25989
Last: 1 @ 25989
Last: 1 @ 25989
Last: 14 @ 25989
Ask: -7 to 6 @ 25990 <----- Ask price level after trade.


RJay

Hi RJay - thanks for responding.

I agree that the Ask price changes after the trades, but my point was that it took 17 contracts (when there was only an Ask volume of 13) before the Ask price changed.

In any case, I sent other examples, including this one:

Bid: 2 @ 25639
Ask: 4 @ 25640
Last: 1 @ 25639
Bid: 1 @ 25639
Last: 20 @ 25639
Last: 1 @ 25639
Bid: 2 @ 25639

Last: 1 @ 25639
Bid: 1 @ 25639
Last: 2 @ 25639
Bid: 3 @ 25638

In this one, it looks like 21 contracts are traded at the Bid, but the Bid volume actually increases by 1.

Reply With Quote
 
  #4 (permalink)
Market Wizard
Birmingham UK
 
Trading Experience: Intermediate
Platform: NinjaTrader
Broker/Data: TST/Rithmic
Favorite Futures: YM/Gold
 
ratfink's Avatar
 
Posts: 3,631 since Dec 2012
Thanks: 16,954 given, 8,228 received

You should only count transacted volume on a Last event, not on the Ask or Bid events, hence the need to store current (last seen) Ask and Bid values in local variables.

Ask and/or Bid can change hundreds of times and mean nothing until the next Last event is seen.

edit: NT8 does pass current e.Bid and e.Ask prices on each event, different to NT7 so local storage not the same requirement, assuming they're updated properly. Transaction volume still only counts on the 'Last' event.

Travel Well

Last edited by ratfink; April 7th, 2019 at 03:02 PM.
Reply With Quote
The following 2 users say Thank You to ratfink for this post:
 
  #5 (permalink)
Oxford UK
 
Trading Experience: Intermediate
Platform: NinjaTrader
Favorite Futures: YM
 
Posts: 10 since Apr 2019
Thanks: 4 given, 5 received


ratfink View Post
You should only count transacted volume on a Last event, not on the Ask or Bid events, hence the need to store current (last seen) Ask and Bid values in local variables.

Ask and/or Bid can change hundreds of times and mean nothing until the next Last event is seen.

edit: NT8 does pass current e.Bid and e.Ask prices on each event, different to NT7 so local storage not the same requirement, assuming they're updated properly. Transaction volume still only counts on the 'Last' event.

Hi ratfink

Thanks for the explanation. Unfortunately, I think I'm still missing something here!

I thought OnMarketData was supposed to give a complete picture of the level one changes. If this is the case, when there is a Last event (a trade), I would expect something to also happen to the Bid or Ask volume, matching the traded volume. This is what is seen in most of the data I have looked at, there is even an instance of this at the beginning of the second example I gave:

Bid: 2 @ 25639
Ask: 4 @ 25640
Last: 1 @ 25639
Bid: 1 @ 25639


Here, there was originally a Bid volume of 2 at a Bid price of 25639, followed by a trade (Last) volume of 1 (also at 25639, presumably the Bid price) and then the Bid volume fell by 1 and the Bid price remained at 25639. This is what I would expect to see normally. So I'm not sure what you mean by "Ask and/or Bid can change hundreds of times and mean nothing until the next Last event is seen." - what is the point of Ask or Bid (volume) then? I understand that the Ask or Bid volume can change as new orders come in or are cancelled, but I want to distinguish these from the traded volume.

Please keep explaining, I might get it eventually!

Cheers

Reply With Quote
The following user says Thank You to Cilla for this post:
 
  #6 (permalink)
Market Wizard
Birmingham UK
 
Trading Experience: Intermediate
Platform: NinjaTrader
Broker/Data: TST/Rithmic
Favorite Futures: YM/Gold
 
ratfink's Avatar
 
Posts: 3,631 since Dec 2012
Thanks: 16,954 given, 8,228 received


Cilla View Post
So I'm not sure what you mean by "Ask and/or Bid can change hundreds of times and mean nothing until the next Last event is seen." - what is the point of Ask or Bid (volume) then? I understand that the Ask or Bid volume can change as new orders come in or are cancelled, but I want to distinguish these from the traded volume.

Hi @Cilla The only point of Ask and Bid volume in OnMarketData is as a snapshot of the inside orderbook levels at any given time. For or a fuller picture you can look at OnMarketDepth.

The orderbook (including the inside levels) can and does change independently or concurrently with transacted volume, so the inside level changes can't in themselves be used to distinguish much of use.

Picture a hundred traders adding and removing best bids and offers every second, then once in a while a market order comes in and hits a limit order causing one unit of transaction volume on one side, but at the same time (in our limited retail timestamp space) the latter may or may not be simultaneously replaced by another limit order on the same side in amongst all the other ongoing changes, so your wished for indication may not be seen.

The only information we have is where in Bid-Ask price space the transaction occurred and, as we have now seen, that is still flawed in NT8 though not as much as it was in NT7.

[https://futures.io/ninjatrader/46543-possible-issue-buy-sell-net-volume-2.html#post712025]

Cheers

Look here if you want to go further down the hole...

https://futures.io/emini-index-futures-trading/46299-market-microstructures-red-pill.html#post707839

Travel Well

Last edited by ratfink; April 8th, 2019 at 05:45 AM.
Reply With Quote
The following user says Thank You to ratfink for this post:
 
  #7 (permalink)
Oxford UK
 
Trading Experience: Intermediate
Platform: NinjaTrader
Favorite Futures: YM
 
Posts: 10 since Apr 2019
Thanks: 4 given, 5 received


ratfink View Post
Hi @Cilla The only point of Ask and Bid volume in OnMarketData is as a snapshot of the inside orderbook levels at any given time. For or a fuller picture you can look at OnMarketDepth.

The orderbook (including the inside levels) can and does change independently or concurrently with transacted volume, so the inside level changes can't in themselves be used to distinguish much of use.

Picture a hundred traders adding and removing best bids and offers every second, then once in a while a market order comes in and hits a limit order causing one unit of transaction volume on one side, but at the same time (in our limited retail timestamp space) the latter may or may not be simultaneously replaced by another limit order on the same side in amongst all the other ongoing changes, so your wished for indication may not be seen.

The only information we have is where in Bid-Ask price space the transaction occurred and, as we have now seen, that is still flawed in NT8 though not as much as it was in NT7.

[https://futures.io/ninjatrader/46543-possible-issue-buy-sell-net-volume-2.html#post712025]

Cheers

Look here if you want to go further down the hole...

https://futures.io/emini-index-futures-trading/46299-market-microstructures-red-pill.html#post707839

Thanks again ratfink. In your "hundred traders" picture, I was thinking that would mean that each time a best bid and offer was added or removed, there would be an individual update of the bid or ask volume. Similarly, I was thinking that each trade should be followed immediately by an individual change to the bid or ask volume. But I wasn't catering for "simultaneous" updates (I take it that this can happen on the futures exchange?) being reported as a single update to the bid or ask volume...

Reply With Quote
 
  #8 (permalink)
Market Wizard
Birmingham UK
 
Trading Experience: Intermediate
Platform: NinjaTrader
Broker/Data: TST/Rithmic
Favorite Futures: YM/Gold
 
ratfink's Avatar
 
Posts: 3,631 since Dec 2012
Thanks: 16,954 given, 8,228 received


Cilla View Post
Thanks again ratfink. In your "hundred traders" picture, I was thinking that would mean that each time a best bid and offer was added or removed, there would be an individual update of the bid or ask volume. Similarly, I was thinking that each trade should be followed immediately by an individual change to the bid or ask volume. But I wasn't catering for "simultaneous" updates (I take it that this can happen on the futures exchange?) being reported as a single update to the bid or ask volume...

At some level in the matching engine(s) they will be individual updates, but all we are ever seeing in NT is a summary of the order book. Only transacted volume is consecutive for us.

Always remember to think of the massive server farms in New Jersey and the microwave and optical data links and micro/nano second speeds rather than the second and millisecond levels that we are seeing.

Cheers

Travel Well
Reply With Quote
The following user says Thank You to ratfink for this post:
 
  #9 (permalink)
Oxford UK
 
Trading Experience: Intermediate
Platform: NinjaTrader
Favorite Futures: YM
 
Posts: 10 since Apr 2019
Thanks: 4 given, 5 received


ratfink View Post
At some level in the matching engine(s) they will be individual updates, but all we are ever seeing in NT is a summary of the order book. Only transacted volume is consecutive for us.

Always remember to think of the massive server farms in New Jersey and the microwave and optical data links and micro/nano second speeds rather than the second and millisecond levels that we are seeing.

Cheers

I guess we get what we pay for. I will try feeling smug that there really is "missing data" somewhere in the system!
Cheers!

Reply With Quote
The following user says Thank You to Cilla for this post:
 
  #10 (permalink)
Market Wizard
Birmingham UK
 
Trading Experience: Intermediate
Platform: NinjaTrader
Broker/Data: TST/Rithmic
Favorite Futures: YM/Gold
 
ratfink's Avatar
 
Posts: 3,631 since Dec 2012
Thanks: 16,954 given, 8,228 received



Cilla View Post
I guess we get what we pay for. I will try feeling smug that there really is "missing data" somewhere in the system!
Cheers!

I think better considered as missing information rather than missing data.

Travel Well
Reply With Quote

Reply



futures io > > > > Missing data from OnMarketData?

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)
 

futures io is celebrating 10-years w/ over $18,000 in prizes!

Right now
 

Thanks for everything -- $250 in Amazon Gift Cards ready to give away

Right now
 

Journal in July -- over $2,500 in prizes w/NinjaTrader

July
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
NT Providing Bad Price & Volume Data in OnMarketData!!! RJay NinjaTrader Programming 18 June 5th, 2017 09:53 AM
NT7 anyone else experiencing drawings on charts missing then doubling then missing! Angelo1 NinjaTrader 2 June 10th, 2016 06:37 PM
NT7 drawings on charts missing then doubling then missing ! Angelo1 Traders Hideout 1 May 31st, 2016 11:00 PM
Using OnMarketData() on Historical data with a recording engine gomi NinjaTrader Programming 115 March 6th, 2011 06:11 PM
Programming Indicators Using OnMarketData in NT7!!! RJay NinjaTrader Programming 3 December 16th, 2010 12:16 PM


Tags
amp, buy, cancelled, cme, code, data, data feed, dow, dtn_iqfeed, emini, es, eurex, futures, information, instrument, iqfeed, kinetick, limit, limit order, ma, market data, market order, micros, ninja, ninjatrader, ninjatrader 8, nt7, nt8, onmarketdata, onmarketdepth, order book, orders, people, ream, size, support, tick, traded, trading, underlying, update, values, volume

All times are GMT -4. The time now is 11:14 PM. (this page content is cached, log in for real-time version)

Copyright © 2019 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432 WhatsApp Business, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts