NexusFi: Find Your Edge


Home Menu

 





MDP 3.0, is CME migrating to a new data comm protocol?


Discussion in Brokers

Updated
      Top Posters
    1. looks_one Big Mike with 17 posts (34 thanks)
    2. looks_two artemiso with 6 posts (2 thanks)
    3. looks_3 LukeGeniol with 6 posts (1 thanks)
    4. looks_4 Malthus with 5 posts (18 thanks)
      Best Posters
    1. looks_one SierraChart with 13.5 thanks per post
    2. looks_two aslan with 7 thanks per post
    3. looks_3 trendwaves with 3.6 thanks per post
    4. looks_4 Big Mike with 2 thanks per post
    1. trending_up 46,109 views
    2. thumb_up 193 thanks given
    3. group 50 followers
    1. forum 86 posts
    2. attach_file 0 attachments




 
Search this Thread

MDP 3.0, is CME migrating to a new data comm protocol?

  #51 (permalink)
 Ruffcut 
South Lyon Michigan USA
 
Experience: Intermediate
Platform: Ninjatrader, AMP, tradeking thinkorswim
Trading: CL, TF
Posts: 61 since Sep 2012
Thanks Given: 617
Thanks Received: 106

Hi,
I noticed your post in the thread "MDP 3.0, is CME migrating to a new data comm protocol?"

I use tick charts frequently as well. You mentioned "Even midday I would have to clean the NT database tick files and cache. "

Is there a proper way to do this?

Thank you!

Monika

I am putting it here for anyone else:
2015-05-09_1145 - Ruffcut's library
2015-05-09_1146 - Ruffcut's library
2015-05-09_1148 - Ruffcut's library
2015-05-09_1149 - Ruffcut's library
2015-05-09_1151 - Ruffcut's library
2015-05-09_1154 - Ruffcut's library

You can do this for minute files as well, but they never seem to clog up the DB like the tickers.

Reply With Quote

Can you help answer these questions
from other members on NexusFi?
Better Renko Gaps
The Elite Circle
Deepmoney LLM
Elite Quantitative GenAI/LLM
Build trailing stop for micro index(s)
Psychology and Money Management
The space time continuum and the dynamics of a financial …
Emini and Emicro Index
NexusFi Journal Challenge - April 2024
Feedback and Announcements
 
  #52 (permalink)
HFF Trader
Minneapolis Minnesota
 
Posts: 14 since Apr 2014
Thanks Given: 8
Thanks Received: 3

There is a lot of misunderstanding about MDP3 here. Please read the documentation on CME's website and you will see that it's not the bundling that is the issue. The issue is that most data providers are not willing/capable to deliver the unbundled trades to you-although CME provides that information in the raw feed.

Reply With Quote
  #53 (permalink)
shots fired
Boise, Idaho
 
Posts: 71 since Feb 2013
Thanks Given: 28
Thanks Received: 10



aslan View Post
Interesting thread, but there seems to be a lot of mis-understandings.

The new protocol sends thru trades using an event based model. So if an aggressor order triggers 5 other orders (i.e. buy 5 @ market triggers 5 sell orders with qty = 1), a single trade entry is sent thru. To me, this is NOT bundling, but instead is correct behavior in that a single aggressor trade filled (or partially filled). The messages for the event also contain the number of orders and the qty that was filled against each order, so that trade could be surfaced as 5 single contract trades, but while that matches todays feed behavior, it seems like that is not the way to go IMO. Would you rather know that someone bought 100 or see 100 1 car entries fly by?

Say you have a single aggressor order that triggers a bunch of orders at different prices. In this case, multiple trade entries are sent thru, one for each price. These trade entries are bundled into the same event and have the same timestamp, but should be represented as multiple trades. Again, the order fill qty is available, so it could be broken down into smaller pieces based on the matched order vs the aggressor order. Again, this is not really bundling.

There are some other fringe cases for spreads, implied trades, and misc events, but I think the above are the major ones that matter.

Another type of bundling is the bundling of multiple messages into a packet that is sent over the network. I have not seen any doc on what kind of time window is used to do this bundling, but it is required to do this in order to have an efficient transfer of data. I suspect the window is rather small so as not to affect latency.

The event based model sends thru trades marked to the nano-second, but really there is not much use for this granularity for the mere mortal, especially when your latency is measured in ms. Also, most charting platforms won't be able to record the ns anyway, so will likely be truncated back to something less granular.

Also, the aggressor is properly marked, so things like delta should be fine.

Translation, please?

Reply With Quote
  #54 (permalink)
 
aslan's Avatar
 aslan 
Madison, WI
 
Experience: Advanced
Platform: ALT
Trading: ES
Posts: 625 since Jan 2010
Thanks Given: 356
Thanks Received: 1,127


shots fired View Post
Translation, please?

People are worried about nothing. The only down side is a tick chart may need to be slightly adjusted in terms of number of ticks to account for a smaller number of avg ticks.

Reply With Quote
  #55 (permalink)
 
trendwaves's Avatar
 trendwaves 
Florida
Market Wizard
 
Experience: Advanced
Platform: NinjaTrader 8
Trading: ES, NQ, CL
Posts: 703 since Dec 2012
Thanks Given: 2,898
Thanks Received: 2,525


aslan View Post
People are worried about nothing. The only down side is a tick chart may need to be slightly adjusted in terms of number of ticks to account for a smaller number of avg ticks.

I wish i could share your optimism.

Seeing how different vendors are already looking at different ways to break apart (or not) these data messages for subsequent delivery to the end users concerns me. CME is taking the position: here is the new protocol, good luck.

There are timing issues involved in how to precisely and accurately break apart and correctly synchronize (insert) the components of a packet into the resulting output stream. This is because the stream is not just trade executions, but is happening in parallel with book update events. It is the correct and accurate ordering of these asynchronous event streams that has in the past caused significant problems for vendors.

In the current protocol, there is a well known race condition between asynchronous trade execution events with the separate stream of order book update events. This race condition, can produce a mis-ordering of events in the resulting stream produced by an invalid vendor algorithm inadvertently causing buy orders to appear as sell orders and conversely sell orders to appear as buy orders in the stream. With each vendor's home brewed algorithm we ended up with vendors implementing a variety of solutions resulting in a ranking list from good to bad of retail data streams with respect to what we call an 'unfiltered' data feed. A significant variance of the accuracy of unfiltered data feed currently exists from vendor to vendor.

This new protocol adds at least one new layer of complexity to this, and perhaps multiple layers of complexity. Based on past performance, I cannot help but wonder if any vendor will 'get it right' this time ?

Having said all that, I personally have always favored viewing the market from the perspective of the initiating trade ( 'agressor' ). I find much more value in seeing the initiating trade, (Mikes 50 lot trade), on the tape. I really don't care about seeing the unending stream of Joe 1-lot fills. I view this as a signal to noise problem. My concern is some (hopefully not all) vendors may attempt to deconstruct the initiating trade execution packet back into a useless stream of resulting 1-lot fills. My hope is the bandwidth savings (and lower implementation complexity) found in the initiating trade stream, as designed by the CME, will persuade vendors to do the right thing and keep it intact for their customers. Sometimes less is more.

Be Patient and Trade Smart
Visit my NexusFi Trade Journal Reply With Quote
  #56 (permalink)
 
cory's Avatar
 cory 
virginia
 
Experience: Intermediate
Platform: ninja
Trading: NQ
Posts: 6,098 since Jun 2009
Thanks Given: 877
Thanks Received: 8,090


trendwaves View Post
... I find much more value in seeing the initiating trade, (Mikes 50 lot trade), on the tape. I really don't care about seeing the unending stream of Joe 1-lot fills. ...

nice analogy!

Reply With Quote
Thanked by:
  #57 (permalink)
 
aslan's Avatar
 aslan 
Madison, WI
 
Experience: Advanced
Platform: ALT
Trading: ES
Posts: 625 since Jan 2010
Thanks Given: 356
Thanks Received: 1,127


HFF Trader View Post
There is a lot of misunderstanding about MDP3 here. Please read the documentation on CME's website and you will see that it's not the bundling that is the issue. The issue is that most data providers are not willing/capable to deliver the unbundled trades to you-although CME provides that information in the raw feed.

This dead on. The info is there, but there are some fringe cases that are not well documented. The entire MDP3 protocol is kind of like that though, very general purpose (actually too general purpose) and not trivial to implement. It is definitely optimized to send thru trades based on the aggressor order.

Reply With Quote
  #58 (permalink)
 ejtrader 
Portland, OR
 
Experience: Intermediate
Platform: Sierra Chart
Trading: ES
Posts: 269 since Jan 2011
Thanks Given: 327
Thanks Received: 180

@SierraChart - Would prefer to have the transfer done in a single trade rather than individual trades which are split. Atleast please have an option to pick this method.


SierraChart View Post
We have done further research and we can break out the individual trades and we will do that.


Reply With Quote
  #59 (permalink)
 
aslan's Avatar
 aslan 
Madison, WI
 
Experience: Advanced
Platform: ALT
Trading: ES
Posts: 625 since Jan 2010
Thanks Given: 356
Thanks Received: 1,127


trendwaves View Post
There are timing issues involved in how to precisely and accurately break apart and correctly synchronize (insert) the components of a packet into the resulting output stream. This is because the stream is not just trade executions, but is happening in parallel with book update events. It is the correct and accurate ordering of these asynchronous event streams that has in the past caused significant problems for vendors.

In the current protocol, there is a well known race condition between asynchronous trade execution events with the separate stream of order book update events. This race condition, can produce a mis-ordering of events in the resulting stream produced by an invalid vendor algorithm inadvertently causing buy orders to appear as sell orders and conversely sell orders to appear as buy orders in the stream. With each vendor's home brewed algorithm we ended up with vendors implementing a variety of solutions resulting in a ranking list from good to bad of retail data streams with respect to what we call an 'unfiltered' data feed. A significant variance of the accuracy of unfiltered data feed currently exists from vendor to vendor.

This really is not the case, as each trade in the feed is marked as a buy or sell aggressor (true for old and new protocol). There are some trades that come thru not marked (i.e. implied trades), and these trades can cause some minor differences between vendors. Now if the vendor does not send that flag thru, then what you say is totally true as they have to be marked based on the current book at the client end which is a crap shoot.

In the old protocol, it was very painful because all updates were interspersed, so you could see a packet with trades and book updates for multiple symbols in any order. So, you are correct there were some serious race conditions to watch for, but it was manageable. In the new protocol, that is mitigated with the event model, so things are not interspersed.

Another major factor is how the vendor gets and processes the data. Each vendor has to decide how to interface to the MDP stream. If you are using an off the shelf component, you may not have access to the info to do the splitting, because you are getting a massaged version of the data. If instead the vendor writes to the lowest level, then they have the info.

I think in the short term, you will see differences between vendors until the CME switches over to MDP only and people decide on showing aggressor trades vs split trades).

Reply With Quote
  #60 (permalink)
 
Big Mike's Avatar
 Big Mike 
Manta, Ecuador
Site Administrator
Developer
Swing Trader
 
Experience: Advanced
Platform: Custom solution
Broker: IBKR
Trading: Stocks & Futures
Frequency: Every few days
Duration: Weeks
Posts: 50,398 since Jun 2009
Thanks Given: 33,173
Thanks Received: 101,537



artemiso View Post
No, the new protocol is simpler. In fact it's a shame that CME is 10 years late to the game to make this change. Among other things, FIX specifies the dissemination of field values in text representation, which contributes to this mess:



For comparison, ARCA's raw feed (trades only) already adopts a binary protocol and on the same day and time window looks like this:



As quoted earlier in this thread, I had been using MDP 3.0 in pre-production since early 2014. A small dump of channel 313 on Mar 10, 2014 just for credibility:

 
Code
4294967236 4294967253 3 0 4294967232 28 125 42 99 65 90 19 47 0 9 0 20 0 1 0 4294967168 4294967258 109 42 99 65 90 19 4294967172 27 0 1 1 49 20 4294967179 9 0 4294967279 9 0 0 1 4294967200 37 38 0 0 0 0 0 112 23 0 0 1 0 0 0
4294967237 4294967253 3 0 64 4294967180 101 62 99 65 90 19 47 0 9 0 20 0 1 0 0 74 86 62 99 65 90 19 4294967172 27 0 1 0 48 63 4294967221 1 0 4294967265 15 0 0 1 4294967232 4294967214 59 40 0 0 0 0 3 0 0 0 1 0 0 0
4294967238 4294967253 3 0 4294967232 4294967291 77 82 99 65 90 19 50 0 9 0 23 0 1 0 64 119 47 82 99 65 90 19 1 30 0 1 0 63 4294967221 1 0 4294967266 15 0 0 4294967232 4294967214 59 40 0 0 0 0 3 0 0 0 2 0 0 0 107 2 0 0 2 33 0 9 0 5 0 1 0 64 119 47 82 99 65 90 19 2 13 0 1 0 63 4294967221 1 0 4294967267 15 0 0 50 7 0 0 47 0 9 0 20 0 1 0 64 119 47 82 99 65 90 19 4294967172 27 0 1 2 48 63 4294967221 1 0 4294967268 15 0 0 1 4294967232 4294967214 59 40 0 0 0 0 3 0 0 0 1 0 0 0
4294967239 4294967253 3 0 0 41 39 102 99 65 90 19 47 0 9 0 20 0 1 0 4294967232 4294967270 23 102 99 65 90 19 4294967172 27 0 1 1 49 47 76 1 0 4294967269 4294967242 1 0 1 64 4294967250 4294967263 3 0 0 0 0 26 0 0 0 26 0 0 0
4294967240 4294967253 3 0 4294967168 4294967192 15 122 99 65 90 19 47 0 9 0 20 0 1 0 64 86 0 122 99 65 90 19 4294967172 27 0 1 0 49 63 4294967221 1 0 4294967269 15 0 0 1 4294967168 99 4294967279 39 0 0 0 0 100 0 0 0 1 0 0 0
4294967241 4294967253 3 0 4294967232 4294967237 4294967272 4294967181 99 65 90 19 47 0 9 0 20 0 1 0 4294967168 4294967171 4294967257 4294967181 99 65 90 19 4294967172 27 0 1 1 48 97 16 5 0 4294967189 2 0 0 1 0 4294967172 4294967255 23 0 0 0 0 26 0 0 0 26 0 0 0
My experience is that it took >100,000 lines of code to build a production quality CME FIX/FAST feed handler (most of it general-purpose FIX/FAST), but only >10,000 to do the same for MDP 3.0. I don't think there's any excuse for a vendor to have issues giving you a production quality MDP 3.0 feed.

Does the new feed have shortcomings?

Yes, but they matter only to very few on this forum and in general the developments are very positive for retail users.

1. For one, XML schemas are still verbose and dated. (Of the >10,000 lines of code I mentioned, about 15-20% had to do just with the XML message schemas.)

2. A positive departure from FIX/FAST is that you can access fields within a message sub-tree in random order. As mentioned in earlier posts, most vendors ignore certain fields from the data feed before disseminating it to you. In a FIX/FAST feed, they would have to search for delimiters to skip each field, whereas in the new feed, they can 'instantly' skip between primitive fields to reduce latency. For those of us in the unfortunate business of being faster than everyone else though, the message tree is nevertheless written in pre-order. You could shave another 20~ nanoseconds in limited use cases if you can access message sub-trees in random order.

Thank you.

Mike

We're here to help: just ask the community or contact our Help Desk

Quick Links: Change your Username or Register as a Vendor
Searching for trading reviews? Review this list
Lifetime Elite Membership: Sign-up for only $149 USD
Exclusive money saving offers from our Site Sponsors: Browse Offers
Report problems with the site: Using the NexusFi changelog thread
Follow me on Twitter Visit my NexusFi Trade Journal Reply With Quote




Last Updated on May 19, 2017


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts