MDP 3.0, is CME migrating to a new data comm protocol? - Reviews of Brokers and Data Feeds | futures io social day trading
futures io futures trading


MDP 3.0, is CME migrating to a new data comm protocol?
Updated: Views / Replies:24,543 / 86
Created: by Malthus Attachments:4

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 4  
 
Thread Tools Search this Thread
 

MDP 3.0, is CME migrating to a new data comm protocol?

  #51 (permalink)
Elite Member
South Lyon Michigan USA
 
Futures Experience: Intermediate
Platform: Ninjatrader, AMP, tradeking thinkorswim
Favorite Futures: CL, TF
 
Posts: 61 since Sep 2012
Thanks: 617 given, 107 received

Monika writes in PM

Hi,
I noticed your post in the thread "MDP 3.0, is CME migrating to a new data comm protocol?"

I use tick charts frequently as well. You mentioned "Even midday I would have to clean the NT database tick files and cache. "

Is there a proper way to do this?

Thank you!

Monika

I am putting it here for anyone else:
2015-05-09_1145 - Ruffcut's library
2015-05-09_1146 - Ruffcut's library
2015-05-09_1148 - Ruffcut's library
2015-05-09_1149 - Ruffcut's library
2015-05-09_1151 - Ruffcut's library
2015-05-09_1154 - Ruffcut's library

You can do this for minute files as well, but they never seem to clog up the DB like the tickers.

Reply With Quote
The following user says Thank You to Ruffcut for this post:
 
  #52 (permalink)
 Vendor: highfrequencyfootprint.com 
Minneapolis Minnesota
 
Futures Experience: Advanced
Platform: NinjaTrader,Sierra Chart
Broker/Data: IQFeed
Favorite Futures: CL
 
Posts: 13 since Apr 2014
Thanks: 6 given, 3 received

There is a lot of misunderstanding about MDP3 here. Please read the documentation on CME's website and you will see that it's not the bundling that is the issue. The issue is that most data providers are not willing/capable to deliver the unbundled trades to you-although CME provides that information in the raw feed.

Reply With Quote
 
  #53 (permalink)
Trading for Fun
Boise, Idaho
 
Futures Experience: Intermediate
Platform: Ninja Trader
Broker/Data: AMP/CQG
Favorite Futures: es
 
Posts: 71 since Feb 2013
Thanks: 28 given, 10 received



aslan View Post
Interesting thread, but there seems to be a lot of mis-understandings.

The new protocol sends thru trades using an event based model. So if an aggressor order triggers 5 other orders (i.e. buy 5 @ market triggers 5 sell orders with qty = 1), a single trade entry is sent thru. To me, this is NOT bundling, but instead is correct behavior in that a single aggressor trade filled (or partially filled). The messages for the event also contain the number of orders and the qty that was filled against each order, so that trade could be surfaced as 5 single contract trades, but while that matches todays feed behavior, it seems like that is not the way to go IMO. Would you rather know that someone bought 100 or see 100 1 car entries fly by?

Say you have a single aggressor order that triggers a bunch of orders at different prices. In this case, multiple trade entries are sent thru, one for each price. These trade entries are bundled into the same event and have the same timestamp, but should be represented as multiple trades. Again, the order fill qty is available, so it could be broken down into smaller pieces based on the matched order vs the aggressor order. Again, this is not really bundling.

There are some other fringe cases for spreads, implied trades, and misc events, but I think the above are the major ones that matter.

Another type of bundling is the bundling of multiple messages into a packet that is sent over the network. I have not seen any doc on what kind of time window is used to do this bundling, but it is required to do this in order to have an efficient transfer of data. I suspect the window is rather small so as not to affect latency.

The event based model sends thru trades marked to the nano-second, but really there is not much use for this granularity for the mere mortal, especially when your latency is measured in ms. Also, most charting platforms won't be able to record the ns anyway, so will likely be truncated back to something less granular.

Also, the aggressor is properly marked, so things like delta should be fine.

Translation, please?

Reply With Quote
 
  #54 (permalink)
Elite Member
Madison, WI
 
Futures Experience: Advanced
Platform: Sierra Charts, ALT
Favorite Futures: ES
 
aslan's Avatar
 
Posts: 614 since Jan 2010
Thanks: 342 given, 1,077 received


shots fired View Post
Translation, please?

People are worried about nothing. The only down side is a tick chart may need to be slightly adjusted in terms of number of ticks to account for a smaller number of avg ticks.

Reply With Quote
The following 5 users say Thank You to aslan for this post:
 
  #55 (permalink)
Elite Member
Florida
 
Futures Experience: Advanced
Platform: NinjaTrader 8
Favorite Futures: YM, ES, NQ, CL, ZB
 
trendwaves's Avatar
 
Posts: 740 since Dec 2012
Thanks: 2,821 given, 2,427 received


aslan View Post
People are worried about nothing. The only down side is a tick chart may need to be slightly adjusted in terms of number of ticks to account for a smaller number of avg ticks.

I wish i could share your optimism.

Seeing how different vendors are already looking at different ways to break apart (or not) these data messages for subsequent delivery to the end users concerns me. CME is taking the position: here is the new protocol, good luck.

There are timing issues involved in how to precisely and accurately break apart and correctly synchronize (insert) the components of a packet into the resulting output stream. This is because the stream is not just trade executions, but is happening in parallel with book update events. It is the correct and accurate ordering of these asynchronous event streams that has in the past caused significant problems for vendors.

In the current protocol, there is a well known race condition between asynchronous trade execution events with the separate stream of order book update events. This race condition, can produce a mis-ordering of events in the resulting stream produced by an invalid vendor algorithm inadvertently causing buy orders to appear as sell orders and conversely sell orders to appear as buy orders in the stream. With each vendor's home brewed algorithm we ended up with vendors implementing a variety of solutions resulting in a ranking list from good to bad of retail data streams with respect to what we call an 'unfiltered' data feed. A significant variance of the accuracy of unfiltered data feed currently exists from vendor to vendor.

This new protocol adds at least one new layer of complexity to this, and perhaps multiple layers of complexity. Based on past performance, I cannot help but wonder if any vendor will 'get it right' this time ?

Having said all that, I personally have always favored viewing the market from the perspective of the initiating trade ( 'agressor' ). I find much more value in seeing the initiating trade, (Mikes 50 lot trade), on the tape. I really don't care about seeing the unending stream of Joe 1-lot fills. I view this as a signal to noise problem. My concern is some (hopefully not all) vendors may attempt to deconstruct the initiating trade execution packet back into a useless stream of resulting 1-lot fills. My hope is the bandwidth savings (and lower implementation complexity) found in the initiating trade stream, as designed by the CME, will persuade vendors to do the right thing and keep it intact for their customers. Sometimes less is more.

Be Patient and Trade Smart
Reply With Quote
The following 4 users say Thank You to trendwaves for this post:
 
  #56 (permalink)
Elite Member
virginia
 
Futures Experience: Intermediate
Platform: ninja
Favorite Futures: ES
 
cory's Avatar
 
Posts: 5,198 since Jun 2009
Thanks: 627 given, 6,285 received


trendwaves View Post
... I find much more value in seeing the initiating trade, (Mikes 50 lot trade), on the tape. I really don't care about seeing the unending stream of Joe 1-lot fills. ...

nice analogy!

Reply With Quote
The following user says Thank You to cory for this post:
 
  #57 (permalink)
Elite Member
Madison, WI
 
Futures Experience: Advanced
Platform: Sierra Charts, ALT
Favorite Futures: ES
 
aslan's Avatar
 
Posts: 614 since Jan 2010
Thanks: 342 given, 1,077 received


HFF Trader View Post
There is a lot of misunderstanding about MDP3 here. Please read the documentation on CME's website and you will see that it's not the bundling that is the issue. The issue is that most data providers are not willing/capable to deliver the unbundled trades to you-although CME provides that information in the raw feed.

This dead on. The info is there, but there are some fringe cases that are not well documented. The entire MDP3 protocol is kind of like that though, very general purpose (actually too general purpose) and not trivial to implement. It is definitely optimized to send thru trades based on the aggressor order.

Reply With Quote
 
  #58 (permalink)
Elite Member
Portland, OR
 
Futures Experience: Intermediate
Platform: Sierra Chart
Favorite Futures: ES
 
Posts: 269 since Jan 2011
Thanks: 326 given, 179 received

@SierraChart - Would prefer to have the transfer done in a single trade rather than individual trades which are split. Atleast please have an option to pick this method.


SierraChart View Post
We have done further research and we can break out the individual trades and we will do that.


Reply With Quote
 
  #59 (permalink)
Elite Member
Madison, WI
 
Futures Experience: Advanced
Platform: Sierra Charts, ALT
Favorite Futures: ES
 
aslan's Avatar
 
Posts: 614 since Jan 2010
Thanks: 342 given, 1,077 received


trendwaves View Post
There are timing issues involved in how to precisely and accurately break apart and correctly synchronize (insert) the components of a packet into the resulting output stream. This is because the stream is not just trade executions, but is happening in parallel with book update events. It is the correct and accurate ordering of these asynchronous event streams that has in the past caused significant problems for vendors.

In the current protocol, there is a well known race condition between asynchronous trade execution events with the separate stream of order book update events. This race condition, can produce a mis-ordering of events in the resulting stream produced by an invalid vendor algorithm inadvertently causing buy orders to appear as sell orders and conversely sell orders to appear as buy orders in the stream. With each vendor's home brewed algorithm we ended up with vendors implementing a variety of solutions resulting in a ranking list from good to bad of retail data streams with respect to what we call an 'unfiltered' data feed. A significant variance of the accuracy of unfiltered data feed currently exists from vendor to vendor.

This really is not the case, as each trade in the feed is marked as a buy or sell aggressor (true for old and new protocol). There are some trades that come thru not marked (i.e. implied trades), and these trades can cause some minor differences between vendors. Now if the vendor does not send that flag thru, then what you say is totally true as they have to be marked based on the current book at the client end which is a crap shoot.

In the old protocol, it was very painful because all updates were interspersed, so you could see a packet with trades and book updates for multiple symbols in any order. So, you are correct there were some serious race conditions to watch for, but it was manageable. In the new protocol, that is mitigated with the event model, so things are not interspersed.

Another major factor is how the vendor gets and processes the data. Each vendor has to decide how to interface to the MDP stream. If you are using an off the shelf component, you may not have access to the info to do the splitting, because you are getting a massaged version of the data. If instead the vendor writes to the lowest level, then they have the info.

I think in the short term, you will see differences between vendors until the CME switches over to MDP only and people decide on showing aggressor trades vs split trades).

Reply With Quote
The following 8 users say Thank You to aslan for this post:
 
  #60 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,237 since Jun 2009
Thanks: 29,350 given, 83,153 received



artemiso View Post
No, the new protocol is simpler. In fact it's a shame that CME is 10 years late to the game to make this change. Among other things, FIX specifies the dissemination of field values in text representation, which contributes to this mess:

Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


For comparison, ARCA's raw feed (trades only) already adopts a binary protocol and on the same day and time window looks like this:

Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


As quoted earlier in this thread, I had been using MDP 3.0 in pre-production since early 2014. A small dump of channel 313 on Mar 10, 2014 just for credibility:

 
Code
4294967236 4294967253 3 0 4294967232 28 125 42 99 65 90 19 47 0 9 0 20 0 1 0 4294967168 4294967258 109 42 99 65 90 19 4294967172 27 0 1 1 49 20 4294967179 9 0 4294967279 9 0 0 1 4294967200 37 38 0 0 0 0 0 112 23 0 0 1 0 0 0
4294967237 4294967253 3 0 64 4294967180 101 62 99 65 90 19 47 0 9 0 20 0 1 0 0 74 86 62 99 65 90 19 4294967172 27 0 1 0 48 63 4294967221 1 0 4294967265 15 0 0 1 4294967232 4294967214 59 40 0 0 0 0 3 0 0 0 1 0 0 0
4294967238 4294967253 3 0 4294967232 4294967291 77 82 99 65 90 19 50 0 9 0 23 0 1 0 64 119 47 82 99 65 90 19 1 30 0 1 0 63 4294967221 1 0 4294967266 15 0 0 4294967232 4294967214 59 40 0 0 0 0 3 0 0 0 2 0 0 0 107 2 0 0 2 33 0 9 0 5 0 1 0 64 119 47 82 99 65 90 19 2 13 0 1 0 63 4294967221 1 0 4294967267 15 0 0 50 7 0 0 47 0 9 0 20 0 1 0 64 119 47 82 99 65 90 19 4294967172 27 0 1 2 48 63 4294967221 1 0 4294967268 15 0 0 1 4294967232 4294967214 59 40 0 0 0 0 3 0 0 0 1 0 0 0
4294967239 4294967253 3 0 0 41 39 102 99 65 90 19 47 0 9 0 20 0 1 0 4294967232 4294967270 23 102 99 65 90 19 4294967172 27 0 1 1 49 47 76 1 0 4294967269 4294967242 1 0 1 64 4294967250 4294967263 3 0 0 0 0 26 0 0 0 26 0 0 0
4294967240 4294967253 3 0 4294967168 4294967192 15 122 99 65 90 19 47 0 9 0 20 0 1 0 64 86 0 122 99 65 90 19 4294967172 27 0 1 0 49 63 4294967221 1 0 4294967269 15 0 0 1 4294967168 99 4294967279 39 0 0 0 0 100 0 0 0 1 0 0 0
4294967241 4294967253 3 0 4294967232 4294967237 4294967272 4294967181 99 65 90 19 47 0 9 0 20 0 1 0 4294967168 4294967171 4294967257 4294967181 99 65 90 19 4294967172 27 0 1 1 48 97 16 5 0 4294967189 2 0 0 1 0 4294967172 4294967255 23 0 0 0 0 26 0 0 0 26 0 0 0
My experience is that it took >100,000 lines of code to build a production quality CME FIX/FAST feed handler (most of it general-purpose FIX/FAST), but only >10,000 to do the same for MDP 3.0. I don't think there's any excuse for a vendor to have issues giving you a production quality MDP 3.0 feed.

Does the new feed have shortcomings?

Yes, but they matter only to very few on this forum and in general the developments are very positive for retail users.

1. For one, XML schemas are still verbose and dated. (Of the >10,000 lines of code I mentioned, about 15-20% had to do just with the XML message schemas.)

2. A positive departure from FIX/FAST is that you can access fields within a message sub-tree in random order. As mentioned in earlier posts, most vendors ignore certain fields from the data feed before disseminating it to you. In a FIX/FAST feed, they would have to search for delimiters to skip each field, whereas in the new feed, they can 'instantly' skip between primitive fields to reduce latency. For those of us in the unfortunate business of being faster than everyone else though, the message tree is nevertheless written in pre-order. You could shave another 20~ nanoseconds in limited use cases if you can access message sub-trees in random order.

Thank you.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
The following 5 users say Thank You to Big Mike for this post:

Reply



futures io > > > > MDP 3.0, is CME migrating to a new data comm protocol?

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Adam Grimes: TBA

Elite only

NinjaTrader: TBA

Dec 7

Linda Bradford Raschke: TBA

Elite only

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Bitcoin protocol vulnerability Big Mike Cryptocurrency Trading 0 November 5th, 2013 05:11 PM
Free CME Globex data KennyK Reviews of Brokers and Data Feeds 2 September 23rd, 2013 11:04 AM
Just need CME mini data Feed Blister Reviews of Brokers and Data Feeds 5 August 31st, 2013 04:00 PM
FBI wants to ban new Internet protocol kbit News and Current Events 1 June 19th, 2012 05:44 PM
Mirus free $500 comm thru end of March 2012 Big Mike Reviews of Brokers and Data Feeds 7 June 17th, 2012 04:54 AM


All times are GMT -4. The time now is 05:48 PM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-11-19 in 0.15 seconds with 19 queries on phoenix via your IP 54.224.18.114