Market Microstructures - The Red Pill - futures io
futures io



Market Microstructures - The Red Pill


Discussion in Emini and Emicro Index

Updated
      Top Posters
    1. looks_one iantg with 20 posts (130 thanks)
    2. looks_two quants00 with 6 posts (11 thanks)
    3. looks_3 options with 4 posts (6 thanks)
    4. looks_4 varaamo with 3 posts (8 thanks)
      Best Posters
    1. looks_one iantg with 6.5 thanks per post
    2. looks_two varaamo with 2.7 thanks per post
    3. looks_3 quants00 with 1.8 thanks per post
    4. looks_4 options with 1.5 thanks per post
    1. trending_up 22,072 views
    2. thumb_up 182 thanks given
    3. group 79 followers
    1. forum 55 posts
    2. attach_file 8 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 150,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

Market Microstructures - The Red Pill

(login for full post details)
  #41 (permalink)
 iantg 
charlotte nc
 
Experience: Advanced
Platform: My Own System
Broker: Optimus
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 405 since Jan 2015
Thanks: 84 given, 1,121 received

Hi Birdyalex,

I appreciate and respect the reconciliation work you put in. You definitely understand this way better than most people.
To your question about the anomalies. As you can imagine the task of un-bundling and classifying the data isn't super linear. Within Ninjatrader here plenty of cases where the level 2 feed and level 1 feed get out of sync pretty bad, and I have built some heuristics into my extraction program that backfill some of the gaps with kind of a one shoe fits all approach based on the standard case of things likely fit. But there are for sure outliers that will completely be off because I didn't capture every permutation.

But as you have seen, the majority of the cases reconcile to the raw feed file I provided. I will also add that the actual extraction program uses a slightly different data model to the raw file I posted, so there will be some differences there as well. This was more of just a high level example of how everything ties.

I won't claim to be the first guy to ever try to un-bundle volume data and do this type of analysis, but I am perhaps the first that has shared this publicly in such a way.

But in general the gaps you found, are all backfilled with logic in my program and at the end I think I am probably accurately capturing 95% of rows the correct way. The remaining 5% won't be material to the analysis I am doing, but I generally think I get 50% of those right. The remaining rows are likely off by just a small margin, because I didn't go through every possible whammy that could happen.

I also have the MBO feeds, which are way larger, and way more accurate / detailed. I did a recon to these just to get an idea of how well this approach was working and it was within an acceptable margin of error for me.




birdyalex View Post
Hi Ian!

First of all thank you for sharing the results of your work and answering questions!
I tried to replicate your "base data" (ES 2-25-19 Limit Order Fill Rate Analysis.xlsx) based on provided raw L1 data and found some strange cases.
Let's be more specific - from the sample of L1 data (ES 2-25-19 Raw L1 Data.csv) one can build something like 238 "macro price levels" and 214 of which are fully aligned with what one can find in the "base data" sheet (ES 2-25-19 Limit Order Fill Rate Analysis.xlsx), but for the rest of 24 examples there are some minor divergences. I guess it should be emphasized that while one can build 238 "macro price levels" from the raw data youíve got almost 310 "macro price levels" in your "base data" sheet (for the same time slice). Letís see by some examples what I mean.

Example #1
your "base data" sheet got the following figures (in green)


based on the raw data these figures should correspond to the following time slice

Ö


the differences in the figures ("base data" (left) vs. reproduced (right))


at least have a look at the "Ask Starting Volume" and "Ask Ending Volume" for example. One can see from the raw data that the figures should be of 200 and 192 respectively while youíve got 199 and 199. Thatís strange.

Letís for example see if there are any differences for the cases preceding and succeeding the case in question (in blue)

My calculations got matched with yours

- - -


Whatís interesting is the rows 3359-3360 and 3362-3363 canít be calculated from the raw data because thereís no such data in there from which it can be calculcated.


Example #2
your "base data" sheet got the following figures (in green)


based on the raw data these figures should correspond to the following time slice


the differences in the figures ("base data" (left) vs. reproduced (right))


here it is clearly seen that for example "Bid Starting Volume" and "Ask Trans Volume" are 189/3 (vs 191/0 in the "base data").

Letís again compare if there are any differences for cases preceding and succeeding the case in question (in blue)


we have got rows #22173-22218 covering all three cases (preceding, in question and succeeding).


for those preceding and succeeding cases I have got fully matched results with yours

- - -


but again for BBOID=3590,3591 (rows #3595,3596 of the "base data" sheet) one canít recover that data from the raw data.

So my question is did you use one extraction program to build your "base data" sheet and some other to dump raw data? Or am I missing something?

Thanks!

Alex


In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
Visit my futures io Trade Journal Started this thread Reply With Quote
The following 4 users say Thank You to iantg for this post:

Can you help answer these questions
from other members on futures io?
NT8 - Footprint Chart adding on code
NinjaTrader
 
 
(login for full post details)
  #42 (permalink)
 iantg 
charlotte nc
 
Experience: Advanced
Platform: My Own System
Broker: Optimus
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 405 since Jan 2015
Thanks: 84 given, 1,121 received

More or less. The extraction program uses the level 1, and level 2 feeds in a specific sequencing way. What I dumped into excel to show the raw data was just to illustrate how it works at a high level. Some of it can be reconciled and some of it can't. But I hope seeing how everything relates is useful. There aren't many people that have ever tried to un-bundle volume data using retail tools to do this type of analysis. But I think it's super useful for modeling and testing.


karun View Post
From what I understand the raw data file was solely to illustrate the format of the data, I think the starting timestamps were different so it doesn't align exactly with the Summary Statistics


In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
Visit my futures io Trade Journal Started this thread Reply With Quote
The following user says Thank You to iantg for this post:
 
(login for full post details)
  #43 (permalink)
 vaibhavkamble 
Parsippany, NJ, USA
 
Experience: Beginner
Platform: NinjaTrader
Trading: Currency Futures, Futures, Treasuries, Bonds
 
Posts: 20 since Mar 2019
Thanks: 52 given, 5 received



options View Post
Here what I was talking about . Price at support , orders adding on a bid side , plus a 100 lot buyer (dimond) . Went market to get this one , easy 5 ticks .

Hello @options, may I know how you get those diamonds, are those block orders? Thank you.

Sent using the futures.io mobile app

Reply With Quote
 
(login for full post details)
  #44 (permalink)
 Nadir22 
Cairo+Egypt
 
 
Posts: 5 since Jul 2020

Excellent info here. Did you figure out things mostly by yourself or you'd recommend some educators ?

Reply With Quote
 
(login for full post details)
  #45 (permalink)
 iantg 
charlotte nc
 
Experience: Advanced
Platform: My Own System
Broker: Optimus
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 405 since Jan 2015
Thanks: 84 given, 1,121 received

I figured everything out that I presented in this thread on my own just by looking at the data. I am a data scientist among other things by trade, so building this type of taxonomy and analyzing the data this way wasn't too much of a leap for me.



Nadir22 View Post
Excellent info here. Did you figure out things mostly by yourself or you'd recommend some educators ?


In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
Visit my futures io Trade Journal Started this thread Reply With Quote
The following 2 users say Thank You to iantg for this post:
 
(login for full post details)
  #46 (permalink)
jjttjj
Brooklyn, New York
 
 
Posts: 2 since Oct 2014
Thanks: 40 given, 0 received

I discovered this thread months ago and it sent me down a path researching microstructure and developing tools to try to understand it. Thanks for this info, it truly was eye opening for me!

I hope this isn't too off topic, but I'm curious on how the concepts you describe in this thread would apply to penny-tick sized and small spread assets such as stocks? Do you have any thoughts on that? The relatively large tick size of futures make natural price levels. Would you instead use some small bars (possibly tick or volume bars) to set the range for price levels, and consider a "price level" break on the breakout of one of these bars? Or would you just look at it exactly the same way as ES and just accept the penny tick size? Would greatly appreciate any thoughts on this.

Reply With Quote
 
(login for full post details)
  #47 (permalink)
 TWDsje   is a Vendor
 
 
Posts: 756 since Apr 2016
Thanks: 20 given, 952 received

If you really want to get into this sort of thing the book trades quotes and prices financial markets under the microscope covers these subjects quite thoroughly.

Reply With Quote
The following 2 users say Thank You to TWDsje for this post:
 
(login for full post details)
  #48 (permalink)
 iantg 
charlotte nc
 
Experience: Advanced
Platform: My Own System
Broker: Optimus
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 405 since Jan 2015
Thanks: 84 given, 1,121 received

Good question, I've tried the NQ and YM before. They are both a bit thinner than the ES. The concepts are largely the same but there are more gaps in the data with multiple price levels getting swept often. So reconstructing a halfway decent model of what happened is a bit more challenging. The signals you get from thinner books tend to be less helpful in terms of finding alpha.

But on the flip side there are other opportunities with thinner markets too... You just have to know what to look for.

99% of this stuff won't help the average retail trader a bit, I might post a few things more applicable for the majority of users on this site at some point though.


jjttjj View Post
I discovered this thread months ago and it sent me down a path researching microstructure and developing tools to try to understand it. Thanks for this info, it truly was eye opening for me!

I hope this isn't too off topic, but I'm curious on how the concepts you describe in this thread would apply to penny-tick sized and small spread assets such as stocks? Do you have any thoughts on that? The relatively large tick size of futures make natural price levels. Would you instead use some small bars (possibly tick or volume bars) to set the range for price levels, and consider a "price level" break on the breakout of one of these bars? Or would you just look at it exactly the same way as ES and just accept the penny tick size? Would greatly appreciate any thoughts on this.


In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
Visit my futures io Trade Journal Started this thread Reply With Quote
The following 3 users say Thank You to iantg for this post:
 
(login for full post details)
  #49 (permalink)
jjttjj
Brooklyn, New York
 
 
Posts: 2 since Oct 2014
Thanks: 40 given, 0 received

In thinking about this a bit more, perhaps the more general concept is "best bid/offers tend to mean revert". A retracement to a "weak side" is a reversion to the mean. When the tick size is large this plays out as a bouncing between the recent levels. The degree of bounciness is probably related to the volatility of the thing being traded, but a large tick size makes for smoother movement because each tick movement has a relatively large barrier to break through. I think I'll look into using some moving averages of tick bars. Perhaps an "average movement per tick" will correspond to some rough price levels.

Reply With Quote
 
(login for full post details)
  #50 (permalink)
 chipwitch 
Nashville, TN
 
Experience: Beginner
Platform: NinjaTrader
Broker: NinjaTrader, Continuum Data
Trading: MES for now... baby steps
 
chipwitch's Avatar
 
Posts: 320 since Feb 2022
Thanks: 229 given, 596 received


@iantg and @birdyalex

Did you need to sort the streamed data by time? I just started messing with the NT OnMarketDepth() event. When I streamed a data sample direct to a text file I noticed that the orders are somewhat out of chronological order? Sometimes by as much as 30 seconds?

Reply With Quote


futures io Trading Community Traders Hideout Emini and Emicro Index > Market Microstructures - The Red Pill


Last Updated on June 20, 2022


Upcoming Webinars and Events
 

NinjaTrader Indicator Challenge!

Ongoing
     



Copyright © 2022 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada), info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts