Market Microstructures - The Red Pill - futures io
futures io



Market Microstructures - The Red Pill


Discussion in Emini and Emicro Index

Updated
      Top Posters
    1. looks_one iantg with 20 posts (130 thanks)
    2. looks_two quants00 with 6 posts (11 thanks)
    3. looks_3 options with 4 posts (6 thanks)
    4. looks_4 varaamo with 3 posts (8 thanks)
      Best Posters
    1. looks_one iantg with 6.5 thanks per post
    2. looks_two varaamo with 2.7 thanks per post
    3. looks_3 quants00 with 1.8 thanks per post
    4. looks_4 options with 1.5 thanks per post
    1. trending_up 22,072 views
    2. thumb_up 182 thanks given
    3. group 79 followers
    1. forum 55 posts
    2. attach_file 8 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 150,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

Market Microstructures - The Red Pill

(login for full post details)
  #31 (permalink)
 torr 
Moscow
 
Experience: None
 
Posts: 14 since Oct 2019
Thanks: 12 given, 10 received

Hello Iantg,

Could you, please clarify about "Bid SS Price", "SS Bid Starting V" in "ES 2-25-19 Summary Statistics" file?

Am I got right the values of these fields calculated as:
case 1. If Bid is SS, then "Bid SS Price"="Bid Price", "SS Bid Starting V"="Bid Starting Volume"
case 2. If Bid is NOT SS, "Bid SS Price" calculated from
case 2.1. NextPriceMacroLevel "Bid SS Price", "SS Bid Starting V"
case 2.2. PreviousPriceMacroLevel "Bid SS Price", "SS Bid Starting V"

For case 2.1, case 2.2 didn't catch on what condition it forks.

Or may be I'm totally wrong and found only spurious correlation.

If I got all correct, case 2.1. take values from NextPriceMacroLevel, and it confuses me a bit (forward looking??).

Thank you.

Reply With Quote
The following user says Thank You to torr for this post:

Can you help answer these questions
from other members on futures io?
NT8 - Footprint Chart adding on code
NinjaTrader
 
 
(login for full post details)
  #32 (permalink)
Andis1994
Stockholmm Sweden
 
 
Posts: 5 since Oct 2019
Thanks: 2 given, 3 received

pretty normal and common imo

Reply With Quote
The following user says Thank You to Andis1994 for this post:
 
(login for full post details)
  #33 (permalink)
 iantg 
charlotte nc
 
Experience: Advanced
Platform: My Own System
Broker: Optimus
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 405 since Jan 2015
Thanks: 84 given, 1,121 received


Again thanks for taking an interest in this. The taxonomy pretty much follows this:

Every price level has one side starting with resting volume that has been accumulating on resting price levels. This side I am calling the "strong side" or SS. The weak side or WS is the side that previously won the last price level and therefore does not have any resting volume to start, so it has to spontaneously back-fill volume.

At a micro level, every price level is a fight between 1 strong side vs. 1 weak side. And on instruments like the ES with thicker books the strong side wins the next immediately price level 95% of the time.

At a macro level, the market will typically stay in a state of impasse between the two adjacent strong sides fighting against each other, with each strong side winning over their corresponding weak side. This fight can take maybe 10 to 20 price levels until one of the two weak sides finally beats a strong side.

So for every move the market makes, the labeling system I came up with was to label the two adjacent strong sides and their corresponding starting volumes. The bet in a nutshell is to determine which strong side would win based on which had more volume to start.

Hope this context helps a little.

Ian








torr View Post
Hello Iantg,

Could you, please clarify about "Bid SS Price", "SS Bid Starting V" in "ES 2-25-19 Summary Statistics" file?

Am I got right the values of these fields calculated as:
case 1. If Bid is SS, then "Bid SS Price"="Bid Price", "SS Bid Starting V"="Bid Starting Volume"
case 2. If Bid is NOT SS, "Bid SS Price" calculated from
case 2.1. NextPriceMacroLevel "Bid SS Price", "SS Bid Starting V"
case 2.2. PreviousPriceMacroLevel "Bid SS Price", "SS Bid Starting V"

For case 2.1, case 2.2 didn't catch on what condition it forks.

Or may be I'm totally wrong and found only spurious correlation.

If I got all correct, case 2.1. take values from NextPriceMacroLevel, and it confuses me a bit (forward looking??).

Thank you.


In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
Visit my futures io Trade Journal Started this thread Reply With Quote
The following user says Thank You to iantg for this post:
 
(login for full post details)
  #34 (permalink)
 AllSeeker 
Legendary Pratik_4Clover
Mumbai, India
 
Experience: Beginner
Platform: TradingView & ZerodhaKite
Trading: Crude, NIFTY, BANKNIFTY
 
AllSeeker's Avatar
 
Posts: 1,236 since Jan 2019
Thanks: 4,458 given, 4,313 received

Went over this thread from morning, help me get clear understanding of somethings and for newbies like me.

1. I understood that using orderflow for scalping is not efficient in its current form or shape that retailers use, is that right?

2. If it is, what way should we improve this scenario? I mean I got the overall fill issue of losers getting filled and winners not getting filled, but lets say I'm a scalper looking to use orderflow as secondary confirmation system, how should I improve this overall? Or is it best to take orderflow out of question for scalpers?

3. Have you had opportunity to test it across different time frames? I mean what about using orderflow to confirm accumulation and distribution on larger time frames technical levels? Does that work?

Sorry if I missed or misunderstood anything, but these were the basic questions nagging me when I read through the info you already provided. Please note that I'm not yet fully trading orderflow due to lack of good data provider in my region, so my understanding is very low, so sorry for any stupid questions.

Visit my futures io Trade Journal Reply With Quote
The following 2 users say Thank You to AllSeeker for this post:
 
(login for full post details)
  #35 (permalink)
 iantg 
charlotte nc
 
Experience: Advanced
Platform: My Own System
Broker: Optimus
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 405 since Jan 2015
Thanks: 84 given, 1,121 received

Hi LastDino,

Thanks for taking an interest. To your questions:

1. I won't say that resting volume analysis has no informational value at all, but I can tell you that the edges that I have presented here are not available to retail traders. I tried testing some entries based on volume signals at the top of the book using Ninja Trader with a VPS with fairly decent location to the exchange and my latency was off by a comical magnitude. This was not related to the proximity hosting or the order execution speed, but rather the lag in the data feed reader. By the time my algo touched the data to start analyzing the top of the book, the real market was already 2-3 price levels somewhere else. So It's pretty much a myth that retail traders can use any informational data from the top of the book. And as you get further and further away from the top of the book, resting volumes tent to be less and less correlated to anything.

2. I am probably not the best person to ask about "order flow". In general this means different things to different people. I don't really use this term much because it's not really specific. I would say that most of the edges I have presented here are based on analyzing the Level 2 resting volume, not transacted volume. For the most part these are not going to be relevant to retail traders. But this is the first and only statistical edge that can be peer tested that I have seen demonstrated on this subject. Virtually no one with real edges will ever share, and by contrast most of what get's shared is speculative nonsense. I shared this because it's mostly futile for me at this point, and a few people might get a kick out of seeing it, but most people wouldn't dig in to see the merit.

3. While I will stop short of saying I don't think accumulation and distribution analysis is very helpful, I don't believe there are any real edges there. And regarding time frames, the data here should be a decent-ish representation of everything. If you look at this structure by macro and micro price level and see the analysis I have done here, you will see just how ridiculous it is to arbitrarily always trade off a chart that updates every 157 ticks, or every 35 seconds. To me it's like intentional deception which I will never understand.

If you are interested in scalping at a retail level I think your best bet is to place a resting stop order several ticks away, and pick an instrument with high volatility. You will have a better shot of an entry that pushes your order 1 to 2 ticks in your direction with a break even or 1 tick positive to start. If you try to use limit orders on a very thick instrument, or something that is heavily arbed, you will only ever get toxic fills that run you several ticks in the wrong direction.


Best of luck

Ian




LastDino View Post
Went over this thread from morning, help me get clear understanding of somethings and for newbies like me.

1. I understood that using orderflow for scalping is not efficient in its current form or shape that retailers use, is that right?

2. If it is, what way should we improve this scenario? I mean I got the overall fill issue of losers getting filled and winners not getting filled, but lets say I'm a scalper looking to use orderflow as secondary confirmation system, how should I improve this overall? Or is it best to take orderflow out of question for scalpers?

3. Have you had opportunity to test it across different time frames? I mean what about using orderflow to confirm accumulation and distribution on larger time frames technical levels? Does that work?

Sorry if I missed or misunderstood anything, but these were the basic questions nagging me when I read through the info you already provided. Please note that I'm not yet fully trading orderflow due to lack of good data provider in my region, so my understanding is very low, so sorry for any stupid questions.


In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
Visit my futures io Trade Journal Started this thread Reply With Quote
The following 6 users say Thank You to iantg for this post:
 
(login for full post details)
  #36 (permalink)
 AllSeeker 
Legendary Pratik_4Clover
Mumbai, India
 
Experience: Beginner
Platform: TradingView & ZerodhaKite
Trading: Crude, NIFTY, BANKNIFTY
 
AllSeeker's Avatar
 
Posts: 1,236 since Jan 2019
Thanks: 4,458 given, 4,313 received


iantg View Post
Hi LastDino,

Thanks for taking an interest. To your questions:

1. I won't say that resting volume analysis has no informational value at all, but I can tell you that the edges that I have presented here are not available to retail traders. I tried testing some entries based on volume signals at the top of the book using Ninja Trader with a VPS with fairly decent location to the exchange and my latency was off by a comical magnitude. This was not related to the proximity hosting or the order execution speed, but rather the lag in the data feed reader. By the time my algo touched the data to start analyzing the top of the book, the real market was already 2-3 price levels somewhere else. So It's pretty much a myth that retail traders can use any informational data from the top of the book. And as you get further and further away from the top of the book, resting volumes tent to be less and less correlated to anything.

2. I am probably not the best person to ask about "order flow". In general this means different things to different people. I don't really use this term much because it's not really specific. I would say that most of the edges I have presented here are based on analyzing the Level 2 resting volume, not transacted volume. For the most part these are not going to be relevant to retail traders. But this is the first and only statistical edge that can be peer tested that I have seen demonstrated on this subject. Virtually no one with real edges will ever share, and by contrast most of what get's shared is speculative nonsense. I shared this because it's mostly futile for me at this point, and a few people might get a kick out of seeing it, but most people wouldn't dig in to see the merit.

3. While I will stop short of saying I don't think accumulation and distribution analysis is very helpful, I don't believe there are any real edges there. And regarding time frames, the data here should be a decent-ish representation of everything. If you look at this structure by macro and micro price level and see the analysis I have done here, you will see just how ridiculous it is to arbitrarily always trade off a chart that updates every 157 ticks, or every 35 seconds. To me it's like intentional deception which I will never understand.

If you are interested in scalping at a retail level I think your best bet is to place a resting stop order several ticks away, and pick an instrument with high volatility. You will have a better shot of an entry that pushes your order 1 to 2 ticks in your direction with a break even or 1 tick positive to start. If you try to use limit orders on a very thick instrument, or something that is heavily arbed, you will only ever get toxic fills that run you several ticks in the wrong direction.


Best of luck

Ian



While I'm no expert, I do promote debating usefulness of tools that retailers use and something like orderflow, which is very investment intensive for retailers (especially when you have to buy it in 3rd world countries) should be most debated. And Level 2 is not even available here easily, let alone me using it to get anything out of it.

Its actually little sad that there aren't many like you. But like you said no one will discuss if they really had edge over others, unless of course its some colorful indicator they want to sell.

This is generally to help improve my understanding of it, so your research here is very important, keep going! I might not get everything in get go, but eventually I'll.

Thank you so much

Visit my futures io Trade Journal Reply With Quote
The following 3 users say Thank You to AllSeeker for this post:
 
(login for full post details)
  #37 (permalink)
adolphus
Glendale + CA/United States
 
 
Posts: 3 since Nov 2019
Thanks: 0 given, 1 received

Thanks for the submission. I am new here and I don't know how to use this excel file to my advantage when it comes to actual trading. Mind explaining how this can be used to scalp?

btw just trying to learn (im 17)

Reply With Quote
The following user says Thank You to adolphus for this post:
 
(login for full post details)
  #38 (permalink)
 iantg 
charlotte nc
 
Experience: Advanced
Platform: My Own System
Broker: Optimus
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 405 since Jan 2015
Thanks: 84 given, 1,121 received

In the simplest form possible, trading is essentially betting which way the market will go. So what is your bet? Why are you putting money on red or black? At some point you have to form a basis for your bet. In sports betting you have statistics on teams as a reference. If the Patriots were playing the Dolphins, you would bet on the Patriots right? They have a 90% win rate right now and the dolphins have a 10% win rate. So when you trade the market you have to have some statistics as well.

A lot of people assume that the market behaves a certain way. If you read trading blogs, books etc, you will often hear buzz words around order flow, buying selling pressure, candle stick patterns, etc. How can you definitively know if any of these things are legit or BS? You need to take raw data and analyze all this stuff.

So this leads to the question of how do you get high quality raw data that you can model and test different things. This is where this data comes in. Test away all your ideas, theories and stuff you find online from various gurus. This raw data will allow you the ability to test everything right inside of excel without needing to waste any of your money trading until you have a proof of concept that at least works statistically on paper.

And to your specific question concerning scalping. This data is a look under the hood of all the forces of the market that will destroy you when you try to scalp. Once you learn the odds that are against you, you will realize how unlikely it is that you can ever make money scalping.

Specifically you can learn a lot concerning the use of limit orders, toxic fills, and understanding the real mechanics of involved to potentially earn a few ticks.


Best of luck!



adolphus View Post
Thanks for the submission. I am new here and I don't know how to use this excel file to my advantage when it comes to actual trading. Mind explaining how this can be used to scalp?

btw just trying to learn (im 17)


In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
Visit my futures io Trade Journal Started this thread Reply With Quote
The following 2 users say Thank You to iantg for this post:
 
(login for full post details)
  #39 (permalink)
birdyalex
Moscow
 
 
Posts: 1 since Nov 2013
Thanks: 1 given, 1 received

Hi Ian!

First of all thank you for sharing the results of your work and answering questions!
I tried to replicate your "base data" (ES 2-25-19 Limit Order Fill Rate Analysis.xlsx) based on provided raw L1 data and found some strange cases.
Let's be more specific - from the sample of L1 data (ES 2-25-19 Raw L1 Data.csv) one can build something like 238 "macro price levels" and 214 of which are fully aligned with what one can find in the "base data" sheet (ES 2-25-19 Limit Order Fill Rate Analysis.xlsx), but for the rest of 24 examples there are some minor divergences. I guess it should be emphasized that while one can build 238 "macro price levels" from the raw data youíve got almost 310 "macro price levels" in your "base data" sheet (for the same time slice). Letís see by some examples what I mean.

Example #1
your "base data" sheet got the following figures (in green)


based on the raw data these figures should correspond to the following time slice

Ö


the differences in the figures ("base data" (left) vs. reproduced (right))


at least have a look at the "Ask Starting Volume" and "Ask Ending Volume" for example. One can see from the raw data that the figures should be of 200 and 192 respectively while youíve got 199 and 199. Thatís strange.

Letís for example see if there are any differences for the cases preceding and succeeding the case in question (in blue)

My calculations got matched with yours

- - -


Whatís interesting is the rows 3359-3360 and 3362-3363 canít be calculated from the raw data because thereís no such data in there from which it can be calculcated.


Example #2
your "base data" sheet got the following figures (in green)


based on the raw data these figures should correspond to the following time slice


the differences in the figures ("base data" (left) vs. reproduced (right))


here it is clearly seen that for example "Bid Starting Volume" and "Ask Trans Volume" are 189/3 (vs 191/0 in the "base data").

Letís again compare if there are any differences for cases preceding and succeeding the case in question (in blue)


we have got rows #22173-22218 covering all three cases (preceding, in question and succeeding).


for those preceding and succeeding cases I have got fully matched results with yours

- - -


but again for BBOID=3590,3591 (rows #3595,3596 of the "base data" sheet) one canít recover that data from the raw data.

So my question is did you use one extraction program to build your "base data" sheet and some other to dump raw data? Or am I missing something?

Thanks!

Alex

Reply With Quote
The following 2 users say Thank You to birdyalex for this post:
 
(login for full post details)
  #40 (permalink)
karun
London
 
 
Posts: 1 since Jul 2017
Thanks: 0 given, 1 received


From what I understand the raw data file was solely to illustrate the format of the data, I think the starting timestamps were different so it doesn't align exactly with the Summary Statistics

Reply With Quote
The following user says Thank You to karun for this post:


futures io Trading Community Traders Hideout Emini and Emicro Index > Market Microstructures - The Red Pill


Last Updated on June 20, 2022


Upcoming Webinars and Events
 

NinjaTrader Indicator Challenge!

Ongoing
     



Copyright © 2022 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada), info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts