Hello Iantg,

Could you, please clarify about "Bid SS Price", "SS Bid Starting V" in "ES 2-25-19 Summary Statistics" file?

Am I got right the values of these fields calculated as:
case 1. If Bid is SS, then "Bid SS Price"="Bid Price", "SS Bid Starting V"="Bid Starting Volume"
case 2. If Bid is NOT SS, "Bid SS Price" calculated from
case 2.1. NextPriceMacroLevel "Bid SS Price", "SS Bid Starting V"
case 2.2. PreviousPriceMacroLevel "Bid SS Price", "SS Bid Starting V"

For case 2.1, case 2.2 didn't catch on what condition it forks.

Or may be I'm totally wrong and found only spurious correlation.

If I got all correct, case 2.1. take values from NextPriceMacroLevel, and it confuses me a bit (forward looking??).

Thank you.

pretty normal and common imo

Again thanks for taking an interest in this. The taxonomy pretty much follows this:

Every price level has one side starting with resting volume that has been accumulating on resting price levels. This side I am calling the "strong side" or SS. The weak side or WS is the side that previously won the last price level and therefore does not have any resting volume to start, so it has to spontaneously back-fill volume.

At a micro level, every price level is a fight between 1 strong side vs. 1 weak side. And on instruments like the ES with thicker books the strong side wins the next immediately price level 95% of the time.

At a macro level, the market will typically stay in a state of impasse between the two adjacent strong sides fighting against each other, with each strong side winning over their corresponding weak side. This fight can take maybe 10 to 20 price levels until one of the two weak sides finally beats a strong side.

So for every move the market makes, the labeling system I came up with was to label the two adjacent strong sides and their corresponding starting volumes. The bet in a nutshell is to determine which strong side would win based on which had more volume to start.

Hope this context helps a little.

Ian

torr 

Hello Iantg, Could you, please clarify about "Bid SS Price", "SS Bid Starting V" in "ES 2-25-19 Summary Statistics" file? Am I got right the values of these fields calculated as: case 1. If Bid is SS, then "Bid SS Price"="Bid Price", "SS Bid Starting V"="Bid Starting Volume" case 2. If Bid is NOT SS, "Bid SS Price" calculated from case 2.1. NextPriceMacroLevel "Bid SS Price", "SS Bid Starting V" case 2.2. PreviousPriceMacroLevel "Bid SS Price", "SS Bid Starting V" For case 2.1, case 2.2 didn't catch on what condition it forks. Or may be I'm totally wrong and found only spurious correlation. If I got all correct, case 2.1. take values from NextPriceMacroLevel, and it confuses me a bit (forward looking??). Thank you.

Went over this thread from morning, help me get clear understanding of somethings and for newbies like me.

1. I understood that using orderflow for scalping is not efficient in its current form or shape that retailers use, is that right?

2. If it is, what way should we improve this scenario? I mean I got the overall fill issue of losers getting filled and winners not getting filled, but lets say I'm a scalper looking to use orderflow as secondary confirmation system, how should I improve this overall? Or is it best to take orderflow out of question for scalpers?

3. Have you had opportunity to test it across different time frames? I mean what about using orderflow to confirm accumulation and distribution on larger time frames technical levels? Does that work?

Sorry if I missed or misunderstood anything, but these were the basic questions nagging me when I read through the info you already provided. Please note that I'm not yet fully trading orderflow due to lack of good data provider in my region, so my understanding is very low, so sorry for any stupid questions.

Hi LastDino,

Thanks for taking an interest. To your questions:

1. I won't say that resting volume analysis has no informational value at all, but I can tell you that the edges that I have presented here are not available to retail traders. I tried testing some entries based on volume signals at the top of the book using Ninja Trader with a VPS with fairly decent location to the exchange and my latency was off by a comical magnitude. This was not related to the proximity hosting or the order execution speed, but rather the lag in the data feed reader. By the time my algo touched the data to start analyzing the top of the book, the real market was already 2-3 price levels somewhere else. So It's pretty much a myth that retail traders can use any informational data from the top of the book. And as you get further and further away from the top of the book, resting volumes tent to be less and less correlated to anything.

2. I am probably not the best person to ask about "order flow". In general this means different things to different people. I don't really use this term much because it's not really specific. I would say that most of the edges I have presented here are based on analyzing the Level 2 resting volume, not transacted volume. For the most part these are not going to be relevant to retail traders. But this is the first and only statistical edge that can be peer tested that I have seen demonstrated on this subject. Virtually no one with real edges will ever share, and by contrast most of what get's shared is speculative nonsense. I shared this because it's mostly futile for me at this point, and a few people might get a kick out of seeing it, but most people wouldn't dig in to see the merit.

3. While I will stop short of saying I don't think accumulation and distribution analysis is very helpful, I don't believe there are any real edges there. And regarding time frames, the data here should be a decent-ish representation of everything. If you look at this structure by macro and micro price level and see the analysis I have done here, you will see just how ridiculous it is to arbitrarily always trade off a chart that updates every 157 ticks, or every 35 seconds. To me it's like intentional deception which I will never understand.

If you are interested in scalping at a retail level I think your best bet is to place a resting stop order several ticks away, and pick an instrument with high volatility. You will have a better shot of an entry that pushes your order 1 to 2 ticks in your direction with a break even or 1 tick positive to start. If you try to use limit orders on a very thick instrument, or something that is heavily arbed, you will only ever get toxic fills that run you several ticks in the wrong direction.


Best of luck

Ian

LastDino 

Went over this thread from morning, help me get clear understanding of somethings and for newbies like me. 1. I understood that using orderflow for scalping is not efficient in its current form or shape that retailers use, is that right? 2. If it is, what way should we improve this scenario? I mean I got the overall fill issue of losers getting filled and winners not getting filled, but lets say I'm a scalper looking to use orderflow as secondary confirmation system, how should I improve this overall? Or is it best to take orderflow out of question for scalpers? 3. Have you had opportunity to test it across different time frames? I mean what about using orderflow to confirm accumulation and distribution on larger time frames technical levels? Does that work? Sorry if I missed or misunderstood anything, but these were the basic questions nagging me when I read through the info you already provided. Please note that I'm not yet fully trading orderflow due to lack of good data provider in my region, so my understanding is very low, so sorry for any stupid questions.

iantg 

Hi LastDino, Thanks for taking an interest. To your questions: 1. I won't say that resting volume analysis has no informational value at all, but I can tell you that the edges that I have presented here are not available to retail traders. I tried testing some entries based on volume signals at the top of the book using Ninja Trader with a VPS with fairly decent location to the exchange and my latency was off by a comical magnitude. This was not related to the proximity hosting or the order execution speed, but rather the lag in the data feed reader. By the time my algo touched the data to start analyzing the top of the book, the real market was already 2-3 price levels somewhere else. So It's pretty much a myth that retail traders can use any informational data from the top of the book. And as you get further and further away from the top of the book, resting volumes tent to be less and less correlated to anything. 2. I am probably not the best person to ask about "order flow". In general this means different things to different people. I don't really use this term much because it's not really specific. I would say that most of the edges I have presented here are based on analyzing the Level 2 resting volume, not transacted volume. For the most part these are not going to be relevant to retail traders. But this is the first and only statistical edge that can be peer tested that I have seen demonstrated on this subject. Virtually no one with real edges will ever share, and by contrast most of what get's shared is speculative nonsense. I shared this because it's mostly futile for me at this point, and a few people might get a kick out of seeing it, but most people wouldn't dig in to see the merit. 3. While I will stop short of saying I don't think accumulation and distribution analysis is very helpful, I don't believe there are any real edges there. And regarding time frames, the data here should be a decent-ish representation of everything. If you look at this structure by macro and micro price level and see the analysis I have done here, you will see just how ridiculous it is to arbitrarily always trade off a chart that updates every 157 ticks, or every 35 seconds. To me it's like intentional deception which I will never understand. If you are interested in scalping at a retail level I think your best bet is to place a resting stop order several ticks away, and pick an instrument with high volatility. You will have a better shot of an entry that pushes your order 1 to 2 ticks in your direction with a break even or 1 tick positive to start. If you try to use limit orders on a very thick instrument, or something that is heavily arbed, you will only ever get toxic fills that run you several ticks in the wrong direction. Best of luck Ian

While I'm no expert, I do promote debating usefulness of tools that retailers use and something like orderflow, which is very investment intensive for retailers (especially when you have to buy it in 3rd world countries) should be most debated. And Level 2 is not even available here easily, let alone me using it to get anything out of it.

Its actually little sad that there aren't many like you. But like you said no one will discuss if they really had edge over others, unless of course its some colorful indicator they want to sell.

This is generally to help improve my understanding of it, so your research here is very important, keep going! I might not get everything in get go, but eventually I'll.

Thank you so much

Thanks for the submission. I am new here and I don't know how to use this excel file to my advantage when it comes to actual trading. Mind explaining how this can be used to scalp?

btw just trying to learn (im 17)

In the simplest form possible, trading is essentially betting which way the market will go. So what is your bet? Why are you putting money on red or black? At some point you have to form a basis for your bet. In sports betting you have statistics on teams as a reference. If the Patriots were playing the Dolphins, you would bet on the Patriots right? They have a 90% win rate right now and the dolphins have a 10% win rate. So when you trade the market you have to have some statistics as well.

A lot of people assume that the market behaves a certain way. If you read trading blogs, books etc, you will often hear buzz words around order flow, buying selling pressure, candle stick patterns, etc. How can you definitively know if any of these things are legit or BS? You need to take raw data and analyze all this stuff.

So this leads to the question of how do you get high quality raw data that you can model and test different things. This is where this data comes in. Test away all your ideas, theories and stuff you find online from various gurus. This raw data will allow you the ability to test everything right inside of excel without needing to waste any of your money trading until you have a proof of concept that at least works statistically on paper.

And to your specific question concerning scalping. This data is a look under the hood of all the forces of the market that will destroy you when you try to scalp. Once you learn the odds that are against you, you will realize how unlikely it is that you can ever make money scalping.

Specifically you can learn a lot concerning the use of limit orders, toxic fills, and understanding the real mechanics of involved to potentially earn a few ticks.


Best of luck!

adolphus 

Thanks for the submission. I am new here and I don't know how to use this excel file to my advantage when it comes to actual trading. Mind explaining how this can be used to scalp? btw just trying to learn (im 17)

Hi Ian!

First of all thank you for sharing the results of your work and answering questions!
I tried to replicate your "base data" (ES 2-25-19 Limit Order Fill Rate Analysis.xlsx) based on provided raw L1 data and found some strange cases.
Let's be more specific - from the sample of L1 data (ES 2-25-19 Raw L1 Data.csv) one can build something like 238 "macro price levels" and 214 of which are fully aligned with what one can find in the "base data" sheet (ES 2-25-19 Limit Order Fill Rate Analysis.xlsx), but for the rest of 24 examples there are some minor divergences. I guess it should be emphasized that while one can build 238 "macro price levels" from the raw data you’ve got almost 310 "macro price levels" in your "base data" sheet (for the same time slice). Let’s see by some examples what I mean.

Example #1
your "base data" sheet got the following figures (in green)


based on the raw data these figures should correspond to the following time slice

…


the differences in the figures ("base data" (left) vs. reproduced (right))


at least have a look at the "Ask Starting Volume" and "Ask Ending Volume" for example. One can see from the raw data that the figures should be of 200 and 192 respectively while you’ve got 199 and 199. That’s strange.

Let’s for example see if there are any differences for the cases preceding and succeeding the case in question (in blue)

My calculations got matched with yours

- - -


What’s interesting is the rows 3359-3360 and 3362-3363 can’t be calculated from the raw data because there’s no such data in there from which it can be calculcated.


Example #2
your "base data" sheet got the following figures (in green)


based on the raw data these figures should correspond to the following time slice


the differences in the figures ("base data" (left) vs. reproduced (right))


here it is clearly seen that for example "Bid Starting Volume" and "Ask Trans Volume" are 189/3 (vs 191/0 in the "base data").

Let’s again compare if there are any differences for cases preceding and succeeding the case in question (in blue)


we have got rows #22173-22218 covering all three cases (preceding, in question and succeeding).


for those preceding and succeeding cases I have got fully matched results with yours

- - -


but again for BBOID=3590,3591 (rows #3595,3596 of the "base data" sheet) one can’t recover that data from the raw data.

So my question is did you use one extraction program to build your "base data" sheet and some other to dump raw data? Or am I missing something?

Thanks!

Alex

From what I understand the raw data file was solely to illustrate the format of the data, I think the starting timestamps were different so it doesn't align exactly with the Summary Statistics