Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
There is a ton of misinformation out there about the usefulness and utility of a lot of trading concepts. A recent poll on here is showing that there is more interest on the topics of order flow / and scalping than all the other trading methods.
What is the best trading strategy and methodology for 2019?
Please vote and discuss. If your method isn't listed in the poll, please add it here in the thread.
Mike
For those interested in this space, understanding the underlying data of the market is a must. Over the last couple of years as a side project I have developed a fairly decent market microstructure model that explains the market in a way that makes sense to me. Most importantly, I can use this base data to test different edges, and ideas with actual statistics.
There are a lot of snake oil salesmen in this business, and particularly within a lot of the newer trading fads. Most people that promote trading ideas do it from a couple of cherry picked charts. It is hard to trust that something would work in the long run when the evidence is limited to a microscopic sample of just a few cherry picked use cases. So what I have done is put together a data model that can allow testing of just about any edge, correlation, or relevant trading method across millions of data points. Case in point this first data set has roughly 20,000 price levels that can be tested to prove out various directional edges.
If you are looking for easy, the holy grail, low hanging fruit, or naive optimism, this maze isn't for you. If you really want to see how deep the rabbit hole really goes, I will show you a few things I have picked up over the years that may either help you, or influence you to quit trading altogether. Fair warning. You will get very discouraged reading this thread as concepts you may have thought you understood come into a different light. That is about all I can promise.
I am not here to sell or promote anything. I am a data scientist / programmer professionally, and just happen to have built a fairly decent model that can test all the myriads of BS that is out there. I just don't want to see people keep banging their heads against the wall. In the scalping / Order flow / Tape / Level 2 data world there are a lot of assumptions people make about how orders might get filled, what the value of posting early might do, how they might be able to participate in order flow dynamics and a whole host of other things. Virtually all of the information that I have seen out there is wrong on these subjects to the point that rather than watch everyone make the same mistakes for the same reasons, I would rather give you all the raw data and let you actually see under the hood before you make these mistakes about the value of the so called edges in these spaces. Again, if you don't want to see under the hood, turn around now, because the reality of this may not be what you want to hear.
For starters I am loading up yesterday's ES 2-16-19 03-19 data.
File 1: is the raw level 1 feed for around 5 to 10 minutes. I loaded this just to illustrate what the raw feed looks like and how the raw data reconciles back to the summary information.
File 2: is the summary level microstructure that I use. This is a decent no frills model that just collects and summarizes some of the more important data elements that will allow for testing.
The data is sequenced by price level. Here are the primary relevant fields.
Notes on a few of these. I am using an extraction program I wrote in C# with NinjaTrader and running this on market replay which was downloaded via Kinetic. The data feed is aggregated, so to get a lot of this information I had to un-bundle it.
Added / Subtracted Volume: Every volume update will have either a positive or negative delta from the prior update. I just run these as two separate buckets throughout the life of the price level. The transaction feed is separate, and at the end I take the total subtracted - the total transacted to get the total canceled.
The Level 2 feed is not perfectly synced with the Level 1 feed. But for every change in the level 1 price level I am also running a snap shot on the level 2 feed at that time. Sync issues will likely make this off every now and then a little, but I think it is around 90% to 95% in sync.
I don't think about price level sweeps, or spread widening due to lag as a real change to the spread. I always maintain the spread at 1 tick on every level in my model. Often times you will see that one side moves up or down, which the other side takes a while to catch up. This is only due to the fact that they only report changes to price when there is a corresponding to change to volume. Naturally the two sides don't move 100% at the same time on each level change. So I smooth this out with my extraction program.
Overall I will say this is directionally accurate, and good enough to kick the tires on most of the edges out there in the retail space.
I will cover some basic definitions and concepts next, and later get into showing you how to test different edges with this data.
I invite discussion on this thread, and will try to be as forthcoming as possible and share as much as possible without giving away any of own strategy. But the more common edges, that are more obvious and less useful to me are fair game and I will demonstrate how several of these work.
Ian
In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
In the analysis spreadsheet, there are two cumulative volume types (SS and WS) one each for ask and the bid been tracked. care to explain how/when they get reset? are they used to determine the immediate bias/trend?
Backstory: There is so much misinformation out there regarding volume / order related trading methods. I have heard these described as volume profile, order imbalance, DOM, Level 2, Order Flow, and countless others. The general idea being promoted is that users can analyze the resting volume on the bid against the resting volume on the ask and find various edges.
I don't believe most of the people that are promoting these ideas fully understand them, or even understand them at all. For starters any trading method trying to exploit short term BBO dynamics at the top of the book wouldn't be available to retail traders. Most price levels clear in < than 100 ms. Also most price levels clear in such a way that there is no real bet that could be made to begin with.
Before I cover the analysis, I will need to cover a few basic definitions. This will be like drinking from a fire hose a little bit, but I created a fairly comprehensive taxonomy within this structure. Here are the key pieces.
1. Strong side vs. Weak side: Every price level can be characterized as a fight between the strong side and weak side. The strong side is the side that gets to drag in resting volume from level 2, and the weak side always starts with 0 volume because it won the prior level. Statistically speaking, the strong side will prevail and win around 90% to 95% of the time.
2. Strong side vs. Adjacent Strong side: Given the previous point, what we are really trying to measure is not the action between the strong side and weak side. We are trying to measure the action between the two adjacent strong sides. It is here that we are making a bet and attempting to place an order. One of these two strong sides will ultimately break. There are clues in the data that can be used to place a bet as to which side will hold up and which will break. I have never seen anyone on this forum or any other place explain this point, but it is very important.
3. Strong side types: Accumulated vs. Resting: The two adjacent strong sides can be thought of as follows:
A: Accumulated: This strong side was a weak side from the previous level and recently overtook the prior levels strong side. In this process it built up volume, and now it is one of the two strong sides in the fight on the next price level. Being that this side had to accumulate volume from the prior level, it is at a disadvantage because it had a shorter time period to gain volume. This strong side typically starts the fight with less volume.
B: Resting: The side that starts the fight with resting volume from level 2 will typically have 10% to 25% more volume than the side that had to accumulative from the prior price level. This volume will typically be waiting for a longer period of time as it has had a chance to develop over many price levels.
Dynamics of the battle: The resting side will be subject to multiple price level sweeps. Because once one level is cleared, the next level will be also resting. The accumulated side has the same risk, but once two or more levels consecutively have been cleared we classify each new level as resting. So from a classification perspective only the resting levels experience sweeps.
In order to measure these events correctly, you need to classify each side as either accumulated or resting and then using different thresholds you will see different results.
To perform this analysis, I used the base data I previously posted and created a pivot table that is easy to trace back. I left in the formulas so everyone can see what is being measured, why and how.
The results present a real edge, albeit not enough alone to make it to the finish line. But this is the first demonstration of how this edge really works. I have seen countless attempts to discuss this from a whole host of pseudo-science angles, using incorrect ideas, mixing make believe Fibonacci or random numbers with their presentation of ideas. I have also seen people completely get these concepts wrong.
Anyway, the analysis speaks for itself. This is how it works
Ian
In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
"The strong side is the side that gets to drag in resting volume from level 2, and the weak side always starts with 0 volume because it won the prior level." - Could you elaborate more on this? I don't know exactly what resting volume means.
"There are clues in the data that can be used to place a bet as to which side will hold up and which will break. I have never seen anyone on this forum or any other place explain this point, but it is very important." - Do you think that these clues can be comprehended in time by a discretionary trader?
The way the market moves and clears levels can be visualized like this:
Buy limit orders can only be placed on or south of the current best bid price. In the enclosed image, the buy orders on the DOM (Depth of market) below 2740.00 are currently outside of the transaction level. So they are "resting" waiting for the market to move towards them.
Sell limit orders can only be placed on or north of the current best ask price which here is at 2740.25. You can see all of the resting orders waiting above this.
So what happens is this: When a price level clears and the bid side breaks. (All of the 35 orders at 2740.00 get filled or cancel), there will be nothing left to hold up this price level so it will break and the market will move down 1 tick to the new price level.
This new price level will start with 2 prices with two corresponding resting volumes.
1. The new bid price in this example will be 2739.75 and it will start the price level with 318 contracts (assuming this number doesn't change from the screenshot).
2. By Contrast, the winning side (The ask in this case), will not be moving towards it's strong side where it has a queue of resting orders on the DOM. So it will start the new price level with literally 0 resting orders. It will have to spontaneously back fill resting volume very quickly to stand a chance in the fight over the next price level.
But as you can imagine, a fight with all other things being even between one side with 318 contracts and one side with 0 contracts is pretty much a done deal 9 times out of 10. So what happens is this: The market will tick down to this level, a few limit orders will pop into the new ask level and prop it up for a bit, a few transactions might hit both sides, but ultimately the ask side will break and the market will tick back up.
When the market ticks back up, the same thing will occur that direction but in reverse. The Ask side will start with heavy resting volume, but the bid side will have 0 volume.
From here what you will see occur in almost a split second is a fight between these two adjacent strong sides as they keep beating and flipping their opposing weak sides over and over. With the ES during the US cash session this usually takes 10 price levels (They will flip back and forth with the strong siding always winning). Until eventually one of the two strong sides will break. And the market then moves to the new next inflection point.
So that is kind of how this thing works under the hood.
To your question of is this an edge for retail traders. I am inclined to generically answer no, but I will give you a little more of a detailed answer.
I think that if you are trading something much thicker, like Treasuries, and you are trading a slow afternoon, or if you were trading the ES on the overnight session you might be able to see this dynamic and potentially react to it with an Algo on a VPS somewhere. But I don't think that pointing and clicking a mouse over the internet on a retail platform that has a baked in 200 MS lag in their application would allow you to participate in this in any way.
To give you a better idea: Look at the time stamps in the Overview data I provided and measure the delta for each macro price level. With a retail platform like NinjaTrader they bake in around 200 MS lag in what you are seeing on the screen. Add to this, your internet delay, your broker delay, and the round trip to the exchange and you are likely up to 1 to 2 seconds before you have even clicked your mouse or started to analyze what to even do.
So the overnight session might be your only shot if you were trying to do something manually.
Hope this explanation helps.
Ian
In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
Fair Warning: This is the part where the Red Pill comes in. If you are pursing a scalping, Order flow related strategy, aiming for 1 to 2 ticks, modeling a strategy that heavily relies on limit orders, or have a backtesting system showing 1 Million in profit but assumes a 100% limit order fill rate, or uses an exotic bar type that a backtester can't even model so it fills every order perfectly, this is where you have to decide.... Do you want to continue down the rabbit hole, or stay content with the hope that there is a brighter tomorrow. Blue pill: Stop reading. Red Pill: Take a look, but it won't be pretty.
Before I put out the next large data set and accompanying analysis, I want to provide a brief overview of this topic witch I will cover in depth in the next post. One of the key pieces of the puzzle in trading and modeling / building a strategy is determining how well one can participate in non toxic limit orders. If you are pursuing any sort of a scalping / order flow based system that aims for just a few ticks then your primary concern and ultimately a large part of the success of your strategy will be based on your non toxic limit order fill rate.
A toxic fill is when your order is filled only as a result of the entire price level breaking. All participants left (that did not cancel out of the way) were filled and the price moved 1 tick against this position. So you will start your trade down 1 tick at least. Often times a multiple price level sweep will occur and your level will get taken out as well as 1 -2 levels beyond your level. In higher volatility periods, or in thinner instruments this will happen very often. As a data point, on the ES with just medium volatility multiple price level sweeps occur around 20% to 25%. And my definition of a sweep is that the bid / ask level clears without a single flip to the opposite side. So the last two rows in this example are what I consider a sweep.
BID ASK
200.00 x 200.25
199.75 x 200.00
200.00 x 200.25 199.75 x 200.00
199.50 x 199.75
A non toxic fill by contrast is what everyone is hoping to get. This is where you both 1. get filled and 2. get 1 free tick to start the trade because it is the opposite side that breaks first.
So the big question is: What is a realistic non toxic fill rate? This is an unknown for most traders, because retail trading backtesters do not model limit order fills well at all. Looking at any scalping strategy under the hood, you have the following 5 problems that you have to solve for to be successful. Here is a break down of the largest obstacles you will face along with some high level probabilities.
1. When you are only getting a few ticks profit (Often 1,2, or 3 ticks only), then it is difficult to make any strategy profitable when you cross the spread and pay 1 tick to enter, cross the spread and pay 1 tick to exit, and pay full retail commissions + full exchange fees. Due to crossing the spread to enter and exit, you will need to get 5 or 6 ticks in your direction just to net your 3 to 4 ticks. By contrast to get to an equal losing position you only need to travel 3 to 4 ticks. So just based off the exchange mechanics alone you have far less than 50 / 50 odds.
2. Assuming that you have 0 directional edge, your mathematical best case is 50/ 50 that you can pick the direction of the next few ticks. But the reality is way worse than this when you understand how adverse selection works relating to limit orders. Start with the standard probability of:
A: P(Long) + P(Short) = 1. So I pick Long, now what do I have? P(Win) + P(Lose) = 1 (Mathematically speaking). So far so good. 50/ 50 odds right...
B: So now you place your limit order, and watch it touch your price 3,5,10 times and tick back and forth and you never got filled, but the market traded the exact direction that you thought it would. So technically you were right in picking direction, yet you never got filled and thus didn't get to participate in the trade. So really what you have to solve for is this:
*** If you look carefully at the formula above you will notice something out of balance and it isn't a typo. You only really have to content with not getting filled on winners. You will always get filled on losers. So filling in the unknown variables with this point we can break down your entire scalping / Order Flow strategy to this:
((P(Win) * P(Get Filled)) + ((P(Win) *(P(Don't get filled) + ((P(Lose) *P(100% You will always get filled on losers)) = 1
For the first two, you obviously hope that your participation rate in winners is high. But what is realistic and how high is high enough to overcome high commissions / exchange fees and everything else? If you had a very high 75% participation rate with winners you would have:
100 Trades * 50 / 50 odds of predicting direction (1 tick profit or loss) = 50 * 75% participation rate in winners = 37.5 winning trades
And opposite this you would have 50 losing trades. Now pay your commissions on all the losers and all the winners alike and you have:
For the ES
Winners: 37.5 trades @ $12.5 - retail commissions of $4.00 = Gross Profit of $8.50 per trade. Total = $318.75
Losers: 50 Trades @12.5 + retail commissions of $4.00 = Gross Loss of $16.50 per trade. Total = $825
Total Net Loss: -$8 dollars per trade. or for every 100 trades you will lose $506.25.
3. It gets worse.... Because momentum moves against you using limit order entries. You will get swept into 2-3 tick losses immediately 20% to 25% of the time. By contrast you won't get an equivalent bump the opposite way for winners. This is mostly due to the fact that you will never be in the front of the queue when the market ticks to your price level, touches it, then moves the opposite way sweeping multiple levels. If the market knocks out 75% - 90% of the queue (What is going to be required to fill your order in most cases), it will rarely have the momentum to go back the opposite direction and knock out multiple levels. So the quick math on this point is:
P(Positive Sweep that you participate in) = 1% to 5% at most
P(Negative Sweep that you participate in) = 20% to 25% for medium volatility. *** This gets higher for higher volatility and lower for volatility.
4. It gets way worse: The actual non toxic fill rate isn't 75%. It is incredibly lower. I will post all the details of the fill rates of 20,000 price levels so you can see what they actually are. But I will leave you with this: Roughly 50% of the time, the price level that wins has 0 transactions (This is during the level 1 fight between the best bid / best ask at the time we are measuring the price level). So filling in a little more information into our probability decision tree we have:
A: Can I pick the right side? With 50/50 odds ( I am actually being generous here, most people aren't right even 50% of the time) P(yes) + P(No) = 1
B: Participation rates: (P(Win) * P(Get Filled =50%)) + ((P(Win) *(P(Don't get filled 50%) + ((P(Lose) *P(100%) = 1
C: So with A + B, the house edge against a retail trader trying to use limit orders to scalp 1 tick profit with a one tick stop loss can be deconstructed to: 25% winners, 50% losers, and 25% picked the right side but didn't get filled. And this assumes queue optimization, good infrastructure, speed and a few other edges to get you the 25% non toxic fill rate.... Which by the way is highly unlikely. Adding to the house edge against you is the fact that every winner is worth 1 tick - retail commissions and every every loser is worth 1 tick + retail commissions. So for this ES this is typically 8.50 for winners and 16.5 for losers.
5. It gets way, way worse. On the last point in 4C above I lightly point out that a 25% nontoxic fill rate would be best case if you start with 50/50 odds, and eat every loser, and catch every winner possible (The catch being 50% of the time the winning side fills 0 of the queue). This assumption is completely unrealistic. The people that do get to participate in the non toxic fills are the the professionals who optimize for this. A retail traders non toxic fill rate will be way, way lower. I will show you just how low.
Detailed Analysis with the actual fill rates (Non Toxic + Toxic) to be posted in the next week.
Ian
In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
Could you elaborate on what "directional edge" means?
I don't plan on scalping the ES. I would rather hold my trades and have a 35%-40% win rate over the course of a year/ ~1:3 R:R or more for every trade. Would this kind of approach revolve around having a "directional edge"?
