Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
Well I have NT8 here and I'm looking at your code on it. I would need a really really compelling reason to learn yet another framework that does less than my own stuff. Maybe such reason will emerge.
I really prefer my stuff. It does not do nice charting and it looks archaic. But it's (very) fast and it does things retail software can't. I have a huge saved history of ticks and dom (admittedly not the rhythmic dom) and other data and I can run backtests against at speeds NT can only dream of. I can model things off this and simulate how they perform.
Not bragging here. Just saying. We're in a world where we need to train our stuff on data before releasing it in the wild.
I'd mock up a "feature" based on an idea (it's called indicator here, but it's the same). I then chart it on multiple days with multiple parameters and see how it looks in a matter of hours, not days or weeks.
If I like what I see I then I'll delve deeper. Add it to an algo, mix it with other features to assess it's predicting power.
All this is unexciting, hard work. But ideas as not easy to come by. Many features are overlaps or combination of other features, so they lack predicting power. So much was tested, tried and arbitraged away.
Sorry, I'm digressing.
I'm wondering why they trade differently. I'm no expert in NQ (I trade mostly DAX).
In DAX you have the big, liquid FDAX (1 point tick, 25euro/tick and ATR between 100 and 1000)
and the two minors (FDXM x5 and FDXS x1). You can clearly see that FDAX and FDXM trade in a similar way while FDXS will follow, sometime just quote updates and no trades.
Is it the same on NQ?
Well it has all to do with arbitraging. Give me half a tick profit across contracts on each lot I trade and I'll make you a billionaire. All the HFT frenzy started this way. Hell they even created a custom order type for folks in the know to profit from this. But I think you're well aware
Would you expand on this concept ?
Not sure what you mean by "full matching": Do you really need to fully match orders ?
I.e. do you keep each order in a bucket then remove from the bucket once an equivalent order from the other side has been executed?
Isn't it the same to just accumulate (i.e. integrate in your lingo) the totals matched on one side vs the other ?
Here's what I'm talking about with an analogous toy example on SMA:
The SMA can be calculated either by keeping all values in a buffer and iterating each time a new value is inserrted or by a simple formula:
NewSMA = OldSMA + (NewValue - OldSMA)/Period
And some food for thoughts on the resilience of MM towards adverse selection:
My experience is with the DAX mostly (I have thousands of hours of screen time).
The MM is wary of adverse selection, but "she" has to play the game. And MM plays it on a very long timeframe: the timeframe of future contract expiration, with unlimited resources.
"She" has the same privilege as we do: Adjust her quotes and chose how much to interact.
And she has one huge advantage: She knows where your positions are. How does she know? Because those positions were dealt by her directly. Think about it as the card dealer at a blackjack table. Except the dealer can see your the cards.
The loading: On the DAX there's the "loading longs" phases. Here the bid drops on very light volume. MM does not want to buy and drops the bid. "She'll" sip a few shares then adjust lower. Then sip some more and adjust lower. At some point some stops are hit. This is where part of the "biglot" volume comes from. Those are stupid retail placing their stops in a cluster. And you get the breakout crowd joining too.
I've tried running models to simulate the MM behavior. It's still a work in progress for me.
The loading process can be brutal. Multiple levels can be taken out. It all depends how much the MM needs. On the DAX you'll usually get 2-3 phases during the day before a reversal occurs But you never know in advance.
And I mentioned levels. Those levels are there for a reason. Each level corresponds roughly to a number of positions dealt by the MM. "She" know how much, we don't. We also don't know how much she needs. Maybe 100'000 shares? Maybe 200'000 ? How much she's open on options? How much stock from the index she has?
So how do you play this?
I'd use statistics. Try to estimate the number of shares MM is scooping. The max, the mean etc. Then act on this knowledge. It won't help with shorter timeframes (seconds) but should work on intraday basis.
I'm not sure if the minimum size filter helps in this scenario or not. Perhaps all trades matter?
And, one last thought: wavelet analysis on the accumulated shares with retention over longer periods. Perhaps this can expose the behavior of MM? Then you choose your wavelet order based on your preferred timeframe and play on it's wave length?
Can you help answer these questions from other members on NexusFi?
@Connor : there's a lot in what you're saying, so I'll formulate a reply
later on today; the problems we're trying to understand are complex,
and I think that level of ambition transcends code that could be presented
usefully in a forum like this; but very thoughtful stuff, and i think whenever
something "goes to the heart" of the nature of trading, then it's
likely to be on the right track
I work with NQ, ... because I enjoy being punished, maybe? ... but, LOL
and seriously; because it's the highest challenge; so if I can go some
way to "taming it" then I think I"m on the right track. But the stuff which
specifially involves "Inventory Analysis" should theoretically apply to just
about any similar symbol's trading patterns.
Currently, I'm studying the relationship between 2 key factors that I
think capture the "essence" of trading: 1) short term Accumulation
and Distribution inventory patterns; and 2) the Size Bias on the inside Market Depth of Market. This fast "inside Bias" is, first of all, difficult to capture;
but, then the predictive relationships need some analysis. And, finally,
if they are going to be used as "retail trade predictors" then Triggering
conditions need to be established. The last of these is the easy part;
and so much R&D goes into conceiving what should be measured, how,
and finding some implementation that works....
Because this is about Time and Sales reading, it will
work across most similar symbols.
I suggest, RAISE the Risk_Threshold so that you see the numerical
Risk levels ONLY infrequently. Those will be good turning points.
[EDIT] YOU MUST USE UPPERCASE FOR THE KEYS, SO MY
lowercase mention ABOVE WON'T WORK; IT MUST BE RISK_THRESHOLD,
and that should not be changed, as a key, on the left side of
the equals sign.... Just sayin'.... sometimes SHOUTING IS GOOD ? LOL
Also your retention interval is 300 sec = 5 min, and you may want
to make that longer. Filtering BigLots under 14 is a bit harsh;
so could reconsider that.
[edit2] SORRY, I misunderstood. BigLot 14 is fine and I see your
minimum volume filter is 3; which is also fine Reasonable choices...
Taper is optional also.
But, that's a good first shot; and the more you use the
facility, the better you will get a "feel" for it.
I use notepad++ and keep the parameters file permanently
in a Tab. Then altering values, and CNTL-S to SAVE, will
effect the changes. If you wonder about an issue, create
a NinjaScript Output window, and select Tab 2, where any
Exceptions should be thrown.
I'm trying a few different settings. FDAX trades relatively infrequently and bigger lots are common. It's not uncommon to see 20 - 50 - 100 lots flying around.
Unfortunately today is not a good day for this type of analysis. We're in a pre-expiration week and we're very far below the monthly and quarterly vwap. MM needs to show profits for the quarter, month and year end and there's lots of ground to cover. Long sequence of buys all day long and this is not retail buying. It's just marking up the price with no trading. The snapshot I posted before was taken in a relatively quiet phase after the first push this morning, around noon and pre-US open.
See below adjusted settings and the corresponding volume profile. Almost no trades and the price is moving in one direction only. You don't want to short this market (yet).
@Connor yes, when you get a constant Rally bias,
it turns out there's only 1 way to measure that; and
it involves using the Market Depth, taking into account
Market Maker's placements away from the current
Market price.
The way this works, is that there is a "Wave of Push Volume"
which appears against the Market, pushing it upwards.
An attempt to measure that *could be attempted* using Level 1 data, just Bid and Ask size by themselves...
However, "reaching outwards" a few Tiers on the Market
Depth is the way to approach it.
Unfortunately, that's something I'm working on at this
very instant !! and it's very difficult to measure; but it is
generally possible to determine a persistent trend like
that, by constant real time measurement...
[edit] I attached a pic of the Nasdaq Recovery Rally today;
and you can see the Retail peeps were sellers, until they finally
realized the Rally wasn't gonna stop ! lol More properly
this should be interpreted as some Big Lot Buyers trying
to jump on board the continued Rally.
[edit nuther pic] Attached also shows the Rally "rolling over"
and how Retail players respond to the down pricing. 15 minutes
Retention Interval here...
Thanks for the insights. It's amazing how the DAX first half of the day was a precursor to the NQ first part. They're almost 100% identical: Push - consolidate - push and No retracement allowed. All shorts trapped and no escape.
Regarding the DOM measurements, my theory is that the information contained in the DOM decreases exponentially with the distance from the inside mkt.
It's way too easy to stuff the levels with fake orders and make them disappear as mkt approaches. There was a retail guy I read recently about, who was
convicted for exactly the same stuff. And... he was retail!
Even the inside bid can be manipulated (iceberg and co.).
I think I understand your "push wave" idea. Does the wave stay outside or does it "touch" the inside quote?
But most interestingly, do you trace what's going on, via the dom ? It it was me, I'd probably measure the rate of arrival/cancels
of orders at a certain distance from the inside, over time. These are several data points.
Then add to the mix the number of shares traded vs the real posted volume (i.e. plain old iceberg order detection), at each price level.
Look at this trough a window and I guess there should be something informative in it.
Don't be like me !!! Don't "over think" this stuff and start with
some complex thing you'll only have to back away from...
Well, generally, yes; the "distance in tiers" from the inside market
does reflect a decreasing relevance; and anti-spoofing is definitely
a big consideration.
We're straying here from the Tape to the Book; but here are some ways
of thinking about aspects of the problem.
SPOOFING. Well, most traders "eyeball" the DOM and hope to get
clues. Fine. I argue often, that if they do find "clues" that they are
unsure how to interpret them. But one thing is for sure: There are
momentary changes to the DOM which are invisible to the naked
eye; and best read by a computer.
So, my approach to "anti-spoofing", in an attempt to "estimate" the
"true" size, is to Sample a Window of time; and then take the
Minimum value seen in that interval. Why? Because if they really
want to plant size; then they won't pull it away. And if they pull
it, even momentarily; it is an indication they have a reduced desire
to transact with the Retail population; when the Market finally reaches
that Price level.
So your primary anti-spoofing tool is code which evaluates size over
a fairly short period of time; and has the ability to "getMin", "getMax",
"getAvg", "getAvgOf2LowestSizes" and concepts like that !!!
But far more significantly is the question of HOW you should Capture
the Size on the Depth of Market ??? The most obvious approach
(spoiler alert: turns out to be wrong) is to Capture data at a Tier
distance (in Ticks) from the Inside Market. So we have an Analyzer
allocated to "watch" BID - 4 ticks, for example. Unfortunately, the
Price which that relative placement represents, is constantly changing,
as the Market Bid price moves up and down.
BETTER approach: Allocate Analyzers for EACH Price (Price-specific Analyzers)
by having a Dictionary keyed by Price converted to Integer or some unique
key for EACH possible Price. When size or volume comes in at THAT PRICE,
then that Analyzer captures it in its "moving window".
BUT THEN WHEN YOU WANT TO EVALUATE the Bias, you then interrogate
the Price-Specific Analyzers, but you take your Snapshot using a Tier offset
from the Inside Market, converted to a Price, to fetch the Analyzer.
So you 1) Capture size with Price-Specific analyzers, but then 2) you
interrogate them to find a "bias" Bid side versus Ask side by using Tier
offsets from the Inside Market.
When evaluating a Range of Tiers, I'd suggest REJECTING any Huge sizes,
and possibly even constraining a Volume relative to its adjacent neighboring
sizes in some linear interpolation approach, to take that "noise" out of
the system.
That only scratches the surface of what has to happen... I have a long history
of messing around with these concepts, by the way.
Since we're talking about the DOM, you could use the LeeLoo service to
gain access to the Rithmic DOM and Market feed; and you can check what they offer; no
commercial advertisement intended.
[EDIT] IF you do intend to dip your toes into the pool of DOM Analysis, then
you'll have to handle "burst" data rates in excess of 1000 events per second,
possibly. That's certainly true for Nasdaq futures. First of all anything >50
tiers out, maybe; you'll reject immediately; but then in order to handle the rates, you'll
CAPTURE each OnMarketDepth event's key callback info, and Queue it up; so that you
can return immediately for the next one. You've only got a millisecond or so...
That way, nothing gets missed; cuz you are Capturing and returning.
Then your Queue will be processing as fast as it can, with a millisecond or so
latency from the Capture time; and if it finds itself unable to keep up; can then
selectively discard less recent data, at those few moments when Congestion
becomes an issue. If you want to know about that stuff, I do it all the time,
and I find my latencies are <1 msec most of the time as the Queue shoves
the captured information into the Analytics. LOL that'll scare anybody away !!
Just tossing out some slightly off-topic ideas there.
No worries. But I'd like to have a general idea of what
I'm trying to achieve. If I'm not focused on a specific task, time
flies and I feel unproductive.
I follow you. This deals with time-spoofing. You might want to augment
this data with a small "series" of the state you're looking at: I.e. the
same as the getAvgOf2LowestSizes idea but more generic.
. Agree. So much scientific literature ignores this simple fact, yet
practitioners are well aware.
You'd run a "horizontal" aggregator on each level. You can include some
non-linearity here (like a log or tan-based function). So you don't just throw data away
but allow it to participate, but not skew your distribution.
T hen you'd combine the result "vertically" with some other "aggregator".
At the end you're looking at a single number with hopefully some meaning in it.
Thanks for the pointer. I'm trying to decide if to open a new battle-front. The DAX
is plenty atm but I'm quick to jump-start a new venture and fund myself, if the
"business plan" works.
Yay, no scare here. I started as a professional programmer and I was
writing code in MASM when C was considered too high-level and too slow
for the job. I'm actually surprised you can do your trading in NT, considering the
amount of layers and "junk" between the data and your code. But I guess you
don't and you only use NT for visualizing.
Anyway, yes, I've been there and I hear you. A thread for each feed, a thread
for managing data and duplicating it to each consumer, which runs in its own thread
because mutexes are slow and memory is cheap. Then shuffling stuff so it fits the
same cache line and fighting with the branch predictor... all the pains and joys
we're both too aware of.
I actually like that. It's not easy to talk at certain level very often.
So thank you from my heart.
