Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
Unfiltered data feeds and HFT, detecting algos with orderflow?
I recently attended a webinar for Shadow Traders. The CIO, Barbara, says that she is primarily a programmer, not from a trading background. And since she has coded for some trading houses has inside knowledge of HFT structure and behavior. Also that trading is no longer a technical analysis game, but a programming game. I can attest that writing strategies using standard TA tools (like stochastics and macd) are very hard to make profitable. It seems like more time is spent trying to stay out of bad trades than getting into good trades.
I'm also a programmer and know where she is coming from. I'm not a Wall Street coder, but I am doing a lot of study about it. Goldman makes $50M a day front running stock quotes. The Quants are PhD physicists, mathematicians, statisticians, etc. They are coding up some nasty looking math equations. But their algorithms work best when the market is acting nice. As soon as things get wonky, their models don't apply any more.
I'd like to hear from any experienced coders from Wall Street. Is it true that the HFT robots can be detected by watching the order flow database? I googled "order flow database" and only got back a single page of hits, most of the references being on people's resumes. Can Shadow Traders really detect when the HFT engines start trading, or is increased volume activity just the outside behavior and they are attributing it to HFT.
I know that an unfiltered data feed is better overall, but is it really a make-or-break scenario? Tradestation has a lot of customers trading every day. Can they really be all losers because they aren't getting rocket speed unfiltered data?
Is watching the order flow database in this fashion only good for scalping? If I want to catch 60 ticks and stop for the day, do I really care if the HFT's are running the market around. I'm looking for setups and building stop movements around the bigger move.
I don't have enough inside knowledge and experience to validate what Barbara is saying. Does anybody here?
Since Shadow Traders have been mentioned in at least three different threads this past week, I caved in and did a search on Youtube:
I am amazed that she sells products; she seems completely clueless. These fraudsters piss me off! Apparently @tigertrader knows who she is, and he did not seem impressed either.
If you have a decent feed, you can get access to cancels. I would guess that she simply looks at increased activity (number of ticks p/s, e.g.), maybe combined with analysis of the fluctuating order depth (doubtful).
As retailers we can just forget about competing on speed. Take a step back and try to catch some of the bigger moves...
This thread will NOT be about Shadow Traders. Use existing threads. All posts made about Shadow Traders will be deleted from this thread. This is not a thread to discuss a vendor.
I don't want to talk about them. I want to know about the order flow database.
How do you look at it from a programmers view inside of ninjatrader? Is there any sample code for looking at transactions, volume, etc that might come close to this?
In MetaStock I created a chart that plotted each tick as a dot and price action looked like a wide spray paint swatch that moved up and down. So that was my first inkling to how transactions were being recorded. But I wasn't programming the platform back then.
I've thought about the bid/ask model and how I might dig into it from ninja. Is there any clues in there that might give an edge?
I don't really know what she means with "order flow database". I assume she is referring to Level III. To be quite honest, I am not really sure how much information certain firms gets from the exchanges. Maybe someone else can chime in on that.
I started looking into the possibilities of pursuing something like this, but I quickly discovered that it was hopeless. Just the cost of an in-memory database like kdb+ is about $200k/year. You also have one-stop shops like OneTick and Deltix. The latter looks really interesting for my purposes, but it costs around $6-8000/mo, excluding data. Add about the same amount for a good data feed and it adds up quickly.
I'm not really a programmer, so I can't really say what people can do with NT or not. But last time I checked, it didn't have CEP capabilities!
I must warn that I am really out of my depth here, and I am simply offering a layman's perspective. However, I did spend a considerable amount of time last year researching various options.
Keep in mind that the feeds we retailers use are crap for such purposes. There are missed ticks and services like IQFeed only timestamps to the second. NxCore does sub-second timestamps and is priced reasonably. I'm not really aware of any other comparable solution in that price range.
As for my personal experience, there used to be a lot of information in the DOM, but it has changed drastically with the advent of HFT. Quote stuffing ruined a lot of my analysis. I export the DOM (along with other data) to Excel, and there were a lot more useful things one could gain from that several years ago. Things have gotten more "primitive" now, but I still make money. Now I am reliant on catching the bigger swings, though.
Portraying ticks as dots, representing the size of the order, is a very intuitive way of looking at price action. @Vegas has made a product that does just that.
Are you a programmer by trade? I became increasingly frustrated with the limitations of all the platforms I've used, I finally decided to take some CS classes this year. It seems that nearly all of my ideas require custom programming not supported by retail platforms, and I am tired of doing things semi-manually in Excel.
i looked at shadow trader before, about 4 years ago. i can not believe they still around. for rookies...anyone know a moving over cross over with fancy background coding
yes, I am a programmer by trade. Even though I've been a software engineer for 30 years, I've been out of the loop as a full time programmer. So there are a lot of technologies that I don't know. I've only been coding on ninja for a short time. But I like C# as a language and it has career enhancing properties.
From my research, I know that most of the HFT programs are running as C++ under Linux on big PC-based hardware hanging off the exchange. This is the fastest it can get.
The London Exchange tried going to a Microsoft infrastructure and it died. They had to scrap it and go to a Linux based structure.
So writing C#/.Net/Windows over the open internet is not a good formula for fast transations. But that's not what we are doing anyway.
I'm also looking into what protocols the HFT's are using, like FIX. This is supported by the CME and many vendors use it. But I'm thinking that to go that route, you have to write a custom C++ app for talking to the exchange directly and not using any visual charts, etc.
There is an open source java-based framework that does a similar thing. No graphics, just algorithms and order management.
Right now I'd just like to experiment with what ninja can do at the bid/ask level and see how far that can go. One interesting example is the indicator that plots the speed of order flow. I can't remember what it's called, but it is freeware. When orders pick up speed, he displays the bars in a different color and I think there is an audio alert.
I would like to do something like measure the orders based on whether they are at the bid or at the ask. This should give indication as to where the transaction flow might be headed.
