Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
Automated Strategy - Processing Each Tick Considerations
Apologies about the length of this post - this is due to including context and assumptions...
I am trying to design an automated strategy so that it will not start to lag in busy markets (when the ticks are coming in very fast). I am trying to understand/map out the logical sequence of events (and related considerations/limitations) that happen in MultiCharts to a strategy during a busy market (e.g. ES at news time). I have looked into the documentation, but I still can’t get it straight in my head how best to design the strategy to cope with fast moving markets.
I would be grateful if anyone can contribute to my understanding of the best methods to mitigate lag in the strategy. If the information is available in the documentation, or has been addressed previously, then I apologise, but would appreciate a pointer to the appropriate location.
Requirements, Assumptions and Context
Assume that the Strategy must be executed on each tick (e.g. it’s a short term strategy).
Assume that there are a few indicators that need to be updated and also some calculations to do on level II data.
Assume that there is no lag in the datafeed, other than the natural lag/jitter from the Exchange to the computer location through the internet (due to distance considerations).
Assume that the computer has sufficient CPU cores, memory and disk speed for the majority of calculations - we are concentrating on what happens when the market gets busy and the ticks are starting to come in too fast to allow the relevant calculations to be carried out for each tick.
Conceptually, each time a new tick comes in and the strategy performs a typical number of calculations. A simple assumption is that each of these strategy loops takes a certain (average) amount of time to perform. That means that there is a certain amount of ticks that can be calculated each second, and if the ticks start to come in faster that that, then something has to give - which one of the scenarios below happens, or does something else happen?
Scenario 1: The application diligently calls the strategy for each tick that comes in, and the strategy diligently performs the same (average) set of calculations, so the strategy starts to lag. Any orders submitted by the strategy while in a state of lag will likely not be applicable.
Scenario 2: The application diligently calls the strategy for each tick that comes in, but the strategy is designed for this, and performs some sort of quick test on each tick to check how long it’s been since the last tick (or something similar) - if it’s it too short a time, then the strategy performs a subset of calculations - thus (hopefully) not falling behind.
Scenario 3: The application knows if the strategy is falling behind and skips data, or signals somewhere (where the strategy can check) that there is a processing lag.
I'm not looking for code or anything, just the logical progression or options available so I can design appropriately
Regards,
Sc1entia
Can you help answer these questions from other members on NexusFi?
I have been pushing Ninjatrader to it's absolute limits pursuing some rather speed sensitive strategies.
So I know what's under the hood over there quite well, as I play on the bid / ask price levels 1 by 1 in my style of trading.
I think I can help you out, field your general questions, and tell you where the real boggy men are at in terms of your largest latency barriers.
To your general question about the data structure: There are three data feeds that NinjaTrader uses.
1. OnBarUpdate: Runs off of the defined data series (Bar / Size)
2. OnMarketData: Updates every level 1 event both transacted volume, bid ask price, and bid ask resting volume
3. OnMarketDepth: Updates every level 2 event including the best bid / ask as well as all levels of the DOM.
To your specific question about which feed to use to see "every tick". This is where I will disappoint you.... There is no good answer unfortunately. I have been working with the guys over there on this very topic and struggling for this very issue myself for a while now.
By design, technically the OnMarketData feed should be the one that you want to use to catch every tick, that's what it's purpose is, there are a lot of indicators that run off of this feed, etc... Sadly though, the OnBarUpdate feed if ran on every tick can capture the current price level with a call to GetCurrentBid() / GetCurrentAsk() and front run the OnMarketData by anywhere from half a second to 2 to 3 seconds if you capture this during a price level change. So that's great, so why not just go with the OnBarUpdate and run it on every tick... Because even targeting that level of granularity it still misses price levels that are included in OnMarketData.
The link I provided, explains this issue fairly well, and provides tons of code snippets that you can use to access the various feeds, and test for yourself different sequences of events.
So the real solutions, and the one I am pursuing is to build a price definition of the "current best bid / best ask" using logic to flip between OnBarUpdate (running on a 1 tick time series) and OnMarketData. At various points one feed is leading and one is lagging and you will need logic in your code to know when to look where. You will either be 1 to 3 seconds behind if you only use OnMarketData or missing price levels often if you only use OnBarUpdate.... I don't get a sense that they have any plans to fix this, as they mentioned in their last post that HFT was not a focus for them, so they are just going to leave this issue out there for the time being and leave guys like me to build work arounds.
In terms of fighting overall latency here are a few notes:
1. Co-locate obviously. If you don't have this in place nothing else matters... You can get around 1 Millisecond to 10 millisecond order execution speeds on some VPS machines for less than $200 bucks a month. I have tested mine with NT on my rig and can hit: (Submitted > Accepted > Working) in < 1 Millisecond around 30% of the time, then 10 for another 25%, the rest between 10 to 50 depending on bottlenecks / traffic etc. The worst bottle neck I ever had from the broker was around 200 milliseconds, but generally this is < 1 MS. If you are using a laptop at home, unless you live 10 minutes from your data provider, you will typically hit 100 to 300 milliseconds, which will cause you to miss 5, 10, 20 price levels between your click and the exchange.
2. The way you should write your code should separate your execution logic from your analytical logic in such a way that analysis does not bog down your executions. Put your execution lines near the top and analytical lines below as a starting point.
3. Kill any indicator references. These can be the largest areas for optimization. Just code it yourself, or peak at the indicator and see what it is doing, but do it faster, better, smarter. The crap I have seen out there for indicators is ridiculous... Recursive looping do while blocks inside of try catch blocks inside of some other garbage and all they needed was a simple average of 5 variables.
4. Don't use OnPositionUpdate at all.... This lags OnOrderUpdate by a few seconds some times and is nothing but a redundancy if you know how to track your position with OnOrderUpdate.
5. I use the Unmanaged approach myself, and couldn't ever imagine going back to managed. There are some real speed tricks and things you can gain here but for some people this is very painful. One of the first times I ran unmanged live, I hit a bad block of code and ended up submitting orders faster than I could cancel them and ended up with 25 orders with no exits in place at all, and Ninjatrader / my broker killed my strategy and left me naked, so I had to manually close the position. It can be a nightmare at times, but building all your own cancel logic, exit and entry logic with no training wheels gives a lot of freedom to pursue some really strange stuff.
Your strategy shouldn't cause you any lag if done properly. I have around 5,000 lines in mine and I can often times hit 1 Millisecond execution speed times.
Hope some of this helps... Any you thought your opening post was long
Ian
In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
I would suggest to raise these questions in the official Multicharts forum directly, as this way you can get an answer from the developers.
In general Multicharts has a feature to queue quotes that come in faster than it can process them. So it should not skip any, but you might experience a slight additional delay.
It will display the current state in the lower right corner of the main window (if the window is blank this means there are no quotes queued at the moment).
Thanks for the information folks - much appreciated.
@Ian - You mention NinjaTrader, but I'm sure that those points are relevant to most of the platforms out there (unless they are dedicated HFT systems). Sounds like you've been on quite the journey
@ABCTG - I will post the query on the Multicharts forum as you have suggested. Last night I could not logon to the Multicharts forum - I have no user name yet and I could not register for some reason (no option to).
For context, I have previously dabbled with NinjaTrader 7 and 8, but am now looking at Multicharts and Sierra - trying to decide which of the two to migrate to (I'm with AMP, so NinjaTrader is not an option for now).
Multicharts is C# similar to Ninja, whereas Sierra is C++....but that's a different conversation. I'm trying to tease through the data feed limits behavior to give me a better insight as to what the right solution for me might be (which might not be right solution for anyone else).
I'm pursuing a somewhat similar path in MultiCharts, just from a different angle.
The first task I assigned myself is to look at the data, tick by tick. To do so, I have a 1 tick Range chart setup. (It's actually a converted Renko chart, but close enough for me.)
So for each tick I look at the time frame and volume data posted by MultiCharts. For volume data it shows UpVol, DnVol and TTLVOL. (How it distinguishes Up from Down volume is still a mystery to me.)
-----
Let's say a "typical" bar will show it took about 10 seconds to form, and has a handful of both up and down volume contracts exchanged.
-----
However, when the market is moving fast some bars form in zero seconds and have zero volume of any type. On occasion, there may be several of these "zero" bars in a row. These "machine gun" ticks will finally end with a large volume printing on either Up or Down, but seldom is there any volume on the other side. For the example below, when the "machine gun" ticks end I might see 1200 for Down volume and zero for Up volume.
Now back to your original post - I too am trying to map out the logical sequence of events when the market is moving fast. All I can tell so far is the ticks seem to appear before the time and volume data can catch up to the chart. This would affect the foundation of any strategy I would think.
