Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
I'm not looking to have anything programmed/created by anyone here, so I hope this thread is in the proper location and can remain . . . apologies if I've err'ed. .
I'm curious as to whether or not anyone has, or knows about or has come across, any utility or program that pulls trade execution data in real-time for importation/processing elsewhere, to allow for statistical display? Ideally for Ninjatrader, as we're still using NT as our platform for executions, for the moment. It'd need to be able to handle hundreds of strategies running simultaneously.
My goal is to create a page of real-time trading statistics, broken down into several categories ranging from the most broad (cumulative total profit/loss across all strategies), to the the most narrow (a specific strategy) and everything in between (groupings by instrument, by market-direction, by instrument and market direction, etc).
For this, of course, I need the hard data of all trade executions as the foundation, and I want this to be collected automatically in real-time. I'm almost certain I'll need custom solution created, so I'm about to head to my tried and true companion 'Upwork' and similar sites, but I'd love to hear if there's anything currently out there that can accomplish the data collection portion of this task. . would save quite a bit of time/effort. Or, if anyone has gone a similar route and has any advice, I'm all ears.
A hearty thanks in advance!
Can you help answer these questions from other members on NexusFi?
The results need to be server-rendered if you want a real-time streaming dashboard. I think the simplest setup is to do this with basic Apache/HTTP. The backend that computes your statistics could be anything, really. Something like Go, Ruby, Python should sit well with a HTML/AngularJS frontend; if you need plots, for these type of statistics, you don't really have to write your own fancy UI code, you can use existing libraries. Aside from how you do threading, I think the number of strategies has very little to do with how well your solution handles this. You can communicate over JSON which will be one of the slower parts of the entire stack, but it will probably still be seamlessly fast to the naked eye if you do it right.
I've coded up something similar to what I've described above in a couple of days that produced real-time analytics down to every trade but also fetched historical summaries. Here's a screenshot of it computing summary statistics on an entire day across multiple data centers. I uncensored 1 data center where the strategies sent out about 1 million orders and canceled about 880k orders in just one data center, which notwithstanding fills would probably be around 0.1% to 0.3% of the CME's order activity across every symbol nowadays.
And I could drill down on the positions and individual orders of individual strategies like this seamlessly as well:
As for how you collect the data, the simplest way to do this is to store this after you serialize the outgoing messages or deserialize the incoming messages. Unfortunately, a third party vendor like NT most definitely uses some kind of binary format behind the scenes that you won't be able to decode easily. So the easiest way you might be able to do this while relying on them is to write a custom strategy or indicator that runs in the background and uses their API to access the incoming fills, and have your trading strategy serialize the data associated with outgoing orders.
You'll want to store only the lowest level information like that and then compute whatever's higher level like strategy PnL in the server backend.
No need to decode anything: the OnOrderUpdate() builtin method can be used to send execution data (using ZeroMQ for example) then have a ZMQ listener (I'll wrote it in Python but it could be anything else) which will store everything, SQL, CSV or whatever.
AngularJS would be a good option for displaying the data, Excel would do the job too (simpler but not that nice).
Excellent, thank you both. . all I needed was a clean way to gather the 'root' data of order execution information on an ongoing basis in real-time, to be processed (in a separate stage) in various ways afterwards, and I think I now have all the info I need to build the proper structure to allow for this. . . appreciated.
If anyone else has done such things and has any additional tips, I'd love to hear 'em, just to assure I don't reinvent any wheels here, but I now have enough to go on
Also been looking for something like this for a long time, even a live equity curve as you're trading intra-day would be awesome, may even find patterns in your own trading around certain times of the day.
