I've been working on and off to build a large scale data distribution system. What do I consider "large scale"? The end implementation needs to support realtime and historical data for several bulge bracket banks, asset management firms, and retail brokerages similar in scale to Fidelity/Robinhood simultaneously.
There's three goals to this thread.
I'm finally getting back into the project but it has been hard given the activity in the markets this year. I figured it would be fun to document the process somewhere as it keeps myself honest to finishing the project, while hopefully benefiting a few readers who endeavor to build their own trading systems.
I found myself writing technical documentation for a different type of audience during this project, which is not something I'm used to. I think exercising that muscle in a retail-oriented community is one way to get better at it.
It will be interesting to get feedback from as many people as possible as I work on this.
Here are some of the philosophical requirements that drive the project:
Pareto-optimal, cost-adjusted performance. In other words, at every cost level to the end user ($0, $10, $100, $1k, ...), there will be no more performant commercial solution. I measure performance by (i) 50/90/99th percentile latency on each event, (ii) percentage of messages lost at the end user. It's more important that the tail and worst case latencies are low, as there's no point getting most messages within 10 milliseconds in NY metro but seeing 10 minute delay or downtime during peak hours.
Full proprietary exchange feeds streamed over internet. This is a substantial challenge. There's a couple of retail-oriented data feeds that boast of streaming the full SIPs and OPRA over internet. However the SIPs only contain top of book for each market center, not the full depth. By my rough estimate in US equities, the SIPs are about 40% the size of the proprietary exchange feeds. So the final implementation needs to be bandwidth-efficient.
Scaling to 100k concurrent clients. This is where most web frameworks or conventional database servers running on a single node start to fail and it becomes necessary to think about active-active clustering. In this context, each client is not a person - a single application or receiving server may establish multiple connections. While the full ambition of the project goes above this target, it's a good intermediate target.
I'm happy to cover any topic of interest, but here's some that touch the data:
Data storage and processing. Primarily an application level, such as feed handling, messaging and serialization formats.
Devops practices. Such as monitoring and deployment.
Systems engineering and infrastructure. From bare metal to higher level matters like colocation and IP transit.
Frontend design. Visualizations and UI.
The project is mostly implemented in a mix of C, Rust, and Python. Python acts as glue code that instantiates much of the performance-sensitive components written in C and Rust.