I'm comfortable enough (or used to be) with the performance and fail safes of my algo that I do not spend any time watching it anymore. It chugs away, and usually makes me money.
Today was a bad day in what's been a really good month. Around 2oclock I reached my algo's daily loss limit :-( it happens.
At the time I had 14 long contracts in the MESH3, and 2 open orders, MIT and STP.
So, my algo calls rithmic BUILT IN function to exit all positions, and then sends another built in function to cancel all open orders.
For context, when I send or cancel orders in my code, I have checks to make sure the order was received without error. And wait for the order to be returned.
When I use the built in functions, I assume it's going to work. I wait for confirmation that the message was received from rithmic servers. Then my software quits!
Welp.....
When you use Rithmics exitPositions() function, it sees all the positions you're in and sends a market order with the opposite sides. So, within microseconds rithmic sent a market order selling 14 contracts.
The thing is, 2oclock was FOMC today, and CME rejected the order with message "Order type not permitted while the market is reserved". Which basically means things are too volatile, we're not accepting market orders.
Like I said before, I wait for confirmation that Rithmic received my message without errors. But do not actually check that rithmic's servers actually do it. There's no callback for that. Only way I can think of doing it is to have my code wait, until positions = 0.
Apparently Rithmic only tries once.... In some alternate reality there is a version of me that lost 2k today (based on the price at close).
However, after I sent the exitpositions message I also sent cancel all orders message. With the intent to cancel the MIT and STP orders. It canceled the MIT order just fine. But for the stop order I got lucky. Looking at Rithmics communication logs with CME. CME received the cancel request, but 78 microseconds (i kid you not) after it received the cancel request, the stop price was triggered.
And for reasons I don't understand but I'm grateful for, CME cancelled the cancellation process and proceeded to fill the stop order. So my positions were liquidated!
I was 78microseconds (not milliseconds) away from losing 2k today. What a world!
I am going to add another fail safe. But imo Rithmic deserves some blame also, the server should have some sort of fail safe for this (CME always accepts limit orders, just not market orders). Or at least a direct way to alert if the exitposition method fails. It's a function that usually gets called for emergencies, imo it should be rock solid.