I agree 100% with you on the full line, and we are not even talking about a ministry here where a day of downtime is not much of an issue, as they take a paper and pencil and continue...
Here we are talking about electronic Futures trading, where people trade with leverage, traders put an order, the stops are rejected (bad fill price reported) and minutes later you are cut-off the market and can't reconnect
Help desk of the broker is overwhelmed with calls, no emergency communication, only very late came some communication..
I have redundancy on my side, production machine and stand-by machine, main internet connection (cable) and backup internet connection (VDSL2), UPS, etc...
All of no use, as the main high-way is down...
Mirus trader was also not accessible during the period
I agree with you, the manager of that migration, can hardly be called a manager,
very poor migration plan and poor risk mitigation.
The following 3 users say Thank You to rleplae for this post:
I do fully agree with you. As occasional trader ( not every day ) i take care that my programs are Always in good order of working, making secure bacups and making image from all those programs.
On 20 november 2013 i received from Mirus futures a mail telling me that i was required to do a mandatory update to download ninjatrader V17.
I did do that but had trouble with the fdax, i had no data any more and so on . I did sent a mail to mirus and
on 21/11/2013 i received a mail telling me follow :" We are currently aware of the issue and working on a solution. Sorry for the inconvenience.
If you are trading the DAX live you can revert back to .16 until the issue is resolved.
Ninja Trader: 7.0.1000.16"
So i reverted back to .16
As i knew other traders received the same mail i warnd them not to make the change to version 17 to avoid they have problems too.
On 27/12/2013 i received a mail to ,upodate before sunday 29/12 to version V18.
Again, i did what they requested.
Again i had trebles with dta and connection. Finally i phoned to the tech of mirus and the did, while on the Phone connect me again.
Today, we had again big problems. The ones we are talking about.
Normally Mirus must had a surveillance on the data as the markets was open from sunday on monday.
A word of thanks to the helpdesk of ninjatrader as they did try to get me back online for the FDAX ( without succes )
Mirus could suspect that there was a problem as there first request for upgrading to V17 was not a succes.
As soon they had news from the problems they should have warnd all the clients of the problems and they should
have warned then again when problems are solved. At least we would have know what there was going on and now traders had the risk of trading there real account so see connection fall out a few seconds later.
This behaviour of Mirus is irresponsible and Mirus was looking away of the problems instaed of warning his clients.
I can only hope nobody lost money and that Mirus have learnd a lesson .
I am hoping that Mr. Deux will talk to Mirus to avoid this sort of problems in the future . A good communication plan would be a good start
The following 2 users say Thank You to crossover for this post:
I don't usually post on forums, but seeing as I paid for another 3 months of NT only last week, I think I am entitled to a rant.
I too worked in IT before becoming a full-time trader. Coincidentally I worked for 15 years developing trading software for a number of high profile investment banks.
As you have said in your post, there are processes in place for QA testing, rolling out new software, migrating data, etc. etc. and contingency plans if it all goes wrong. Change control and the ability to roll back a software release is a big part of it. I wonder if Ninjatrader have even heard of SourceSafe?
I contacted Ninjatrader today and they blamed the connectivity to Zenfire. However, part of the QA process for every trading system I have ever worked on is the integration with upstream and downstream systems, i.e... Zenfire.
The fact that Ninjatrader are blaming Zenfire worries me a lot, because they obviously don't have a clue that it is both parties responsibilty to ensure connectivity works. Also as mentioned by Wolfie, not being able to roll back the software release and treating users (who have real money on the line) as Beta testers stinks to high heaven.
If this ever happened at one of the banks I worked at, the traders would be livid and heads would roll.
The following 3 users say Thank You to tradersimon for this post: