System failure at the Malaysian Stock Exchange
Thursday, 10 July 2008
The Malaysian stock exchange (Bursa Malaysia) had to suspend trading on 3 July for an entire day due to system failure.
Sounds like all the planets were in alignment, two hard disk failures, one cpu failed, and connectivity issues. The secondary system couldn’t even start up in time. Uh uh, this would be costly.
Transcript:
I just want to go through the sequence of events of the trading system of the [Malaysian] stockmarket. There was one hard disk which was faulty. This was replaced as per standard procedure. About half and hour later, we found that the new disk was not able to recover data.
We have a system we have two disks running in parallel. This then triggered the failure of the second disk as well as one of the CPU. Half an hour later, we managed to restart the system and went about trying to reconnect with the brokers. We found that the brokers were unable to connect to our system. This normally takes about half an hour. By 8 am, more than half of the brokers were unable to connect to our system. We have a policy within Bursa Malaysia, where if more than one third of brokers were unable to connect to our system or to trade, we would suspend the market. This decision we took at 8:30am, and that point in time we took the decision to activate our backup site. This normally would take 3 hours. By 1 pm we noticed that the start up process at the back up site was taking longer than expected.
At that time, together with our vendors, we had resolved the problem at the primary site.
We tried to resume the afternoon session from the primary site, and this was targetted at 3 pm, and we were aiming for the market to resume at 3:15pm.
At 3:15 pm we noticed there were some preopening orders which were being keyed in. But we noticed there many brokers site which were unable to connect to our site. There were connectivity issues being encountered.
By 3:30pm, we were still unable to resolve the connectivity issues and we decided to suspend the market for the rest of the day.
However, because some pre-opening orders were keyed in at 3:15. We noticed there were some orders which have been matched. We subsequently cancelled these matched orders. …
