How a loud noise brought a data center to its knees

A data center's operations completely collapsed as a rare phenomenon cut off service for 10 hours.

Getty Images/iStockphoto

A Romanian bank experienced serious disruption to payment services and ATM withdrawals for roughly 10 hours due to fire extinguishers.

In a peculiar and rare phenomenon, the loud noise created by inert gas being released during a planned test of fire extinguisher systems not only forced the bank's main data center in Bucharest, Romania, offline, but also managed to destroy dozens of hard drives in the process, causing serious and irrevocable damage.

Last week, Daniel Llano, Head of ING Retail Banking admitted to customers that the "serious technical problems" were caused by Inergen flooding.

Inergen is a kind of fire extinguishing system which relies on gas rather than traditional foam or liquid. Suitable for enclosed spaces, Inergen, stored in cylinders as compressed gas, is dispersed through hoses and nozzles evenly across a small space to wipe out fires.

Usually, this kind of fire protection would be best suited for data centers -- especially as foam and liquid would damage valuable and delicate equipment -- but in this case, something went horribly wrong.

When the gas was released through the nozzles, the pressure was too high, which in turn created an incredibly loud noise when Inergen was released.

An ING spokeswoman told Motherboard that "the drill went as designed, but we had collateral damage."

A source also told the publication that the noise produced by the testing was louder than expected -- being about as high as the bank's equipment was able to monitor at over 130dB. Unfortunately, sound causes vibration, which transferred to the cases of hard drives stored in the data center and damaged internal components.

Motherboard was told that "putting a storage system next to a jet engine" was a fair comparison to make of the tests.

Servers and data storage were affected, as well as card transactions, ATM activity, internet banking, email and the bank's website. While services have been restored and customers can now withdraw money from ATMs and conduct card transactions, the company has been forced to rely on a backup data center.

"I activated emergency procedures and recovery plans provided for such situations," Llano said. "But because of the magnitude and complexity of breakdowns, unfortunately, the time required to restore activity by the back-up was longer than the tests we perform regularly."

The bank was forced to perform a restart of services and to take as many precautions as possible ING has also made an additional copy of the firm's databases. An investigation into the disaster has also begun in the hopes the scenario -- which has likely cost the bank a fortune in repairs -- will not happen again.