The lawsuit
One, such vendor disputes are usually settled out of court. Two, the suit hinges on the failure of one piece of storage hardware. Three, it comes at a time when
Neither BB&T — which is expected to complete its merger with SunTrust Banks soon — nor Hitachi agreed to an interview. Both offered statements cited below.
The case against Hitachi
Hitachi is being sued for breach of contract, unfair and deceptive trade practices, and gross professional negligence over how it sold, installed and maintained a G1000 Storage Disk Array.
“The system outage that began Thursday, Feb. 22, 2018, was caused by an equipment malfunction in one of BB&T’s data centers,” the bank said in its statement. “This lawsuit references a critical hardware component that failed and then affected many of our other systems. We believe the vendor who manufactured, installed, and maintained the component was responsible for the equipment failure.”
The G1000 is a well known and commonly used piece of equipment in data centers. BB&T used the same unit in several of its data centers.
In the Zebulon, N.C., data hub where the outage occurred, the Hitachi storage array was connected to four mainframe computers. When it failed, everything those mainframes handled failed: online banking, mobile banking, phone banking, ATMs and wire transfers.
A big issue in the lawsuit involves the fiber optic cables connecting the storage disks to the unit’s data controllers. If improperly installed, these cables can suffer undue physical stress until they eventually fail, the bank said.
“Hitachi did not properly loop and secure the excessive slack in the fiber optic cables, but instead recklessly wadded up the slack in the cables and shoved them into a narrow cavity within the data controller cabinets,” the bank said in its complaint.
The bank said the storage disk array failed nine times between June 2017 and February 2018, and that this is extraordinary “based on BB&T’s experience with similar data management systems." The bank said it had no access to the internal components of the storage disk array and could not perform any maintenance on the storage unit. Hitachi was responsible for all maintenance.
According to the bank, a Hitachi engineer who responded to an alert on the G1000 replaced a failed cable and reported that all the fiber optic cables were tangled. Hitachi did not address the problem, and a Hitachi supervisor told bank staff everything was fine, according to the lawsuit. When memory cache cards failed and left the G1000 vulnerable to a complete system outage, Hitachi did not respond at all to an alert about this, BB&T said. The system shut down completely ten days later.
This is not the first time a storage unit has caused a bank outage.
“There are instances in the financial services space where storage has been the root cause of problems and the vendors have faced the ire” of clients, said Naveen Chhabra, senior analyst at Forrester Research. “If the storage failed so frequently, it should have been diagnosed and a remediation should have been put in place.”
But it is uncommon for such a case to go to court.
“Usually the contracts banks have with these service providers provide for specific service standards, auditing and examinations,” said Quyen Truong, a partner at Stroock & Stroock & Lavan who has held senior positions at the Consumer Financial Protection Bureau, the Federal Deposit Insurance Corp., and the Federal Communications Commission.
“There would be robust indemnification provisions in there as well," she said. "If there is a breakdown, you generally have the parties work it out, investigate the breakdown and to take corrective action and address cost issues. It is less common for those situations to require a lawsuit.”
Questions the bank might get asked
In its statement, Hitachi said “Hitachi Vantara is trusted by 85% of Fortune Global 100 companies, and we take the satisfaction of our customers very seriously. We are in the process of reviewing the allegations made by BB&T. Hitachi Vantara believes it provided high-quality service to BB&T at all times, including service delivery and advice regarding best practices for maintaining high system availability."
In court, the bank may be questioned about its information technology architecture and its single point of failure at the storage unit.
“It is not normal, not recommended, and not practical for any large enterprise, especially a financial services organization, to depend on just one class of storage array,” Chhabra said. “Systems like core banking that support hundreds of millions of accounts and transactions daily need dedicated high performance, highly resilient storage systems.”
The bank’s point about tangled cables sticking out of the storage disk array might also be challenged.
“These cables would always find themselves behind the systems in the cabinets, sometimes very systematically drawn and most of the times not,” Chhabra said. “Bent cables can be an issue in storage, but the question is could the Hitachi team not resolve that for eight months? That seems doubtful.”
What’s next
The bank carries the burden of proof to support the allegations in its complaint and prove what happened, Truong said. Hitachi will rebut the evidence the bank presents. Hitachi could move to dismiss the complaint.
Much of this case, including the liability and calculation of damages, will hinge on the terms of the master agreement between the two companies.
If BB&T prevails, it could seek a great deal of money from Hitachi. The bank has said it lost about $15 million in deposit service charges and incurred $5 million in higher operating expenses due to the outage. It is asking for compensatory damages, punitive damages, treble damages and attorneys’ fees.
If Hitachi wins, banks will be on notice that they have to take a more active role in the running of their data centers, even when they outsource pieces of them to vendors.
