Visa blames European outage on systems it had been phasing out

Visa, which was criticized for its vague explanation of a June 1 outage in Europe, was in the process of installing better technology, but the project was not complete at the time of the incident and won't be finished until the end of this year.

Charlotte Hogg, Visa's Europe's chief executive, sent a letter to Nicky Morgan, a Conservative MP and chair of the House of Commons' Treasury Select committee, answering the questions Morgan had posed to Hogg following the incident. The outage took place during the evening rush on a Friday, causing rejected payments, lines at ATMs, makeshift paper-based payment records at supermarkets, and stranded travelers.

The nine-page letter discusses how the breakdown happened and what Visa is doing about it. There's also some unlucky timing, as Visa's letter includes details of an unrelated project that, once in place, would have had a better chance of preventing the kind of outage that happened in the U.K.

Visa is combining technology and other systems as part of its acquisition of Visa Europe, which used to operate separately from the U.S.-based Visa. As part of this process, Visa is migrating its European processing to VisaNet, the card brand's global processing system. VisaNet is an existing processing system with multiple data centers in disparate locations and is capable of handling more capacity and scale than the local European system that was in place during the time of the outage, according to Hogg's letter.

More than 99.9995% of transactions were processed over VisaNet over the last "several years," covering a total of 319 billion transactions. This European migration to VisaNet, which has been underway since February, is not a cause of the June 1 incident and is expected to be finished by year end.

"VisaNet is more resilient in its ability to detect and recover from partial malfunctions of this type that impacted our system on 1 June," Hogg's letter says.

Visa did not return a request for comment on Tuesday afternoon New York time.

"The letter does seem to imply that VisaNet is more resilient than the processing systems Visa Europe built when it remained a bank-owned association, while the rest of Visa went public," said Zil Bareisis, a senior analyst at Celent. "Now that Visa is one company again, migrating to a single global processing system makes strategic sense."

Regarding the legacy European processing system, Hogg's letter details what it calls a "partial degradation" that contributed to the June 1 outage.

This type of problem normally triggers a response to switch operations between the two redundant data centers in the U.K. that are synchronized and designed to take over for each other instantly in the case of an emergency.

But on June 1, a component in a switch in Visa's primary U.K. data center suffered a partial failure which prevented a backup switch from activating. As a result, it took longer than normal to isolate the problem at the primary data center, during which time the malfunctioning system at the primary center tried to synchronise messages with the secondary center. This created a backlog of messages which slowed down the entire processing chain.

As a result, payments were not processing normally at both data centers until just after midnight on Friday, or about 10 hours after the glitch began.

Under the same circumstances, VisaNet is able to "isolate and remove a failing component with one command, taking mere minutes to remove the malfunctioning component from the processing environment," Hogg wrote. "By doing so, our operating systems at other data centers can ignore a malfunctioning system and continue to operate seamlessly."

At the time of the outage Visa attributed it to an unspecified "hardware failure," an explanation that drew heat because its vagueness raised questions about Visa's ability to manage spikes in digital transactions. In Visa's letter to Parliament, the card brand said it has replaced components of its switch and plans a software upgrade to automate monitoring and shutdown for future malfunctions.

Hogg wrote the length of the incident was "unacceptable," but added that at no time during the incident was there a full system outage. Over the course of the entire event, 91% of transactions processed normally, with individual 10-minute and 50-minute peak periods in which 35% of transactions failed to process.

During the total period of disruption, 16 million cards were used and 1.7 million were impacted. There were a total of 27.6 million transactions sent to Visa for processing during the window, and 2.4 million failed, or about 9%. Consumers should not be charged for transactions that did not complete.

Visa is communicating with issuing banks regarding any reimbursement that is necessary and will make decisions on a case by case basis. Visa has also engaged EY to conduct an independent review of the incident.

Morgan's initial letter raised the possibility Hogg and other Visa executives could receive a subpoena to testify before Parliament. But The Financial Times reports Morgan is satisfied with the initial answers, though the committee expects to see the findings of EY's external review.