Tech issues afflict banks, Microsoft after critical CrowdStrike glitch

Early Friday morning, a buggy software update issued by CrowdStrike began causing some Windows users to boot their computers into the Blue Screen of Death. The problem appears to also be affecting Microsoft Azure, the company's cloud services offering.

Australian and New Zealand banks reportedly experienced outages to online banking services during the day Friday. Airlines experienced a spike in delays and flight cancellations at the onset of the errors just after midnight East Coast time. NBC News, Sky News and several Australian broadcasters temporarily stopped broadcasting live content.

The problem seems to have a mixed impact on U.S. banks, but the financial services sector has not suffered any systemic impact, according to the Financial Services Information Sharing and Analysis Center, or FS-ISAC, a consortium of roughly 5,000 financial services companies advancing cybersecurity and resilience. The consortium's members collectively have $100 trillion in assets.

"Core functions, including banking and payment processing, are largely functioning with some scattered effects," the FS-ISAC spokesperson said. "CrowdStrike has posted solutions that are already being implemented by many customers. FS-ISAC will continue to assess additional impacts to financial services."

According to one analyst, the impact of the CrowdStrike bug is historic, adding that some businesses were not able to transact because of it.

"This issue has the most far​-​reaching impact we have ever seen with a security tool," read a note from Barclays Capital. "We have seen multiple security tools have issues, whether it's vulnerabilities or even cyber-attacks, but the impact of those have not seemed as big as this."

Root cause a buggy CrowdStrike update

CrowdStrike CEO George Kurtz said the issue has been identified, and a fix has been deployed, so the company is now working with impacted customers to ensure that their systems are back up.

"Today was not a security or cyber incident," Kurtz said. "Our customers remain fully protected. We understand the gravity of the situation and are deeply sorry for the inconvenience and disruption."

The root cause of the many problems appears to be a defect in a single update pushed by CrowdStrike to Windows hosts overnight. The company has provided updates on the defect and workarounds, which involve booting Windows into Safe Mode or Windows Recovery Environment and deleting a single system file.

While the root issue is not a cybersecurity issue, the Cybersecurity and Infrastructure Security Agency, CISA, said Friday that it had observed threat actors "taking advantage of this incident for phishing and other malicious activity."

"CISA urges organizations and individuals to remain vigilant and only follow instructions from legitimate sources," read the release, which linked to CrowdStrike's guidance. "CISA recommends organizations to remind their employees to avoid clicking on phishing emails or suspicious links."

Banks see mixed impact

Customers of TD Bank, Bank of America, JPMorgan Chase, Wells Fargo, U.S. Bank and Arvest Bank reported significantly higher volumes of disruptions via Downdetector, a platform where users can report disruptions to digital services across industries.

JPMorgan Chase employees have reportedly had trouble gaining access to their workstations, according to the New York Times. The bank did not immediately respond to a request for comment.

To a lesser extent, Citi customers reported problems on Downdetector — mainly problems logging in to their accounts online. However, the bank has experienced no material impact from the outage, and all of the bank's systems are currently operating on a normal basis, according to a company spokesperson. 

Truist customers also reported issues logging in to their mobile apps. But, "branches, care centers, and digital banking services are open to serve clients," according to a company spokesperson.

U.S. Bank has since reported that it does not use CrowdStrike. Arvest, a $26 billion-asset bank in Fayetteville, Arkansas, posted on X, the site formerly known as Twitter, that its banking systems were not directly impacted by the CrowdStrike global outage, and that customer issues logging in to online and mobile banking have been resolved.

TD Bank acknowledged the issue on its website. A banner on the homepage reads, "TD has been impacted by the global technology disruption that has affected organizations around the world. Teams are working hard to restore digital systems. TD customers can visit Stores or ATMs." A similar message appeared when customers tried logging in to online banking.

TD did not respond immediately to a request for comment.

The outage hindered some Fifth Third Bank employees from logging in to their computers. 

"We are working to address the issue and reinstate access," said a Fifth Third spokesperson. "At this time, impact to our customers and branch network is minimal and our digital and self-service channels are operational."

Synovus Financial, in Columbus, Georgia, said in a statement that it was impacted by CrowdStrike's software update. 

"Our teams implemented contingency plans to minimize disruption to our clients, and we have restored connectivity," reads the statement. "We'll continue to closely monitor systems to ensure the safety and integrity of our operations."

Some smaller banks have acknowledged outages due to the CrowdStrike-Microsoft glitch. 

"The global Microsoft/CrowdStrike service interruption is impacting all branches and bank offices, with some branches unable to complete full-service transactions," reads a banner on the website of Canandaigua National Bank, a $5 billion institution in Canandaigua, New York. "Our mobile and online banking platforms are all functioning as normal. However there may be delays in transaction processing." The bank also said its call center is affected, and customers should expect longer wait times.

The Milford Bank, a $598 million-asset institution in Milford, Connecticut, posted its own warning on X: "Attention — Some of our systems have been impacted by the CrowdStrike worldwide computer outage. We are working to restore connectivity as soon as possible."

Neither Canandaigua nor The Milford Bank immediately responded to a request for comment. The Milford Bank updated customers on X a few hours later that all bank systems were up and running again.

PCBB, the correspondent bank based in Walnut Creek, California, wrote in an email blast early this morning that its systems were impacted by the CrowdStrike outage, and "our teams are working diligently to bring our platforms back online." It has since reported via email that it is fully operational. 

The National Bankers Association heard directly from minority depository institution CEOs during its strategic planning retreat on Friday. 

"The recent CrowdStrike outage significantly impacted several of our MDI member banks," said Nicole Elam, president and CEO of the National Bankers Association. 

While some don't use CrowdStrike for cybersecurity, the impact on their vendors meant customers could not always access certain applications, including digital accounts. Some operations, such as accounts payable, were disrupted while other services were down. 

"These service interruptions highlight the ripple effects one key vendor can cause directly and indirectly," said Elam.

The American Bankers Association referred American Banker to FS-ISAC. Maggie Leung, media relations manager of the Canadian Bankers Association, said in a statement, "Like many other companies, banks in Canada are reviewing the situation based on updates from their technology partners. Canadians can be reassured that our country has a well-protected banking system. Any current impact on banking services would be temporary."

A spokesperson for the Independent Community Bankers of America said, "We are monitoring the IT outage closely, in coordination with our partners and officials in the financial services sector, to gather additional details and assess the impact on the industry."

Some capital markets firms were affected. The Charles Schwab website ran this warning: "Due to a third-party, global, industry-wide issue, certain online functionality may be intermittently slow or unavailable. We're actively monitoring the issue. Phone services may be disrupted and hold times may be longer than usual." The London Stock Exchange's website posted a similar note early Friday morning, but later in the day told Reuters the problems had been resolved and services restored.

Payments largely unaffected

Though some news reports said consumers were having trouble making payments, Visa and Mastercard said they were unaffected. A Visa spokeswoman said the organization's systems are operating normally, but added, "we are aware of reports of people being unable to make payments and are working with our financial institution clients to understand any impact on their services to cardholders and merchants." A Mastercard spokesman said, "There is no indication that these issues impacted our systems."

An American Express spokesperson also reported that the company is processing transactions normally, but added, "due to higher than normal call volumes in some areas, some American Express customers may be experiencing longer-than-usual wait times."

Discover Financial Services relayed in a statement that it does not use CrowdStrike and was not directly impacted by the outage. The company said it does have third-party vendors who were impacted by the outage, "which resulted in some of our banking features being temporarily unavailable," the statement continued. "These issues have been resolved."

Repairs will take time

For physical machines running Microsoft, companies will need to implement a manual fix for each one, according to multiple experts, including Andras Cser, principal analyst and VP at consultancy Forrester.

"Because of the way in which the update has been deployed, recovery options for affected machines are manual and thus limited: administrators must attach a physical keyboard to each affected system, boot into Safe Mode, remove the compromised CrowdStrike update, and then reboot," Cser said.

Cser added that some system administrators will have an additional roadblock if the hard drive for affected machines are encrypted by BitLocker, a Windows security feature.

"Some administrators have also stated they have been unable to gain access to BitLocker hard drive encryption keys to perform remediation steps," Cser said. "Administrators should follow CrowdStrike guidance via official channels to work around this issue if you're impacted."

While remediating the immediate issues will require "significant effort," according to Cser, the good news is that tech vendors that have been impacted by similar incidents in the past have shown that their "operations, product testing, and communications strategies only get better" afterward.

The fix is possible to automate for virtual machines, according to steps outlined by CrowdStrike. The simplest option is to roll back virtual servers to snapshots (backups) taken before Friday morning.

Kim Kirk, the chief operations officer at Queensborough National Bank & Trust Company in Louisville, Georgia, has been holding calls every two hours to communicate plans and feel out further impacts on servers and employee workstations. She prioritized the issues she wanted her $2 billion-asset bank's managed information technology service provider to resolve. These included problems with the VPN server, the interactive teller machines in the bank's drive-through lanes, systems that process ACH and wire transfers and branch technology. 

Online and mobile banking, ATMs and the bank's core provider are all operational. 

"It has been a great test of our business continuity processes," she said.

Some customer reports complex to diagnose

Some spikes in customer complaints about accessing their banking accounts may not reflect problems with banks' systems. The spikes may be caused by the sudden popularity of the Downdetector platform amid the disruptions, or by IT infrastructure not belonging to the banks whose customers have reported issues.

For example, users of Comcast's telecommunications business Xfinity reported more outages than usual coincident with the start of the CrowdStrike and Microsoft problems. The company operates a content delivery network, or CDN, and domain name servers that, if affected by the CrowdStrike and Microsoft problems, could cause problems that customers might not specifically attribute to Xfinity.

Penny Crosman contributed to this article.

Update (2:00 p.m.): This article has been revised to include statements from FS-ISAC, Barclays Capital and Arvest.

Update (4:30 p.m.): This article has been updated to include commentary from the National Bankers Association, Charles Schwab & Co., Queensborough National Bank & Trust and Synovus Financial.