SAN FRANCISCO — Brenden Smith, the chief information security officer of FirstBank, hired a group of
Over the course of three years, the attackers made numerous attempts to sneak their way in. At first, the attacks were difficult for FirstBank staff to detect; the hackers largely exploited previously undiscovered vulnerabilities in the software and devices used by FirstBank, which is headquartered in Lakewood, Colorado, and had $28 billion in assets at the end of 2023.
In one case, Smith said during the panel at the cybersecurity-focused RSA Conference, the hackers made an initial breach into FirstBank's systems that took 102 days for his team to detect.
"The good news," he said, "is that they didn't accomplish any of their objectives."
But the months the hackers lurked silently in FirstBank's system "felt uncomfortable," according to Smith, who had hired them to levy the attacks against his team as a way of finding the team's weaknesses so they could improve. The professional cyber threat his team faced was benign, but it was designed to simulate a real threat — malicious hackers who could conceivably target FirstBank one day.
This professional team worked for a company called Randori, which
IBM's Randori is one of many so-called attack surface management vendors. Microsoft offers its own such service, as do CrowdStrike, Halo Security, Palo Alto Networks, Verizon and
Smith advises other companies to conduct their own due diligence before choosing a red team vendor.
"I think it's a bit foolish when someone picks a vendor based on a talk without any additional due diligence," he said. He added that, given how long FirstBank has worked with Randori, the bank plans to reevaluate the market soon.
Red team exercises can be critical to validating the assumptions of the blue team, according to Keith Mularski, a managing director in Ernst & Young's cybersecurity practice. These assumptions might be about the security measures a CISO might assume their institution implements adequately. A red team can help validate that these measures are indeed sufficient.
In one example, Mularski described a client retaining EY for a red team exercise, and his team was able to find a way to bypass the multifactor authentication system the client had set up.
"From the blue team's perspective, they knew they had multifactor authentication in place, and they thought it was good," he said. "But the red team comes in and says, 'yes you have this in place, but let's see how we can get around it.'"
To combat cybercriminals, banks and credit unions turn to white-hat hackers to uncover system vulnerabilities. But what motivates these computer whizzes can be surprising.
For two years after Smith hired Randori, his blue team won. The red team made initial attempts to break into FirstBank's systems but was never able to achieve any substantial objectives with those footholds. Smith described a three-part cycle that repeated itself during the exercise.
In the first part of the cycle, the red team would conduct a scan of FirstBank's attack surfaces — the parts of its system exposed to the internet or otherwise accessible to attackers. As it detected potential ways in, it would launch scattershot attacks that were typically easy for FirstBank to detect but miss every once in a while. Once the red team gained initial access, it would attempt to move laterally within the system, at which point the blue team would detect their presence and remove them, restarting the cycle.
But one day, three years into the exercise, the red team finally won — albeit with a lot of help from Smith. The red team targeted an internet of things, or IoT, device owned by FirstBank with a zero-day vulnerability. Smith did not say what the device was because of a nondisclosure agreement; when hackers disclose zero-day vulnerabilities to device and software creators, those creators often bind the hackers to nondisclosure agreements to ensure they do not share any details of the vulnerability, even if it gets patched.
Once the red team compromised the IoT device, they used knowledge they had gained over the three years prior to evade FirstBank's threat detection scanners. This was one of the bits of help Smith admitted the red team had; while some threat actors are persistent, few would spend years learning the ins and outs of a bank's computer systems the way the red team had.
Not only had the red team been paid to focus on FirstBank for so long, but Smith had even opened the metaphorical door for them a few times. In one case, Smith allowed the team to hide a small device in a tissue box in an infrequently-used part of a FirstBank building during the pandemic, when everyone was working from home. Even in that case, FirstBank staff caught the intruders before they could further infiltrate the system, and a staff member eventually found the device and turned it over to the security team.
But after three years of the cat catching the mouse and letting it go, the mouse had studied the cat well enough to know its next move.
Once the red team compromised the IoT device, they used that access to find a computer that could send internal emails, thereby bypassing FirstBank's phishing filters. The red team conducted two phishing campaigns to collect two types of credentials. One worked; in the other case, an employee turned the email over to the security team, and the team stopped the attack.
Even though the security team had detected the phishing attack and determined it was coming from somewhere inside FirstBank's system, they couldn't pinpoint where. Smith chalked this up to the fact that the red team was operating from an IoT device using a zero-day vulnerability; these exploits, he repeated, are "very hard to detect."
So, the blue team recommended activating FirstBank's incident response retainer — a team of outside cybersecurity experts who come in to clean up after a cybersecurity attack. Smith said he was proud of his blue team and that the moment was a win for them. "They knew they were out of their element," Smith said.
He denied the request to activate the incident response team, as he didn't want to spend money having them respond to an exercise. So he told his team that the people behind the IoT device attack were members of a red team and challenged them to do their best responding themselves.
At that point, the red team went silent, to try to avoid detection, all the time maintaining the access they had gained — and that Smith allowed them to keep. After months of lying dormant, they planned an attack using knowledge of FirstBank's Active Directory system, which is a service that authenticates and authorizes users in enterprise Windows systems.
The attack worked. Smith spared the details, but at a basic level, the red team forced a password reset on one user; that user had control over the account of a second user with domain administrator privileges; and "within 15 minutes," the red team gained full control over the system. At least, hypothetically; the red team didn't exercise this control because they were hired by Smith for the exercise.
Smith said that, even though the blue team eventually lost to the red team in the end, the failure provided innumerable lessons that were valuable to the institution as a whole, which he shared with the session attendees. For starters, the users who fell for the phishing attacks and password reset attacks would be far less likely to make those same mistakes again, he said.
But the broader takeaway, he said, was how valuable this kind of long-term red team exercise can be. Many times, banks and other firms enter into two-week contracts with red teams to save time and money while trying to learn where their vulnerabilities are. But in Smith's case, it took years for the red team to compromise just the outer perimeter of FirstBank's computer systems.
Continuous red team exercises also emulate attackers more realistically, Smith said.
"Attackers aren't on two-week contracts," he said in his presentation. They can — as the red team did — go inactive for months on end once they gain initial access to avoid detection.
Ultimately, continuous red team exercises make the prospect of a cyberattack against a bank much more real for the people defending it, Smith said. Cyberattacks become more than something that just happens in the news; they become something that can actually happen to the bank.
"It just changes the way people think," he said. "Nobody can say, 'It's happened in the news to other companies, but it couldn't happen here.' It will happen here, because we're paying someone to do it 24/7."
