How the Stress Tests Have Reshaped Banking Supervision

Second of two parts

WASHINGTON — The stress tests started as an ad hoc test designed to shore up faith in the banking system during the financial crisis but have become the "cornerstone of a new approach to regulation and supervision," in the words of a Federal Reserve Board Gov. Daniel Tarullo.

Though stress tests are widely viewed as a successful and critical exercise, there are growing concerns that regulators — and the banks themselves — may have become too reliant on them, overshadowing other aspects of the supervisory process.

"It's a tremendous change in the examination and regulatory system," said Andrew Metrick, deputy dean of Yale School of Management. "I think that most of the regulatory community believes that it is the stress tests that are the binding constraints for the large banks, not ordinary examinations and Camels ratios and things like that."

There are also fears that the stress tests, the first results of which will be released on Thursday with another round out on March 11, force banks to prepare for scenarios that are unlikely to happen, while not being prepared enough for those that do occur. And there is an overall worry that many banks are beginning to view them as a rote exercise.

With each passing year, there is a perception that the Comprehensive Capital Analysis and Review — the formal name for the primary stress test exercise — becomes more routinized and predictable.

"What I do fear is that there is a real risk that CCAR is going to devolve into a compliance exercise. The stakes are really high, so the focus is on the outcome — passing if you will — rather than the process," said Til Schuermann, a partner at Oliver Wyman and former senior vice president at the New York Fed, at a conference in January. "Time and again the question comes up, 'Look, just tell me what I need to do to keep the regulator happy,' rather than... 'What are my strengths and weaknesses and how should that inform my strategy?' "

But regulators counter the stress tests have become an important part of the supervisory process and that they work hard to ensure that the tests never become a rote exercise, or one that institutions can easily game.

Simply put, stress tests are analyses of how a bank or other financial institution's balance sheet performs under hypothetical market conditions. In the U.S., there are two main stress tests — the Dodd-Frank Act Stress Test and CCAR, which has both qualitative and quantitative assessments (For more information about how stress tests work, read this.)

Stress tests were initially developed by the banks themselves in order to assess their stability — so called "bottom-up" stress testing — and were initially mandated for the largest banks by the 2004 Basel II accords, which said banks should consider "at least the effect of mild recession scenarios." That mandate went unobserved in the U.S., however, because Basel II was never implemented. 

The collapse of Lehman Brothers in 2008 and the fall of Bear Stearns, Merrill Lynch, Wachovia and other financial institutions during the crisis hastened the creation of a "top-down" stress test directed by financial regulators.

That initial stress test, called the Supervisory Capital Assessment Program, or SCAP, was conducted jointly by the Fed and other financial regulators in 2009 at the height of the government's efforts to arrest the crisis' downward trajectory. SCAP was included as part of the 2008 Troubled Asset Relief Program and assessed 19 bank holding companies books under three scenarios of increasing severity, much the way CCAR does. The test found that 9 of the 19 banks required no additional capital, and most of the remaining ten only needed to hold more Tier 1 common capital to meet the required 4% buffer.

SCAP demonstrated that a "top-down" stress test could work in practice and formed the starting point for DFAST and CCAR when Congress began formulating what became the Dodd-Frank Act. But the new stress tests included notable improvements.

In a speech last year, Tarullo pointed out that CCAR, unlike SCAP, has "independent supervisory models" to assess bank balance sheets, rather than relying on banks' own estimates.

Over the ensuing years, CCAR has also built more expansive and more independent estimates of bank revenues and losses and has substantially improved its scenario modeling, Tarullo said. But the Fed also remains committed to constantly refining and improving the tests to ensure they remain effective.

"We do not regard the supervisory stress test and CCAR as finished products," Tarullo said. "In fact, we should never regard them as finished products."

It is precisely that commitment to evolution that makes the banks nervous.

Wayne Abernathy, vice president of regulatory affairs at the American Bankers Association, acknowledged that the CCAR tests are improving as time goes on. But he said the fundamental reliance on scenario-based stress testing can lead to "model error" — where reliance on projections can make institutions overly prepared for one case and unprepared for another. The precipitous and enduring collapse of oil prices in the latter half of 2014 is a prime example of such an error.

"You can have banks chasing after things that just aren't going to happen," Abernathy said. "And yet because [regulators] take the stress tests so seriously — they use them to affect what banks are allowed to do and not allowed to do — the scenarios are being treated as if they are reality, and that's a structural conceptual problem that needs to be fixed."

Abernathy suggested that the modeling process should be opened up for public scrutiny in order to make the scenarios as realistic as possible — or at least reduce the potential that banks build up defenses in the wrong areas.

Banks are not the only critics of CCAR's scenario testing model.

Anat Admati, a professor at Stanford Business School, said the tests are not as rigorous as they are given credit for, and leave banks appearing safe and secure if their capital plans are approved, even though there are far more possible sources of systemic risk that remain untested.

"I believe they give us a false sense of security," Admati said. "They are fooling themselves to think that these answers are meaningful. They're not admitting to what they don't know."

The scenario-based aspect of stress tests is frequently viewed as its weakest link. The CCAR scenarios in this way are somewhat akin to annual flu vaccines — regulators consider what strains of market contagion could plausibly infect the system with severely adverse conditions, but at its heart the scenarios are nothing more than an educated guess.

Yale's Metrick said the Fed's inability to predict the future is not as significant a weakness for CCAR as its inability to predict how institutions are interrelated. The failure of Lehman Brothers was one thing, Metrick said, but the subsequent run on Money Market Mutual Funds was totally unexpected and had a far more devastating systemic impact.

"Those interconnections are the thing that we need to make the most progress on," Metrick said. "I know people at the agencies are well aware they don't have all the answers to that, but the interrelationships part is the biggest unknown."

CCAR also derives much of its strength from not only the Fed's scenarios, but from the central bank's oversight of banks' internal stress scenarios — the so-called "qualitative" stress test. Metrick said that the best way to make CCAR stronger is to compel banks to expand and diversify the types of risks it knows about and can protect itself against.

"If we're going to get to the point of maximal usefulness ... we're going to have to just get better and better at coming up with scenarios that can really hurt us that we didn't think of," Metrick said.

Regulators, too, see it less as a number-crunching exercise than a way for bank to tests their systems and make thousands of big and small decisions about how they handle risk. If a bank's board and management must consider multiple bad events that could conceivably hurt their firm, it can force them to rethink how they address risk.

That "qualitative" testing in the U.S. is a tool that regulators in other countries do not necessarily possess. Stefan Walter, director general of microprudential supervision at the European Central Bank, said in January that the European stress test differs in important ways from CCAR.

Like CCAR, it is a mix of "top-down" and "bottom-up" testing, but the elaborate top-down process is not conducted annually. By necessity, Walter said, the ECB will have to rely more on banks' own internal reports, though it hopes to improve some of its benchmarking models in the interim.

But CCAR's ability for banks to pass quantitatively but fail qualitatively "is an important aspect" that "would likely be an integral part of another stress testing exercise.

"In the near term, given what we'll be able to do on data development, we think we will probably have to stay with a hybrid approach," Walter said. "Over time we think we can build out better some of the top-down capabilities, but no matter where we go … we need to have a bottom-up focus."

There are some areas of risk that are more difficult to test than others. Liquidity risk, Walter said, continues to be a modeling challenge for European regulators, as is operational risk, which is notoriously unpredictable.

Operational risk is also a problem for U.S. regulators. David Lynch, assistant director for quantitative risk management at the Fed, said last month that the central bank continues to struggle with quantifying operational risk in a reliable way.

But Loretta Mester, president of the Federal Reserve Bank of Cleveland and former head of research at the Philadelphia Fed, said despite its limitations, CCAR has given regulators invaluable insight not only into the stability of financial institutions but how market contagion can move throughout the financial system. Almost none of that analysis existed before stress tests were introduced into the regulatory toolkit, she said.

"We weren't doing any of that before," Mester said. "It was very hard to identify the financial stress that's actually in the system."