The Risks of Oversimplifying Stress Test Models

One of the objections we've heard from banks' internal and external validators and the Federal Reserve during the last Comprehensive Capital Analysis and Review is that stress-testing models need to be very simple.

The notion is that since stress-testing results are to be woven into risk management processes, humble folks, such as C-level executives at major banks, need to understand exactly how the models work. We do not disagree with the thrust of this sentiment, though we have somewhat more faith in the ability of bank managers to understand well-thought-out econometric models than it appears some others have.

By "simple" we mean, specifically, an over-reliance on too few macroeconomic drivers. We do not question the imperative to avoid complex functional forms, unnecessary nonlinearities, and counterproductive levels of portfolio segmentation in developing stress-testing models. In terms of these three features, we come down squarely in favor of the standard "Keep It Simple, Stupid" maxim.

The way the Supervisory Stress Scenarios are defined suggests that very simple models involving only two or three economic drivers will fail to adequately represent the effect of each scenario on the bank's credit losses and pre-provision net revenue calculations. The easiest way to see this is to examine the "adverse" scenario juxtaposed with its "severely" adverse counterpart. Given that the severely adverse installment ostensibly represents a repeat of the Great Recession–though, admittedly, coming without an intervening boom period–a model that tracks a small handful of key performance measures should do a reasonable job of projecting performance under renewed stress.

The merely "adverse" scenario is another matter altogether. This event, while clearly milder than the severely adverse scenario, has a distinct nature that is fully at odds with anything that has occurred in the U.S. economy since at least the 1980s. Put simply, the scenario is stagflationary–elevated inflation and interest rates coupled with a very weak real economy, rising unemployment, and falling house prices.

In terms of only the last two concepts mentioned, the adverse scenario is roughly half as bad as the severely adverse event (see charts above). 

Now suppose that our credit-loss model was built using only these two variables, i.e. housing prices and unemployment.  Many banks that are currently stress-testing are using models with exactly this feature. Credit-loss projections will, necessarily, follow contours that closely match either a linear or nonlinear combination of these two drivers alone. The question is, given this situation: How can stagflation possibly be reflected in the adverse scenario projections developed using these models? To answer this question in a way that even C-level executives will understand: It cannot be.

But inflation has an important impact on many aspects of credit performance. For one thing, the real repayment burden for borrowers with fixed-interest loans will fall as inflation causes nominal income to rise quickly but does not affect repayment amounts. Though the ability to prepay may be harmed as interest rates rise, the desirability of refinancing loans whose real repayments are falling in an environment of rising interest rates is likewise reduced and fewer prepayments equates to a larger denominator for the loan portfolio, further diluting the loss rate. Applying these fairly uncontroversial statements to the problem of credit-loss assessment, we should expect losses, in real terms, to be lower than suggested by the dynamics witnessed in unemployment and house prices alone. This implies that real capital charges, under the adverse scenario, should also be lower than otherwise indicated by the simple model. At the very least, stagflation presents for banks an interesting research question that, clearly, cannot be addressed with a model involving only two concepts.

The apparent desirability of very simple models probably stems from the academic forecasting literature. If I have, say, a probability-of-default model that is built using 10 economic drivers, and I need a baseline forecast for future default rates, I must generate separate or related projections for all 10 variables to compute my output. Since forecast errors will be incurred on all 10 economic forecasts, these errors will accumulate, and my baseline probability-of-default predictions will likely be too volatile and thus unreliable. Under these circumstances, there is a clear imperative to simplify the model, reduce the input forecast errors, and thus generate probability-of-default projections with lower rates of error.

The CCAR stress test, however, is not a baseline forecasting exercise. Rather, the Fed asks banks to produce–conditional on three specific, fully defined economic scenarios–projections of various aspects of bank performance. Under these circumstances, input forecast errors are not possible, since the Fed has told us exactly what economic conditions are going to be. In building a stress-testing model, therefore, we do not need to be concerned with the issue of accumulating input forecast errors. This suggests that the ideal CCAR-style stress-testing model will be bigger, involving many more economic drivers, than the standard baseline forecasting engine.

Building the model with a few more economic drivers may make it slightly harder for humble bank managers to understand the models they are using. It is, however, important that models used for stress-testing be capable of representing the nuances of the questions they are being asked to answer. We feel that the adverse, stagflation scenario is a very interesting test for bank risk managers to apply; it holds the potential to yield deep insights regarding the performance of bank portfolios under stress and management actions that may be available to mitigate the situation. 

These insights would be lost, however, if stress-testing models are kept too simple.

Tony Hughes is a senior director for consumer credit at Moody's Analytics.