JPMorgan Changes VaR Model Again, But It's Still a Black Box

Tucked away in a footnote to the financial supplement to JPMorgan Chase's third-quarter earnings release was the revelation that it made yet another change to the value-at-risk model for its chief investment office and investment bank portfolios.

The way in which the company reported this change says a lot about the lack of transparency that continues to cloud our understanding of the risks of our largest and most systemically important financial institutions. It also signals that our reliance on complex models of socioeconomic phenomena must be tempered by the reality that such tools are only as good as the data and assumptions underlying them. 

JPMorgan reported that the investment bank and credit portfolio VaR using the new model was now lower by $28 million than under the previous VaR model, which implies about a 19% drop in the combined VaR for the investment bank trading and credit portfolio, to $122 million.

Very little additional information was forthcoming about the drivers of this change, and there lies the problem.  The company stated that it "believes the new model … more appropriately captures the risks of the portfolio."  We have to take JPMorgan at its word on this, for there is nothing else to go on in the disclosure.  But switching out three VaR models in less than a year raises a number of issues regarding not only the stability of such models but also how risks across such institutions can be compared.

First, comparing the changes in the model over time is impossible, as JPMorgan did not restate figures from earlier periods using the new VaR model.  Some will rightly point out that the impact of the change is inconsequential; after all, we are talking about a relatively small dollar value for a company of JPMorgan's size.  Whether it is $100 million or $100 billion is not the point as much as making sure investors and other relevant parties have confidence in the stability and accuracy of such risk measures and that they have some means of comparing them across firms consistently.  And it isn't just VaR models where such a lack of consistency exists across the industry. 

With the industry in the depths of the financial crisis, models used to estimate loan loss reserves were scrutinized carefully by external observers for signs of weakness among firms.  But the underlying assumptions of such models, such as the confirmation period for a loss event, are not fixed according to Generally Accepted Accounting Principles. Rather, the period can and will vary across companies.  This and other technical differences make it extremely difficult to compare loan loss reserve estimates across institutions.

That's why Vikram Pandit's call for a bank risk index approach makes a lot of sense.  The (former, as of Tuesday morning – Ed.) Citigroup CEO suggested that regulators create a model portfolio of hypothetical assets, for which banks would have to disclose their risk assessments – essentially showing the public how they think, so investors could compare banks' methodologies on an apples-to-apples basis. (Full disclosure: I worked at Citi in risk management from 2008 to 2009.)

We already see some movement toward achieving model consistency in the Federal Reserve's stress test requirements of the largest bank-holding companies where the same assumptions are applied and where the Fed has been developing its own models to assess these risks on a comparable level.  Still, even here, the Fed follows the same path as the industry to hold back on providing more details on key assumptions, parameters and relationships used in estimating bank risks. 

It is not sufficient to provide these risk estimates without having more information on the sensitivity of the results to changes in key economic variables.  The VaR model applied now by JPMorgan may very well be appropriate for measuring risk under some market conditions, but is it susceptible to major errors if markets move just a little from where they are today?  We simply do not know. 

The industry's track record with risk models in general during the crisis has been poor in part because they rely so much on historical data, which becomes less meaningful when market conditions change abruptly. Credit default and property valuation models for instance were notoriously off in their assumptions regarding the level of home prices.  However, the level of conviction surrounding the validity of such models was no less present in 2004 than it is today. And, the lack of model transparency and consistency in assessing risk across the industry remains about where it was back then as well. 

Given the increased reliance on models for risk management at our largest banking institutions, investors and other industry observers should demand greater disclosure of the key assumptions, inputs and sensitivities used in these models as well as efforts to bring greater consistency in the assessment of risks across firms.

Clifford V. Rossi is the Executive-in-Residence and Tyser Teaching Fellow at the Robert H. Smith School of Business at the University of Maryland.