Why Cleveland Fed should retract its online lending study

The Marketplace Lending Association is calling upon the Federal Reserve Bank of Cleveland to temporarily retract and revise its report on online lending due to what we see as serious flaws in the authors’ reliance on certain underlying data.

In our view, this paper — “The Taste of Peer-to-Peer Loans” — and its accompanying materials show that a lack of precision and understanding of subject matter can result in significant inaccuracies. The report’s authors presented findings that seemed to reflect issues with the P-to-P industry, but they actually relied on data from a much broader category of loans. The result was a misleading and brutally critical report about the P-to-P industry that was actually based in part on data from more traditional loans.

The Cleveland Fed has admitted in a written statement to American Banker that there may have been issues with the way the report was branded as an examination of the P-to-P industry.

“Our goal was to contribute to research around this dynamic industry, but we understand use of the term ‘peer-to-peer’ as shorthand for a set of loans originated by peer-to-peer and other online lenders has led to some confusion around the study,” the Cleveland Fed said in the statement.

But the authors have still not yet changed the name of the report.

The researchers must acknowledge that the loan group they studied was not an exclusive set of online loans that could be fairly separated from, and compared to, more traditional loans. And TransUnion, which provided loan data for study, has now confirmed that the data set that the researchers drew from did actually include traditional loans.

A recent American Banker article quoted Ezra Becker, a senior vice president at the credit bureau, who “said that TransUnion sent the data to the Cleveland Fed a long time ago for a different study, and that none of the data the firm provided distinguished between peer-to-peer loans, so-called fintech loans and traditional personal loans.”

In the article, Becker said, "We have no understanding of how the Federal Reserve Bank of Cleveland could have used our data to reach the conclusions they did," and "We need to talk with [the Cleveland Fed] further to better understand their methodology."

The credit bureau representative “added that TransUnion has conducted its own research on fintech loans, and that its findings do not align with those of the Cleveland Fed.”

Our belief is that the Cleveland Fed relied on a data set covering 2007 to 2012 that was largely made up of traditional storefront personal loans. The researchers appear to have grabbed 90,000 of these borrowers for their analysis, labeled them peer-to-peer with no evidence that these were peer-to-peer or even online loans, and then released a misleading report criticizing the relatively new peer-to-peer lending industry with their findings.

Using data on traditional lenders makes sense if you are doing an analysis of different types of traditional lenders (traditional bank versus traditional nonbank loans, for instance). But the point of the study was to compare a group of online peer-to-peer loan borrowers to a group of similarly situated consumers who did not take out such loans. Yet that exercise would be pretty futile if the data set contained traditional loans in the peer-to-peer group, which is apparently what happened in this case.

The researchers never checked with the online peer-to-peer lending industry to strengthen their sense of the size or scope of the industry, or to ask any other clarifying questions. Anyone from the industry would have pointed out to them that data points from the working paper are off by such a large magnitude that it clearly represents a broad consumer loan market, including many traditional loans as well.

For example, the report included a chart titled “Growth of Peer-to-Peer Lending” with data from TransUnion. The key data point in the chart was for a broad category of “Consumers with a personal loan.” In reality, the actual P-to-P loan balances covering the same period, according to publicly available information, were just a fraction of that total. For example, the chart shows the industry with $45 billion in loan volume at the end of 2012. In fact, the real outstanding peer-to-peer loan balances at the end of 2012 (when the peer-to-peer industry was effectively just Lending Club and Prosper) were roughly $1 billion.

The number of loans is also off by a similar magnitude. Beyond TransUnion’s public statements, this is the other way we can observe that there is a data set problem: for the researchers’ “peer-to-peer” data set on these loans to have been so large, the data must have actually included lots and lots of traditional finance company data. There simply were not $45 billion of online consumer loan balances in 2012, a time period when people were still operating the original iPhone and most of the largest online lenders didn’t even exist yet.

It is ironic that the data set the Cleveland Fed used likely included some of the exact products that peer-to-peer loans seek to disrupt by providing better, lower-cost online credit options.

What’s more, the study is in conflict with findings by other Fed districts. The Federal Reserve Bank of Philadelphia and the Federal Reserve Bank of Chicago recently conducted a study focusing on Lending Club — the largest marketplace lending platform — and reached opposite conclusions from the researchers that authored the Cleveland Fed working paper.

The earlier study, titled “Fintech Lending: Financial Inclusion, Risk Pricing, and Alternative Information,” found that for a given level of risk, Lending Club charged borrowers lower prices than bank credit cards. Here is the key quote from the Philadelphia and Chicago Fed researchers:

“The results indicate that given the same credit risk (i.e., for borrowers with the same expected delinquency rate), consumers would be able to obtain credit at a lower rate through the LendingClub than through traditional credit card loans offered by banks.”

But the damage has already been done by the Cleveland Fed’s study. The peer-to-peer lending industry’s reputation has been unfairly tarnished. The standard for accuracy in research associated with an entity as well respected as the Federal Reserve cannot be so low that it unfairly impugns an entire industry.