As financial institutions chase the benefits of generative AI, a cottage industry has cropped up to provide technology that can test, validate and put risk controls on those models. Companies such as Ethos, Corridor Platforms and H20.ai have taken on clients ranging from big banks to small fintechs.
The potential rewards of generative AI are promising. In this week's
But banks also have to keep any generative AI models they deploy in check. All algorithmic models banks use, AI-based or not, are subject to the Federal Reserve's SR 11-7 model risk management guidance.
"Banking organizations should be attentive to the possible adverse consequences (including financial loss) of decisions based on models that are incorrect or misused, and should address those consequences through active model risk management," a 2011 Fed
Other AI regulations are coming. The European Union's AI Act, which was published last summer, takes effect in August 2026.
"The AI Act, for the first time in my knowledge, is a regulatory act that says you have to automate [AI risk management] capabilities, otherwise they will break down and create issues," Manish Gupta, CEO of Corridor Platforms, told American Banker.
On top of this, generative AI presents risks that banks need to mitigate.
The risks
The most important risk is that an AI model will be wrong, according to Agus Sudjianto, former head of model risk management at Wells Fargo, who is now senior vice president of risk and technology at H2O.ai in Mountain View, California.
"That's always top of mind, because a model can be harmful when it's wrong, either to the institution in terms of reputation or to its customer," Sudjianto told American Banker. "When we use models, we are always taking model risk, and before deploying any model, we need to know all the risks and what to do about them."
For this reason, every bank needs to have model validators who are independent and report to the risk management organization, not the business side, he said.
"Their job is to find what can go wrong with a model," Sudjianto said. "At the end of the day, for every single application, we need to understand what are the risks? What can go wrong? You need to test it to failure, so you know how the model will fail."
For example, if a bank is using a generative AI model to summarize customer complaints, that summary could be wrong. The model could hallucinate, adding information that's not there, or it could give an incomplete summary. People need to find out in which situations the model will hallucinate or provide an incomplete answer, and the harms of both so they can be addressed.
Another risk is that large language models like OpenAI's ChatGPT are so accessible to workers that it's hard for any company to know who is using what, said Kristen Fisher, Ethos' head of risk innovation, who formerly led model validation and model risk management at several financial institutions.
So visibility into user activity is a big challenge. Another is understanding what data is being used in models.
"That explainability, that understanding, where the limitations of that data is and how it impacts your models" is another often missing element, Fisher said.
"You need to educate your team members on what the risks are, being able to get the proper input from all the proper parties and aligned on who's using it, what are they using it for. And then be able to govern that," she said. "And then on the highest level, you want your C-suite to be able to understand those risks and then make strategic decisions that will benefit the company while preventing financial losses."
How tech companies are addressing this
On Thursday, H2O.ai released a model risk management framework for generative AI for regulated industries like financial services. It is intended to help banks and others conduct model testing and evaluation at scale, using artificial intelligence and human calibration. It can work with any large language model, according to Sudjianto.
Having humans test and evaluate models is "very, very time consuming," Sudjianto said. Having large language models do it is not traceable, he said. He advocates for automated testing that's transparent, traceable and explainable, calibrated with human judgment.
H2O.ai has worked with AI practitioners, risk teams and model validators at banks including Wells Fargo, U.S. Bank and Fifth Third, the company said.
Ethos in late February secured $6 million in seed funding for its model risk management system for banks and fintechs. The funding round was led by Canapi Ventures, with participation from Capital One Ventures and Better Tomorrow Ventures.
Jett Oristaglio, CEO and co-founder of New York-based Ethos, formerly worked at AI software company DataRobot with heavily regulated clients including banks.
"I saw the hunger in financial institutions around innovation and also the challenges that they have managing those risks for advanced modeling and AI," Oristaglio said. This led him to co-found Ethos.
Oristaglio refers to Ethos' technology as a unified platform for model risk management.
"We help financial institutions audit their models and other decision making systems," he said. "That includes AI and generative AI, but that also includes machine learning models, more traditional statistical models, and even things like spreadsheets that need to be tracked and part of risk management."
Ethos' model risk management software has what he calls governed inventories — a single source of truth for models, validations and change controls and risk score cards that need to be tracked and moved through a model risk management cycle. It also has a workflow engine that companies use to build, execute and orchestrate custom model risk management workflows such as model registration and model validation, including documentation, testing and reporting to others such as bank directors and a broader risk management team.
Ethos can integrate with monitoring tools that would do things like analyze model outputs for errors, hallucinations or bias. One company that fits in this category is FairPlay, which provides bias and accuracy testing of AI-based lending models. The Los Angeles company is working on developing technology that would help de-bias large language models.
Ethos is working with institutions that range from top 10 banks to $5 billion fintechs that are bank partners. It's building a solution called Ethos Core that it will sell to smaller institutions.
On a larger scale, Corridor Platforms in Haworth, New Jersey,
Manish Gupta, CEO and co-founder of Corridor Platforms, said GenGuardX is in production at two of the Tier 1 banks and several smaller banks and fintechs.
"What we help to do is, once you have figured out the model design and what you want to use it for, the ability to test each and every component, and then how you stitch it together, and then what you need as guardrails, and then test the whole thing, through analytics or through human in-the-loop testing," Gupta said. "Then we let you challenge it in your own safe environment and try to break it. And that is also a very important control these days for large language models, where you can try to break it and give feedback and strengthen the whole pipeline and guardrails."
