Many people are jumping on the AI bandwagon without asking the most basic questions. FICO Chief Analytics Officer Scott Zoldi, who holds a Ph.D. in theoretical and computational physics from Duke University, has been in the business long enough to see the warning signs. Join Dr. Zoldi and Daniel Wolfe, American Banker content director, in a detailed discussion on how to bring accountability into your company's AI strategy while removing risks and biases.
Transcription:
Transcripts are generated using a combination of speech recognition software and human transcribers, and may contain errors. Please check the corresponding audio for the authoritative record.
Daniel Wolfe (00:09):
Hello everyone and welcome to Leaders. I am Daniel Wolfe with American Banker and I am here with a fascinating gentleman, Dr. Scott Zdi. Scott, thank you for joining us today.
Dr. Scott Zoldi (00:20):
It's my pleasure to be here.
Daniel Wolfe (00:21):
Daniel Scott is the Chief analytics Officer at fico and he's also one of American Bankers Innovators of the year. And so before we begin, actually I want to just advise anybody if you have questions, you don't actually have to wait until the end type them in. We'll get to them as we see them. And I really do hope you make this a dynamic and interesting discussion. So Dr. Zdi, can you tell us a bit, you are one of our innovators of the year. Just before we get into the weeds of AI here,
Dr. Scott Zoldi (00:50):
What
Daniel Wolfe (00:51):
Does innovation mean to you, particularly in this industry and in your role?
Dr. Scott Zoldi (00:55):
So for me, innovation is really looking at problems that we'd like to solve in entirely different ways. And so that could be inventing new algorithms, it could be leveraging learnings from different areas and applying it to a new area to solve problems. I look at almost every problem as an opportunity to rethink of how we did something in the past and how can we maybe improve it. And as technology's evolving around us, there's lots of options to do that.
Daniel Wolfe (01:21):
So tell us a bit about your history, your career path and how you got to where you are today, and especially also how that informed your views on AI and how it's implemented in the industry.
Dr. Scott Zoldi (01:32):
So I had an interesting journey. I'm trained as a physicist, so I have a PhD from Duke University in computational theoretical physics. And I was studying chaos theory. Chaos theory is all about finding patterns amongst a huge amount of complexity. I found that fascinating where you work with lots of data and looking for these patterns. From there, I went to Los Amos National Laboratory. I was a postgraduate fellow there. And the same thing, I was looking at complexity. Eventually I got to a point where I said some of the best data to understand complexity and behaviors and patterns where it was actually financial data. And so I joined a company called Agency Software, which was later acquired by fico where we did just that. We were working on fraud problems, examining huge amounts of payment data and looking for patterns of what is likely fraud or what is likely non fraud. And I've been now at fi o for 25 years. So I started as a scientist and now I'm a chief analytics officer at fico. And throughout that time, constantly asking that innovation question about how to make things different or better and what sort of algorithms do we need to create because they don't exist today.
Daniel Wolfe (02:41):
And AI has been a large part of what you've been working on, right? It's
Dr. Scott Zoldi (02:44):
Been almost the entire focus of my 25 years at fico. Yes.
Daniel Wolfe (02:48):
So for those of us who are newer to ai, maybe people brand new, they bought the latest iPhone and they're like, oh, AI can make new emojis and stuff like that. That's not your experience with it. And for those of us who might be joining in and seeing it as something a bit more playful, what can you caution us about? What can you advise us about as, and of course not just for making emojis, but as we incorporate this into our day-to-day jobs, our work, what is your advice to somebody who's newer to this technology?
Dr. Scott Zoldi (03:20):
So AI and machine learning has phenomenal properties, properties to allow us to learn from data in a superhuman way. And so that's where this tremendous amount of opportunity, at the same time, it's very complicated. Most machine learning models don't explain how they work to the user. They can be biased and learn things in data that they should not and it's a good and a bad thing. More of us have access to AI and machine learning, but also means that some of us that have access to this technology may not know how to build models responsibly and we could do more harm than good. And so my advice is essentially to really focus on what is responsible use of AI and machine learning. And for me that comes down to having a responsible AI sort of playbook or standard that we will enforce when we build models. I think the average person that's getting into AI today just needs to understand that it is phenomenal. But if you're going to use it to make decisions that impact people, there's a whole nother level of standards that you need to set yourself to when building those models. Because they're not typically transparent, they can be unethical and they can act in sort of odd and bizarre ways at times, which you need to be able to anticipate or put guardrails around.
Daniel Wolfe (04:40):
So that was a point you raised at the beginning that we don't always know how it works.
Dr. Scott Zoldi (04:44):
And
Daniel Wolfe (04:45):
So as the people who are watching us are developing new uses of ai, what do they need to know? What do they need to be documenting? What do they need to take into consideration so that the users understand how it works, why it's performing the way it is, and what are the pitfalls that you're hoping to avoid in taking that more careful approach?
Dr. Scott Zoldi (05:07):
So one needs to have a AI model development sort of standard or process. And the very first step in that is to understand the data on which you're training the model. And many models that exist today, people don't know the data that was used to train it. So they don't know what the model learned. And at the end of the day, an AI or machine learning model is just a representation of data. So if you put in data that is biased or has errors in it, so will your model and it'll reflect that, right? And so the very first step is document exactly what data you're using, right? Second step is what are you trying to achieve with that data and do you have the right sort of outcome? So if you want to predict fraud versus non fraud or credit risk versus not credit risk or someone propensity to buy something or not, you need to make sure you can label that data appropriately.
(05:55):
Once you have the data prepared and the problem sized up, then it comes down to selecting the algorithm. And this is critical today, many people, we look at large language models and all that they can achieve. There's also deep learning models, there's traditional neural network based models. You have to choose the algorithms that are going to be meeting your business need. And so for many of us in financial services environments, those are interpretable machine learning models, models that are transparent and we can go test what was learned and understand if it's exhibiting bias or not. But beyond that, selecting that algorithm, then we also then have to put in the right sort of requirements around testing for bias, testing for explainability. And then it has to be recorded. And so for me, for example, at IO we have an AI blockchain and that's how we do it. So each of these critical decisions are made recorded to the blockchain so that years from now we can go look back and know exactly what were the decisions that got made and what's baked into these models. Otherwise they become black boxes and that's not going to be responsible in a business context.
Daniel Wolfe (06:57):
And you have a patent probably more than one relating to the blockchain in this context.
Dr. Scott Zoldi (07:02):
We have two in AI blockchain. So for us what we call it's AI blockchain, and it's around model development governance and it's around AI auditability. And so I think one of the things that people don't always recognize is that sometimes we go and build AI models and it's like, I dunno, it's like back in school, it's like a classroom experiment, but the severity of the decisions that get made require that level of sort of scrutiny. And so we get down to this concept of governance and audit very, very quickly. And so yes, two patents, those areas more to come because I think it's a really critical part of establishing trust in the use of these models both by the businesses that want to use them but also by the consumers that are impacted by them.
Daniel Wolfe (07:46):
And so the innovators package that we published, which you were part of that came out I think something like five months ago. And it feels like this space moves so incredibly fast. And I do encourage people to look up that story and that package and read about what we were discussing then What would you say has changed in those five months that is relevant to bring up now that you couldn't get by just looking at our past coverage?
Dr. Scott Zoldi (08:14):
So one of the things that has changed is AI regulation is becoming more and more concern across the board. So we have the EU AI Act, we have our own state legislations that are coming up with AI regulation. And so this concept of being held to a standard and demonstrating that a standard's met means that many organizations are now focused on this concept of how do I meet regulation or do I anticipate regulation and am I taking the care and steps necessary to document that work? So that's one major change and I think I'll continue to strengthen because AI trust is really at a low right now. People are concerned about the UCS technologies. The other major thing that's changed in those last four or five months is that where we are with these large language models, we've all been very impressed with 'EM for a couple of years, but more and more we're getting to this concept and something that I'm driving from a research perspective called responsible generative ai. How can we make sure that we can apply the concepts of responsible ai, which is around robust models, explainable models, ethical models and audible models to generative ai. And so there's a big focus on people understanding that the current generation of large language models will have to evolve to something that is more responsible, more safe to use in a business context. And I see a huge amount of research in those areas right now.
Daniel Wolfe (09:41):
So not so much about generating emojis on our phones,
Dr. Scott Zoldi (09:44):
Not about the emojis.
Daniel Wolfe (09:46):
So another thing in terms of what's changed since then is today I was watching President Biden's address to the general assembly of the UN, and he made AI a big theme of that, whereas in his previous address in 2021, it got barely a passing mention. And he said, with just such gravity, there may be no greater test of our leadership than how we deal with ai. So do you think that's an overstatement understatement or how do you see this now on a global scale that it's not just technologists who care about it, but political leaders or just whoever else the audience is for that sort of address?
Dr. Scott Zoldi (10:26):
So I think it's spot on from my perspective, what has changed over time is just to access this technology. So a couple of examples, our reliance on the fact that these models were built with real data and correct data means that we have this concept of what is truth in data. And so this is a big issue. We have a lot of misinformation and DeepFakes and other sort of technology where entire sort of groups of citizens can be misled, right? Because they're all dependent upon on an output of an ai, which may or may not been trained properly or might have data injected into the training corpus to change a perspective or to provide a particular political view. So I think it's a very serious concern for society. Moreover, we see we build a lot of fraud models and we protect a lot of the world from fraud and scams that people are being scammed expertly because these criminals have access to this technology. And so we see a lot of AI that can be used for not good purposes, and that's why AI regulation is starting to bubble up. And I think that's really where we are as are we going to put some boundaries around it, sensible boundaries, ones that we can understand how to limit its use to make sure that we don't take those harms. But having this of concept that we're going to just use the technology without those restrictions around is probably one of the biggest concerns because people will push the boundaries.
(12:06):
I think we're all tired of let's break it and we'll fix it later. When that impacts political sort of discussions or outcomes, that's pretty serious business. So I think there is a crisis here and has to be addressed.
Daniel Wolfe (12:18):
So the flip side of that is the people who are misusing this technology, they don't care about the boundaries, they don't care about the regulations, they're going to do what they want to do and any fraud or anything like that, it becomes an arms race. So how do you compete with that when you have to follow rules that the bad guys or won't?
Dr. Scott Zoldi (12:39):
Yeah, Daniel, it is a great question and it's a fair question. I think one of the things that I would say is that when I've seen regulation and other things applied to different domains, I see better innovation. So coming back to that innovation theme, if we say, listen, we want to solve this problem, but we want to do it in a way that we can understand it and we can audit it and make sure that it's done faithfully correct, technology will evolve maybe a little bit slower than let's say those that are using it in an irresponsible fashion. But we have to hold ourselves to that standard and we will evolve to that point. Moreover, we are not spending enough time rooting out where badness is. And so there's a lot of other work we can do to bring it right to those that are using AI and in responsible ways or in harmful ways, and kind of targeting that and not just having these two sort of moving in parallel, but strengthening one so we can make sure decisions are made safely. And then second, finding ways to maybe audit that information that's being generated to make sure that it's faithful.
Daniel Wolfe (13:43):
And speaking of world leaders like the US president, another prominent world leader, Taylor Swift
(13:49):
Recently had a very compelling statement on ai. She released her own presidential endorsement and as part of her message on that, she said that she was concerned about ai. What she wrote on Instagram was that seeing her image misused through AI to suggest an endorsement she hadn't made, brought me to the conclusion that I need to be very transparent about my actual plans for this election. The simplest way to combat misinformation is with the truth. Now for Taylor Swift who has a massive following, everybody knows her verified Instagram account, it's easy for her to get the truth out, but what about for somebody who's not public facing? What about for somebody in the banking and payments industries who is combating misinformation as just part of their day-to-day? How do we make sure that the truth prevails and how do we make sure that the information we're working with even is the truth?
Dr. Scott Zoldi (14:42):
Yeah, yeah, it's a tough one. I think one of the things that we want to do is make sure that we are questioning what we're seeing. And so for example, hallucinations for some of these large language models can be as large as 10% depending on the questions you ask, and particularly in the legal spaces. And so one is when we're leveraging tools like large language models, we have to be constantly aware of the fact that there's mistakes and hallucinations made, which goes back to having this sort of validation of information. And so that's one of my biggest fears is that people just go on autopilot and they'll just use the technology and not question it. So that's one we should not secede our decision-making just to the ai. We have to have this human and AI sort of relationship. The other aspect of this is around establishing for certain types of transactions, let's say in payments, what is truth?
(15:35):
And part of this would be things like using blockchain technology to say, okay, this is from a verified source that this is the correct balance in s Scott's account, or these are five transactions that summarize the way this payment card was used in the past. Those we can establish through these sort of contracts of truth through blockchain. And I think more of this sort of focus on what data can I trust and not trust and make the decision, we will start to leverage things like blockchain, which is I'm a big fan of because I think it does establish that sort of credit one source of truth, which has always been a challenge all the way for the last three decades. This concept of why do we have chief data officers? One of the reasons we have chief data officers is because trying to figure out what is the data that's allowed to use, how many copies are there, what's the most fresh data? Is there data being injected? And so I think part of that is to say we have to, for certain types of individuals or certain types of transactions, we need to establish what is the one source to get that information from Scott, because there's probably going to be lots of other sources that may be inaccurate or even just outdated, but still has the same effect.
Daniel Wolfe (16:39):
And I like what you said about the human being involved in the decision. As a human myself, I find that reassuring.
Dr. Scott Zoldi (16:45):
Yeah, I think there should always be a role for us. If we're going to secede that, then I think the society and the way we interact will be very different. But in all seriousness, I think people that will win in this space will find the right combination. We're still in this sort of space where FIO, we launched our first AI machine learning model, excuse me, in 1992. It's been working seamlessly in protecting payment cards for fraud for more than 30 years. We want to get every AI application down to that aspect, which is we have a machine that does a certain specified task, it's been refined to do it very, very well. And then yeah, there's a huge amount of humans that are looking at what that machine outputs because in fraud, for example, I can detect fraud, but also there's lots of false positives. So there's always that sort of role. And so we just have to figure out what that sort of augmentation looks like and it's not going to be one or the other. And we're still trying to find that happy medium between the two human and the ai.
Daniel Wolfe (17:49):
I do encourage, again, folks who are watching to submit questions to us. I am eager to see what they are. But before we get to that, I want to switch gears a bit, and I was hoping you could tell me a bit about the F ICO Educational Analytics Challenge. It's in its second year now, right?
Dr. Scott Zoldi (18:02):
It is, yeah. It's something that's really, really important to me. So the F ICO educational challenges is F ICO's recognition that we need to see more diversity in data science. And so we are focused, this challenge is a three year program and we're focusing on HBCUs. And so the concept is is that we work with the historically black college universities to bring in students that are interested in data science or in a major around data science and to get them to work on real world problems. And the concept is that I give a lecture there and then we mentor the students through a semester long project to really give them that hands-on sort of experience with what it means to build AI that actually gets deployed and that actually sees all the real world problems that they're going to encounter as they become practitioners in data science.
(18:55):
Last year, year one was all about racial and bias within lending data. A very serious sort of problem today is to go look at this historical data. And part of it was not to say that there are issues with that. I think people generally are concerned about ethics associated with lending, but it was also to get their opinions on what to use, what not to use, and also to get their understanding that it's really tough. It's really hard to solve that problem in one go and it might be an iterative process. So we did five different classes last year. This year it's about fraud and it's around operationalization of ai. So one of the biggest problems we have in the industry, Daniel, is that we developed these AI models. People can't operationalize them. And so today, like FICO has an analytic decisioning platform that allows us to run these AI models at scale, but many of those machine learning models can't be deployed and operationalize. And so we want to have the students focus then on what does it mean to operationalize ai? It's very different than what's in the classroom. And so this is about getting that diversity of thought and focus on getting more and more of the HBCU students to take careers in data science and really make sure we have a representative voice in ai, which is incredibly important.
Daniel Wolfe (20:14):
How many colleges are involved in this?
Dr. Scott Zoldi (20:17):
So we have three this semester, and they are Morehouse, Louis State and Delaware State. And then we're going to have another four next semester.
Daniel Wolfe (20:29):
Wow. I remember when you first told me about this, I thought, this is a great idea. What data are you working with? Maybe we can do some kind of internal data journalism project on this and train people to use data better. And then I was told that that was way, way, way, way reaching way too high. I just want to emphasize this is no small task that you're asking these students to take on.
Dr. Scott Zoldi (20:51):
It's not, it actually challenges them a lot. And so we didn't hold back. We had all the same sort of questions like, wow, the data's huge and it's messy and it's dirty, and yeah, it is. That's what it is. That's what life is like when you're data scientist. But you know what? Their creativity, their energy will allow them to pivot and to learn from it. So I have this sort of view that let's make sure they have a clear view of what those challenges are and we'll help them overcome it. And that's why we have mentors that work with these students and they're members of my staff that are professionals solve these problems day in and day out. And those problems start to dissolve a way and they see path forward. But yeah, it is a serious sort of project. We want to make sure that they have that real world experience.
Daniel Wolfe (21:34):
And you don't sit still, you were just traveling for the past, what week? Two weeks, three weeks, five weeks, five weeks touring these colleges for future participants or current participants?
Dr. Scott Zoldi (21:46):
Current. And we're focused on who's going to be there in the spring and then also defining what's going to be year three and year three. I don't know what it's going to be yet. It might be responsible generative ai, but we'll see. We will see. That's
Daniel Wolfe (21:57):
A good one. That's a good one, a good one. It's timely. So, so again, I encourage folks to submit questions, but as we are starting to run towards the end, I wanted to ask what should I have asked that I haven't yet? Is there anything that you think is important for our audience to know about these topics or specifically how to apply them to their own jobs, these lessons?
Dr. Scott Zoldi (22:20):
I think one of the things that we could talk about is just simply what is each organization's sort of readiness for the future? So President Biden talked about AI and how important it is that we look at that as potentially a threat and something that we need to manage. I think every business has to sit down and ask themselves the question, how do we build AI models today? And what is even AI in my organization? And I think that's a big issue. And so who's in charge? Does every data scientist get to build it the way they want to or do we have a standard that's applied? I think as regulation or even if regulation doesn't come and doesn't impact a particular financial institution, just public opinion about the use of AI continues to be a concern. Each organization has to say, well, are we in a good position or not a good position?
(23:12):
We did a survey a year or two back, and more than 50% of financial institutions did not have an AI standard for how they could develop it. And that's a lot of risk. And so to get prepared is to basically go and say, this is how AI models get developed today. These are the people that are in charge. Do we have a standard or not? And if you don't start developing it, right, because once some of the regulation and other things start to hit, right, it'll be very, very critical that you have that standard. But moreover that you can apply and enforce a standard.
Daniel Wolfe (23:43):
So just to be clear, they're using AI but they don't have that standard.
Dr. Scott Zoldi (23:47):
Correct?
Daniel Wolfe (23:47):
Correct. Okay, that's scary.
Dr. Scott Zoldi (23:49):
Yeah, it is, right? Because Jack versus Jill could build a model in very different ways, and this is software, I think this is what people don't understand is software engineering has much more structure around it than AI development. And I think there's this perception, let's just put the smart people in a room and they'll come up with this marvelous model. But no, they have to be held to the same sort of standards. And I have more than a hundred data scientists that are working on models. I can't afford to have each of 'em build it a different way. In fact, we have to at fi CO, make sure that we have that standard and these are the three algorithms you're allowed to use and the rest you're not. And if you want to use a different algorithm down the road, that's great. That's a research sort of direction where we'll explore that innovation and decide if we bring it into the production fold and we have to stop treating AI as a research project, we have to treat it as really around operationalization of that technology in a way that meets the standard.
Daniel Wolfe (24:43):
Okay. So one thing that I find interesting is just as we have been reporting on banks' uses of ai, we get some more creative stuff. A lot of folks are focused on fraud as well, they should be.
Daniel Wolfe (24:54):
But
Daniel Wolfe (24:54):
Every now and then you see a more inventive use that I don't think any of us would've predicted. The one that I had brought up before the cameras went on was Commonwealth Bank of Australia. They had noticed that a lot of the messages and digital payments were indicative of potential domestic abuse or financial abuse of people who were as one side of that transaction.
(25:18):
And it's not something that's usually explicit. It's something that you need to observe as a pattern, which of course is what AI is good at. And so they developed a tool that would actually allow them to highlight and block transactions. And as of when we covered this, which was actually close to a year ago, they had already blocked a million transactions that they had flagged as indicative of domestic or financial abuse. And this is something that they shared with the community. Any bank can use the same system. And I don't know how that would even be possible to be able to identify at least at that scale by hand, any sort of thing. And it's not even something that I normally think of as a bank's responsibility, although of course people at the branch level will be able to spot certain things and maybe try and nudge people into accidentally, or not accidentally, but unintentionally withdrawing their life savings to buy iTunes gift cards for somebody who told them that they were their boss or something like that. So within these constraints that you advise people to use, is there still that opportunity for creativity to apply this technology responsibly in ways that may not be as traditional as what we think of as a banker or technologist role?
Dr. Scott Zoldi (26:35):
That's a great question. And the short answer is there are, right? So we have this sort of technology called interpretable neural networks that provide that visibility to what's being learned in new data. The use case that you mentioned is an excellent one. Another one that I focus on is financial inclusion data from open finance, for example. And if you think about it, a lot of the problems we solve, we've been solving for 30 years. We know what are the right sort of features to use and how to perfect that score. But when you bring in new types of data, that's pretty smart, looking at the memo information and try to understand whether or not there's domestic violence associated with the payment, that's a new data source. And machine learning plays a great role there because it can figure out what are the words or sequences that might be more prevalent for domestic abuse or not.
(27:23):
When that's learned, we need to choose technologies that expose it so that we can make sure that we agree with that assessment. But fortunately, a lot of my work over the last seven years has been around this sort of responsible AI and temporal neural networks, which will expose what those learnings are. And as long as we have that visibility and transparency where we can explain what's being learned and we can go look at it not just data scientists, but lawyers and business people and say, yeah, this all makes sense and we're going to move forward on it, and we will audit it and we'll record it, then we can essentially have our cake and eat it too. We just have to choose the right algorithms and have the right process around it. And then the world is kind of the oyster because there's lots of data sources that we just haven't even explored yet.
(28:08):
And the problem statement you just mentioned is a phenomenal one. And so you can ask questions to this data and then use machine learning to find those patterns. And as long as it's exposed and you know what it's doing right, there's tons of new problems we can solve with the technology. And I think that's really the promise for AI and machine learning in general. We just want to make sure, I want to make sure that while we solve those problems, we're not doing it in a way that causes harm so that people back away from AI and machine learning, which some financial institutions are.
Daniel Wolfe (28:37):
So we're at time, but I did want to get to this audience question that came in while I was asking my last rambling question. Can you share any best practices for preparing documentation repositories to optimize their use in knowledge management generative AI applications particularly to ensure the quality and relevance of the data?
Dr. Scott Zoldi (28:58):
Okay, yeah. So the way I would answer that is that we want to make sure when we use documentation, whether it be for things like context in a large language model development that we spend a lot of time curtailing the data. So for example, a lot of times there'll be a document and people will embed this document in a vector database and it's entire document, but most things, if you read most documents, there's a few sentences that really matter or maybe three or four, and we want to capture those. And so the quality of use of these documents, and to do it in the best possible way is to really go through that document and figure out what passages have the right context. And so that means, again, human in the loop again
Daniel Wolfe (29:42):
To
Dr. Scott Zoldi (29:42):
Go review those documents. Or maybe use smart AI tools that can make suggestions around what those contexts are that are really important, but then to embed those pieces because the rest just adds noise in terms of what that value is. And so I think it just comes back down to if we think documents have value, let's figure out what in those documents have value, let's capture that information, embed that information such that it can be retrieved in terms of let's say a use in a large language model, or maybe in the way it would answer the questions around that documentation. Otherwise, we were just going to be taking all the worlds sort of documentation, embedding it, and hoping something matches. And generally that's where things start to fall down. So be more prescriptive on what you're trying to do with that model and curtailing the context in that document that matters is probably one of the best ways to do with that.
Daniel Wolfe (30:30):
Alright, so I think we're over time now. So I wanted to thank you very much for this conversation for coming out here to New York to engage with us and I wanted to thank everybody who participated as well remotely. It was a great experience and I hope you enjoyed it too.
