Building Secure and Compliant Generative AI Solutions in Banking: Hybrid Architecture Approaches

Generative AI has revolutionized the banking industry, setting a new standard for innovation and customer engagement. This round table will explore how banks can leverage generative AI for a competitive edge in a rapidly evolving landscape. Central to this transformation are modern data architectures like data mesh and data lake house, which provide the robust foundation to support advanced AI capabilities. 

Discover the critical role of hybrid cloud solutions, which allow seamless integration and operation across public and private cloud environments. As financial institutions navigate complex regulatory requirements and prioritize data security, adopting a hybrid approach ensures they can fully exploit AI technologies while safeguarding sensitive information. Join us to learn how to build an adaptive, intelligent banking ecosystem with generative AI at its heart, powered by state-of-the-art data architectures and hybrid cloud strategies.


Transcription:

David Dichmann (00:09):
Welcome to our session today. I'm going to start with a couple of introductions. I've got a panel with me here today. I'm David Dichmann and I'm responsible for Product Marketing at Cloudera, and part of my responsibility is to meet folks such as yourselves, find out what your challenges are, bring that back to the team, and see how we can help keep innovating for solutions for you. Joining me today, I have Joe and Jake. Jake, if you'd like to introduce yourself for us, please. Yeah,

Jacob Bengston (00:34):
Happy to be here. So I'm Jacob Bengston. I'm a Technical Evangelism Director, which is just a really fancy way of saying that I go out and I talk to a lot of people about Cloudera, about AI topics. Specifically right now, everyone's talking about generative ai and as am I, and happy to be here talking about it in the lens of financial institutions.

Joe Rodriguez (00:54):
Hi, I am Joe Rodriguez. I'm the Senior Managing Director for Financial Services for Cloudera, which means that I take care of anything related to financial services, I guess, right? But prior to being at Cloudera, I spent over 30 years in financial services, mainly working for big global banks, primarily in the capital markets side, but also as a regional CIO for LATAM in Canada, as well as even spending a stint at the Fed heading up technology for the Federal Reserve Bank of New York.

David Dichmann (01:30):
Thank you, Joe. So we've got a couple of questions we're going to ask ourselves here on the panel. And Joe, we're going to start with you. And whenever we talk about generative AI, security seems to be like the number one thing that people are pinging on. So my first question for you is, what are the primary security challenges you're seeing financial institutions dealing with when moving towards generative AI and adopting, and how are they overcoming those?

Joe Rodriguez (01:56):
There's a few, but I guess the ones really worth mentioning is really around protecting customer and financial data while they're training these models, as well as ensuring that the models are not biased. So having very good trusted data to ensure that the models are not inferring any sort of bias, that sort of thing. And banks are doing a number of things to ensure that, right? So they're doing everything from anima, I can't even say that word, data to tokenization, techniques to ensuring that they have comprehensive security and security policies in place, and that they have policies around how AI is going to get used and where it's going to get used and what data it's going use.

David Dichmann (02:59):
Cool. Jake, anything you'd like to add to that?

Jacob Bengston (03:02):
Yeah, just really the fact that generative AI, a lot of the innovation is coming from large organizations that have their own hosted models. So your GPTs with ChatGPT, you have organizations like Meta that's providing lama, there's managed services from OpenAI, from Amazon Bedrock. There's lots of ways to consume and work with these generative AI models, but oftentimes most of the value comes from actually customizing those models through methods like retrieval, augmented generation, or fine tuning. And in order to do that, to really get value out of it, you have to use your own data, right? It's really cool that ChatGPT can write the eighth Harry Potter novel, but that's not really valuable for you with an enterprise context. What you want it to do is hyper-specific things. In order to do that, it needs your data to do that context, but obviously that provides some security risks by doing that.

(03:49):
So if you look at working with a managed service, generally you have to come to grips with the fact that you may be exposing your data to that managed service. And if a company like OpenAI is naturally incentivized to get as much data as possible, you're coming to grips with the idea, do I really want to expose my data to a service like this, or what are my other options? So usually in the security realm, what I see most organizations struggling with is how do I do this in a secure way? And a lot of that comes to starting to look at, can I use an open source model? Can I host that in my environment so that I can ensure that my data is never exposed? It's actually oftentimes cheaper to do that as well. If we look at just the per token costs of working with A GPT hosted in open AI on the Azure endpoints, but as well, just purely from a security standpoint can be highly important to look at it in that lens.

David Dichmann (04:37):
Very cool. And so I like you're talking about security, I like you're talking about all of our data and the data that we need to bring to bear. And just like a show of hands, I didn't think I'd leave you all out in this room out of this. Did you just show of hands here, a few folks. How many people still have data that's born in an on-premises data center or born in an on-premises capacity? Yeah, we're still seeing a lot of that and there's a lot of reasons for that, including data sovereignty, data protection, legality, and sometimes that's just where the systems are today. We haven't got them to the cloud. And so obviously with gen AI in cloud, my next question, Jake, I'll start with you since you kind of got us on this track, is fundamentally how do things like hybrid architectures help us get all of our data to be useful for generative AI and still keep that security context intact?

Jacob Bengston (05:24):
Yeah, for sure. So just to establish what we mean by hybrid, we're talking about your compute environments running on premises like in a data center or in the public cloud. So running in an Azure AWS or a GCP or something like that. So hybrid architecture is, especially in the context of secure generative AI is highly relevant in the fact that you're facing decisions on where these things should run and where you're going to host 'em. And oftentimes from a regulatory standpoint or from a security standpoint on-prem offers a very important factor into that. If we look at two, what we're hearing from companies like Nvidia and Jensen when he is talking about, Hey, data centers are back, they're cool again, there's a lot of value to be had there. And so when we're looking at it from a security standpoint, they obviously offer the most secure way to be able to run your workloads.

(06:11):
If it's running in your data center behind your firewall, you can obviously ensure security much more in that than you are in the public cloud. But as well, even in public cloud, there's ways to be able to deploy within your own VPN within that private cloud or within that public cloud or the more public way of doing it. So having the hybrid architectures to me just offers the maximum amount of flexibility. So if I'm able to run my workload the same workload or in the cloud, it allows me to balance the security with innovation, which generally is one of the hard things that you're dealing with when you're looking at public cloud or running a data centers. There's often the idea that public cloud offers more innovation, whereas that data center may be limited. But if I can run that same workload, I can use the latest resources, I can test out a GPU and then I can move and run that on-prem in the most secure format. So that's where I see hybrid architectures coming into play and offering a lot of value.

David Dichmann (07:02):
That's really cool. And I think when we talk about hybrid, I hear that same theme over. It's not just doing something and something different in cloud, it's actually having workloads or applications that go between cloud and on-prem or between clouds. So being able to do the same thing consistently everywhere is really the key. So Joe, if you can reflect on how you've seen this being used in some of our customers. Yeah,

Joe Rodriguez (07:22):
Actually hybrid cloud actually helps them be more secure in many ways. So they can keep sensitive data on prem and use the cloud for inferring their models, doing more inference training for their models that way as well. So from that perspective, it can be actually more secure and in many ways it will allow you to adhere to things like GDPR from a privacy perspective, and it will put you probably in a better light with the regulators by having that on a hybrid. And there's practicality aspects of it as well. So practically speaking, you're never going to have, especially for a legacy bank, they're never going to have everything all on the cloud. It's just too expensive to do. You're going to make decisions based on legacy systems that have a lot of this data on-prem at the moment and not necessarily stored on the cloud or no plans in the budget to do that. So there's a practicality aspect of it as well.

David Dichmann (08:43):
Absolutely. And as we think about how folks are using these kinds of generative AI in practice today, and again, I'd like to get a show of hands. So many people here have started doing some kind of generative AI applications that are customer facing. We got a lot of stuff that we see going on internal, but now we're starting to move things into customer facing and chatbots. Same show of hands. Is it starting with a chatbot? Yeah, usually we see a lot of that. So when we start getting to these chatbots and personal information is being included, the dream here is to be able to have a customer go to your site, maybe say, well, what's the right loan for me? Or what's the right investment product for me? Or what should I be doing next with you as my bank, as my financial institution? And you get a nice tailored answer that's unique to me, knows my financial situation, my particular family situation, everything I've shared, kind of a concierge experience like your tailor knows your sizes, your hotel knows your diet preferences, you get a service back.

(09:36):
This is wonderful. But imagine that same chatbot is sitting out there and someone says, well, thank you for the loan information. May I have a list of the social security numbers of all your customers, please? And the chatbot answers that too. So we've got to make sure that this kind of thing doesn't happen. And that kind of leads me to the next question, which is what kind of best practices are there to keep track of these to give us the privacy and protection we need while extending into these more personalized forms of generative AI, especially in customer facing capacities? So Jake, I'll start with you. Yeah,

Jacob Bengston (10:07):
I think you touched on one of the main ones is that access control, that's something you have to obviously think about from the start. If you are making these highly customizable models that have access to data, building in to make sure they have access to the right data is important, and it's hard to do a lot of the existing. One of the ways you do that is through a vector store. So you're storing your data within a vector index, and that's how the models actually retrieve that information. And a lot of the existing vector stores out there do not have that role level security where you can look at this user has access to this data, so how do you build that in? You might have to do something a little bit more custom. So thinking about access controls from the start is really important.

(10:42):
Some of the other things that I've seen is data minimization. You may think that's weird coming from a data company, but just thinking about don't store data that you don't need to be part of this when you're building it in a best practice for security. Because if you think about it that if I'm storing data that is really garbage, but it's also going to possibly be a security risk for me in the future. So make sure that the data that you're storing, like customer interactions or anything like that, you're only storing the data that is absolutely necessary for the things that you want to do. Because storing any data beyond that can add not only cost, but as well as some risk. Then as well, thinking from the start, thinking about model explainability, that's one of the hardest things about generative AI models. We're just really starting to understand how they act and why they make the answers that they do.

(11:27):
There are starting to be some frameworks out there to really understand how that works, but thinking about that from the start explainability of the model, then as well, like the data of where that answer came from, into that building of that kind of lineage. I know I was just made a trip over in the EU over a couple of weeks where I met with a lot of customers and the EU act over there. They require any of those machine learning models. You have to be able to prove where the data came from to be able to feed that. So start thinking about AI in that way that I can actually reproduce these results. I can create explainable ai. Thinking about that from the start will really help a long ways in your security best practices.

David Dichmann (12:03):
Very cool. And Joe, I'm sure a lot of our customers have been talking to you about this topic as well. Yeah,

Joe Rodriguez (12:08):
And actually I'm glad that you mentioned the EU AI Act because it actually provides us with a framework for doing all this. And a lot of the references in the EU AI Act is really around explainability of those models and ensuring that there are no biases, ensuring that data is protected or personal information is protected within these models. So definitely that, and thank you for bringing that up, Jake. So I guess the advice there would be go read the EU AI Act even though your bank may not be acting within the EU, but they've already done a lot of the heavy lifting around what the framework should be.

David Dichmann (13:00):
Very cool. I want to dig deeper into the notion of regulatory compliance because we deal with a lot of highly regulated industries, not just banking at Cloudera, and there always seems to be a big push to innovate quickly, but always kind of like the wet towel is the regulations and other things that we need to keep track of. And how many folks here saw the keynote this morning? One of my favorite takeaways from that keynote was when the gentleman was on stage and he says he was talking with the CEO and the CEO said, your job is to go innovate. My job is to put the brakes on it just to make sure that it matches. And I think it's important to be able to dream and extend the art of the possible and try new things and then see how that can be managed properly as we roll these things out to organizations such as in banking. And the other trend that we're seeing in a lot of cases is we do a lot of stuff internally where those regulations perhaps aren't as heavily applied as they would be in interbank or to customer transactions. But the question I've gotten, I did click the button, so I have to check that. How do you balance innovation with regulatory compliance and make sure that we're doing generative AI in a way that is going to be successful and capable in the banking industry? So this one, Joe, I'll start with you. Yeah,

Joe Rodriguez (14:18):
Actually, the presentation this morning was excellent. He touched on a number of things that absolutely are best practices. I really liked the fact that they're educating everyone within the bank about what AI is and how it should be used, what are some of the guardrails around it, et cetera. They obviously need to have comprehensive governance around it and have strict policies and procedures around what kind of data gets used and what doesn't get used they need to do. I think testing in this regard is probably the biggest factor as well. And being able to thoroughly test these models. It's funny because testing is one of the first things that gets AIed, if that's a word. So with AI testing, I guess testing those models very thoroughly using things like synthetic data as well. And from an innovation perspective and regulation, right, it is just keeping in tune and in contact with the regulators themselves, right? Because developing their regulations for all this as we go along, they don't always get it right the first time. And collaboration is super important. So don't shut out your regulators, talk to them. It's really important. It's really important to maintain that relationship with them and to take their input as well.

David Dichmann (15:58):
Fantastic. And Jake, anything you want to add for that? Yeah,

Jacob Bengston (16:01):
I think something I've heard actually one of our customers talk about is compliance by design. Just that idea from the start is making sure your innovative teams, there is a goal for them to not just say, go out and do whatever. There's obviously some advantages in that, but really putting the onus on them to say, Hey, we want to from the start, build in this compliance. So step one, you have to understand what those regulatory processes are, who we have to adhere to. Looking at things like we already talked about the EU AI Act. Maybe looking at something that is a little bit more forward thinking, even though it's not regulated for you yet, thinking about what's going on in the broader sense, and then try to be forward thinking in your regulatory design and then building that right from the start. If you start out and just saying, I'm just going to innovate for this however I want.

(16:49):
And then at the end of the day you get this application and then you start to think, okay, how are we going to make it compliant? It's pretty hard to do retroactively to add in that compliance. So to me, being compliant by design is a way of thinking. And really just doing that from the start rather than doing it retroactively is an important part of it. And the second part is, I already kind of talked about that, is just being proactive in your engagement with regulatory bodies, making sure that you're spending that time making sure that the groups that are doing this are doing that as well.

David Dichmann (17:18):
Absolutely. I think the expression might've started at Facebook, the move fast and break things that only applies when the things that you are breaking aren't your customers, aren't your goodwill, aren't your security, these things. Obviously speed alone is what's going to get you into trouble. So you always have to temper that. And when we think about the industry as a whole, and especially as where we're seeing generative AI being applied safety first, the more commodity workloads on commodity data kind of stuff, we're seeing a lot of that happening first. But we know that that next wave of generative AI is going to come from using our proprietary data, using our sensitive data and using that in customer facing and public facing applications. So we know that's where we're going to differentiate. We know that's where we're going to be able to make a difference amongst others in our markets to be able to use generative AI in those creative ways. And so Jake, the last question I have for you and Joe is can you give us some examples of where generative AI has been used in the industry as a whole, where we're seeing some of the market using this in financial institutions that are doing so in such a way that it's both creative but also meeting regulatory compliance?

Jacob Bengston (18:33):
So one example we have that is publicly referenceable is OCBC, which is a bank over in Singapore. So this was actually something you touched on really early on, is saying that most people have started with these internal use cases. How can I enable my employees, make them more successful, make them more efficient? And then after they figured that out, they've moved on to doing more of a customer facing stuff. It's easier, there's less risk involved when you're enabling your employees as opposed to working with external customers. So OCBC very much took that point of view. They analyze and looked at a lot of the different use cases that are out there and decide on ones that were good for them. They're actually, they have some generative AI use cases that have been in production for over a year, which I'm sure that blew my mind when I first heard that they're in production with these, because most of our customers and most people I talked to, everyone's stuck in this pilot stage.

(19:23):
They're trying to figure out what is the best way to proceed with this. We can do something, but there's a lot of these risks involved, and anyone that's worked with generative AI models, it's pretty amazing. They really can't, in some ways think for themselves. All they're doing is predicting the next word one at a time. And so oftentimes it's hard to really put control around them. So it's hard to really know how they're going to act. So a lot of people, they have a use case, they build it out and find out, Hey, this is, it does work sometimes the way we want it to, but sometimes it doesn't. But OCBC over a year, they've had some of these use cases in production. So some of the first ones already mentioned internal employee drivers. So there was a copilot for their coding. So for all their developers, they have about over 2000 developers with an OCBC, and they started out using a third party service to do a coding assistance.

(20:10):
So they're actually using GitHub copilot. They found that it worked well, but they thought it was a little bit expensive. And then as well, they felt like they could maybe do it a bit better. They could customize a model with their own way of doing things and get better experience from that. So using Cloudera actually, so they went from three days, which was the most impressive part from idea to application of actually rolling this out in a pilot use case. So they took a look at some different models that were out there. They selected, it was actually STAR Coder was the open source model that they used for that, and they hosted it within public cloud within Azure using Cloudera. And they rolled out this initial use case to about 200 users. It was very successful right from the start. Since then, they've actually deployed on-prem.

(20:53):
So bringing in the hybrid architecture into play, they found that they could do it even less expensive if they hosted this on-premise. They had some GPUs available to do that. And they have now since rolled it out to all 2000 developers. They have it actually embedded directly into their ides that they use there for all these developers. They find their findings is so far, their developers are about 20% more efficient at their work. So if you think about it, they have 2000 developers now. It's like they have 2,400 developers. They haven't let go those developers because they're more efficient. They just find that they can do more use cases and get more value out of that. So it's been a huge win for them. They also found that they saved about 80% working with an open source model as opposed to a paid endpoint. I already kind of touched on that a little bit, but then as well, just from a security standpoint, they could run that in their own environment on premise.

(21:40):
So all their developers, they know that this isn't being exposed. One of the first security breaches that happened with ChatGPT is that some Samsung developers had put proprietary code in ChatGPT asking it to find errors or find efficiencies. And that data actually got leaked, it got added into the ChatGPT training data and other people were able to see that. So having that model within their own environment gave them the security and gave 'em the confidence to know that they could deploy these types of use cases. And then beyond that, they then rolled out other use cases. So they have some that are customer facing now for customer services that has language translation as part of it, as well as sentiment analysis to define and find out customers that are possibly have silent complaints. So then they go back and retroactively find those customers based on the recommendations of their internal processes.

(22:27):
And then as well, probably the most common one that all of our customers and they're doing as well is like a document summarization, like a knowledge-based copilot. So really taking internal documentation that you have and allowing your users to be able to converse with that data. So ask it questions and it has that context of your documentation, which is super valuable. They say that they have a 95% improvement in time to understand a document essentially. 95% is a lot. I think people still read the documents eventually, but if you just think about it, they can find the document that has the answer to their question, they can summarize it quickly, they can get to the meat of their question very fast to get the answer.

David Dichmann (23:07):
Absolutely. And I like one of the things you pinged on there. When we see generative AI helping improve productivity, typically that's allowing people to get more done and being used as a tool, not as an excuse to start downsizing, reducing efficiencies in other places. It's about optimizing the skill sets. One of my favorite questions that usually doesn't get asked is, well, how much did that AI cost you? When you introduce that savings and efficiencies, are you gaining enough back in efficiencies to cover the AI cost? And that's where bringing things into on-premises and finding other ways to keep your AI costs down, really help amplify those improvements in productivity and gains that you're going to be able to make.

Jacob Bengston (23:44):
Maybe one more thing that was important for them to understand, I think for most is just the idea that this is an assistant, a copilot. It's not a pilot. So any of the things that they're rolling out, they're very much having the mindset of a trust but verify. So for their users or developers, there's a lot of training that goes into it for them to understand this can write code for you, but this shouldn't be the only thing that's writing code for you. You should use this to look at test out yourself and make sure it makes sense for you. For more advanced users, like if I'm using ChatGPT, I can understand that it's not always going to have the right answer. I might have to iterate on a few times, but maybe someone that's less technical may just blindly trust the output that comes from it and then copy and paste and use that. And that's not the experience that you want right now with generative AI. It's meant to be an assistant for your users not to replace your users. And so just building in the training and the mindset within your employees in order to use it that way is very important. Otherwise, you may get some nonsense out there.

David Dichmann (24:41):
Really where the human in the middle matters. Yeah, exactly. So Joe, anything you want to add on to that or any final comment from you today?

Joe Rodriguez (24:48):
Just that one of the cool things about this job is that we get to see all the innovation that our customers are doing. And there's some really fascinating things going on right now. So we have a bank in Indonesia, for example, that is committed to financial inclusion for their population, and they've completely revamped the way that they do things like credit scoring. They have a whole product line aligned to that demographic as well. That includes things like financial education. It's just really, really cool stuff. We have another client in the Middle East that essentially wants to be the first or one of the first because there might be others financial services, super app. And they're very committed to being that first financial services super app. So it's beyond going beyond open banking, going beyond frictionless banking to really being the stickiest place out there for financial services.

David Dichmann (25:57):
Thank you, Joe. And I think one of the things that you just pinged on I think is really important here. Especially when it comes to things like generative ai. One of the things that we're seeing that's quite clear is there's a lot of room for industries firsts and industries bests to be born out of this work. There's a lot of room for new innovation, there's a lot of room for every organization to do something that no one else is doing and do something in a really powerful and creative way. So we know that the demand is there, we know we need to throw caution, but we know we need to do caution correctly so that we don't lose on innovation and move forward. So I want to thank you all for joining us today. Thanks to my panel here to Jake and Joe for their answers to my crazy questions. And if you have any questions for us, we're going to be here in the room. We have our booth right outside the door here. We're also going to be in this room when the lights turn off on us, and we'll be more than happy to answer any questions you may have here or any other time during your time at the show. So thanks again for joining us and enjoy the show. Thank you.