How Capital One drives returns on its AI investments

Generative AI is rewriting IT in just about every way. But the head of AI strategies at $39 billion Capital One sees many CIOs miscalculating gen AI costs to the extent that they are making “irreversibly bad decisions” about their generative AI strategies.

Prem Natarajan, EVP and head of enterprise AI at Capital One, says the economics of gen AI costs are changing so dramatically that attempts to use traditional financial tools to calculate and project AI ROI is the wrong way to go about it.

“In the last 22 months, the cost of inference has come down by more than a factor of 1,000 on a performance equivalent basis,” Natarajan tells CIO.com. “Something that cost you $10 to do inference on [two years ago] is now costing you one cent. In that environment of dramatically changing costs, any focus on near-term robust prediction of ROI as a justification for investing in gen AI” is likely to fail.

Natarajan, who described the economic changes as being “on the throes of a generational inflection,” believes that CIOs taking that approach to projecting ROI “are making irreversibly bad decisions that will make them fall behind” given their “obsession of calculating ROI in the face of transformative technologies.”

Natarajan joined Capital One in March 2023 from Amazon, where he spent almost five years as vice president for Alexa AI. He estimates that at Capital One he oversees “several hundred petabytes of data that will approach exabyte scale” in “a couple of years.” That data trove is a key asset for Capital One in making the most of AI, according to Natarajan, who sees data governance and accessibility as additional keys to AI success.

Jason Andersen, a vice president and principal analyst tracking AI for Moor Insights & Strategy, says Natarajan’s take on the ROI issue for CIOs is valid. “Enterprises [such as Capital One] are starting to get really smart about how they are deploying AI and building AI applications,” Andersen says. “The reality is that we haven’t seen a trend in technology ever move this fast — ever.”

That speed has caught many IT executives off guard as techniques that have always worked for them stop working, Andersen adds. “With this absolute velocity, you are seeing the old norms of trying to figure out how much to invest, those are no longer useful tools,” he says. “If you use traditional methods, you just don’t get it.”

Although Andersen agrees that inference pricing has gone down significantly, “the reality is that we are asking for more sophisticated tasks, queries that are perhaps 1,000 times more complicated” today as compared to two years ago, he says.

Capitalizing on cloud and data

When Natarajan joined Capital One in March 2023, ChatGPT was barely four months old. Despite having been used for about 15 years at that point, generative AI didn’t take off in terms of C-suite and board mindshare until OpenAI introduced ChatGPT.

“My first eight to nine weeks [at Capital One] were incredibly exhilarating. We were meeting three times a week minimum for multiple-hour meetings,” Natarajan says. “By the end of May, we had aligned on the path forward. By June, we actually started fingers on keyboards.”

With almost any form of AI, he says, “your data advantage is your AI and ML advantage.”

“The amount of proprietary data we had was an important asset to be brought to life in building generative AI applications and capabilities that would be differentiating for us,” Natarajan says, stressing that their evaluations showed that they “could not use closed-source models, because you cannot meaningfully customize those models.”

The Capital One AI team eventually opted to use Meta’s open-source Llama LLM and set about building AI solutions atop its public cloud foundation, established prior to Natarajan’s tenure at the financial services company.

“Capital One was the first bank — and to date, the only bank — that is all in on the public cloud. They shut down all their data centers over a period of two years and moved everything to AWS,” Natarajan explains. “We became cloud-native developers.”

At Capital One, which employs roughly 14,000 IT specialists, talent is critical but so too is data — perhaps more so, Natarajan says.

“You can create a top-notch tech organization, but you cannot create three decades of data overnight. You can’t buy it. Who would sell it?” he asks. “We are talking about very deep data — not just transaction data you can get from Visa or Mastercard.”

Data quality and discoverability are also chief concerns, according to Natarajan. There may be quadrillions of insights locked in an enterprise’s data stores, but if the data is not clean or discoverable, those insights are beyond reach.

Unstructured data in particular “has remained largely outside the reach of traditional machine learning algorithms,” he says. “So all of the insights and capabilities buried in unstructured data have remained buried until now” with generative AI.

As for the data reliability problems that plague many gen AI deployments, including hallucinations, Capital One sidesteps most of those issues by severely limiting permitted queries to information that exists within the company’s own data.

“We don’t have to answer the world’s curiosity for random information,” Natarajan says. “We are [only] trying to answer questions for customers who are interested in learning very specific things and those answers come from our curated knowledge bases and our curated databases. We are only answering questions that we have an answer to.”

The company also customizes data guardrails, rigorously checks AI outputs with humans-in-the-loop, and extensively leverages retrieval augmented generation (RAG).

“We are trying to improve the number of times the human changes the answer. Ideally, we want to get it to zero,” Natarajan says. “Every time a human makes a correction, you try to learn from it.”

And Capital One’s agent servicing model is already paying dividends, helping customer service agents resolve customer questions more efficiently, he says.

“For example, if a customer calls to ask about whether a declined card transaction will count against their daily card limit, our agents can use this tool to quickly search for the relevant information in real-time, enabling them to deliver reliable information more quickly than ever before,” he explains. “With this tool, we saw a shift from 84% to 93% highly relevant search results as compared to the legacy, non gen AI tool.”

The company has also deployed an agentic AI tool called Chat Concierge to help car dealerships offer auto loans through Capital One.

“Chat Concierge consists of multiple logical agents that work together to mimic human reasoning and not simply provide information to the customer, but take action on their behalf based on their requests,” Natarajan says. “In a single conversation, Chat Concierge can perform tasks like comparing vehicles to help car buyers decide on the best choice for them and scheduling test drives or appointments with salespeople. Some car dealers are reporting up to 55% increase in customer engagement with the tool, and we’ve reduced its latency by 5X since deployment earlier this year.”

Priming data for success

For Natarajan, the secret to success is all about data — its governance, quality, reliability, and discoverability.

“A lot of these companies that are trying to do a gen AI transformation today will not succeed if they don’t pivot their attention bigtime to getting their data ready,” Natarajan says. “Gen AI techniques are so wonderful at learning subtle patterns across massive collections of data. [But they] are fundamentally limited when you don’t give them massive amounts of data. They can only learn patterns across the things that they are exposed to.”

Making sound architectural decisions also helps influence gen AI project outcomes, Moor’s Andersen says, noting that, in Capital One’s case, the decision to deploy an open-source model was shrewd.

“It shows a level of evolved thinking that we are not seeing in a lot of enterprises today,” he says, adding that retaining flexibility in the models available is also vital, which Capital One has emphasized by selecting AWS for its cloud platform.

“Amazon has said that [they] support a lot of different models. They are not invested in their own internal models in the way that Google and Microsoft are,” Andersen notes, adding that the AWS approach delivers the “most flexibility and adaptability.”