In the summer of 2024, JPMorgan Chase deployed its internal AI platform LLM Suite, launching it very differently than most do: The company didn’t force anyone to use it.
When LLM Suite arrived at its first major division, asset and wealth management, employees were asked to think of it as a research analyst: someone to ask for data, a draft, or an idea. Leadership didn’t set usage objectives or provide a formal mandate.
Access was rolled out in phases and only to those who requested it, and the bank allowed the tool to circulate through word-of-mouth recommendations among colleagues. While half the industry rushed to count users and publish adoption rates, JPMorgan gave up on pursuing that number.
It became flooded with users. In eight months, 200,000 employees had signed up without a single order being issued, out of a workforce of over 300,000. In time, the bank established more than 450 use cases in production.
Two years after that summer launch, JPMorgan had everything to boast about. It had established itself as a global leader in the use of AI: It was the top bank on the Fortune AIQ 50 list, and the third company overall, ahead of all the tech giants except Alphabet.
It was then that the bank’s head of analytics, Derek Waldron, the person best positioned to sell the success, pointed out what still wasn’t working: There was a gap between what the technology was capable of doing and what the bank was actually capturing in its business results.
That gesture is what distinguishes JPMorgan. Although it has much to celebrate, it knows what it lacks, it says so publicly, and it keeps searching for it. Behind that statement lies a way of innovating and measuring that the bank has been developing for years.
Giving up the number everyone was chasing
The first thing JPMorgan did right was not to make adoption the goal. By not forcing anyone, it turned platform usage into a barometer. If a tool worked, it was filled without any campaign; if it didn’t, it was emptied, and that emptiness provided valuable information. If adoption had become a target to be pursued, the organization would have optimized the number instead of understanding what the number represents.
The bank itself acknowledges that if a tool is broadly used, it means it’s popular, but not necessarily effective. To determine its effectiveness, something more was needed. The answer came from two decisions that only work together: linking each project to a business outcome, and creating the metrics to demonstrate that outcome.
First, to find initiatives that could have a real impact, instead of creating an agenda from the top down, the bank surveyed its business units, asking where there was a problem to solve. Within a few weeks, an internal portal gathered, according to the bank’s figures, nearly a thousand ideas. Of these, only a few hundred moved forward and reached production. An organization doesn’t open a funnel of that size if it expects most ideas to survive; it anticipates that many will be discarded.
The funnel’s filtering method was also different. Before launching each test, the outcome that would ensure the experiment’s survival was defined, along with the steps to be taken the day after the decision. By planning future actions in advance, indecision and the perception of failure were avoided.
A clinical approach to AI experimentation
But setting a threshold for each experiment requires verification, and that’s where the bank encountered an unexpected obstacle. Metrics have their own cycles. Bank customers conduct business on Mondays, not Sundays. They receive their paychecks at the end of the month. In August, they disappear. When an initiative generates a change and a figure rises the following week, there’s no way to know whether it increased due to the change or the calendar.
The solution was borrowed from clinical trials. Instead of rolling out the change to all users, it was rolled out to half, chosen at random. The other half (the control group) operated on the same Monday, the same payroll, and the same August, so that the experimental contribution (the attribution) could be separated.
The next step was to industrialize the experiments. Doing it properly required a specialist sitting alongside each product team, and with that method, they reached eight per year. A self-service platform increased the figure to around 300 tests annually.
The results are concrete. For example, tens of thousands of the bank’s engineers have gained between 10% and 20% efficiency thanks to an internally developed programming assistant.
Finally, the bank discovered that a figure can be accurate and yet mean nothing. Its head of analytics explained this with a simple example. They measure the hour that AI saves one employee, and the three hours it saves another. They add them up, and the result is accurate. But in a process that goes from beginning to end, those saved minutes often don’t appear on the bottom line: They merely shift the bottleneck to the next one.
It’s easy to get stuck on partial metrics because they’re more immediate and produce more impressive numbers. JPMorgan’s discipline consisted of not accepting a metric as valid until verifying its impact on the business at the end of the process.
The question then remains on Monday morning: What can a company that has neither the size nor the budget of a bank take home?
The method is what best exports
What’s most interesting about JPMorgan isn’t what it has done with AI, but how it has done it . Any company can replicate this approach, because it doesn’t depend on proprietary data, scale, or budget.
The following are some best practices that don’t require a €20 billion annual budget. They do require making decisions before starting and are within reach of any company:
Launch far more initiatives than will survive, and announce this clearly. If the organization discovers halfway through that most of its projects will be canceled, it may misinterpret this as a planning failure; if it knows from the outset, it understands it as the natural selection process. This is what makes making mistakes quick and cheap.
Decide in advance the threshold that will shut down a project and plan the next steps. Both aspects are necessary, not just the metric. If a certain figure isn’t reached, the team needs to know what will happen next. Applying a threshold without future planning leaves the team in limbo, and they’ll have to find a reasonable reason to wait another quarter before shutting down.
Work on business outcome metrics from the outset, not just when they’re requested. This tracking not only guides the initiative but also prevents having to reconstruct months of poorly documented decisions. Adoption, by the way, is the number the CFO won’t ask for. It serves as a signal while no one is pursuing it, and it ceases to be useful the day it becomes a target.
How to get it right
Whether metrics mean anything depends on where you focus your attention. It’s best to start with scope, because that’s the most common mistake. Saving three hours in one stage isn’t the same as improving time-to-market: If the entire process isn’t shortened, what you have is freed-up capacity, which is also valuable, but it’s something different, and it’s advisable to make that distinction clear.
Then it’s important to consider that value leakage occurs in two directions. The first is outward: The savings are passed on to the customer in the form of lower prices or better service. The second is inward: The savings in personnel are replaced by spending on computing. If these items fall into different budget categories, it’s easy to overestimate the actual savings.
Finally, there’s an excessive focus on cost savings, at the expense of revenue opportunities. Jamie Dimon, CEO of JPMorgan, put it more bluntly to his analysts than any consulting firm: No one benefits uniquely from AI. In other words, competitors will eventually incorporate those savings. The greatest potential for differentiation lies in revenue: using AI to uncover unmet demand.
The question a CIO will have to answer in a year’s time won’t be how much AI their company uses. It will be which of projects are still alive because they work, and not because no one has bothered to test them.
Read More from This Article: What JPMorgan does differently with AI that any company can apply
Source: News