Architecting the AI backbone of intelligent insurance: How to engineer a scalable and performant enterprise AI platform

I spent years at Meta engineering large-scale systems for billions of users, delivering sub-second latency and five-nines (99.999%) uptime. When we started Outmarket AI, I brought that same lens: scalability, reliability, sustainability. Not buzzwords but real engineering.

Commercial insurance turned out to be a different planet. Some departments were still on pen and paper, going through manila folders. Others had systems built on COBOL, mainframes from the 80s to handle claims. Nobody wants to touch them because the guy who understood the code retired years ago and didn’t leave notes. Underwriters, brokers, marketing, customer reps — everyone going through thousand-page policy documents, making million-dollar calls for businesses. According to McKinsey’s State of AI research, 78% of organizations are using AI in at least one business function. Insurance has been slower to change the way it operates day to day.

We started building AI products for a few lines of commercial business: workers’ comp, general liability and property coverage to better understand all the pain points. Consider workers’ compensation, which in itself is a beast. A human has to analyze injury claims, workplace risk factors, OSHA reports, medical records, claims histories and state regulations that differ wildly. For general liability, one has to dig through premises risk, operations exposure, vendor agreements and similar hassles for property coverages. Meaning a single policy decision might need someone to pull together dozens of documents from different sources and spend more time on clerical work as opposed to the real deal

Within weeks, our first client wanted it for every other line of business. Not just one department, but the entire organization. The pattern repeated with every new client as they quickly realized the same AI infrastructure could transform how they handled all of their commercial policies. That moment crystallized something for the founding team. We weren’t building a feature, but instead building an AI-backed infrastructure and I knew exactly what that meant from my time engineering at scale.

The AI part wasn’t what kept me up at night. Large language models (LLMs) can handle dense insurance documents. That’s been proven. What worried me was everything underneath.

First, scale. How do we build something that grows with more clients? And scale by users per client? What about seasonality when commercial insurance policy renewals peak? Q4 is a mess. Traffic doesn’t grow linearly. It spikes ~10x.

Second, reliability. We started with one LLM provider. It worked fine, but what will happen when traffic spikes? Everyone’s slamming the same LLM. That’s a nightmare of hitting rate limits, token limits. What if third-party systems go down? We all have seen this in action when ChatGPT went down

Third, data isolation. No insurer would tolerate its proprietary underwriting data bleeding into a competitor’s context window. Every client needs their own guardrails.

So we weren’t just building a system. We were building a beast that can’t flinch under pressure, can’t go dark when a provider fails and can’t leak data between clients.

We attacked each problem head-on.

For isolation, we went single-tenant. Every client gets their own instance, their own database, their own AI context boundary. No shortcuts.

For reliability, we designed the load balancers of AI agents that look at everything and most importantly, latency, cost, accuracy needs, provider health and make a call in real time. If one provider is down, it is now obvious that the traffic has to shift.

This orchestration layer was the breakthrough that can scale infinitely now.

Why is insurance the ultimate stress test for AI infrastructure?

Think about a mid-sized restaurant chain buying commercial insurance. They need workers’ comp for kitchen staff, general liability for slip-and-fall incidents, property coverage for equipment, outdoor dining coverage, liquor liability, theft protection. Probably a dozen policies total. And these are all thousands of pages of dense legal language, exclusions, endorsements, coverage schedules and many more.

Before AI, someone had to read all of this manually. Risk managers spent weeks on it, sometimes months, comparing quotes from various carriers, hunting for gaps, trying to catch redundancies and all of this manually. The mental load was brutal and mistakes were inevitable. I have seen claims denied because of a coverage gap buried on page 847 that no one saw. The policy looked fine. The exclusion that mattered was hiding in plain sight. When that happens, insurers fall back on their errors and omissions coverage (E&O) to protect against mistakes made by their employees while reviewing insurance. That’s how broken the manual process is and can easily lead to millions of dollars in claims.

A typical policy bundle containing 2K pages can now be ingested in 10 to 15 seconds. Even though speed is a big win, what’s more exciting are things that were not possible before. Quotes from various carriers can now be compared side by side in real time. AI flagging gaps automatically before they turn into claim denial. Underwriters can type questions in plain English. “Does this cover water damage from a burst pipe in an unoccupied building during winter?” Answer with citations to gain more trust and confidence. No human can process with that speed and accuracy. The humans are now reviewers and decision-makers and not document processors.

Surviving seasonality: Engineering for 10x traffic spikes

Insurance has a brutal seasonality problem. Policy renewals cluster around year-end. As soon as Q4 hits, traffic is expected to spike by ~10x. An architecture that runs fine in March can collapse in December if we haven’t planned for it.

Three things kept me up. First, caching. LLM caching is not like a typical web caching. Take these two questions, for example:

1. “What’s my deductible for property damage?”
2. “How much do I pay out of pocket for building damage?”

Both are basically the same question. How do we recognize that and not waste compute power?

Second, scaling. When renewal season hits, the largest client might need 10x the capacity overnight, but I don’t want to pay for that capacity year-round.

Third, routing. Not every query to LLM needs the biggest and the best model. A simple policy lookup doesn’t need the same horsepower as a complex one. Sending everything to one model means simple queries wait behind heavy ones.

We tackled each one.

For caching, we have semantic matching algorithms at multiple levels.

1. At the embedding level: We cache vector representations so re-injection would re-use the same embeddings.
2. At the query level: We use locality-sensitive hashing to spot similar questions and serve cached responses. If a question is already answered, then a similar question can use the same response without burning the compute power twice.

For scaling, each worker process can auto-scale horizontally based on queue length and current latency in-place. The largest client might go from 4 workers to 40, then scale back as soon as traffic drops. The key here is that scaling can be reactive, but for seasonalities, it can be predictive. If client X’s renewal rush started October 15th last year, then we can technically pre-warm their infrastructure on October 10th this year.

For routing, we built a classifier that examines incoming requests and sends them to the right model. A simple lookup can use a small, fast model; however, a complex coverage analysis workflow can be routed to more sophisticated models. This can cut cost by about 40% and actually improve P95 latency because simple queries are not jammed behind complex ones.

Now let’s put this together and we see users getting sub-second responses irrespective of quiet Tuesdays or chaotic Decembers. That consistency is what turns AI to what people can use at scale.

AI hallucinations kill trust; domain knowledge fixes it

Large Language Models (LLMs) fail in ways that regular software does not. In traditional software engineering, a database either returns the right row or throws an error, but an LLM will always return plausible-sounding nonsense and any system will happily pass it downstream unless we build detection mechanisms. Research published in Nature has shown that detecting these “confabulations” (arbitrary and incorrect generations) requires measuring uncertainty about the meanings of responses, not the text alone.

The root cause depends on how all these models learn. General-purpose LLMs train on public data crawled from the internet. They’re capable of broad reasoning without any domain expertise. If we ask a general LLM about insurance policy structure, it will give a reasonable-sounding answer drawn from insurance data that exists in its training set, which may or may not reflect the actual terminology, coverage structures and regulatory requirements that clients operate on. In any insurance, a reasonable-sounding yet wrong answer can lead to denial of claims or even regulatory violations, leading to millions of dollars in losses

Research on fine-tuning LLMs for domain knowledge graph alignment has demonstrated that when models are tuned to domain knowledge, then it can perform multi-step inferences while minimizing hallucination. So we built out our own knowledge graph for insurance, which holds definitions of how the industry actually works. Coverage types, policy structures, regulations, carrier-specific terms, claims workflows, how everything connects. It took years of domain expertise to build it and we are still fine-tuning it every time we run into a weird edge case. What we found out was that when our models were fine-tuned against this custom graph, they stayed inside verified boundaries instead of inventing plausible-sounding answers from pre-trained public data.

In practice, this makes a huge difference. If a user asks for coverage exclusions, then the system no longer hallucinates. It uses a knowledge graph as a source of truth. Any missing knowledge in the graph means uncertainty rather than confabulating an answer.

No system is perfect, though. Even with the knowledge graph, things slip through. I call it hallucination tripwires, an automated check that can catch AI when it’s making stuff up.

Model claims a coverage limit that’s nowhere in the source document? Tripwire.

Model references a policy section that doesn’t exist? Tripwire.

Model pulls a number that’s way outside expected ranges for that policy type? Tripwire.

An ACM survey on LLM hallucinations categorizes hallucination detection techniques into two: factuality and faithfulness approaches. When a tripwire triggers, a smart system won’t just log an error and move on. It will fall back to a secondary model for verification purposes. And when that fails, it will escalate to a human for review, depending upon the severity and confidence scores.

Hallucination detection is one piece. The other is model drift. Models can get worse over time and shift away from training data and accuracy drops. We track this constantly, checking against human-verified samples. When we see the numbers trending down, we fine-tune or adjust our prompts. Observability isn’t a nice-to-have; it’s a must for enterprise applications to stay reliable and win clients’ trust.

Databricks popularized a concept called medallion architecture, where raw ingestion produces what we call “bronze” data, minimally processed, potentially messy. AI-driven normalization transforms this into “silver” data with consistent schemas and validated fields. Further enrichment and cross-referencing produce “gold” data that’s ready for downstream analytics and reporting. This tiering can help serve different use cases appropriately. Real-time policy queries can work against silver data, whereas regulatory reporting and actuarial analysis must work off of gold-tier data with full audit trails.

Engineering principles that made the difference

A few principles that stand out when I look back in time.

AI is an infrastructure. Treat it that way from day one. Don’t bolt on scalability later. The early decisions on single-tenant v/s multi-tenant, sync or async, one LLM provider or several will compound. Unwinding them later will be painful and expensive as it may eat up a good amount of engineering resources and time and even get new features to stand still for weeks to months.

Build for failures. Providers go down, models hallucinate and demands can go up at any time, especially when we least expect it, so build the fallback paths before entering panic mode.

Observability is not optional. In regular early-stage software, we can skip the fancy monitoring, but in later stages, especially with AI systems, we can not afford to do that. No observability will mean shipping a broken output and being blindfolded about it.

Commercial Insurance has built its traditional processes around human limits, especially reviewing speed and mental bandwidth. AI can lift those limits up if and only if the infrastructure holds up to its expectations reliably, at scale and under pressure.

The difference between an AI demo and an enterprise AI system is not the AI models but the backbone, the infrastructure that doesn’t flinch.

This article is published as part of the Foundry Expert Contributor Network.
Want to join?