Why AI needs contextual intelligence — not just bigger models

A product manager on my team recently asked me where we were seeing the most issues across the engineering team. Instead of guessing, I had an engineering lead point Claude at our Jira via an MCP connector and look at the bug patterns himself.

One team had a wildly disproportionate share of tickets — about 50% of their sprint time was spent on “bugs,” versus roughly 25% for everyone else. The headline number suggested a quality problem.

It wasn’t. When we layered in the context around those tickets, almost none of them were bugs. They were manual workarounds for a missing product capability: customers asking us, one request at a time, to restore items they had accidentally deleted. Not shipping an item restore feature was burning roughly 1.5 engineers’ worth of capacity. I went back to our product team and said, “Build this, and you reclaim a person and a half.”

The analysis took 45 minutes. It was only possible because our data was already organized, tagged by team, connected to contributors, accessible through MCP and protected by role-based access. None of that is “AI.” All of it is the layer underneath AI that almost nobody invests in first. That’s probably because the investment is unglamorous: updating data dictionaries, access controls, team taxonomies, system-to-system mappings. Most of the work has been the same for twenty years. AI just raised the cost of skipping it.

The intelligence underneath the models

I keep coming back to the value of context data layers as a CTO in the middle of an AI rollout. I have started calling that value proposition contextual intelligence because I haven’t found a better name. Anthropic’s engineering team has been calling this kind of work “context engineering” since late 2025, and CIO ran its own feature on the term shortly after. Whether you describe it as contextual intelligence or context engineering, it’s the part of the stack where the actual programming work still lives.

If business logic is your company’s official org chart, then contextual intelligence is knowing who actually gets things done, how decisions are actually made and what the unwritten rules are. One is theory. The other is reality.

Most enterprise systems capture the theory. The systems that capture how work actually happens — what people do, how teams operate, where decisions get stuck — are rarer and harder to build. And modern LLMs, it turns out, are useless without both.

I learned this the hard way at a recent company hackathon. Nine engineering teams, one prompt: make our operational dataset more usable through AI. My team built persona-based chatbots (CFO, CIO, sales manager) on top of an MCP server backed by Postgres and our enrichment data. Other teams built dashboard generators, Looker conversational analytics and workflow agents.

The initial demos all had the same problem. Claude could talk to our data, but the answers were either generic or confidently wrong. The CFO persona would happily report a “spend trend” that quietly conflated two distinct cost categories across two different tables. The CIO persona would answer questions about team productivity, but the averages across roles should never have been aggregated. The sales manager persona returned answers that were technically correct against the schema and completely wrong against the business. The raw data was rich. The context layer around it didn’t exist yet. Chatting with raw data is not an AI product. It’s a demo.

One of my senior engineers spent the second day ripping out the agent’s direct database connection. He stopped trying to prompt-engineer the LLM to understand our business and instead codified that logic into the data pipeline. Working backward from the failed CFO answers, he mapped out the implicit knowledge an experienced controller relies on: Explicitly defining which legacy tables actually represent ‘spend,’ writing the rules for currency normalization and hardcoding our fiscal time windows. He built a series of semantic SQL views to enforce these rules and restricted the MCP server to exposing only this curated layer. When we pointed the same model at those same questions, it returned completely different answers. They were specific, evidence-based and grounded in our actual business reality. The model didn’t get smarter. The engineering beneath it did.

The same pattern shows up everywhere I look right now

McKinsey keeps publishing that software development tops enterprise AI use cases, with companies reporting 30–50% productivity gains in pilots. The pilot numbers are real. They rarely translate to top- or bottom-line impact in production. Our own company data tells the same story: Between Q1 2025 and Q1 2026, our total AI tool usage grew by 328% (over 4x). Over that same period, PR throughput grew by just 49%.

That gap — adoption way up, outcomes inching along — is the context gap. Plug a generic agent into raw, uninterpreted data, and it will act inefficiently at best, harmfully at worst. An agent optimizing sales without your customer segmentation or product hierarchy will confidently recommend the wrong thing. Anthropic framed the shift directly: building with language models is becoming “less about finding the right words and phrases for your prompts, and more about answering the broader question of what context configuration is most likely to generate our model’s desired behavior.” That second question — what context configuration  — is the entire game. Most organizations are still answering the first one.

Where the work actually lives

A growing number of CTOs I talk to are shifting their AI investments accordingly. Less attention on the model. More on the layer between the model and the data.

When peers ask me what that actually looks like day-to-day, I tell them I give every engineering role the same mandate: the LLM should never see raw, uncontextualized data.

In practice, that breaks down to three pieces of work, none of them glamorous.

The first is semantic middleware. We need code that transforms raw data into business-meaningful signals before it ever reaches the model. Our feature stores hold things like “employee code velocity on critical-path features,” not “X logged 50 Git commits.” The work of figuring out what “critical-path” means in our product, in our org, on this team is the work. It does not get cheaper because the model has gotten better.

The second is multi-agent design. Instead of one omniscient orchestrator, we run smaller agents scoped to specific domains, each with rules that catch the failure modes the main model is known for. We pair them with RAG that retrieves precomputed insights, with their rules attached, rather than raw documents. Validation checkpoints sit between steps and flag suggestions that violate known constraints, such as averaging productivity across completely different job functions. The guardrails are not there to be clever. They are there because we already watched the model make those exact mistakes.

The third is evaluation that takes business logic seriously. When I look at a model, general benchmark accuracy is the least interesting number. I want to know whether it respects our constraints and integrates cleanly with our existing architecture. That sometimes means fine-tuning our patterns, sometimes constitutional approaches to embed principles, sometimes hybrid systems where deterministic rules sit alongside the probabilistic ones. The throughline is the same: validate against reality, not against the benchmark.

Why this matters now

The reason this matters more now than it did six months ago is that adoption is moving faster than measurement, let alone integration. Model Evaluation & Threat Research’s (METR) developer productivity work tells the story in a way they didn’t intend. In early 2025, they ran a controlled study and found AI tools slowed experienced open-source developers by 19%. When they tried to repeat the study in late 2025, the experiment broke. Thirty to fifty percent of developers refused to submit tasks under the no-AI condition. They wouldn’t accept working without their tools. METR is now redesigning the study because the original methodology no longer holds up against how developers actually work. That’s how fast adoption moved. But I’d be willing to bet the organizational scaffolding required to convert that adoption into outcomes — context layers, workflow redesign, retraining around new tools — moved nowhere near as fast.

Get ahead with context

The teams I’ve seen succeed with AI built the context layer first. The teams I’ve seen struggle eventually built in context anyway, just at higher cost and with more scar tissue. Raw data is the new currency. But raw data without a context layer is cash sitting in a vault. It cannot act on anything. The difference between insight and noise is a layer of code that understands what your data means.

That layer is the work. It is where the next decade of competitive advantage will sit. And in my experience, the organizations that build it first are the ones that will actually get the productivity gains the rest of the market keeps promising.

This article is published as part of the Foundry Expert Contributor Network.
Want to join?