Token efficiency: Building AI that spends intelligence wisely

Last year, my company decided to rebuild our enterprise AI platform, which our customers had been using for over a decade. But instead of simply updating it, we started over with a different ambition: to build an AI-native platform using AI.

Using AI to rebuild a platform of that size gave us first-hand experience with AI economics. We employed AI throughout the entire engineering process to write code, review changes, generate tests and accelerate development. It helped us reduce years of engineering to just a few months but also forced us to keep close watch over every single token we consumed and truly consider why we were consuming it.

This experience challenged one of our key assumptions: It turns out that token consumption is influenced as much by architectural decisions as by choice of model.

What token sprawl actually looks like

Indeed, we found that most of the token consumption waste wasn’t due to complex engineering tasks, but the architecture around them; this is something that is referred to as the architecture tax in enterprise AI.

Recursive agents kept spawning subtasks and subagents, and although they technically completed every task, each handoff introduced new prompts and context. Furthermore, frontier reasoning models were being wasted on routine classification tasks that deterministic code could have completed at a fraction of the cost. And, because we hadn’t clearly defined ‘success,’ retry loops continued firing and burning tokens to chase marginal gains, instead of improving business outcomes.  Finally, we experienced context inflation, with workflows carrying information from earlier steps long after they had stopped contributing to the decision being made.

None of these decisions seemed expensive in isolation. Things like a few extra prompts, an unnecessary reasoning step or a larger context window barely impacted the cost of a single workflow. Across millions of executions, however, they grew into a level of token consumption that no model upgrade alone could offset.

Designing for token efficiency

Once we figured out where all our tokens were going, we had to determine how to spend them more deliberately.

First, we accepted that not every step in an AI workflow needs the same amount of intelligence. Some tasks legitimately require reasoning, but others are structural, deterministic or repeatable, and are better handled through software, business logic or tools that are reused rather than reasoned through every execution.

Even when a model is required, not all tasks need the same amount of reasoning. This concept is better known as Reasoning Effort Optimization. If you’re designing systems, exploring alternatives or making difficult decisions, high reasoning effort has value. However, once you’ve established the approach, you can usually produce the same outcome with far less reasoning.

It’s comparable to organizations that structure work as a pyramid of expertise. The most experienced engineers tackle the most difficult problems, make important architectural decisions and handle any ambiguity that arises. Then, once their work is complete, they split it into small, well-defined tasks that other engineers and machines can execute consistently.

Allocation is only part of why this works, though; persistence is equally important. A senior engineer’s judgment gets built into interfaces, schemas and conventions, allowing everyone downstream to build on those decisions without having to derive them again. The same applies to AI systems. If you fail to capture their reasoning as a durable artifact like a graph, specification or compiled skill, the system repays for it every single time it runs.

We apply the same principle to AI systems. Advanced reasoning should only be used when judgment is required. Smaller models can handle well-defined implementation work, and deterministic software and reusable tools can execute tasks where the outcome is already known. The goal is to apply the right level of intelligence to each stage of the workflow.

Architect once, inherit everywhere

So, how much of an AI system should teams rebuild for every new workflow?

In our experience, very little. Architectural capabilities like evaluation, orchestration, guardrails, context management and auditability aren’t unique to a single AI application; every enterprise AI system eventually needs them. Rebuilding them for each workflow is a waste of time and money that forces every AI system to solve the same infrastructure problems independently.

It’s much more efficient to move these capabilities into a common harness. Orchestration, governance and evaluation can then be developed, tested and improved at the platform level rather than separately inside every workflow. Each new AI system can reuse this foundation while teams focus on the business logic that makes the application distinct.

This architectural discipline extends to the execution layer. The amount of computation an AI system performs before producing an outcome depends heavily on the context provided, how its instructions are designed and how its agents are structured.

A 2026 study found that consolidating three or four agents into a single call reduced token consumption by 53.7% and latency by 49.5%, with comparable accuracy. Most of these savings came from not repeatedly passing the same context between agents.

The study also demonstrated that, as the number of tasks assigned to each agent increased, the agent became less reliable at choosing the best way to approach each task. However, increased routing helped recover much of that performance. In short, AI efficiency depends on how the work is structured and routed, not just how many agents are involved.

Optimize throughout the lifecycle

Building the right architecture creates the foundation for continuous optimization.

Model selection, for example, shouldn’t be a one-time decision. As models improve and workloads evolve, it’s worth re-evaluating every AI workflow to determine whether the same outcome can be achieved with less reasoning and lower cost. Demonstrating that system design can outweigh raw model capability, Andrew Ng, a computer scientist, AI entrepreneur and Stanford University adjunct professor, highlighted that while zero-shot GPT-3.5 scored 48.1% on the HumanEval coding benchmark and GPT-4 scored 67.0%, wrapping GPT-3.5 in an iterative agentic loop pushed its score to 95.1%.

The same applies to cost visibility. Token economics should be visible across every workflow, model and task so teams can identify ways to reduce costs before they accumulate.

AI is much less expensive than it used to be, but lower costs encourage more use. Organizations can run more workloads and take on more ambitious problems, so it’s even more important to know where the AI is actually creating value.

Allocate intelligence by the value it creates

Most organizations start with fixed token budgets, because they are easy to administer. However, not every AI task has the same value.

Think about AI as an engineering resource, rather than a shared utility. The teams that consistently choose the right model for the right task and avoid unnecessary reasoning should get more compute because their tokens are producing better outcomes.

Over time, I expect enterprises to manage AI the same way they manage any scarce engineering resource — by using it where it has the greatest impact.