Smaller, smarter, safer: How to build agentic AI on the right foundation

When it comes to building an effective AI stack, context is king and power isn’t everything it’s cracked up to be.

“Smaller, smarter, safer — this is a bet our company has taken in how we deploy AI internally,” said Ricky Thakrar, head of sales and account management at Zoho, provider of a suite of popular cloud-based software solutions for sales, marketing, and finance.

“I’m on the business side, and so decisions made by our CIO and IT folks affect me directly, and my teams’ workflows and processes,” he added.

Speaking to a room of tech leaders at the CIO 100 Leadership Live New York event last week, Thakrar explained that every company wants the speed of AI-generated work wedded to the quality of human work, even though these two are diametrically opposed. No amount of model upgrades or spend will close that gap, so the only way forward is to architect your way out. Thakrar encapsulated this idea in a simple formula:

• Smaller: Stop deploying maximum firepower on every task. Many tasks don’t need it.
• Smarter: The system around the model decides more than the model does.
• Safer: Verify at the point a mistake gets locked in, not just downstream of it.

He noted that organizations that win with AI won’t be those deploying the biggest, most powerful models or the most sophisticated architecture, but the ones that figure out that the model is the easy part and the right architecture is harder. That means understanding the hardest element, and the biggest differentiator, is building a human system that learns and compounds alongside agentic systems.

To get it right, organizations need to prioritize the context layer. The size of frontier models like the GPT series, Claude, and Gemini mostly exist to compensate for missing context, Thakrar explained. Without enough context, models need to be able to reason harder and infer more about what a user actually means because it doesn’t know the user’s account, process, or history. A rich context layer makes it possible for enterprises to run workloads on much smaller, lower-power models.

“The intelligence moves from the model into the architecture around it,” he said.

A steep learning curve

One of Zoho’s earliest AI agents was a churn management agent to help the account management team detect churn in customer subscriptions. So when a subscription became inactive, the agent would collect context from notes, meeting recordings, and Zoho’s data enrichment tool, then create a summary of reasons the account might have churned, and schedule a call.

“What happened was I got this churn agent a couple months later, already embedded in our CRM, and within a week my team no longer trusted that agent,” Thakrar said. “The reason is we forgot to collect one very key point.”

In Zoho’s CRM, when a customer buys a bundle of products, that bundle is represented as a single line item. That means the status of any products the customer may have previously purchased individually changes to inactive as they’re moved to the bundle. That’s not churn, but it was interpreted it that way. Zoho fixed it in the second version of the agent.

Then a new problem arose. Many potential customers first purchase Zoho products as pilots or sandboxes. As those customers move from pilot to live instance, they close down the pilot versions. And again, the CRM would record that as subscriptions going inactive.

“The trust deteriorates again because everyone got excited for version 2,” Thakrar said.

Sometimes, a certain product might not be the best fit for a customer and Thakrar’s team will suggest the customer move to another product. That’s deliberate churn, not a churn risk.

“You may have a similar story like this where the agent sounds so good, it’s going to do something quick and add value, but it’s missing context from the account managers, and there are so many more pieces we’re still building out,” Thakrar said. “It’s been almost a year and the problem I have is my team still doesn’t trust it. They’ll see [a message from the agent] and go out and do all the research anyway to make sure it gave the correct answer.”

The team is more on top of potential churn, though, but the promised productivity gains have yet to materialize because the agent has to earn back lost trust due to a lack of context.

“My goal for this year is having an AI-assisted customer journey from sales to account management where the handoff is clean, the context flows, and every piece of information we gather about a customer is weighed, identified, and coached so the sales team can close more deals,” he said.

Zoho’s early experience with agents has led to the idea that constrained, context-rich, deterministic architectures consistently outperform expensive models bolted onto fragmented systems. It all comes down to three pillars: routing, harness, and specialization.

Routing

Routing is about sending workloads to the proper model for the job, which entails providing enough context to a given task that a small, cheap model can handle it without the need for spare reasoning capacity to fill gaps.

Frontier models are expensive and companies can burn through a year’s budget worth of tokens in months. But most tasks can be handled by much smaller, more constrained models at a fraction of the cost.

“You don’t always have to pay the frontier guys for every task,” he said. “We’ve observed with some clients that we could save them 95% with a 3 billion parameter model.”

Harness

An AI agent harness is the software infrastructure scaffolding around an LLM that differentiates an agent from a chatbot. It’s what enables an agent to act on tasks rather than simply respond to prompts. A model reasons through a problem and decides what to do about it. The harness connects the model to the tools, systems, memory, guardrails, and execution environments required to perform the actions determined by the model. The term is frequently used more or less interchangeably with orchestration layer.

“It’s the process around the model, which matters way more than the model itself,” Thakrar said.

In benchmark tests, a superior harness on a less powerful model produces better results than an inferior harness on a much bigger model.

For the best results, Thakrar said, it’s essential to understand the deterministic and non-deterministic elements of a given workload, and build that into the architecture. Machines can read, organize, and validate, and they excel at deterministic tasks. Humans, on the other hand, are exceptional at non-deterministic tasks like judging, synthesizing, and deciding.

Those non-deterministic tasks in a process are the ideal point for AI agents to incorporate a human in the loop, what Thakrar calls human harness. He pointed to a stakeholder mapping agent Zoho built for sales as an example, which takes the context of an initial meeting and third-party enriched data like a LinkedIn profile, weighs probabilities, and makes an educated guess about the stakeholder map.

“The initial goal was just to eliminate that task completely from the human workflow,” he said. “The stakeholder map is done, it’s in the folder, and you can look at it.”

But the agent would struggle to capture nuance. The meanings of titles in organizations always vary, and the politics and dynamics of any given meeting can be difficult for an AI agent to discern. Rather than keep feeding the agent data to try to make it intelligent enough to make those determinations, it was simpler and more efficient for the agent to create a proposed stakeholder map and hand it over to a human who could make changes and explain why those changes were necessary.

Ultimately, Thakrar said the agent still saved human team members time because the stakeholder map was usually pretty close, and the corrections also helped the model grow smarter by adding richer context.

Specialization

Specialization is transitioning a process from testing on a frontier model to production on a much narrower, smaller model. Once you’ve proven that an agent can do a job well, you want to stop paying master-craftsman rates to keep doing that one job well.

Specialization is all about capturing your subject matter experts’ best judgement and pattern recognition to build an open-weight, open source, trained, and fine-tuned model that can be deployed in your own data center.

“The true enterprise bet is to keep that orchestration layer, which is your IP and knowledge, in house,” Thakrar said. “You don’t want to host that on someone else’s model. The goal of everyone in enterprise should be to run, train, and host their own models.”