Skip to content
Tiatra, LLCTiatra, LLC
Tiatra, LLC
Information Technology Solutions for Washington, DC Government Agencies
  • Home
  • About Us
  • Services
    • IT Engineering and Support
    • Software Development
    • Information Assurance and Testing
    • Project and Program Management
  • Clients & Partners
  • Careers
  • News
  • Contact
 
  • Home
  • About Us
  • Services
    • IT Engineering and Support
    • Software Development
    • Information Assurance and Testing
    • Project and Program Management
  • Clients & Partners
  • Careers
  • News
  • Contact

Your agentic AI results can’t be checked. Here’s how to fix that

Read 50 published accounts of agentic AI in production and a pattern emerges fast. Escalations down. Handling times down. Conversion up. Onboarding compressed from weeks to days.

Now read them looking for how those numbers were produced, and the field thins out. The window is rarely stated, the denominator almost never and concurrent initiatives that could have produced the same movement go unnamed. You get a number and an implied causal claim, with nothing a skeptic could object to.

This usually isn’t dishonesty. Nobody has told practitioners what a defensible result looks like, so careful and careless accounts read identically.

The cost lands on CIOs. If you’re weighing a major agentic investment, you can’t tell a result achieved at peak trading volume from one achieved in a quiet quarter with the window picked afterward. Both show up as a percentage.

I recently completed an 18-month deployment of 25 production agents across a large travel and cruise group. I’ve written about the strategy and the domain-tuned small models behind it, but not how far its numbers can be trusted. Here is the standard I now hold them to, and the operational failure that undermines every metric if you miss it.

Five ways deployments corrupt their own measurement

  1. The denominator moves. An agent reduces escalations. So does a quiet quarter. A rate without its volume is a way of declining to say which happened.
  2. The window is chosen after the fact. A steadily improving agent always has some recent window in which it looks better than any earlier one, and research on analytical flexibility shows how easily that produces apparently strong findings from nothing. The tell: when one account reports three outcomes over three different periods, the periods were fitted to the outcomes.
  3. The baseline contains slack. Automate a manual process and you usually clear its backlog at the same time, so the comparison measures a faster process and a one-off vanished queue at once.
  4. The change is never alone. Organizations willing to deploy autonomous agents are usually already in motion: a platform rebuild, a repricing, a reorganization. Year-over-year comparison controls for seasonality, and for none of that.
  5. The definition is proprietary. A “Tier 1 escalation” in your own platform means whatever your queue classification says, and isn’t comparable with a competitor’s figure, though both appear as percentages.

CLEAR: Five tests for a reportable result

The NIST AI Risk Management Framework and ISO/IEC 42001 tell you to measure, not what a reportable number looks like. CLEAR sits beneath them and needs only ordinary operational records and one discipline: decide before you know the result.

  1. C: Concurrent changes registered. What else changed in the same window, and could it independently produce the result?
  2. L: Locked window. Was the period fixed before the result was known, and does it include your hardest trading conditions?
  3. E: Evidentiary custody. Which system produced the count, who defines the category and who reviewed the figure?
  4. A: Attributable baseline. How much of the gain came from re-engineering the process, and how much from clearing accumulated slack?
  5. R: Reference volume. Did the underlying volume hold steady, and is that stated?

Four tests ask for disclosure, which is cheap. The locked window asks for pre-commitment, which is costly because it forecloses discovering later that a different period tells a better story. In my experience, it does most of the work.

Medicine and ML research already have reporting standards such as CONSORT-AI and REFORMS; agentic deployments don’t. CLEAR isn’t a validity standard, though: a result that passes all five is still a single-organization observation. It gives you an account a skeptic can attack.

The CLEAR standard: Five tests for a reportable agentic AI result.

The CLEAR standard: Five tests for a reportable agentic AI result.

Naren Gangavarapu

Applying it to my own results

We used three outcomes internally to justify further investment. Two came out weaker under CLEAR.

Tier 1 support escalations fell 34%

This largely survives. The six-month window was fixed to include peak season, contact volume was flat year over year and counts came from the system of record, reviewed by finance rather than the agent team. With volume flat, falling demand can’t explain the drop. What doesn’t survive: Tier 1 is our own classification, so the absolute figure can’t be compared across organizations.

Operator onboarding fell from 23 days to 3

The baseline isn’t attributable. The 23 days included time in a standing backlog, and part of the gain came from clearing that queue, not shortening the process. It shows a process re-engineered and a queue eliminated, not a like-for-like comparison.

Booking conversion rose 24%

The weakest of the three. A marketplace rebuild and opt-in dynamic pricing ran in the same window as the AI-native search capability credited with most of the uplift. Reported without that caveat, it would be the most impressive number here and the least informative.

Two of three carrying material qualifications is probably typical. A result that survives disclosure of its weaknesses is worth more than one never tested.

The failure that makes everything else moot

CLEAR assumes the operational record is complete. In agentic systems, one gap quietly breaks that assumption.

The most important control in our deployment cost nothing to build. Any line-of-business manager, from the wholesale desk to the cruise operations floor, could stop an agent within ninety seconds. No ticket, no approval. If the manager was wrong, we restarted the agent and discussed it at the next recalibration session. Being wrong cost an hour of throughput; being slow could cost a customer.

A manager using that control is the system working as designed. The failure is the override that isn’t logged. The record that would have shown a tolerance band was miscalibrated never exists, and the agent’s statistics are computed over a population from which a conscientious human has removed the hard cases.

Every downstream metric improves. Escalation rates fall because escalations were intercepted first. Confidence scores look strong because low-confidence decisions were pulled out by hand. Exception reports look clean. The organization believes its agents perform better than they do, and its results become unfalsifiable, because the basis was corrupted before anyone thought to write it down.

This echoes Lisanne Bainbridge’s 1983 “ironies of automation”: the human handles what the designer couldn’t automate, with no way to feed that judgment back into the design. No care at write-up time recovers a record that was never captured. Measurement in agentic systems is a daily operating discipline, not a reporting exercise.

Six practices that keep the record intact

We treated agents as high-volume junior team members: fast, tireless and capable of significant harm if poorly supervised. You wouldn’t put a new hire on the floor without onboarding and an escalation path, and agents are no different. Every manager with agents adopted six habits:

  1. A five-minute morning review of overnight agent activity, treated as a floor walk.
  2. A weekly scorecard written for operations, not technologists: completion, escalation rate, decision confidence.
  3. An exception-first culture that asks what the agent flagged as uncertain, not only what it did.
  4. Explicit human-agent teaming norms: when a human overrides, when they defer and how it’s logged.
  5. Scenario playbooks for the ten likeliest failure modes, owned by operations.
  6. A monthly recalibration session reviewing tolerance limits against incident data.

The fourth carries the most weight and gets the least attention. The sixth is how the organization learns, by asking what fired that shouldn’t have and what didn’t fire that should have. Both are answerable only if the override record is complete.

One design decision supports all six: we allocated autonomy by reversibility, not risk appetite. Non-technologists can classify reversibility at design time, and it maps straight to a control: the less reversible the action, the earlier a human must stand in the loop. Of our 25 agents, seven could act without confirmation. Thirteen either required it or were barred from autonomous external communication.

Where this won’t work

If you ship agent changes weekly, there’s no stable intervention to lock a window around; report rolling comparisons and say so. If the agent does work nobody did before, name the tests that don’t apply instead of quietly dropping them. Disclosure has commercial cost: we withheld volumes but disclosed the comparison basis. And disciplined reporters will look weaker than those who aren’t, a coordination problem no one practitioner can solve.

What to do on Monday

Check whether human overrides of agent decisions are logged. Inspect the log; don’t just ask whether one exists. If overrides aren’t captured, no performance figure you hold about your agents means what it appears to.

Then, before the next reporting cycle, lock your measurement window and write it down: the period, the denominator and everything else changing in the same interval. That costs nothing in advance and is impossible afterward. Then ask of every result: what would make a reader disbelieve this, and have I said so?

The organizations that scale agentic AI won’t have the best models; capability is converging and procurable. The operational habits around it aren’t, and they took us eighteen months to build. The technology was the easy part. The measurement was the work.


Read More from This Article: Your agentic AI results can’t be checked. Here’s how to fix that
Source: News

Category: NewsNovember 18, 2026
Tags: art

Post navigation

PreviousPrevious post:What is a semantic knowledge graph, and does your AI need one?NextNext post:After Black October, OffSec Turns PHPH2Shell and Other New Vulnerabilities Into Hands-On Labs

Related posts

Microsoft keeps failing. Why can’t its customers leave?
November 19, 2026
FAA ground stop offers IT leaders stark modernization lessons
November 19, 2026
The dark side of AI in hiring: When job applicants become security threats
November 19, 2026
AI leadership: 3 success lessons from CBA’s chief AI officer
November 19, 2026
Trusted AI by design, not by default
November 18, 2026
Keep skyrocketing AI costs in check
November 18, 2026
Recent Posts
  • Microsoft keeps failing. Why can’t its customers leave?
  • FAA ground stop offers IT leaders stark modernization lessons
  • The dark side of AI in hiring: When job applicants become security threats
  • AI leadership: 3 success lessons from CBA’s chief AI officer
  • Trusted AI by design, not by default
Recent Comments
    Archives
    • November 2026
    • October 2026
    • September 2026
    • August 2026
    • July 2026
    • June 2026
    • May 2026
    • April 2026
    • March 2026
    • February 2026
    • January 2026
    • December 2025
    • November 2025
    • October 2025
    • September 2025
    • August 2025
    • July 2025
    • June 2025
    • May 2025
    • April 2025
    • March 2025
    • February 2025
    • January 2025
    • December 2024
    • November 2024
    • October 2024
    • September 2024
    • August 2024
    • July 2024
    • June 2024
    • May 2024
    • April 2024
    • March 2024
    • February 2024
    • January 2024
    • December 2023
    • November 2023
    • October 2023
    • September 2023
    • August 2023
    • July 2023
    • June 2023
    • May 2023
    • April 2023
    • March 2023
    • February 2023
    • January 2023
    • December 2022
    • November 2022
    • October 2022
    • September 2022
    • August 2022
    • July 2022
    • June 2022
    • May 2022
    • April 2022
    • March 2022
    • February 2022
    • January 2022
    • December 2021
    • November 2021
    • October 2021
    • September 2021
    • August 2021
    • July 2021
    • June 2021
    • May 2021
    • April 2021
    • March 2021
    • February 2021
    • January 2021
    • December 2020
    • November 2020
    • October 2020
    • September 2020
    • August 2020
    • July 2020
    • June 2020
    • May 2020
    • April 2020
    • January 2020
    • December 2019
    • November 2019
    • October 2019
    • September 2019
    • August 2019
    • July 2019
    • June 2019
    • May 2019
    • April 2019
    • March 2019
    • February 2019
    • January 2019
    • December 2018
    • November 2018
    • October 2018
    • September 2018
    • August 2018
    • July 2018
    • June 2018
    • May 2018
    • April 2018
    • March 2018
    • February 2018
    • January 2018
    • December 2017
    • November 2017
    • October 2017
    • September 2017
    • August 2017
    • July 2017
    • June 2017
    • May 2017
    • April 2017
    • March 2017
    • February 2017
    • January 2017
    Categories
    • News
    Meta
    • Log in
    • Entries feed
    • Comments feed
    • WordPress.org
    Tiatra LLC.

    Tiatra, LLC, based in the Washington, DC metropolitan area, proudly serves federal government agencies, organizations that work with the government and other commercial businesses and organizations. Tiatra specializes in a broad range of information technology (IT) development and management services incorporating solid engineering, attention to client needs, and meeting or exceeding any security parameters required. Our small yet innovative company is structured with a full complement of the necessary technical experts, working with hands-on management, to provide a high level of service and competitive pricing for your systems and engineering requirements.

    Find us on:

    FacebookTwitterLinkedin

    Submitclear

    Tiatra, LLC
    Copyright 2016. All rights reserved.