Before you automate anything, learn to measure it

There is a question that quietly embarrasses more automation projects than any technical failure: How much did it actually save? Ask it a year after launch and watch what happens.

The team is confident the new system is better. Everyone remembers how tedious the old way was. And yet nobody can produce a number, because nobody measured the old way before replacing it. The improvement is probably real, but it remains an article of faith rather than a demonstrated result.

I’ve spent much of my career automating workflows within large organizations — whether contributing to intelligent assistance features in Microsoft Office, architecting AI expert systems such as the Intelligent Filing Manager (INTELLIFM) across client-server and web platforms, or engineering financial automation such as the Bloomberg Valuation Service (BVAL) for the finance industry. Across every domain, I learned early that the measurement question isn’t a formality that follows the engineering. It comes first.

Automation only pays off if you can prove it did, and proof begins before a single line of code is written.

Manual work hides its costs well

The case for measuring first starts with an inconvenient property of manual workflows: their true costs are almost invisible from above. A process that “takes about a day” rarely takes a day of active labor. It usually consumes three hours of actual effort (touch time), spread across a week of organizational latency (elapsed time) — waiting for approvals, waiting for handoffs or waiting for someone to notice an item sitting in their queue. The costs that matter most are precisely the ones no one tracks: rework when a document comes back with errors, delay while a request sits between steps, and inconsistency when five people perform the same task in five different ways — with each believing theirs is the standard.

Baselining exposes all of this, which is why it so often surprises the people who commissioned it. When you map a process end to end and attach numbers to it — elapsed time, touch time, error rates, variance between performers — you routinely discover that the workflow everyone thought they understood behaves quite differently in reality. That discovery has independent value: more than once, careful baselining has revealed steps that shouldn’t be automated but eliminated.

There’s no point perfecting a task that shouldn’t exist. To capture an accurate baseline, examine historical audit trails, system logs and ticket completion timestamps rather than relying solely on self-reported estimates, which are vulnerable to recall bias. When direct observation is necessary, account for the Hawthorne effect, in which people may behave differently because they know they are being observed.

The honest method: Baseline, then measure

The honest way to judge automation is embarrassingly simple to state: establish a quantified baseline first, then measure against it after. What makes it rare isn’t difficulty but timing. Once the new system ships, the old process may disappear or change substantially, making its true cost difficult to reconstruct. The cleanest opportunity to establish a baseline often closes at launch. Miss it, and every efficiency claim afterward becomes harder to defend.

This matters beyond intellectual honesty. Automation initiatives compete for budget against everything else the organization could do, and the initiatives that can say, “cycle time fell 40 percent against a measured baseline” win those arguments over the ones that say, “everyone agrees it’s much better.”

A measured result also protects the project when leadership changes or budgets tighten — sentiment is easy to dismiss; a baseline comparison isn’t. And there’s a subtler benefit: defining the measures up front forces the team to agree on what the automation is intended to do. A surprising number of projects discover in the middle of the metrics argument that stakeholders were pursuing different goals under the same project name. Better to have that argument before the build than after.

The measures themselves should be few and meaningful. Elapsed time from request to completion. Active effort consumed. Error and rework rates. Consistency across performers and cases. Resist the temptation of “dashboard theater” and the burden of 30 metrics. A small set tracked honestly over time beats a massive spreadsheet tracked sporadically, and every vanity metric you add creates collection fatigue after the initial excitement fades.

Agentic automation requires an additional layer of measurement. An agent may merely advise, act with human approval or act autonomously. For agents permitted to act, you must measure the health of that autonomy. Gartner recommends governance proportional to an agent’s autonomy and access because controls that are too restrictive or too permissive create different operational risks.

A practical scorecard should track three indicators: the Autonomous Completion Rate, which measures eligible work completed correctly without human intervention; the Escalation Rate, which measures cases handed off to a person; and the Reversal Rate, which measures actions a human had to undo or correct. A high reversal rate is particularly damaging because it creates additional work. The agent performs the step incorrectly, and a human must then diagnose and repair its output. An agent whose actions are routinely reversed has not earned its autonomy. Only rigorous tracking will reveal this.

Gains erode unless someone sustains them

Here’s the part the launch celebration never mentions: automation gains decay. This does not necessarily happen because the software itself degrades, but because the operating environment changes. The process resembles system entropy in software: exceptions, edge cases and manual workarounds accumulate until the workflow drifts away from its intended design. As this happens, users quietly revert to old habits or bypass automated steps. Volumes shift, and a year later the workflow is partly automated and partly folklore — the gains, never remeasured, have silently surrendered some of the value they once created.

Agentic systems also raise the stakes of this decay. When software merely suggests, a person sees every output and complaints surface early. However, when software acts, fewer eyes fall on each action, and erosion loses its last natural alarm.

The defense against backsliding is twofold. First, standardization turns the improved process from a local achievement into the documented and expected way of working. As a result, the gain does not depend on the memory of the people who happened to be there at launch. Second, continuous improvement, supported by scheduled remeasurement, treats efficiency as an ongoing practice rather than a one-time milestone.

The organizations that sustain their gains schedule follow-up reviews before the project closes. They recognize that once the project team disbands, attention goes only where the calendar sends it.

Preventing this silent value erosion requires moving from a reactive mindset to a structured operational model. As McKinsey’s research on skills in the AI age observes, realizing AI’s economic potential depends less on new inventions than on how organizations redesign workflows and how quickly skills adapt.

Quantifying automation the right Way

Four disciplines, applied in order, make the difference.

1. Map the current process and capture a baseline before you change anything. Walk the workflow end to end as it actually happens, not as the procedure manual describes it. Record elapsed time, touch time, handoffs, error rates and variance. Expect surprises; they’re the point. The baseline is the one measurement you can never take later.
2. Pick a small number of meaningful measures and track them consistently over time. Choose measures that reflect what the automation is for — speed, quality, consistency, cost — and hold them stable so trends stay comparable. Consistency over months beats sophistication in month one. If a measure requires heroic effort to collect, it will stop being collected; design for sustainability, not impressiveness.
3. Standardize the improved process so the gains don’t quietly slip back. Document the new workflow, train to it and make it the path of least resistance. Fold exceptions into the standard deliberately instead of letting workarounds breed in the shadows. A gain that lives only in the launch team’s habits leaves when the team does.
4. Revisit the numbers on a schedule. Put remeasurement on the calendar — quarterly is a reasonable default — and treat erosion as a signal, not a failure. Sustaining efficiency takes attention, not just at launch. A review that detects a 10 percent decline this quarter can prompt timely corrective action. That is far less costly than discovering two years later that a once-celebrated automation has gradually become yet another tedious process.

None of this diminishes the engineering. It completes it. The organizations that measure before they automate know what their improvements are worth, can defend them when budgets tighten, and catch the erosion while it is still cheap to reverse. The ones that skip the baseline have something weaker than a result. They have a story, and stories, unlike baselines, cannot be audited.