The hard truth of IT metrics

You might have heard that if you can’t measure you can’t manage. You might have believed it, too. And if you think you need metrics to manage you might be feeling guilty about not having enough of them.

Guilt no more. Good metrics are hard to craft, harder to manage, expensive to maintain, and perishable besides.

Bad metrics, in contrast, are easier all the way around, but that doesn’t matter. Bad metrics are worse than no metrics. If a bad metric isn’t lying to you outright, it’s directing your attention to what doesn’t deserve your attention.

Not to mention that knowing what’s going on isn’t the primary purpose of establishing metrics in the first place. That’s a fringe benefit.

What metrics do for you is a two-edged sword. Edge #1 tells employees what “better” means and lets them know whether they’re achieving it.

It’s two-edged because without metrics, employees will do what they think makes sense to make a given situation better. But with metrics, they will do what they think will move the needle, regardless of whether moving the needle is a good idea.

Misled by metrics

Metrics are tools, and like any tools they’re as easily misused as used. To help avoid misuse I’ve come up with a simple set of guiding principles:

Lewis’s 1st Law of Metrics: The primary purpose of any metric is to alter employee and team behavior. Letting you know how you’re doing? That’s nice too.

This is followed by Lewis’s 2nd Law of Metrics: You get what you measure – that’s the risk you take.

Which in turn is followed by Lewis’s 3rd Law of Metrics: Once you get employees in the habit of moving the metrics, they won’t do anything that doesn’t move a metric. Anything you don’t measure, you don’t get.

The 7 Cs of IT metrics

Clearly, if you’re going to establish a system of metrics to help you manage performance, you need good metrics. But what’s the difference between good and bad metrics, you ask? Good metrics adhere to what I call the 7 Cs. Okay, its name might be annoying, but if you ignore any of the 7 Cs you’ll end up with metrics that are, under the best of circumstances unhelpful, and under the worst actively destructive.

Good metrics must be:

1. Connected: Good metrics must be connected to something important — usually a business goal, or outcome of a process or subprocess. If a metric isn’t connected to something important it tells you nothing useful. And if the goal, process, or subprocess isn’t important, don’t bother establishing a metric to assess it.
2. Calibrated: No matter who measures what you’re measuring, they must record the same result. With this caveat: Some measures are subjective. Many aspects of employee performance are, for example, in the eye of the beholder. That doesn’t mean calibration doesn’t matter. It means it’s sometimes hard to achieve.
3. Consistent: When whatever it is you’re measuring improves, the metric always moves in one direction. When it deteriorates the metric always moves in the other direction.
4. Complete: It’s Lewis’s 3rd Law. Anything you don’t measure, you don’t get. Incomplete metrics ignore goals that matter.
5. Continuum: Okay, okay. As Cs go, this is a stretch. It should have been “Scaled,” but “6 Cs and an S” doesn’t sing in the shower. Good metrics fall on a known continuum, so you know if the number is good, bad, or indifferent.
6. Communicated: Go back to Lewis’s 1st Law. Uncommunicated metrics won’t alter behavior because employees and teams won’t know what they say.
7. Current: Goals change. If you don’t keep your metrics in sync, the organization will continue to achieve your old goals. Employees won’t help you achieve your new ones because you aren’t measuring them.

Putting IT metrics to the test

Aren’t convinced the 7 Cs framework works as advertised? Let’s test it against a popular industry metric: the “total cost of ownership” (TCO). It’s well-known and widely cited. Surely it must exemplify what quality metrics look like. Here goes:

Connectedness: Economizing is usually a worthwhile goal. TCO is connected.

Calibration: Every TCO component can be specified; in principle, the cost of each component can be tied down to well-documented spending; and the rest is simple arithmetic. TCO is calibrated.

Consistency: Uh oh. When TCO increases, does this mean the situation is worse than it was before and vice versa? Sometimes yes; sometimes no. Take, for example, the cost of connecting a PC to your network. Don’t do this and TCO goes down. But your PCs are pretty much worthless.

The TCO of almost anything increases the more it’s used. Conversely, you can bring TCO to zero through the simple expedient of turning everything off.

TCO isn’t consistent.

Completeness: TCO is as incomplete as it is inconsistent, because in business, cost doesn’t matter. Value is what matters, the net of benefit minus cost (or divided by cost if you prefer ratios). TCO ignores benefit and value, and anything you don’t measure you don’t get. TCO is incomplete.

On a Continuum: In principle, researchers could survey the product category’s marketplace, measure every competing product’s TCO, and report where each product’s TCO falls in the range of measured values. TCO can be placed on a continuum, but in practice it’s unlikely.

Communication and Currency: Unlike the first five Cs, these are about your metrics practices, not the metrics themselves. Whether TCO is communicated and current is a matter of choice, not of TCO itself as a metric.

TCO Metrics Success Metric: Taking practicalities into account, TCO fails three of the 7 Cs — consistency, completeness, and the need for a continuum. Which leads to the question, are yours any better? And if not, what steps do you need to take to fix them?

Wrapping it all up

If you measure well, you can probably manage better than if you don’t — so long as, that is, the cost and difficulty of measuring well don’t exceed the benefits.

Maybe we need to compute the Total Cost of Measurement. Or, better yet, the Total Cost of Management.

Any volunteers?