Misled by metrics: 7 KPI mistakes IT leaders make

Mark Twain famously remarked that there are three kinds of lies: lies, damned lies, and statistics. Today, many CIOs feel the same way about metrics.

Metrics are only as good as their source. “Too often, technology companies pay consulting or analyst firms to create metrics based on the best characteristics of their offerings,” says Judith Hurwitz, CEO of Hurwitz Strategies, an emerging technology consulting firm. “Therefore, CIOs must be cautious about taking metrics at face value [and] leaders need to understand the data behind the metrics.”

Metrics interpretation is essentially a numbers game, and as with any numbers game, it’s possible to win or lose. Here are seven ways IT leaders are often misled by key performance indicators (KPIs) and other critical business and IT metrics.

1. Not considering the source

When studying a metric, it’s important to know who created it and the data source. Results may be based on a survey, for instance. If so, ask how many people were surveyed and the roles they played in their respective organizations. Check as well to see whether the metrics are based on a well-proven methodology. “It’s important to understand the research and data behind the metrics,” Hurwitz says.

Also consider the metric’s purpose. Will it be used as a planning tool? If so, will it help determine a business strategy, a technology selection, or some other need? “Metrics are only one tool for decision-making,” Hurwitz notes. “Therefore, approach metrics with skepticism.”

2. Failing to collaborate with front-line personnel

By now, most enterprises have reached data maturity. “If your company has data, you’re definitely leveraging it and trying to use insights from analytics to drive positive business outcomes,” says John Loury, president and CEO of Cause + Effect Strategy, a business intelligence consulting firm. “It’s 2022, we’re past the age of DRIP — data rich, insight poor.”

Loury believes that most organizations don’t dig deep enough when communicating with the front-line business personnel who will ultimately use collected metrics to make decisions and drive actions. Before building analytics, he recommends collecting business requirements from all involved parties. This means distilling metrics down to the data points most relevant to drive outcomes, Loury notes. “Prioritize what most directly impacts the business decision your user is trying to make.”

Loury advises building and honing communication skills to convey metrics-based insights to team members. “Modern CIOs and analytics leaders need to be adept at pulling together the key metrics that will drive the most impact for a team and presenting them in a way that makes sense to the user and will help guide their behavior,” he says.

Loury adds that it’s also time for CIOs to task their teams with truly understanding their users and building them tailored, effective analytics solutions. “The days of data leaders and their teams scrambling to build something — anything — and ship it to business teams are behind us,” he explains. “We’re living with the results of those days, where teams are inundated with wall-to-wall dashboards that tell them everything and nothing.”

3. Overlooking the importance of ownership, involvement, and balance

Metrics present an excellent opportunity for ownership and staff involvement, as well as continuous improvement and process control. “The key to correctly interpreting metrics is to engage your whole team and use the metrics to collectively improve processes,” says Paul Gelter, coordinator of CIO services at business and technology consulting firm Centric Consulting.

When evaluating metrics, Gelter believes it’s essential to strike a balance between cost, quality, and service. Cost metrics, for example, could be tracked in completed tickets per individual, yet ticket quality could be degraded by rework/repeated tickets. “Service could then be impacted by the response time, backlog, and uptime,” he notes. It’s all about obtaining an optimal balance.

4. Chasing the wrong numbers

Time really is money, so don’t squander precious hours scrutinizing irrelevant metrics. Clearly identify all goals before deciding which metrics to study. In most cases, metrics that don’t support or reflect future decision options are unnecessary and, worse yet, distracting and time-wasting.

Once the goal has been fully defined, allocate sufficient time to understanding the factors that cause individual metrics to fluctuate, suggests Alex Levin, co-founder of technology and design studio L+R. Next, investigate how individual metrics are tied to one another, and what’s likely to happen during different stages within an initiative’s or project’s lifecycle that might directly affect the KPIs being tracked.

Meanwhile, don’t waste staff time by concealing or hoarding conclusions. Levin advises sharing study results with your team, ensuring that each individual can use metric-driven insights to improve performance and/or outcomes.

5. Going it alone

Metrics research and study shouldn’t be a solitary endeavor. Mike Capone, CEO of analytics and data integration platform developer Qlik, and a former CIO, recommends working with functional area owners at the outset to gather and apply valuable contextual details. “These inputs and relationships give the CIO and the IT team the right level of understanding of what’s actually happening in the business … to support operational goals in the short- and long-term,” he explains. Capone also recommends building strong advisory partnerships with C-suite and other key enterprise leaders.

6. Trusting the numbers too much

A heathy dose of skepticism can keep you from being led down the path to faulty conclusions. Remember Twain’s quip about statistics and lies. There’s always the possibility that the collected data is itself flawed in some way.

Data can be flawed in many ways. The sample size could be too low, the time scale could be off, or whoever collected the data might have their own conclusion to promote. “It’s vitally important to make sure you completely understand how the data is collected and what’s included in the scope before you can make a determination on what it’s telling you,” says Brian Winters, CTO at ERP software developer ECI Software Solutions.

In fact, any metric can be misleading, especially if you don’t have a good overall understanding of the data. “System metrics can be particularly misleading because they often provide metrics for a very small part of a large, complex system,” Winters notes. “That narrow view can easily lead you down a rabbit hole.”

7. Failing to see beyond the stats

Metrics, while typically insightful and valuable, may not tell the entire story. In fact, taking any metric at face value can occasionally lead to utterly wrong conclusions. “Sometimes, you have to dig deeper with other, less obvious metrics, to determine what’s really happening,” explains Adi Gelvan, CEO and co-founder of database software developer Speedb.

For example, a high memory utilization level reading might imply that an application is overloading memory. “But something completely different may be at issue — perhaps a component that’s not cleaning up the memory fast enough,” Gelvan says. Further investigation can point to the real bottleneck, which may not be in the memory at all. “For example, if the storage engine cannot effectively dump the data to disks while I/O consumption is high, memory will fill up fast and impact the performance of the system.”

To protect against misleading insights, learn to think critically and don’t immediately leap to what seems to be the most obvious conclusion. As business processes and data architectures grow larger and more complex, many things can go wrong, and finding the root cause can be tricky. “The best approach is to surround yourself with a diverse team of subject matter experts to consult with before making decisions,” Gelvan advises.

ROI and Metrics