Is it worth measuring software developer productivity? CIOs weigh in

Most enterprises are committed to a digital strategy and looking for ways to improve the productivity of their workforce. At the same time, developers are scarce, and the demand for new software is high. This has spurred interest around understanding and measuring developer productivity, says Keith Mann, senior director, analyst, at Gartner. “Organizations need to get the most out of the limited number of developers they’ve got,” he says. “Gartner’s surveys and data from client inquiries confirm that developer productivity remains a top priority for software engineering leaders.”

Dominic Titcombe, CIO at Delta Dental of California, adds that recent advances around generative AI have inspired new ways of working, and there’s been much discussion on applying AI to accelerate software creation. “There are clearly tremendous tools in this space like GitHub Co-Pilot that developers can use to enhance and augment their productivity,” he says.

Angelic Gibson, CIO at accounts payable automation software and payment solutions provider AvidXchange, agrees that removing friction in the developer workflow can enhance agile innovation. “Focusing on innovation and tech deployment helps pinpoint and eliminate obstacles that impede tech teams,” she says, adding that while measuring software development production is essential for IT digitalization, it also requires a careful rollout to maintain a healthy team dynamic. “Connected teams exhibit greater ownership and commitment, resulting in enhanced productivity,” she says.

Streamlining to optimize productivity

Agile software development is essential to innovate and retain competitiveness. Therefore, engineering leadership should measure software developer productivity, says Mann, but also understand how to do so effectively and be wary of pitfalls. “Done properly, measuring productivity gives insights into how development teams can deliver more value to users and customers, and that’s what leads to positive business impacts,” he says.

Titcombe similarly believes that assessing software developer efficiency is worthwhile, noting how it helps IT reach its goals to deliver great products for end consumers. “It behooves any business division to look for productivity improvements and to find ways to do more with less,” he says. “A key part of building experiences for our customers is doing it quickly and cost-effectively while delivering a great product.”

However, delivering great digital products can be challenging if software development teams are not set up for success. Too often, IT grapples with big backlogs of features, hindering greenfield development, says Gibson. “Once IT’s in the backlog, the time it takes to get it back into production is crucial,” she says.

Development teams also often experience bottlenecks that prevent smooth workflows, adds Gibson, including intricate code, complex architecture, or insufficient automation and testing. Since friction within the software development process diminishes productivity, she says gaining insight into these roadblocks is imperative to avoid what’s holding teams back.

Friction can slow the speed of innovation, which can impact a company’s overall revenue and bottom line. “Just as Netflix innovated against Blockbuster with seamless technology development, companies that streamline this process can accelerate market innovation, boosting revenue and profitability,” says Gibson.

Not all executives are convinced that developer productivity measurement can have actionable outcomes, however. Instead, it might be this emphasis on streamlining processes that matters most. “The focus on developer productivity is a fool’s errand,” says Kyle Campbell, CEO and founder at code testing platform CTO.ai. “A more experienced and hands-on leader knows that a team’s output is directly related to the level of support they have to focus on doing their best work.”

Therefore, he recommends empowering development teams by critically thinking through how their developer workflows, such as CI/CD, can be optimized, and empirically measuring the developer experience in these areas.

Measure business outcomes, not lines of code

There are various measurement points throughout the software development lifecycle (SDLC), from idea generation to production stages, that should be monitored to ensure a smooth flow. “If businesses aren’t enhancing the efficiency of these stages or the deployment of commercial technology, they risk falling behind their competitors,” says Gibson.

The desire, however, to measure software developer productivity is itself confronted with obstacles. Though there are many schools of thought about how to do it accurately, the general sentiment from technology leaders is to avoid measuring contributions at the microscopic, individual contributor level.

“Counting lines of code generated per day per developer can lead to false productivity measurements,” says Titcombe. Instead, it’s good to examine the speed of new feature delivery. “An overall better measurement of how effective developers are is if we can get tools and experiences in our customers’ hands quicker, which will have an overall greater benefit,” he says.

One big caveat of some productivity measurements is that some can lead to false positives, or cause engineers to game the system. “As soon as a developer realizes they’re being measured on a certain metric, they’ll aim to artificially inflate that metric,” says Titcombe. “A better one is an enterprise productivity metric that focuses on outcomes delivered to customers.”

At Gartner, they’re seeing interest from clients in implementing certain developer productivity frameworks, says Mann. One such framework is SPACE. Proposed by GitHub researchers, SPACE augments the DevOps Research and Assessment (DORA) framework with more qualitative measurements based on satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and workflow. Another framework Mann has seen in use is DevEx.

Attributes within these frameworks can help measure developer productivity, some more objectively than others, notes Mann. Yet, he encourages leadership to be purposeful with their intent when implementing them. Ideally, measurement activities should unveil roadblocks preventing positive business outcomes and not be used to place specific contributors on a pedestal.

“The purpose of the measurement isn’t to determine whether one developer is better or worse than another by comparing their metrics,” says Mann. “Instead, the purpose is to diagnose which factors might lead to higher or lower productivity for the developer in question.” For instance, he recounts how a client using the SPACE framework unveiled a communication breakdown, which was successfully mended to reduce quality problems and rework.

These sorts of small fixes, exposed by intelligent productivity monitoring, can enable quicker turnarounds that pay out dividends. “When it comes to productivity, it’s about how quickly businesses can progress from conceptualizing an idea and defining its specifics, to planning the architecture, says Gibson. “Productivity directly translates to speed in both market entry and innovation, ultimately impacting the bottom line.”

Teamwork yields productivity gains

Improving software development productivity doesn’t have to be encouraged through metrics alone. Another significant contributing factor to overall productivity is the sense of ownership and commitment a developer has with their team.

“Team connectivity is a cornerstone of productivity,” says Gibson. “In order to have highly productive teams, people need to feel connected, and have a sense of belonging and cohesion with the teams they’re working with,” she says.

Getting a better grasp on productivity could also mean reimagining the concept in its entirety, since the typical industrial definition of what it means to be ‘productive’ simply doesn’t transfer well to the fluid software design and development process. Software isn’t like producing some mechanical widget, whose manufacturing process remains identical for each unit, says Mann. Software is more nuanced and ever-changing, and the end value differs from component to component, complicating traditional productivity measurement techniques.

“Every piece of software is unique and has a unique value,” says Mann. “It’s meaningless to say, ‘We produced twice the number of pieces of software as we did last week, so we were twice as productive,’ because this week’s software may be half the value.” Therefore, productivity measurements can often be illusions without real tangible benefits. “What we need to do is understand productivity as the amount of value we’re delivering per unit of time or cost,” he says.

The other implication is the realization that software is not created in isolation — it’s a collaborative process with many stakeholders involved in each sprint. “Most software is produced by teams of developers, not individual developers,” says Mann. Therefore, leaders should seek to evaluate the team’s productivity — Mann describes productivity as ‘value per unit time’ — over extended periods to truly gauge whether productivity improvements are effective.

“If you can assess value consistently across teams, then you can even compare their productivity,” notes Mann. However, that’s a big ‘if,’ he adds, since value depends highly on the business domain in question and varies greatly across stakeholders.

Of course, value isn’t always easy to measure, underscoring the need for a flexible approach, especially when comparing team dynamics. Therefore, instead of relying on a specific universal metric, it may be more beneficial to reveal trends relative to the teams in question.

“It’s more meaningful to compare and understand trends and use those as the basis for deeper questions,” says Mann. “For example, if one team’s productivity is trending upward while a similar team’s isn’t, we might ask what the first team is doing differently.” Asking such questions could expose knowledge to be shared throughout the company, which would help other teams improve.

In the context of developer experience, key areas to focus on take on slightly different characteristics. “When we talk about development outputs anecdotally, it’s critical to assess the key components of a developer’s experience with feedback from teams,” says Campbell. He groups these components into Clarity (How do I deploy), Ease of use (What are the minimal steps to deploy), Functionality (Is there an existing workflow, API, or SDK I can extend), and Stability (Can I be sure this won’t break in the middle of the night if I deploy it now).

By listening to the feedback from engineers in these areas, leadership can “quickly develop a level of empathy for what areas they need support to do their best work,” says Campbell. With this in hand, IT can best invest in improvements that enhance productivity and positively affect the business.

The developer and customer experience

Technical leaders should be cautious about measuring developer productivity, and if they do attempt it, results must based on tangible value to end consumers.

“Executives should ensure that productivity measures focus on customer experiences and outcomes, and that teams are agile while supporting new opportunities as they arise,” says Titcombe. “We want to prioritize ways in which technology can help us take care of patients now and in the future,” he adds.

Leaders should also remember that mental energy has limitations, and burnout is a real possibility for knowledge workers. As such, when measuring performance, Gibson says it’s essential to focus on processes rather than individuals to avoid instilling fear. “By emphasizing the effectiveness of the overall process and assessing the efficiency of the measurement process itself, the focus shifts to how well individuals are operating within that framework,” she says.

For others, measuring developer productivity alone can be a red herring. Instead, Campbell encourages fostering a culture of continuous improvement and discovering strategies to better instrument developer workflows, and from there, measure this toolchain to garner actionable development insights. “Just as we measure the impact our software has for end users trying to achieve a goal, we can also measure the impact our internal tools have for our objective, too,” he says.