Anthropic highlights productivity gains from use of Claude

A report from Anthropic has highlighted the types of savings that it says could be achieved through the implementation of its AI, Claude, throughout an enterprise.

According to the report, Estimating AI productivity gains from Claude conversations, organizations that have implemented Claude can expect to see staggering productivity gains when implementing a series of tasks such as those required for curriculum development for teachers, invoice production, or financial analysis.

The company used Claude to evaluate 100,000 anonymized transcripts of interactions to estimate the productivity impact of its use. “Based on Claude’s estimates,” it says, “these tasks would take on average about 90 minutes to complete without AI assistance, and Claude speeds up individual tasks by about 80%.”

Based on Anthropic’s observations, the report indicates a number of places where advantages could be achieved through the use of AI, and some where it would not be of use. For example, it says, for software developers, AI speeds up the processes involved in software development, testing, documentation, and manipulating data. But the company does not currently see “meaningful” AI use in the coordination of system installation or the supervision of the work of other technologists or engineers. For teachers, it says, “We see that AI assists with lesson and activity planning, but not with sponsoring extracurricular clubs or enforcing rules in the classroom.”

Analysis has limits

Extrapolating its estimates, Anthropic suggests that current-generation AI models could increase US labor productivity growth by 1.8% annually over the next decade, roughly twice the improvement rate in recent years. But, it notes, “where AI makes less of a difference, these tasks might become bottlenecks, potentially acting as a constraint on growth.”

It also observes that its results are only based on Claude conversations, so don’t reflect the full spectrum of AI uses, and that, historically, the largest productivity gains in organizations have been achieved through restructuring business operations to incorporate new technologies.

In addition, Anthropic points out other possible sticking points in its findings. “Our analysis has limits. Most notably, we can’t account for additional time humans spend on tasks outside of their conversations with Claude, including validating the quality or accuracy of Claude’s work,” the report states.

And, it notes, “our approach doesn’t take into account the additional work people need to do to refine Claude’s outputs to a finished state, or whether they continue iterating on the work product across multiple sessions, both of which would result in smaller time savings.”

The report does have an element of Claude marking its own homework, and it’s easy to be cynical about its assumptions, especially given Claude’s instincts for self-preservation, demonstrated in an Anthropic report in May this year in which Claude resorted to blackmail when threatened with replacement.

A ‘nuanced’ report

However, Tarek Nseir, founder of recently-launched AI consultancy Valliance, said that the Anthropic report was quite nuanced. “They have done a pretty good job at self-declaring the issues,” he observed. “The numbers that they’re describing aren’t outperforming what we’ve been seeing. They’re obviously cherry-picking tasks, concentrating on long-form stuff, but it’s a well-structured, transparent report.”

But, Nseir said, while some of the basic assumptions are sound, Anthropic is underestimating the cumulative effect on a series of tasks. “When you get an inaccuracy in one task, but that is just one part of a chain, you’re going to see any errors compounded over time,” he noted, adding that, while Anthropic’s estimates on time savings for individual tasks are accurate, the aggregate savings that they’re talking about would be hard to achieve.

As for Claude’s propensity for blackmail, Nseir said, “we’re not seeing that sort of behavior in the field. Besides, genAI is improving all the time. The development curve and the safety improvements curve have been very sharp.”

For CIOs looking to implement AI throughout their organizations, regardless of whose technology is under consideration, Nseir had this advice: “I would recommend two approaches: A People First one, where you give employees the right sort of tools to improve their productivity, and a Value First, where you look at the business as a whole, and how the processes fit in.”