Generative AI copilots: What’s hype and where to drive results

GitHub first launched its copilot in 2021, and Microsoft 365 Copilot became generally available a few months ago. These  AI assistants often use the term copilot to indicate how generative AI capabilities embedded in workflow tools can augment and assist people in performing tasks and prompting for information more efficiently.

The term copilot has caught on, and now many platforms have added embedded AI assistant capabilities as copilots, including Appian AI Copilot, Check Point Infinity AI Copilot, Creatio Copilot, ExtremeCloud IQ CoPilot, Freshservice Freddy AI, Nice Enlighen Copilot, OpenText Co-Pilot for ALM, PagerDuty Copilot, Planview Copilot, Salesforce Einstein Copilot,  SAP Joule, and Tableau. Many other platforms, such as Coveo’s Relative Generative Answering, Quickbase AI,  and  LaunchDarkly’s Product Experimentation, have embedded virtual assistant capabilities but don’t brand them copilots.

Many employees want to experiment with  AI assistants like Microsoft Copilot, while CIOs are under pressure from their CEOs to realign digital transformation priorities and deliver business value with generative AI capabilities. CIOs must also partner with CISOs, legal, human resources, and business leaders to build awareness of policies and develop a generative AI risk management strategy.

CIOs and IT leaders are at the center and must decide what copilots to test, who should receive access, and whether experiments are delivering business value. Given how fast technology platforms release copilot functionality, sorting out what’s working today and can scale, what features have limited functionality, and what capabilities are marketing hype can be time-consuming.

Copilot benchmarks show productivity improvements

Early benchmarks show that people using copilots are more productive and use time savings to focus on higher-level functions.

GitHub released data on its copilot’s impact, with 88% of surveyed developers stating improved productivity, 74% focusing on more satisfying work, and over 87%  saying they complete tasks faster. In one benchmark, GitHub asked developers to write an HTTP server in JavaScript; the developers using copilot completed the task over twice as fast compared to those not using copilot.

Microsoft’s benchmarks show that 70% of Copilot users said they were more productive, 68% reported it improved the quality of their work, and 67% used the time saved to focus on more important work.

This data shows the business potential, but you have to dig deeper to find how people actually use copilot capabilities and which ones deliver business value today.

Summarizing meetings, emails, and documents

Microsoft 365 will set the benchmark on copilot capabilities, given the speed at which it introduced embedded AI assistant features and the platform’s wide user base.

So, what delivers on the productivity promise today? Summarization and content transformation capabilities — available across the Microsoft 365 product line — can be important productivity drivers.

“In Microsoft Teams, you can do meeting notes, summarizations, recaps, and even ask Microsoft Copilot questions like what was the meeting about,” reports Roman Dumiak, executive-in-residence at DePaul University.

David Kleinman, chief digital and information officer of Mission Veterinary Partners, elaborates, “Some people send five-paragraph emails in long email chains. Will you spend 20 minutes reading it, or can you read a summary? Microsoft Outlook does a pretty good job at email summarization, but to use this functionality, you have to use the web version, or you have to use the beta version of Outlook desktop.”

“Microsoft Copilot excels in generating summaries for Microsoft Teams meetings, assists in efficient email composition, can convert MS Word files into presentations, and simplifies Excel data analysis by enabling users to create visualizations and predictive models from their data,” adds Kiba Polk, chief solutions architect at Dynamic Solutions Consulting.

DePaul’s Dumiak adds, “While in Excel, I can ask Microsoft Copilot to summarize tables and give me charts, and suddenly, it’s created pivot tables without ever having to learn the command generation sequence.”

But there are some limitations in Copilot for Microsoft Excel, as the data must be in a table for it to work.

While that’s a limitation, there are reports of promised functionality not yet available.

“Microsoft says Microsoft 365 Copilot is a general release, but it seems like it’s still in beta with features they advertise on their website that it doesn’t actually do yet,” says Kleinman. “They advertise a feature where you can follow a meeting, and then Copilot will join and take notes for you.”

Kleinman believes executives will want this added workflow functionality to make it easier for people to sit virtually in a meeting they want to be summarized, especially when they aren’t going to be active contributors. The ambiguity of what’s working today and which users will benefit is driving some CIOs to ask whether adding Copilot licenses to Microsoft 365 is worth the price.

Microsoft is heavily investing in AI capabilities and workflow integrations, so CIOs should expect and plan for improved capabilities. The biggest question CIOs should help answer is where to experiment and learn Copilot’s impacts on workflow. CIOs should seek out departments and employees that are heavy Microsoft 365 users and create opportunities for them to learn, try, and report on Copilot’s capabilities and benefits.

Who benefits from software development copilots

The benefits of using Microsoft Office 365 Copilot may lie in setting realistic expectations and evaluating whether the results improve productivity. For software developers, the benefits of using copilots and other generative AI capabilities may be more about who is using it and the cost-benefit of validating code results.

IT leaders are exploring how different gen AI tools transform the software development lifecycle. Many are preparing a new world of developers as AI agents, with software development being closer to a manufacturing process. Today, top AI-assistant capabilities delivering results include generating code, test cases, and documentation.

GitHub’s research shows that users accept 30% of code its Copilot suggests and that less experienced developers have a greater advantage with AI. The research claims developers are faster and more fulfilled when using Copilot.   

Ali Dasdan, CTO of ZoomInfo, says, “In just three months, nearly all of our individual contributors were onboarded to GitHub Copilot. We saw near-immediate success, as we accepted tens of thousands of lines of code suggested by Copilot with an accuracy north of 26%.”

Other tools for gen AI in coding include Amazon CodeWhisperer, Seek, and Tabnine.  

“I lead a team of 20 developers regularly leveraging generative AI as a coding copilot, and each one has seen productivity improve by 20% on the low end to 100% on the high end,” says Mike Finley, CTO and co-founder of AnswerRocket. He shares the allure of using gen AI, “I often just write a comment indicating what I want the next few lines to do, and AI fills it in,” but also the reality that they still need to review the code.

“We produce a lot of code, so additional efficiency and finding ways to improve the speed of how we develop solutions is crucial for us,” says Luis Ribeiro, head of engineering and digital solutions at CI&T. “Tabnine has boosted developer productivity, and our developers accept 90% of the tool’s single-line coding suggestions resulting in an 11% productivity increase across projects.”

Some CIOs I spoke with say they see fewer benefits in giving junior developers access, largely because of the skills required to prompt and validate Copilot’s code. CIOs may also want to consider each application’s usage, security, and risks to decide which devops teams should experiment with AI copilots.

“The secret to CTOs leveraging gen AI copilot tools is finding the right balance between leveraging AI assistance and maintaining human oversight and control to ensure optimal outcomes,” says Anurag Malik, President and CTO of ContractPodAi.

To drive results with copilots, IT leaders should weigh in on who should experiment, which business functions, what compliance considerations, and which AI gen tools. As copilot technology capabilities are changing rapidly, leaders should frequently identify metrics and evaluate strategies.