A 4-step framework for generative AI success

Before the advent of generative AI, we at Rest — one of Australia’s largest superannuation funds — had already embarked on a strategy to simplify the retirement investment experience for our members.

With the launch of ChatGPT in November 2022, however, the landscape shifted dramatically, and we recognised the potential of this technology to further drive efficiencies for our members — as well as potential to introduce additional risks to our highly regulated organisation.

Around 50% of our members are under the age of 30 and many work in part-time and casual roles, meaning they often have lower account balances than the national average. It is crucial, therefore, that we operate efficiently, while ensuring strong returns on our investments. Generative AI presented a great opportunity to achieve this. But, as many organisations have found, realising meaningful business value with gen AI can be challenging.

To achieve our goals from a technology standpoint, we needed a pragmatic, controlled approach to unlocking the benefits of gen AI that aligned with our organisational strategy and risk appetite.

To address this need, we developed a framework based on the Lean Startup method. Our model — “Test, Measure, Expand, Amplify” — is designed to guide and scale generative AI projects, while mitigating risks and achieving measurable business outcomes.

Organisations looking for a practical project management approach to delivering value-oriented generative AI solutions may find guidance in adopting or adapting the following four-step framework we constructed at Rest.

Step 1: Test — start small to validate ideas

Unlike the Lean Startup method, which focuses on “Build” as its first step, our framework opens with experimentation. In many cases, gen AI models are already “consumer-ready” and don’t require significant software development to get started. But before committing significant resources to an AI initiative, starting small, and validating the idea, is essential.

We tried a few experiments in this phase, including the introduction of RestGPT to improve employee productivity. In our first release, we used ChatGPT’s “engine” running on enterprise infrastructure in a safe environment with data in our own separate tenant.

To ensure we followed a controlled and structured approach, we established guardrails, including a responsible use policy where employees agreed to use gen AI in line with our risk and governance approach. 

We then set up a working group to act as advocatesacross the company to help build interest in the project. We chose a few key use cases to test, which aligned with our goal to drive efficiencies that benefit our members.

To make this a true experiment, we set clear benchmarks for each use case. It was critical to understand how much time employees spent on each task before introducing gen AI so that we could measure any real improvements. 

This Test phase allowed us to validate gen AI in a controlled way, ensuring it aligned with business needs while systematically managing risk. 

Step 2: Measure — define metrics that matter

The Measure phase in our framework evaluates each use case based on defined metrics established during the Test step. This is a particularly important step, as it is where the project team will make the critical decision on whether to continue investing in a use case or to stop and focus on higher-value opportunities. 

At its height, around 90% of our employees were using the RestGPT tool. But as IT and project leaders know, usage of a tool is just an indicator — not a KPI in and of itself. It’s essential to measure productivity gains aligned to strategic goals, in our case driving efficiencies that benefit our members.

As an example, one use case we tested with RestGPT was with our finance team for analysing market insight reports. With RestGPT’s assistance, the time needed to perform this analysis was reduced by around 85% — a significant time savings for our analysts during the pilot period.  

This is exactly the type of result to look for in the Measure phase: a clear, quantifiable efficiency gain, as it serves as a strong indicator of value and justifies scaling that use case. 

Step 3: Expand — scale what works

In the Expand phase of our framework, weidentify additional use cases to scale gen AI’s impact in areas where our experiments have been successful. However, we also learned that not every use case will scale as expected. 

Having confirmed RestGPT drove productivity improvements during our Test phase, we then looked beyond chat-based AI and began exploring enterprise-wide AI integration.

We were also running a parallel test with a chat automation tool designed to support employees by generating recommended responses for online chat. The tool provided AI-generated replies that agents could copy, edit, and send in live chat interactions with members. 

On paper, the results looked strong, with the tool providing a large number of highly accurate response recommendations. But when we analysed actual adoption, only a fraction of the recommendations had been used by our employees. They simply weren’t comfortable relying on AI to craft responses in real-time. Rather than pushing ahead with a solution that wasn’t being used, we paused the initiative after just two and a half weeks to adjust our approach. 

This was a key learning moment: Not every gen AI use case scales successfully, even if it passes initial testing.Adoption is just as important as accuracy. 

In contrast, other pilots in the call centre such as speech-to-text transcription and analysis of call data showed immediate value, reducing post-call work time by 50%. 

The Expand phase will allow you to refine how and where generative AI delivers the greatest impact. By staying flexible and focusing on adoption, you can pivot as necessary.

Step 4: Amplify — unlock full potential

The final step in our framework is where project teams take a step back, assess progress, and identify further use cases that will have the greatest impact.

At this stage, project teams focus on use cases that deliver the most value at scale. In our case, we evaluate this based on two key factors:

• Impact: What is the Net Present Value (NPV) associated with the expansion of the use case as calculated based on the cost of the implementation versus “employee hours saved” or “increase in quality”?
• Practicality: How feasible it is to implement the project at scale, considering integration with existing systems, availability of ready-made solutions and potential risks?

Based on this approach, we found two clear areas where generative AI could significantly enhance our strategy:

AI Assistant for Rest Employees. Our first Amplify initiative involved upgrading RestGPT to expand its use across all 800-plus employees. By leveraging an enterprise platform we could integrate into many of our back-office systems such as ServiceNow, Atlassian, M365, and Desk booking, which allowed us to centralise knowledge retrieval and task automation, including IT requests.

By upgrading to an enterprise platform, we can now track which type of employees are using the tool and for what purpose. Tracking actual hours saved was a game changer for us. This enabled us to identify time saved by anyone from a junior analyst to a senior executive and give us confidence in the value we were realising. 

Conversation Assist in the Call Centre.  Our second initiative focused on enhancing the member experience in the call centre. By combining AI with human expertise, we delivered tailored guidance to our employees when speaking to our members.

Partnering with a gen AI platform, we saw an opportunity to improve efficiency across 1,600 calls per day. By reducing handling times by an average of 2.5 minutes per call, we estimated a total annual savings of 20,000 hours. This allowed our call centre employees to help more members each day.

Our Amplify phase led to another key learning moment: With gen AI, you can uncover unforeseen benefits. For example, we had underestimated the value of being able to analyse call data and are now using that data to get deeper insights on the topics our members are most focused on.

Using gen AI to deliver results

At Rest, this framework has been instrumental in navigating the complexities of gen AI adoption, ensuring that our initiatives align with our strategic goals and deliver tangible value to our members.

As you consider your own generative AI journey, we hope adopting a framework like “Test, Measure, Expand, Amplify” will be useful in helping you to develop value-oriented use cases — and to scale them effectively to an enterprise level.