Shell sees AI as fuel for its sustainability goals

Energy giants are under significant pressure by governments and consumers to reduce carbon emissions. For multinational oil and gas company Shell, artificial intelligence may be a key catalyst for fulfilling that long-term goal.

The London-headquartered energy company’s ongoing digital transformation, fueled by a hybrid cloud platform and Databricks data lake house, includes a mix of AI technologies aimed at optimizing business efficiencies and profits and, over time, reducing its carbon footprint.

“AI has become a very core part of our overall digital transformation journey,” says Shell’s chief AI guru Dan Jeavons, noting that Shell works with several AI companies, including Microsoft and C3.ai, but has been in a close partnership with Databricks since 2015. Roughly 20 Databricks employees are assigned to the Shell account.

Jeavons, who has served as vice president of computational science and digital innovation at Shell for just six months, is the former general manager of data science at Shell and has been knee deep in data science since 2015.

In his new role, reporting to Shell Group CIO Jay Crotts, Jeavons is tasked with employing AI as well as emerging technologies such as blockchain, IoT, and edge computing to overhaul Shell’s future technology strategy and help steer its commitment to reduce its carbon footprint to become a net-zero emissions energy business by 2050.

Gartner AI analyst Anthony Mullens says Shell’s AI implementations are beyond what most other companies are doing. “Shell is over the hump in terms of initial experimentation right across the organization,” says Mullens, pointing to Shell’s Center for Excellence and participation in OpenAI.

Jeavons’ group has several hundred data scientists using AI — mostly on Databricks’ Spark-based platform — writing algorithms to execute tasks such as improving the cycle times of subsurface processing, optimizing the performance of assets, predicting when and if various pieces of equipment might fail, as well as improving offerings to customers.

“Given the threat of climate change, we need to move to a lower carbon energy system and digital plays a key role in that,” Jeavons says, noting many of the CO2 monitoring data streams will flow through Databricks AI platform. “Digital technology is one of the core levers that we can pull in order to significantly reduce the CO2 footprint of the energy system.”

According to Jeavons, Shell’s use of digital technology reduced the CO2 emissions of one liquefied natural gas (LNG) facility by as much as 130 kilotons per year — equivalent to removing 28,000 US vehicles off the road for a year.

“Many of the people that work for us have a sense of compelling purpose actually applying AI to try to accelerate energy transition,” he says. “But I’m not going to pretend it’s easy.”

Data is the foundation

As part of its digital transformation, Shell relies on two public clouds, Microsoft Azure and AWS, as well as Docker and Kubernetes containerization technologies, to run increasingly advanced workloads for various aspects of its $210 billion oil and gas business.

A key facet of that strategy, Jeavons says, is the company’s foundational data layer — a pool from which multiple tools and technologies can access data systematically.

“Having a dual-cloud strategy means you need some consistency as to how you want to manage and integrate your data. Now of course, not all data is going to be in one place. You have a variety of databases; everybody does,” Jeavons says. “But from an analytics perspective, more and more, we’re consolidating certain types of data into an integrated lake house architecture based on Databricks.”

On the analytics side, integrating data into a common layer in Databricks’ Delta Lake and using Python in a common platform allows simple queries and classical reporting query integration with visualization tools such as Power BI.

But on the AI front, it “also allows you to run the machine learning workloads all on the same platform,” Jeavons says. “For me, that’s been a step change.”

For example, Shell has integrated all its global time-series data — information such as temperature, pressure, a particular piece of equipment — into a common cloud based on Delta Lake, enabling the energy giant to keep its finger on the pulse of most global assets, including data from refineries, plants, upstream facilities, winds farms, and solar panels. “It’s 1.9 trillion rows of data aggregated today, which is a huge amount globally,” Jeavons says. “We measure everywhere.”

Shell’s AI efforts also include performing failure predictions and assessing the integrity of its energy assets by using machine vision to identify corrosion. “We’re also using AI to develop technology which can optimize the assets and make them run more efficiently at scale and optimize based on historical performance,” Jeavons says, noting that, while much of Shell’s AI magic is due the implementation of its data lake, none of it could be achieved without cloud advancements.

“Really, the key thing has been the maturing of the clouds and the ability to remove some additional layers that we had [in order] to take data directly from the plants and stream it into the cloud. That’s been helpful in driving both data analytics but also the AI strategy,” he says.

The road ahead

In total, Shell has about 350 professional data scientists and roughly 4,000 professional software engineers working remotely and/or in one of Shell’s hubs in Bangalore, India; the UK; the Netherlands, and Houston, Texas.

Aside from the cloud and data lake house, Shell has also moved to advanced development tools such as Microsoft Azure DevOps and is integrating GitHub into its developers’ ways of working. It is also deploying more mature code screening tools for the cloud, running “proper” CI/CD workflows and monitoring “north” of 10,000 pieces of equipment globally using AI as part of its remote surveillance centers, Jeavons says.

But it is the development of a common lake house architecture that has made the most difference, giving Shell “an integrated data layer that provides visibility of all the data across our business” in a consistent way, Jeavons say.

“We were a very early adopter of Delta,” he says. “For a while, it was more in proof-of-concept mode than in deployed at scale load. It’s really been in the past 18 months where we’ve seen a step change and we’ve been running quite hard.”

Change management, however, remains one of the company’s biggest challenges.

“How do you embed the technology into the business process and make it usable and a part of what happens every day and developing algorithms that work? I’m not going to underplay how difficult it is. It’s non-trivial,” Jeavons says. “It’s tougher to develop the adoption [of AI] at scale. It’s still very much a journey and we’ve made some strides but there’s a lot more to do.”