Between sustainability and risk: why CIOs are considering small language models

As GPT-4 passes the Turing test, Microsoft pushes its AI assistant Copilot into enterprise products, and as Google announces the availability of the Gemini app on mobile phones in Italy, CIOs are studying gen AI technology to stay up-to-date — but without being distracted by either technological excitement or commercial propositions.

“Generative AI can bring many benefits, but it can’t be adopted without appropriate considerations,” says Massimo Carboni, CTO and head of the infrastructure department at GARR, the broadband network dedicated to the Italian research and education community. “The hype is very strong, but the risk of overestimating its possibilities is equally high. In the digital world, we must be increasingly careful, and the first risk with AI and generative AI is to trust too much.”

Moreover, Gartner recently estimated that global business spend on gen AI technologies is of little importance. Out of a total of $5 trillion of IT investments expected this year, up 8% compared to 2023, gen AI won’t account for much. Spending will be driven, instead, by more traditional forces like classic IT services, which will be worth more than $1.5 trillion, a 9.7% increase year on year.

In contrast, large service providers are multiplying their spend on tech to support gen AI projects, and in anticipation of an upcoming boom, AI application servers will represent almost 60% of the total investment in hyperscalers’ servers in 2024. Enterprises are more cautious, though. Gartner sees a “story, plan, execute” cycle for gen AI as it was discussed in 2023, planned to implement in 2024, and projected to execute in 2025. 

Generative AI under the CIO’s scrutiny

Edoardo Esposito, CIO of inewa, member of the Elevion Group, a certified ESCO active in biogas and biomethane generation and energy efficiency, is currently in the planning phase testing Copilot, since inewa’s IT is all on Microsoft systems, and this gen AI product integrates perfectly with the Office suite. His experiments are carried out together with other managers like the CFO, the director of legal, and the director of institutional relations and regulation.

“We’re testing uses in finance, like financial analysis of income and expenses,” Esposito says. “I think that’s where the biggest opportunities are. I don’t see the use in legal as promising at the moment, but we’re trying to use gen AI to manage contracts and study laws.”

Of course, AI doesn’t give legal advice, but it helps navigate the vast amount of rules constantly being updated or changed.

“Even a simple bullet point summary of a new law generated with AI to send to an executive for review helps,” he says. “In the end, for us as a small business, at $30 a month, it’s like having an extra person in the office.”

But while he has no qualms about automating simple tasks, Esposito isn’t convinced gen AI can fully automate complex tasks, among other concerns. “These models don’t seem sustainable to me. They have huge parameters and require a lot of energy to train,” he says.

The unsustainability of AI

Carboni also emphasizes how energy-intensive AI is and how it adds to the already high costs of the technology.

“ICT in the world accounts for 9% of total energy costs, or about $300 billion in 2023,” he says. “This share has increased by up to 60% in the last 10 years and is destined to grow further.”

Then there’s a problem in training, according to Carboni. “Generative AI is overturning the traditional human-centered approach,” he says. “Instead of people training models, which then change the company organization, today it’s people who have to adapt to the models that come from the market. This represents a risk for me. The more the generative AI players decrease, the more it creates a dependency and a loss of control on the part of companies.”

In addition, adds Carboni, AI risks limiting the functioning of digital to a few subjects that determine behaviors and costs because the entry threshold to gen AI is high and most companies can only buy services without the knowledge to distinguish the differences between one product and another. There’s little choice and the risk is the standardization of products for everyone. “So in my opinion, it’s always better to continue building something in-house.”

Companies competing with big tech

Competition between companies is increasing and many, including Carboni, feel how the big suppliers sell its models is unfair in many respects because some market players have capabilities that others don’t.

“Companies like Microsoft and Google have ecosystems of products, and this oligopoly that controls up to 80% of the data market has a huge advantage over other companies,” he says. “Big tech strategies also aim to incorporate startups that allow them to strengthen their dominance over data.” So it’s difficult to think of new entrants that can compete. Startups that offer alternative products certainly exist and are a good way to develop algorithms, but these are not enough for success.

For Carboni, this doesn’t mean a failure of gen AI, but a desire to study it in depth and govern it. “I believe AI is very relevant and we will work on it at GARR because we have a lot of data to exploit,” he adds. “The intention is to derive a generative AI model to better define our internal knowledge base. This isn’t currently public, but if we wanted to expose it, it would have to be developed for external reading. And we could use a small language model for this purpose.”

SLMs: The CIO searching for control

Small language models (SLMs) are ML algorithms trained on much smaller and more specific data sets than LLMs, the large deep-learning models on which products like GPT are based. Initial tests show they’re more efficient, less expensive and more accurate in their task. In fact, Esposito also follows the evolution of SLMs and considers them much more promising for business uses, and more sustainable. Large products have excellent training, but are generic, while companies need vertical applications.

“Using large gen AI models via APIs to train your own gen AI products with your own data requires significant energy resources,” says Esposito. “It’s like bringing a digital colleague into your home, but a colleague who costs a lot. You have to train him with your specific company information, and constantly provide him with new data to keep him updated. You also have to power him with a lot of electricity. This is why I’m not fascinated by large language models but find small language models very interesting. Companies need something more targeted and with less risk of bias and privacy violations.”

For example, Esposito says, IT can isolate a narrow language task, take an SLM, put it in its cloud, and give it access only to the corporate document database. From there, it asks the model only questions related to those documents.

“From the first experiments, it seems that not only energy consumption is reduced, but also the probability of hallucinations,” he says. “After all, companies’ AI models don’t have to know everything, but only respond to certain applications. SLMs can still do translations, perform market trend analysis, automate customer service, manage IT tickets, create a business virtual assistant, and more. It seems more efficient to me to limit the domain and specialize it, keeping it under IT control.”

Weighing gen AI business and small models

Control is key. Alessandro Sperduti, director of the Augmentation Center of the Bruno Kessler Foundation (FBK), says in AI, we risk the domination of private companies. “In the past, the most important AI systems in the world were developed in universities, while today they’re not because private technological giants have emerged with a spending power with which the public can’t compete,” he says.

In the scientific community, in fact, some would prefer a political intervention to bring AI back under the control, as what happened for high-energy physics and the establishment of CERN, the body that brings together several countries to collaborate in the theory and experimentation of particle physics. But other researchers don’t see risks from the hegemony of some private actors, as long as governments regulate the use of AI tools, as has been done in the European Union with the AI ​​Act.

“The difference with what happened in the world of physics is that there’s no big business there, while in AI there is huge profit,” Sperduti says. “This is why companies like Microsoft and Google are fiercely competing today. Every day we read about new goals achieved that surpass previous ones. Startups in the sector exist, but compared to other sectors, they’re few because the investments needed are enormous. I don’t believe, therefore, they can truly threaten the predominance of current players and create a strong competitive dynamic.”

On the smaller models, however, Sperduti highlights the presence of retrieval augmented generation (RAG) systems, which use LLMs to answer questions about documents stored in local databases. In this way, the documents remain private and aren’t given to the organization that provides the LLM. RAGs give companies more control over the data and cost less.

“But they need to be managed locally,” he emphasizes. “You can also use open-source language models locally, which are smaller than LLMs but have lower performance, so these can be considered SLMs.”

On the sustainability of costs, Sperduti says LLMs are managed by big tech as a utility service, as if we were buying electricity, while having an SLM means keeping the turbine at home to generate electricity. “Therefore, an economic evaluation must be carried out,” he says. “This could even be favorable if the use of the model is intense. But it’s a choice that must be made after careful analysis, considering the cost of the model, its update, the people who work on it, and so on.”

The CIO at the helm: governance and expertise

Carboni also warns that if you opt for an SLM, the IT task is greater and the CIO’s life is not necessarily simplified.

“In LLMs, the bulk of the data work is done statistically and then IT trains the model on specific topics to correct errors, giving it targeted quality data,” he says. “SLMs cost much less and require less data, but, precisely for this reason, the statistical calculation is less effective and, therefore, very high-quality data is needed, with substantial work by data scientists. Otherwise, with generic data, the model risks producing many errors.”

Furthermore, SLMs are so promising and interesting for companies that even big tech offers and advertises them, like Google’s Gemma and Microsoft’s Phi-3. For this reason, according to Esposito, governance remains fundamental, within a model that should remain a closed system.

“An SLM is easier to manage and becomes an important asset for the company in order to extract added value from AI,” he says. “Otherwise, with large models and open systems, you have to agree to share strategic company information with Google, Microsoft, and OpenAI. This is why I prefer to work with a system integrator that can develop customizations and provide a closed system, for internal use. I don’t think it’s wise to let employees use the general purpose product by putting company data in it, which may also be sensitive. Data and AI governance is essential for companies.”

Equally important is the competence of the CIO.

“In my work, I consider it important not only to evaluate the cost of accessing a service, but also my ability to impact a service,” says Carboni. “The CIO must build his own background of technological knowledge and equip himself with a team of capable people, including a good share of young people, capable of operating in modern contexts, with cloud-native technologies. In this way, the CIO doesn’t limit himself to buying a product and expecting a performance, but acts and impacts that product or service.”

So the CIO remains at the helm. Whatever the development trajectory of gen AI, it’s the head of IT who wants to decide directions, applications, and objectives.