Skip to content
Tiatra, LLCTiatra, LLC
Tiatra, LLC
Information Technology Solutions for Washington, DC Government Agencies
  • Home
  • About Us
  • Services
    • IT Engineering and Support
    • Software Development
    • Information Assurance and Testing
    • Project and Program Management
  • Clients & Partners
  • Careers
  • News
  • Contact
 
  • Home
  • About Us
  • Services
    • IT Engineering and Support
    • Software Development
    • Information Assurance and Testing
    • Project and Program Management
  • Clients & Partners
  • Careers
  • News
  • Contact

For IT leaders, operationalized gen AI is still a moving target

The rate of companies that have either already deployed generative AI or are actively exploring it is accelerating to the point where, combined, there are very few holdouts. 

The use of gen AI in the enterprise was nearly nothing in November 2022, where the only tools commonly available were AI image or early text generators. But by May 2023, according to an IDC survey, 65% of companies were using gen AI, and in September, that number rose to 71%, with another 22% planning to implement it in the next 12 months.

Even in its infancy, gen AI has become an accepted course of action and application, and the most common use cases include automation of IT processes, security and threat detection, supply chain intelligence, and automating customer service and network processes, according to a report released by IBM in January. Plus, when you add in cloud-based gen AI tools like ChatGPT, the percentage of companies using gen AI in one form or another becomes nearly universal.

And this doesn’t include the gen AI that’s now being embedded into platforms like Office 365, Google Docs, and Salesforce.

However, getting into the more difficult types of implementations — the fine-tuned models, vector databases to provide context and up-to-date information to the AI systems, and APIs to integrate gen AI into workflows — is where problems might crop up. Building enterprise-grade gen AI platforms is like shooting at a moving target, and AI progress is developing at a much faster rate than they can adapt.

“It makes it challenging for organizations to operationalize generative AI,” says Anand Rao, AI professor at Carnegie Mellon University. “There are different tools, models, and vector databases evolving, and new papers coming out, which makes it very challenging for a company. They need stability. Tell me what to do for the next three months; don’t change everything every two weeks.”

Due to the complexity of this challenge, plus the cost involved and the expertise required, only 10% of organizations were actually able to launch gen AI models into production last year, according to findings released by Intel’s cnvrg.io in December.

But that doesn’t mean enterprises should just wait for things to settle down. To help regain some initiative, there are best practices that can be applied now to start building gen AI platforms — practices that will allow them to adapt quickly as the technology changes, including building robust and modern data and API infrastructures, creating an AI abstraction layer between their enterprise applications and the AI models they use, and setting up security and cost policies, usage guardrails, and ethics frameworks to guide how they deploy gen AI.

Data and API infrastructure

“Data still matters,” says Bradley Shimmin, chief analyst for AI platforms, analytics, and data management at London-based independent analyst and consultancy Omdia. Yet according to the IBM survey, data complexity was the second biggest barrier to adoption after lack of expertise, while the cnvrg.io survey said infrastructure was the single biggest challenge for companies looking to productionize large language models (LLMs).

Another setback is enterprises unable to keep up with business demands due to inadequate data management capabilities. An overriding issue, though, is that most organizations don’t have a plan, says Nayur Khan, a partner at McKinsey & Company. “They try to do something and see what sticks.” But with gen AI models being delivered as a service, in the form of, say, OpenAI APIs, there are use cases where companies can skip right ahead to deploying AI as a service.

“Now it becomes a service I can call and I don’t have to worry about training,” says Khan. “That’s fine, but language models are great for language. They’re not great for knowledge.” Knowledge sits inside organizations, he says.

A retail company, for example, might have a 360-degree view of customers, which is all fed into analytics engines, machine learning, and other traditional AI to calculate the next best action. Then the gen AI could be used to personalize the messages to those customers. So by using the company’s data, a general-purpose language model becomes a useful business tool. And everyone is trying to build these types of applications.

“I’m seeing it across all industries,” says Khan, “from high tech and banking all the way to agriculture and insurance.” It’s forcing companies to move faster on the digital front, he adds, and fix all the things they said they were going to do but never got around to doing.

And not only do companies have to get all the basics in place to build for analytics and MLOps, but they also need to build new data structures and pipelines specifically for gen AI.

When a company wants to fine-tune a model or create a new one in a particular subject area, it requires data architecture, critical choices about which model or type of model to pursue, and more. “It quickly adds up in complexity,” says Sheldon Monteiro, EVP at Publicis Sapient, a global digital consultancy.

Even a simpler project, like adding an external data source to an existing gen AI model, requires a vector database, the right choice of model, and an industrial-grade pipeline.

But it all begins with data, and it’s an area where many companies lag behind. Without a single and holistic strategy, every department will set up its own individual solutions.

“If you do that, you’ll end up making a lot more mistakes and re-learning the same things over and over again,” says Monteiro. “What you have to do as a CIO is take an architectural approach and invest in a common platform.”

Then there’s the hard work of collecting and prepping data. Quality checks and validation are critical to create a solid base, he says, so you don’t introduce bias, which undermines customers and business.

So if a particular data set excludes the highest-value transactions because those are all handled manually, then the resulting model could potentially have a bias toward smaller, less profitable business lines. Garbage in, garbage out applies to the new era of gen AI as much as it did in previous technological periods.

For companies that have already invested in their data infrastructure, those investments will continue to pay off into the future, says Monteiro. “Companies that invested in data foundations have a tremendous head start in what they’re doing with generative AI,” he says.

Still, these traditional data foundations originally designed for advanced analytics and machine learning use cases only go so far.

“If you want to go beyond the basics, you’ll need to understand some of the deeper subtleties of generative AI,” says Omdia’s Shimmin. “What’s the difference between different embedding models, what is chunking, what is overlap? What are all the different methodologies you can use to tokenize data in the most efficient way? Do you want high or low dimensionality to save space in your vector database? The MLOps tools we have weren’t built to do that. It’s all very complicated and you can waste a lot of time and money if you don’t know what you’re doing.”

But MLOps platforms vendors are stepping up, he says. “Companies like Dataku, DataRobot, and Databricks all have retooled to support LLMOps or GenAIOps. All the little pieces are starting to come into place.”

Analyzing the abstraction layer

Last November, OpenAI, the go-to platform for enterprise gen AI, unexpectedly fired its CEO, Sam Altman, which set off a circus-like scramble to find a new CEO, staff threatening to walk out, and Microsoft offering to take everyone in. During those tumultuous days, many companies using OpenAI’s models suddenly realized they put all their eggs into one unstable basket.  

“We saw a lot of OpenAI integrations,” says Dion Hinchcliffe, VP and principal analyst at Constellation Research. “But the whole management issue that happened with OpenAI has made people question their over-commitment.”

Even if a company doesn’t go out of business, it might quickly become obsolete. Early last summer, ChatGPT was pretty much the only game in town. Then Facebook released Llama 2, free for most enterprise customers followed by Anthropic’s Claude 2, which came out with a context window of 200,000 tokens — enough for users to cut-and-paste the equivalent of a 600-page book right into a prompt — leaving GPT-4’s 32,000 tokens in the dust. Not to be outdone, however, Google announced in February its new Gemini 1.5 model can handle up to 10 million tokens. With that, and greater speed, efficiency and accuracy across video, audio, and written copy, there were virtually no limits.

The number of free, open-source models continues to proliferate, as well as industry-specific models, which are pre-trained on, say, finance, medicine or material science.

“You’ve got new announcements every week, it seems,” says Publicis Sapient’s Monteiro.

That’s where a “model garden” comes in, he says. Companies that are disciplined about how they select and manage their models, and architect their systems so models can be easily swapped in and out, will be able to handle the volatility in this space.

But this abstraction layer needs to do more than just allow a company to upgrade models or pick the best one for each particular use case.

It can also be used for observability, metering, and role-based access controls, says Subha Tatavarti, CTO at technology and consulting firm Wipro Technologies.

Wipro, with 245,000 employees, has no choice but to adopt gen AI, she says, because its customers are expecting it to.

“We’re foundationally a technology company,” she says. “We have to do this.”

Broadening perspectives

Observability allows a company to see where data is going, what models and prompts are being used, and how long it takes for responses to come back. It can also include a mechanism to edit or obfuscate sensitive data.

Once a company knows what’s happening with its models, it can implement metering controls — limits on how much a particular model can be used, for example — to avoid unexpected spikes in costs.

“Right now, the way the metering works is the token consumption model,” Tatavarti says. “And it could get very expensive.”

In addition, for FAQs, companies can cache responses to save time and money. And for some use cases, an expensive, high-end commercial LLM might not be required since a locally-hosted open source model might suffice.

“All of that is fascinating to us and my team is definitely working on this,” she adds. “This is imperative for us to do.”

And when it comes to access controls, the fundamental principle should be to never expose native APIs to the organization but instead have a middle layer that checks permissions and handles other security and management tasks.

If, for example, an HR platform uses gen AI to answer questions based on a vector database of policies and other information, an employee should be able to ask questions about their own salary, says Rajat Gupta, chief digital officer at Xebia, an IT consultancy. But they shouldn’t be able to ask questions about those of other employees — unless they’re a manager or work in HR themselves.

Given how fast gen AI is being adopted in enterprises across all different business units and functions, it would be a nightmare to build these controls from scratch for every use case.

“The work would be enormous,” he says. “There’d be chaos.”

Gupta agrees enterprises that need to build this kind of functionality should do so once and then reuse it. “Take everything they need in common — security, monitoring, access controls — and build it as part of an enterprise-level platform,” he says.

He calls it an AI gateway, with the open source MLflow AI Gateway being one example. Released last May, it’s already been deprecated in favor of the MLflow Deployments Server. Another tool his company is using is Arthur AI’s Arthur Shield, a firewall for LLMs. It filters prompt injection attacks, profanity, and other malicious or dangerous prompts.

And then there’s Ragas, which helps check a gen AI response against the actual information in a vector database in order to improve accuracy and reduce hallucinations.

“There are many such projects both in the open source and the commercial space,” he says.

Third-party AI platforms, startups, and consultants are also rushing in to fill the gaps.

“The way the AI ecosystem is evolving is surprising,” says Gupta. “We thought the pace would slow down but it’s not. It’s rapidly increasing.”

So to get to market faster, Xebia is weaving these different projects together, he says, but it doesn’t help that AI companies keep coming up with new stuff like autonomous AI-powered agents, for example.

“If you’re using autonomous agents, how do you actually measure the efficacy of your overall agents project?” he asks. “It’s a challenge to actually monitor and control.”

Today, Xebia hobbles agents, curtailing their autonomy and allowing them to carry out only very limited and precise tasks. “That’s the only way to do it right now,” he adds. “Limit the skills they have access to, and have a central controller so they’re not talking to each other. We control it until we have more evolved understanding and feedback loops. This is a pretty new area, so it’s interesting to see how this evolves.”

Building guardrails

According to the cnvrg.io survey, compliance and privacy were top concerns for companies looking to implement gen AI, ahead of reliability, cost, and lack of technical skills.

Similarly, in the IBM survey, for companies not implementing gen AI, data privacy was cited as the barrier by 57% of respondents, and transparency by 43%. In addition, 85% of all respondents said consumers would be more likely to pick companies with transparent and ethical AI practices, but fewer than half are working toward reducing bias, tracking data provenance, working on making AI explainable, or developing ethical AI policies.

It’s easy for technologists to focus on technical solutions. Ethical AI goes beyond the technology to include legal and compliance perspectives, and issues of corporate values and identity. So this is an area where CIOs or chief AI officers can step up and help guide the larger organizations.

And it goes even further than that. Setting up gen AI-friendly data infrastructures, security and management controls, and ethical guide rails can be the first step on the journey to fully operationalize LLMs.

Gen AI will require CIOs to rethink technology, says Matt Barrington, EY Americas emerging technologies leader. Prior to gen AI, software was deterministic, he says.

“You’d design, build, test, and iterate until the software behaved as expected,” he says. “If it didn’t, it was a bug, and you’d go back and fix it. And if it did, you’d deploy it into production.” All the large compute stacks, regardless of software pattern, were deterministic. Now, other than quantum computing, gen AI is the first broadly known non-deterministic software pattern, he says. “The bug is actually the feature. The fact it can generate things on its own is the main selling point.”

That doesn’t mean the old stuff should all be thrown out. MLOps and Pytorch are still important, he says, as is knowing when to do a RAG embedding model, a DAG, or go multi-modal, as well as getting data ready for gen AI.

“All those things will remain and be important,” he says. “But you’ll have the emergence of a new non-deterministic platform stack that’ll sit alongside the traditional stack with a whole new area of infrastructure engineering and ops that will emerge to support those capabilities.”

This will change how businesses operate at a core level, and moving in this direction to become a truly AI-powered enterprise will be a fast-paced shift, he says. “Watching this emerge will be very cool,” he says.

Artificial Intelligence, Business Operations, CIO, Data Architecture, Data Governance, Data Management, Enterprise Architecture, Generative AI, IT Leadership, IT Operations
Read More from This Article: For IT leaders, operationalized gen AI is still a moving target
Source: News

Category: NewsFebruary 28, 2024
Tags: art

Post navigation

PreviousPrevious post:Atos deal to sell its legacy service business falls throughNextNext post:Whether your technology is new or old, lifecycle management is key

Related posts

휴먼컨설팅그룹, HR 솔루션 ‘휴넬’ 업그레이드 발표
May 9, 2025
Epicor expands AI offerings, launches new green initiative
May 9, 2025
MS도 합류··· 구글의 A2A 프로토콜, AI 에이전트 분야의 공용어 될까?
May 9, 2025
오픈AI, 아시아 4국에 데이터 레지던시 도입··· 한국 기업 데이터는 한국 서버에 저장
May 9, 2025
SAS supercharges Viya platform with AI agents, copilots, and synthetic data tools
May 8, 2025
IBM aims to set industry standard for enterprise AI with ITBench SaaS launch
May 8, 2025
Recent Posts
  • 휴먼컨설팅그룹, HR 솔루션 ‘휴넬’ 업그레이드 발표
  • Epicor expands AI offerings, launches new green initiative
  • MS도 합류··· 구글의 A2A 프로토콜, AI 에이전트 분야의 공용어 될까?
  • 오픈AI, 아시아 4국에 데이터 레지던시 도입··· 한국 기업 데이터는 한국 서버에 저장
  • SAS supercharges Viya platform with AI agents, copilots, and synthetic data tools
Recent Comments
    Archives
    • May 2025
    • April 2025
    • March 2025
    • February 2025
    • January 2025
    • December 2024
    • November 2024
    • October 2024
    • September 2024
    • August 2024
    • July 2024
    • June 2024
    • May 2024
    • April 2024
    • March 2024
    • February 2024
    • January 2024
    • December 2023
    • November 2023
    • October 2023
    • September 2023
    • August 2023
    • July 2023
    • June 2023
    • May 2023
    • April 2023
    • March 2023
    • February 2023
    • January 2023
    • December 2022
    • November 2022
    • October 2022
    • September 2022
    • August 2022
    • July 2022
    • June 2022
    • May 2022
    • April 2022
    • March 2022
    • February 2022
    • January 2022
    • December 2021
    • November 2021
    • October 2021
    • September 2021
    • August 2021
    • July 2021
    • June 2021
    • May 2021
    • April 2021
    • March 2021
    • February 2021
    • January 2021
    • December 2020
    • November 2020
    • October 2020
    • September 2020
    • August 2020
    • July 2020
    • June 2020
    • May 2020
    • April 2020
    • January 2020
    • December 2019
    • November 2019
    • October 2019
    • September 2019
    • August 2019
    • July 2019
    • June 2019
    • May 2019
    • April 2019
    • March 2019
    • February 2019
    • January 2019
    • December 2018
    • November 2018
    • October 2018
    • September 2018
    • August 2018
    • July 2018
    • June 2018
    • May 2018
    • April 2018
    • March 2018
    • February 2018
    • January 2018
    • December 2017
    • November 2017
    • October 2017
    • September 2017
    • August 2017
    • July 2017
    • June 2017
    • May 2017
    • April 2017
    • March 2017
    • February 2017
    • January 2017
    Categories
    • News
    Meta
    • Log in
    • Entries feed
    • Comments feed
    • WordPress.org
    Tiatra LLC.

    Tiatra, LLC, based in the Washington, DC metropolitan area, proudly serves federal government agencies, organizations that work with the government and other commercial businesses and organizations. Tiatra specializes in a broad range of information technology (IT) development and management services incorporating solid engineering, attention to client needs, and meeting or exceeding any security parameters required. Our small yet innovative company is structured with a full complement of the necessary technical experts, working with hands-on management, to provide a high level of service and competitive pricing for your systems and engineering requirements.

    Find us on:

    FacebookTwitterLinkedin

    Submitclear

    Tiatra, LLC
    Copyright 2016. All rights reserved.