Be honest: Are your company values and AI aligned?

In 2003, Oxford University professor Nick Bostrom asked what happens if you ask a smart AI to make as many paperclips as possible. Say, a paperclip manufacturing CEO deployed an AI with no guardrails, checks, or balances, and asked it to apply its superintelligence to the paperclip problem. According to Bostrom, the AI starts transforming first all of earth, and then increasing portions of space, into paperclip manufacturing facilities.

But even an innocuous instruction, like “be helpful,” can cause problems if the AI helpfully assists a hacker to break into corporate systems, or give discounts it’s not supposed to give.

We’ve already seen many examples of corporate AIs not performing as intended. Late last year in California, for instance, a ChatGPT-powered chatbot promised a Chevrolet of Watsonville customer a 2024 Chevy Taho for $1, adding “and that’s a legally binding offer — no takesies backsies.”

The dealer didn’t honor the chatbot’s offer but in a similar case involving an Air Canada discount, the customer sued, and the courts found the company had to honor the AI’s promises.

Obviously, accuracy is the biggest problem here. The chatbots are giving false information to customers. But there’s also a question of core values. Whose interests is the chatbot putting first: the customer they’re interacting with or the company deploying the chatbot? Maybe it’s the vendor who made it. So what can CIOs do about this?

Hopefully, no company or government will create an AI that doesn’t have “don’t destroy the whole planet” as one of its core values. And whether it does or not, that’s a question for ethicists and government regulators, not CIOs deploying enterprise AI solutions.

“This isn’t something I worry about,” says Derek Leben, professor at Carnegie Mellon University’s Tepper School of Business. “When I teach MBA students, we’re more worried about the risks of AI in the here and now.”

For enterprises, this means understanding their interests perhaps compete with those of other firms in the market, vendors, and customers.

“You want to make sure the AI system is promoting your interests rather than the interests of competitor firms,” he says. “But at the same time, you want the AI to be respectful of consumer interests, and competitor interests as well.”

It’s a fine balance you want it to walk, he says, and that this is the kind of alignment problem we should try to develop solutions for.

And when the AI is coming from a vendor that has interests which are different from its customers, this can be a challenge, he adds.

“It’s a difficult question and I don’t have any good answers,” Leben says. ” In the long term, what we want is to have more leverage about the kinds of behaviors a model is producing.”

But options for an enterprise customer can be limited in terms of changing the way its vendors do business, especially if those vendors have significant market power.

“There’s only so much you can do with a prompt if a model has been heavily trained to go against your interests,” says JJ Lopez Murphy, head of data science and AI at software development company Globant. “It’s an issue that’s not easy to solve.”

But there are some steps CIOs can take to help protect their companies, including identifying alignment risks, continuously monitoring model outputs, putting guardrails in place, and building model-agnostic infrastructures. And the earlier they start, the more power they’ll have.

Not all alignment risks are equal

Business leaders understand the value of doing AI the right way — ethically, transparently, and responsibly. Nobody wants their company to be the one whose AI destroys the world, or teaches their customers how to make poison gas.

According to an Avanade survey of over 3,000 business and IT executives released in late 2023, 48% have put in place a complete set of policies for responsible AI. And

one company that’s all in is Blue Cross Blue Shield Michigan, the state’s largest HMO with more than 10,000 employees and $36 billion in annual revenues.

“We’ve committed, through our board of directors, to AI principles,” says Bill Fandrich, the company’s EVP of technology and operations. For example, he says, an AI model can’t make a decision on a person’s access to healthcare. A human has to make that decision.

There are also guidelines for transparency, security, and third-party AI.

“If you’re a vendor using AI, we need to understand what you’re doing,” he says. “If it affects our members or customers, we need to know.”

Everybody wants to go fast, he adds, but if the foundation isn’t in place, it’s difficult to do so effectively and impossible to scale. “A lot of people want to skip those foundational elements, but if you’re not mature in those things, your ability to get value is very limited,” he says. “So we’ve been investing in that.”

There’s also an opportunity for the public sector to make a difference. “I’m a strong believer that we desperately need government and other regulatory bodies to put the right guardrails in place,” adds Fandrich.

But for companies in non-regulated fields, or those only deploying gen AI in low-risk use cases, the situation can be very different. In fact, according to an April Deloitte survey of nearly 2,000 business leaders, 72% said their organization’s trust in all forms of AI has increased since gen AI emerged in late 2022.

Another survey released in May by Coleman Parkes, conducted on behalf of Alteryx, showed that out of 2,000 global IT leaders, only 5% said they saw significant negative impact from gen AI, possibly because early deployments focused on the lowest-risk use cases, of which there are plenty where the risk of alignment conflict is relatively small. Say, for example, a marketing team is using an LLM to summarize an article, and a human reviews the work, says Priya Iragavarapu, VP of data science and analytics at AArete, a management consulting firm. “Then I don’t think we need to worry so much about the LLM itself being fully aligned with your organization and culture,” she says.

But if the gen AI is deeply embedded into a core business process and operates without human oversight, then the risk is high, she says, and the alignment needs to be meticulously considered.

Transparency and accountability

A model’s alignment starts with its training data, the weights, and how it was fine-tuned. This information is typically not available at all for proprietary, commercial platforms like OpenAI. Even with open source models, not all of them make this information public.

“The biggest challenge is the lack of transparency,” says Iragavarapu. “If I don’t know what data the model was trained on, or the fine tuning that was done on the model, I wouldn’t trust it to be in alignment with my company values.”

AI doesn’t have a point of view or discriminatory intent, adds Dave Prakash, head of AI governance at Booz Allen Hamilton. “It’s just an equation,” he says. “It’s just math and statistics.” But even with the best of intentions, an AI can produce some unfortunate results.

Prakash himself started out as a doctor, and he’s seen plenty of examples of this in the medical field. For example, a large insurance company used the cost of health care as the basis of an algorithm to evaluate severity. “It was putting African-Americans at a lower severity score than much healthier white Americans,” he says, “because some sub-populations don’t have the same access to care, they work multiple jobs, don’t have health insurance, and don’t rack up the same medical bills.”

But it can be difficult to figure out what’s going on inside an AI model when it’s a black box commercial system.

One place enterprise customers can start is by looking at the vendor’s history: what else have they done, have they respected privacy, or have they been transparent in what they do?

There are other questions CIOs should ask as well, like how to audit an LLM to see its degree of bias, and how well does it comply with executive orders on AI from Washington or the EU AI act?

It’s not always easy to find this information, Prakash says, and the tools needed to determine some of these things are still under development.

“I look forward to the day when we can efficiently audit these models for our values,” he says.

Regulators, non-profits, vendor groups, and industry groups are just beginning to work on these issues. “It’s in everyone’s interest to create standards, because it gives people confidence in how to adopt these technologies,” he says. “Standards are a great opportunity for both the vendor and the end user.” And as autonomous AI agents — systems powered by gen AI that go beyond simple use cases and achieve complex goals — start to see mass adoption, transparency and accountability are going to be even more important.

These systems can create a plan for reaching a goal, delegate tasks, check if those tasks have been completed successfully, adapt, and iterate until the goal is reached. That requires numerous calls to the AI, and even small alignment issues can escalate quickly, so companies have to learn to evaluate the collective performance of these autonomous systems.

“We’re looking at the risks they pose,” Prakash adds. “That’s not to say we can eliminate all risk; nothing worth doing is without risk. But anyone with a modicum of ethics and sense would spend some time evaluating this technology before deploying it.”

Fine-tuning and guardrails

Even when companies use big commercial AI vendors, they’re not completely locked into what the vendor wants the model to do.

Even “black box” vendors like OpenAI allow enterprises to fine-tune their LLMs, which allows companies to add training data in Q&A form. It’s kind of like training a new employee by giving them examples of how to do things, says AArete’s Iragavarapu.

“With a new person, you hire them for their skills,” she says. “But when you onboard them, you explain your culture and how you do things so the new person can work within that understanding. So this is onboarding of your LLMs and it’s crucial for organizations and enterprises.” Finetuning needs a data set the size of between 0.5% and 1% of a model’s original dataset in order to meaningful impact models, she says.

With GPT 4 reportedly coming in at over a trillion parameters, even 1% is a large amount, but enterprises don’t need to consider the entire data set when fine tuning.

“You can’t say you’ve written 10 questions and answers and fine-tuned a model and claim it’s now fully compliant with my organization’s values,” Iragavarapu says. “But you also don’t have to fine-tune it on everything. You only have to on a specific business process or culture. It’s really about digging deep into one small area or concept, not addressing the entire breadth of the LLM.”

With the right fine-tuning, it’s possible to overcome a model’s core alignment, she says. And to find out if the fine-tuning has worked, the LLM needs to be tested on a large number of questions, asking the same thing in many different ways.

So far, there isn’t a good automated way to do this, or an open-source LLM designed specifically to test the alignment of other models, but there’s definitely a crucial need for it.

As simple Q&A use cases evolve into autonomous AI-powered agents, this kind of testing will become absolutely necessary. “Every organization needs this tool right now,” Iragavarapu says.

Vendor lock-in

When a company has no choice but to use a particular AI vendor, maintaining alignment will be a constant battle.

“If it’s embedded in Windows, for example, you might not have that control,” says Globant’s Lopez Murphy. But the task is a lot simpler if it’s easy to switch to a different vendor, an open-source project, or a home-built LLM. Having options helps keep providers honest and puts power back in the hands of the enterprise buyers. Globant itself has an integration layer, an AI middleware, that allows the company to easily switch between models. “It can be a commercial LLM,” he says. “Or something you have locally, or something on [AWS] Bedrock.”

And some organizations roll their own models. “That’s why some governments want to have their own sovereign Ais so they’re not relying on the sensibilities of some Silicon Valley company,” says Lopez Murphy.

And it’s not just governments that require a high degree of control over the AIs they use. Blue Cross Blue Shield Michigan, for example, has some high-risk AI use cases involving cybersecurity, contract analysis, and answering questions about member benefits. Because these are very sensitive areas, and highly regulated, the company built its AI systems in-house, in a secure, controlled, and dedicated cloud environment.

“We do everything internally,” said Fandrich. “We teach and control the models in a private segmented part of the network, and then decide how and whether to move them into production.”