Skip to content
Tiatra, LLCTiatra, LLC
Tiatra, LLC
Information Technology Solutions for Washington, DC Government Agencies
  • Home
  • About Us
  • Services
    • IT Engineering and Support
    • Software Development
    • Information Assurance and Testing
    • Project and Program Management
  • Clients & Partners
  • Careers
  • News
  • Contact
 
  • Home
  • About Us
  • Services
    • IT Engineering and Support
    • Software Development
    • Information Assurance and Testing
    • Project and Program Management
  • Clients & Partners
  • Careers
  • News
  • Contact

10 things to watch out for with open source gen AI

It seems anyone can make an AI model these days. Even if you don’t have the training data or programming chops, you can take your favorite open source model, tweak it, and release it under a new name.

According to Stanford’s AI Index Report, released in April, 149 foundation models were released in 2023, two-thirds of them open source. And there are an insane number of variants. Hugging Face currently tracks more than 80,000 LLMs for text generation alone and fortunately has a leaderboard that lets you quickly sort the models by how they score on various benchmarks. And these models, though they lag behind the big commercial ones, are improving quickly.

Leaderboards are a good place to start when looking at open source gen AI, says David Guarrera, generative AI lead at EY Americas, and Hugging Face in particular has done a good job benchmarking, he says.

“But don’t underestimate the value of getting in there and playing with these models,” he says. “Because they’re open source, it’s easy to do that and swap them out.” And the performance gap between open source models and their closed, commercial alternatives is narrowing, he adds.

“Open source is great,” adds Val Marchevsky, head of engineering at Uber Freight. “I find open source extremely valuable.” Not only are they catching up to proprietary models in performance, but some offer levels of transparency that closed source can’t match, he says. “Some open source models allow you to see what’s used for inference and what’s not,” he adds. “Auditability is important for preventing hallucinations.”

Plus, of course, there’s the price advantage. “If you have a data center that happens to have capacity, why pay someone else?” he says.

Companies are already very familiar with using open source code. According to Synopsys’ open source security and risk analysis released in February, 96% of all commercial code bases contained open source components.

As a result of all this experience, companies should know what to do to make sure they’re using properly-licensed code, how to check for vulnerabilities, and how to keep everything patched and up-to-date. Some of those rules and best practices, though, have particular nuances that companies might overlook. Here are the top ones.

1. Weird new license terms

The landscape of different open source license types is complicated enough. Is a project safe for commercial use, or only for non-commercial implementations? Can it be modified and distributed? Can it be safely incorporated into a proprietary code base? Now, with gen AI, there are a few new wrinkles. First, there are new license types that are only open source under a very loose definition of the term.

Take the Llama license, for example. The Llama family of models are some of the best open source LLMs out there, but Meta officially describes it as a “bespoke commercial license that balances open access to the models with responsibility and protections in place to help address potential misuse.”

Enterprises are allowed to use the models commercially, and for developers to create and distribute additional work on top of the base Llama models, but they’re not allowed to use Llama outputs to improve other LLMs unless they are themselves Llama derivatives. And if enterprises — or their affiliates — have more than 700 monthly users, they have to request a license that Meta may or may not grant. If they use Llama 3, they have to include “Built with Llama 3” in a prominent location.

Similarly, Apple just released OpenELM under the “Apple Sample Code License,” which is also invented for the occasion and covers only copyright permissions while excluding patent rights.

Neither Apple nor Meta use commonly accepted open source licenses, but the code is, in fact, open. Apple actually released not just the code, but also the model weights, the training data set, training logs, and pre-training configurations. Which brings us to the other aspect of open source licensing. Traditional open source software is just that — code. The fact it’s open source means you can see what it does and if there are potential problems or vulnerabilities in it.

Gen AI, however, isn’t just code. It’s also the training data, model weights, and fine tuning. All of those things are critical to understand how a model works and identify potential biases. A model trained on, say, an archive of flat earth conspiracy theories will be bad at answering science questions, or a model fine-tuned by North Korean hackers might be bad at correctly identifying malware. So do open source LLMs release all that information? It depends on the model, or even on the specific release of the model since there are no standards.

“Sometimes they make the code available, but if you don’t have the fine tuning, you could spend a lot of money getting to comparable performance,” says Anand Rao, professor of AI at Carnegie Mellon University and former global AI lead at PwC.

2. Skills shortages

Open source is often a do-it-yourself endeavor. Companies can download the code, but then they need in-house expertise or hired consultants to make everything work. This is a big problem in the gen AI space. Nobody has years of experience because the technology is so new. If a company is just starting out with gen AI, or if it wants to move quickly, it’s safer to start with a proprietary platform, says Rao.

“It takes expertise to download the open source version,” he says. But once a company’s done its proof of concept, deploys the model into production, and the bills start piling up, then it might be time to look at open source alternatives, he adds.

The lack of industry expertise also creates another problem for the open source gen AI space. One of the key advantages of open source is many people look at the code and can spot programming errors, security vulnerabilities, and other weaknesses. But this “thousand eyes” approach to open source security only works if there are, in fact, a thousand eyes capable of understanding what they’re seeing.

3. Jailbreaking

LLMs are notoriously susceptible to jailbreaking, where a user gives it a clever prompt that tricks it into violating its guidelines and, say, generating malware. With commercial projects, there are highly motivated vendors standing behind them who can identify these loopholes and close them as they pop up. In addition, vendors have access to the prompts that users send to the public versions of the models, so they can monitor for signs of suspicious activity.

Malicious actors are less likely to purchase enterprise versions of the products that run in private environments, where the prompts aren’t shared back to the vendor to improve the model. With an open source project, there might not be anyone on the team whose job it is to look for signs of jailbreaking. And bad actors can download these models for free and run them in their own environments in order to test potential hacks. The bad guys also get a head start on their jailbreaking since they can see the system prompt the model uses and any other guardrails that the model developers may have built.

“It’s not just trial and error,” says Rao. Attackers can analyze training data, for example, to figure out ways to get a model to misidentify images, or go off the rails when it comes across a prompt that looks innocuous.

If an AI model adds a watermark to its output, a malicious actor might analyze the code to reverse-engineer the process in order to take the watermark out. Attackers could also analyze the model or other supporting code and tools to find areas of vulnerability.

“You can overwhelm the infrastructure with requests so the model won’t,” says Elena Sügis, senior data scientist and capability lead at Nortal, a global digital transformation consultancy. “When the model is part of a larger system, and its output is used by another part of the system, if we can attack how the model produces the output, it’ll disrupt the whole system, which could be risky for the enterprise.”

4. Training data risks

Artists, writers, and other copyright holders are suing the big AI companies right and left. But what if they believe their IP rights are being infringed on by an open source model, and the only deep pockets around are those of the enterprises that have incorporated that model into their products or services? Could enterprise users get sued?

“It’s a potential issue and nobody really knows how some of the pending litigation is going to play out,” says EY’s Guarrera. We might be headed toward a world where there’ll have to be some compensation for the data sets, he says. “The large tech players are better positioned to have the money to spend on that and weather the storm that may come around copyright.”

The big commercial vendors don’t just have money to spend on buying training data and fighting lawsuits, they also have money to spend on curated data sets, says Sügis. Free, public data sets contain more than just copyrighted content used without permission. They’re also full of inaccurate and biased information, malware, and other materials that can degrade the quality of output.

“Many model developers are talking about using curated data,” she says. “And this is more expensive than if you throw the whole internet at it to train it.”

5. New areas of exposure

Since a gen AI project is more than just the code, there are more areas of potential exposure. An LLM can be attacked by bad actors on several fronts. They could infiltrate the development team on a poorly-governed project and add malicious code to the software itself. But they could also poison the training data, fine tuning, or the weights, says Sügis.

“Hackers might retrain the model with malicious code examples, so it invades user infrastructure,” she says. “Or they can train it with fake news and misinformation.”

Another attack vector is the model’s system prompt.

“This is usually hidden from the user,” she adds. “The system prompt might have guardrails or safety rules that would allow the model to recognize unwanted or unethical behavior.”

Proprietary models don’t reveal their system prompts, she says, and having access to it could allow hackers to figure out how to attack the model.

6. Missing guardrails

Some open source groups might have a philosophical objection to even having guardrails at all on their models, or they may believe that a model will perform better without any restrictions. And some are created specifically to be used for malicious purposes. Enterprises looking for an LLM to try out might not necessarily know which category their models fall into. There’s currently no independent body evaluating the safety of open source gen AI models, says Nortal’s Sügis. Europe’s AI Act will require some of this documentation, but most of its provisions won’t go into effect until 2026, she says.

“I would try to get as much documentation as possible, and test and evaluate the model and implement some guardrails inside the company,” she says.

7. Lack of standards

User-driven open source projects are often standards-based since enterprise users prefer to have them and interoperability. In fact, according to a survey of nearly 500 technology professionals released last year by the Linux Foundation, 71% prefer open standards, compared to 10% who prefer closed. Companies producing proprietary software, on the other hand, might prefer to have their customers trapped inside their ecosystems. But if you’re expecting open source gen AI to all be standards-based, you’d be wrong.

In fact, when most people talk about AI standards, they’re talking about things like ethics, privacy, and explainability. And there’s work happening in this area, such as the ISO/IEC 42001 standard for AI management systems, released in December last year. And, on April 29, NIST released a draft plan for AI standards which covers a lot of ground, starting with creating a common language for talking about AI. It also focuses largely on risk and governance issues. But there’s not much yet when it comes to technical standards.

“It’s an incredibly nascent space,” says Taylor Dolezal, CIO and head of ecosystems at the Cloud Native Computing Foundation. “I’m seeing some good conversations around data classifications, about having a standard format for training data, for APIs, for the prompts.” But, so far, it’s just conversations.

There’s already a common data standard for vector databases, he says, but no standard query language. And what about standards for autonomous agents?

“That I haven’t seen, but I’d love to see that,” he says. “Figuring out ways to not only have agents go about their specific tasks, but also how to tie that together.”

The most common tool for creating agents, LangChain, is more of a framework than a standard, he says. And the user companies, the ones who create demand for standards, aren’t ready yet, he says. “Most end users don’t know what they want until they start playing around with it.”

Instead, he says, people are more likely to look at the APIs and interfaces of major vendors, such as OpenAI, as nascent de-facto standards. “That’s what I’m seeing folks do,” he says.

8. Lack of transparency

You might think open source models are, by definition, more transparent. But that might not always be the case. Large commercial projects may have more resources to spend on creating documentation, says Eric Sydell, CEO at BI software vendor Vero AI, which recently released a report scoring major gen AI models based on areas such as visibility, integrity, legislative preparedness, and transparency. Google’s Gemini and OpenAI’s GPT-4 ranked the highest.

“Just because they’re open source doesn’t necessarily mean they provide the same information about the background of the model and how it was developed,” says Sydell. “The bigger commercial models have done a better job around that at this point.”

Take bias, for example.

“We found that the top two closed models in our ranking had quite a bit of documentation and invested time exploring the issue,” he says.

9. Lineage issues

It’s common for open source projects to be forked, but when this happens with gen AI, you get risks you don’t get with traditional software. Say, for example, a foundation model uses a problematic training data set and from it, someone creates a new model, so it’ll inherit these problems, says Tyler Warden, SVP of product at Sonatype, a cybersecurity vendor.

“There’s a lot of black box aspects to it with the weights and turning,” he says.

In fact, those problems may go several levels back and won’t be visible in the code of the final model. When a company downloads a model for its own use, the model gets even further removed from the original sources. The original base model might have fixed the issues, but, depending on the amount of transparency and communication up and down the chain, the developers working on the last model might not even be aware of the fixes.

10. The new shadow IT

Companies that use open source components as part of their software development process have processes in place to vet libraries and ensure components are up to date. They make sure projects are well supported, security issues are dealt with, and the software has appropriate license terms.

With gen AI, however, the people supposed to do the vetting might not know what to look for. On top of that, gen AI projects sometimes fall outside the standard software development processes. They might come out of data science teams, or skunkworks. Developers might download the models to play with and end up getting more widely used. Or business users themselves might follow online tutorials and set up their own gen AI, bypassing IT altogether.

The latest evolution of gen AI, autonomous agents, have the potential to put enormous power in the hands of these systems, raising the risk potential of this type of shadow IT to new heights.

“If you’re going to experiment with it, create a container to do it in a way that’s safe for your organization,” says Kelley Misata, senior director of open source at Corelight. This should fall under the responsibility of a company’s risk management team, she says, and the person who makes sure that developers, and the business as a whole, understands there’s a process is the CIO.

“They’re the ones best positioned to set the culture,” she says. “Let’s tap into the innovation and all the greatness that open source offers, but go into it with eyes open.”

The best of both worlds?

Some companies are looking for the low cost, transparency, privacy, and control of open source, but would like to have a vendor around to provide governance, long term sustainability, and support. In the traditional open source world, there are many vendors who do that, like Red Hat, MariaDB, Docker, Automattic, and others.

“They provide a level of safety and security for large enterprises,” says Priya Iragavarapu, VP of data science and analytics at AArete. “It’s almost a way to mitigate risk.”

There aren’t too many of these vendors in the gen AI space, but things are starting to change, she says.

Artificial Intelligence, CIO, Data Management, Data Quality, Generative AI, IT Leadership, IT Skills, IT Training , Open Source, Risk Management, Software Licensing


Read More from This Article: 10 things to watch out for with open source gen AI
Source: News

Category: NewsMay 15, 2024
Tags: art

Post navigation

PreviousPrevious post:3 ways to break out of AI ‘pilot purgatory’NextNext post:NASA appoints first chief AI officer as the technology’s importance rises

Related posts

Barb Wixom and MIT CISR on managing data like a product
May 30, 2025
Avery Dennison takes culture-first approach to AI transformation
May 30, 2025
The agentic AI assist Stanford University cancer care staff needed
May 30, 2025
Los desafíos de la era de la ‘IA en todas partes’, a fondo en Data & AI Summit 2025
May 30, 2025
“AI 비서가 팀 단위로 지원하는 효과”···퍼플렉시티, AI 프로젝트 10분 완성 도구 ‘랩스’ 출시
May 30, 2025
“ROI는 어디에?” AI 도입을 재고하게 만드는 실패 사례
May 30, 2025
Recent Posts
  • Barb Wixom and MIT CISR on managing data like a product
  • Avery Dennison takes culture-first approach to AI transformation
  • The agentic AI assist Stanford University cancer care staff needed
  • Los desafíos de la era de la ‘IA en todas partes’, a fondo en Data & AI Summit 2025
  • “AI 비서가 팀 단위로 지원하는 효과”···퍼플렉시티, AI 프로젝트 10분 완성 도구 ‘랩스’ 출시
Recent Comments
    Archives
    • May 2025
    • April 2025
    • March 2025
    • February 2025
    • January 2025
    • December 2024
    • November 2024
    • October 2024
    • September 2024
    • August 2024
    • July 2024
    • June 2024
    • May 2024
    • April 2024
    • March 2024
    • February 2024
    • January 2024
    • December 2023
    • November 2023
    • October 2023
    • September 2023
    • August 2023
    • July 2023
    • June 2023
    • May 2023
    • April 2023
    • March 2023
    • February 2023
    • January 2023
    • December 2022
    • November 2022
    • October 2022
    • September 2022
    • August 2022
    • July 2022
    • June 2022
    • May 2022
    • April 2022
    • March 2022
    • February 2022
    • January 2022
    • December 2021
    • November 2021
    • October 2021
    • September 2021
    • August 2021
    • July 2021
    • June 2021
    • May 2021
    • April 2021
    • March 2021
    • February 2021
    • January 2021
    • December 2020
    • November 2020
    • October 2020
    • September 2020
    • August 2020
    • July 2020
    • June 2020
    • May 2020
    • April 2020
    • January 2020
    • December 2019
    • November 2019
    • October 2019
    • September 2019
    • August 2019
    • July 2019
    • June 2019
    • May 2019
    • April 2019
    • March 2019
    • February 2019
    • January 2019
    • December 2018
    • November 2018
    • October 2018
    • September 2018
    • August 2018
    • July 2018
    • June 2018
    • May 2018
    • April 2018
    • March 2018
    • February 2018
    • January 2018
    • December 2017
    • November 2017
    • October 2017
    • September 2017
    • August 2017
    • July 2017
    • June 2017
    • May 2017
    • April 2017
    • March 2017
    • February 2017
    • January 2017
    Categories
    • News
    Meta
    • Log in
    • Entries feed
    • Comments feed
    • WordPress.org
    Tiatra LLC.

    Tiatra, LLC, based in the Washington, DC metropolitan area, proudly serves federal government agencies, organizations that work with the government and other commercial businesses and organizations. Tiatra specializes in a broad range of information technology (IT) development and management services incorporating solid engineering, attention to client needs, and meeting or exceeding any security parameters required. Our small yet innovative company is structured with a full complement of the necessary technical experts, working with hands-on management, to provide a high level of service and competitive pricing for your systems and engineering requirements.

    Find us on:

    FacebookTwitterLinkedin

    Submitclear

    Tiatra, LLC
    Copyright 2016. All rights reserved.