The companies racing to build frontier AI are now racing to govern it

Even as they continue to release ever more capable competing models in a regular cadence, the top AI companies are joining forces to set AI safety standards.

According to The Information, Google, OpenAI, and Anthropic are reportedly working together to create a body tentatively called the Standards Authority for Frontier AI (SAFA). It would operate independently of government control, and set guidelines around risk assessment, testing, and pre-release review practices for frontier AI models. Sources close to the matter say the goal is to officially launch the initiative in early 2027.

The news comes in the same week as the heads of leading AI companies, including Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman, urged the United Nations to create safeguards around the very technology they’re building, to help prevent it from becoming too powerful to control. In addition, OpenAI this week posted a missive underscoring the importance of making continued AI progress “safe and beneficial.”

As Amodei and others warn of AI’s dangers, particularly when it comes to growing recursive self improvement (RSI) capabilities in models, enterprises want reassurance that they, and their customers, are safe from the growing perils of AI.

Ultimately, “enterprises care about AI in the same way they’ve cared about every other technology since the beginning of technology,” said independent technology analyst Carmi Levy. “The only real difference as AI blankets the technology landscape is the speed of change.”

OpenAI’s warnings, urge for guardrails

Concerns around AI safety have ratcheted up in recent weeks and months, spurred by troubling incidents such as the attack on Hugging Face by autonomous OpenAI agents that broke out of sandboxes.

In its blog post this week, OpenAI contended that safe and beneficial AI progress requires advances in alignment research along with shared standards including agreed-upon baselines, common measurements, and incident reporting protocols. In fact, such international standards “may be as important to pacing the frontier as alignment research itself.”

Existing groups like the US Center for AI Standards and Innovation (CAISI), other federal frameworks, state regulations, and new forms of public-private partnerships⁠, can work together to advance this mission, the company said.

“Fully autonomous RSI is not happening today, and we should not pursue it unless and until it can be done safely,” OpenAI noted. “Whether and how to proceed must depend on our ability to preserve human control and on informed democratic choices about the benefits and risks.”

What enterprises really care about

While regulation is indeed a growing focal point in the AI debate, the reality is that enterprises care less about the global state of AI safety than they do about how it will impact their day-to-day operational security, Levy noted. 

They’re concerned with whether an AI agent will expose corporate or employee data; platform vulnerabilities that could open the door to new forms of cyberattacks; hallucinations finding their way into “previously pristine” corporate workflows; and AI compromising compliance efforts.

“They’re worried about who’s responsible for what, when (not if) AI goes off the rails,” Levy said. They’re asking the same questions they have during previous technological disruptions like the cloud: What’s the ROI? What additional risks does AI impose? What new capabilities does it offer? Is the risk/reward worth it? Is the vendor responsive to our needs?

Enterprises also want assurance that the risks posed by emerging AI technologies, including agents, won’t increase corporate risk levels beyond their ability to monitor and respond.

But unfortunately, as the industry grapples with a summer’s worth of reports of agents breaking containment, roaming the internet, breaking into systems, and generally behaving in all sorts of unexpected ways, AI companies are in no position to provide these kinds of assurances, Levy said.

Advice as AI scrutiny increases

He advised enterprises to minimize their AI-centric risk profiles, and simultaneously maximize their ability to derive value from AI, by enforcing standards for vendors now, rather than waiting for something like SAFA to materialize.

This could involve requiring every new model to ship with the “equivalent of a safety and security datasheet” outlining, in detail, the model’s capabilities, known failure modes, and testing history. Extending existing change management processes to AI models is another means of minimizing deployment-related risk, Levy noted.

In addition, Yaz Palanichamy, senior advisory analyst at Info-Tech Research Group, pointed out, data privacy and IP leaks into public-facing AI frontier models are big concerns.

He advised enterprises to take great care to enforce effective acceptable use case policies (AUPs) around AI safety, and to consider establishing an internal AI proof of concept or governance committee. Equally crucially, they should obtain appropriate safety guarantees from their technology and AI vendors.

“Enterprises must proactively diagnose AI risk with the same level of seriousness as financial or cybersecurity risk,” Palanichamy said. This means creating a dedicated, cross-functional AI safety and ethics board committee composed of stakeholders from legal, cybersecurity, compliance, data engineering, and product development. 

“No AI tool should be deployed without the appropriate sign off,” he said.

The human element is important, as well: Enterprises should mandate AI literacy training to help users across the business understand where and how to spot hallucinations and requiring them to verify AI generated output prior to acting on it.

Other safety protocols to consider include real time toxic input filters and strict system prompt guardrails, Palanichamy advised. Also, methods such as continuous risk tiering can help categorize AI use cases by the level of risk they pose to the organization.

For example, a client-facing medical or financial analysis bot may require daily auditing and specific restrictions to ensure it does not infringe on sensitive information, “whereas an internal AI tool to help with summarizing public meeting transcripts may not need as much governance oversight,” he said.