US government agency to safety test frontier AI models before release

The Center for AI Standards and Innovation (CAISI), a division of the US Department of Commerce, has signed agreements with Google DeepMind, Microsoft, and xAI that would give the agency the ability to vet AI models from these organizations and others prior to their being made publicly available.

According to a release from CAISI, which is part of the department’s National Institute of Standards and Technology (NIST), it will “conduct pre-deployment evaluations and targeted research to better assess frontier AI capabilities and advance the state of AI security.”

The three join Anthropic and OpenAI, which signed similar agreements almost two years ago during the Biden administration, when CAISI was known as the US Artificial Intelligence Safety Institute.

An August 2024 release about those agreements indicated that the institute planned to provide feedback to both companies on “potential safety improvements to their models, in close collaboration with its partners at the UK AI Safety Institute (AISI).”

Microsoft said Tuesday in a blog about the latest agreement that it, and others like it, are essential to building trust and confidence in advanced AI systems. As AI capabilities advance, it said, so too must the rigor of the testing and safeguards that underpin them.

A shift toward proactive security

Fritz Jean-Louis, principal cybersecurity advisor at Info-Tech Research Group, said the CAISI agreements signal a shift toward proactive security for agentic AI by enabling government-led testing of advanced models before and after deployment.

This should, he said, “help strengthen visibility into autonomous behaviors while accelerating the development of standards to mitigate risks. By combining early access, continuous evaluation, and cross-sector collaboration, the initiative pushes the industry toward security-by-design for increasingly autonomous AI systems.”  

However, added Jean-Louis, “there are a few potential hurdles to consider, for example: how would intellectual property be protected under this approach? Regardless, I believe this is a positive step for the industry.”

Executive order ‘taking shape’

Following the announcement from CAISI, a published report on Wednesday indicated that the White House is on the verge of preparing an executive order that would see the creation of a vetting system for all new artificial intelligence models, key among them Anthropic’s Mythos.

Bloomberg reported, “the directive is taking shape weeks after Anthropic revealed that its breakthrough Mythos model was adept at finding network vulnerabilities and could pose a global cybersecurity risk.”

Significant change in policy direction

Carmi Levy, an independent technology analyst, said, “it is patently obvious that this week’s announcement that establishes the Center for AI Standards and Innovation as the testing ground for frontier AI models is directly linked to the potential executive order that would lead to a vetting system for AI models.”

It isn’t coincidental, he said, “that the announcements were made in rapid succession, and it reinforces the growing urgency for governments in the US and elsewhere to tighten partnerships with key AI vendors to maximize AI-related security and minimize the potential for systemic risk.”

This latest flurry of activity from Washington marks a significant shift in policy direction from an administration that up until recently had been following a more laissez-faire approach to regulation, Levy pointed out.

Concerns around Anthropic’s Claude Mythos model, and the relative ease with which it could discover and exploit vulnerabilities in digital systems, “might have helped shift the federal government’s position on AI-related regulation, particularly around the renewed push to enforce standards for AI-related deployments across government infrastructure,” he said.

AI vendors like Google, Microsoft, and xAI, Levy added, “must walk a political highwire of sorts as they balance the need to release models into the marketplace in a timely, cost-effective manner with increasingly defined rules around AI-related cybersecurity and safety. The industry can’t afford a scenario where vendors themselves make up the rules as they go along.”

At the same time, he said, the recent showdown between Anthropic and the Pentagon illustrates why the vendors might be forgiven for viewing the federal government’s growing interest in AI testing and regulation with at least a certain degree of caution.

According  to Levy, “while the administration’s efforts to centralize testing and oversight should streamline the go-to-market process for vendors and accelerate the development of best practices around frontier model development, the political overtones of recent government-industry partnerships cannot be ignored.”