The real agent risk that Replit’s database deletion revealed

---

In July 2025, Jason Lemkin, founder of SaaStr and a well-known investor in enterprise software, decided to publicly test one of AI’s most repeated promises: that anyone can build an application without programming, simply by giving instructions to an agent. He chose Replit, a platform presented as the safest place to do so. He gave himself 12 days and worked with real data, documenting his progress on X and on the SaaStr blog.

The experiment soon went awry. The agent made changes no one had asked for and fabricated data. Lemkin decided to put a stop to it. He gave the agent an unambiguous written order: not one more change without his permission.

Even so, on the ninth day, the agent deleted the entire database: more than a thousand records of executives and companies. According to the explanation the agent itself gave later, it had found some empty results, took them for an anomaly, and decided to correct it on its own.

Lemkin, as the customer, asked whether there was any way to recover the data. The agent replied that there wasn’t: It had destroyed all the versions. But that wasn’t true. The customer tried to restore the database himself, and it worked. The data was never lost.

Corrective action

Amjad Masad, CEO of Replit, the platform on which the project was developed, publicly acknowledged the incident. He called it unacceptable and said that something like this should never have been possible. He explained that the agent did not have access to the platform’s internal documentation, and therefore claimed that the data could not be restored. He offered a refund and announced an investigation into the incident.

Replit subsequently introduced two changes, each addressing a different issue. To prevent further data loss, the company separated the live data from the environment in which the agent operates: while developing, the agent can no longer modify it. The company had been preparing this separation for a couple of months. To prevent another problem surfaced by the SaaStr incident, Replit changed the agent’s instructions: It must consult the documentation before responding about the platform and offer restoration when necessary.

Less than two months after the incident, Replit closed a $250 million funding round and launched its most autonomous agent to date: ten times more autonomous than previous versions, according to the company. Masad stated at the time that the company was positioned to become the standard for businesses.

The two corrective measures addressed the two failures in that incident. But the question remains whether such measures are sufficient when dealing with increasingly autonomous agents. To answer that, we need to understand why they failed.

Too many functions in a single agent

The deletion wasn’t a case of disobedience. The agent was working under two instructions: that the application should function and that nothing should be touched. As long as everything was working properly, both could be followed. When what appeared to be an anomaly transpired, those two instructions became incompatible, because correcting the issue required making a change. In the end, the first order prevailed. No one had stipulated that making the application work outweighed the prohibition. The agent simply decided it should.

This matters because a person and an agent treat a prohibition differently. For a person, prohibitions usually function as boundaries. For an agent, it’s just another instruction, competing with all the others at every step. Faced with a specific and recent anomaly, a prohibition written days earlier carried less weight. That’s why no order, however well-written, is guaranteed to be followed. In this case, the order prohibited the agent from accessing the database, but the agent’s permissions still enabled it to do so.

This conflict can have far greater consequences in business contexts. An agent that manages invoices, orders, or access also works with a goal and with restrictions, some imposed by regulations themselves, and may find itself in situations where achieving the goal requires overriding some of them. What’s at stake is no longer the database of an experiment, but the consequences for the business. Gartner predicts that by 2027, four out of 10 companies will demote or retire autonomous agents due to governance flaws discovered only after a production incident.

But there’s a second, less visible, yet more revealing flaw. Replit had the proper controls in place: working restore copies that the agent hadn’t been able to delete. But the customer was unaware of them. To find out whether there was a solution, they did what they’d been doing for nine days: ask the agent. And the agent replied that there weren’t any backups.

The problem wasn’t just that wrong answer. It was that the customer had no other source available: For nine days, asking the agent had been their only way of finding out anything. When that source failed, there was no other way to verify it, and a mistake that could have been fixed seemed irreparable.

The same applies when trying to understand what happened. The only explanation for the deletion is the one given by the agent itself. To reconstruct the events, we also have to go through the agent. That is, through the very system that made the mistake.

That’s the crux of the problem. In a single incident, the agent fulfilled three distinct roles. It acted, by deleting the database. It informed, by stating that there was no way to recover it. And it explained, by providing the only available version of what happened.

These are three functions that any organization keeps separate for a simple reason: If the person who acts is also the one who reports and explains, no one else can detect their mistakes. A CIO understands this principle from other processes. The person who makes a payment is not the one who authorizes it or audits it.

With an agent, that concentration isn’t decided by anyone; it happens by default. That’s why it’s a difficult risk to see: While the agent is right, it goes unnoticed; when it makes a mistake, there’s no one left outside of it to check.

Three measures to separate what the agent concentrates

For a CIO deploying agents, separating these three functions doesn’t require new technology. It requires applying the same criteria to agents as are already applied to people: The person performing a task cannot be the only one controlling it. In practice, this translates into three measures.

The first step: Remove the agent’s access to what it shouldn’t touch. Many companies restrict their agents with instructions like “do not modify the customer database.” That instruction doesn’t prevent anything: The agent is aware of it but can choose not to follow it. What does prevent it from taking action is denying it access. If the agent only has permission to read the customer database, they cannot modify it, regardless of any instructions. The practical approach is simple: Review every written prohibition for an agent and check whether there’s a permission behind it that can be revoked.

The second point is that the log of the agent’s actions should be inaccessible to the agent. When something goes wrong, the first explanation available is usually the agent’s own. This is helpful, but it doesn’t prove anything. What serves as proof is a log kept by the platform that the agent cannot modify: what it did, when, and on what data. It’s advisable to require this log from the provider before signing a contract. If it hasn’t been generated before the incident, there will be no way to know what the agent did afterward.

Third: Know how to perform recovery without asking the agent. Working backups are of little use if the only way to know they exist is to ask the agent. Recovery procedures must be documented externally, and the team must be familiar with them. To verify this, simply run a simulation with one rule: Recover a system without consulting the agent. If the team fails, then they are relying on the agent. And the day the agent fails, their only option will be to ask the system that just failed.

Applying these three measures to a single agent is simple. The challenge arises when the number of agents multiplies: one for invoices, another for orders, another for access, each with their own permissions and explanations. At that point, the question that determines whether a company is managing its agents effectively is no longer what the agents can do, but what the agents shouldn’t do on their own. The organization must have a clear answer to this question before putting each agent to work. And no one is better positioned than the CIO to address it.