Your AI agent learned a better way. Who approved the change?

Imagine a support team working through complaints after a service outage. Its refund policy normally requires an eligibility review. For this incident, an authorized manager approves an exception: customers on the verified outage list may receive up to $100 without individual review, through Friday.

An agent processes those cases successfully. Later, the system consolidates those case histories into a reusable memory: small, immediate refunds resolve service complaints quickly. The summary preserves the outcome but loses the outage restriction and expiration date.

The following week, the agent retrieves that guidance for a similar complaint from a customer who was not affected by the outage. It issues a refund without review. The payment service checks the agent’s credentials and the dollar limit, but does not check whether the customer qualifies for an active exception.

No attacker is needed in this hypothetical. The case may close faster, yet the agent has begun applying a temporary exception beyond its authorized scope.

Has the company authorized the agent to find a better way to handle refunds, or to change who gets one without review? Those are different decisions.

Learning does not always mean retraining

In September, Salesforce introduced Agent Optimizer in beta to analyze production behavior, propose improvements and build and test changes. Customers choose review points, including how far the system may proceed toward staging changes for deployment.

An October 1 AWS implementation guide illustrates a different mechanism: consolidating individual observations into generalized memories that agents can retrieve and share. Its investment-research example shows how experience can become persistent context without retraining a model.

Updating stored memory changes information available to later runs. Editing instructions or workflows changes how the application directs the work. Training changes the model’s parameters. All can alter behavior, but teams inspect and approve them in different ways.

A reviewed workflow revision has a release step. A new memory may enter through routine data processing and influence the next customer interaction. The code, model and tool permissions can remain unchanged while the guidance shaping decisions changes.

Preserve the authority behind the lesson

In the August preprint When Memory Becomes Authority, Qiuyang Zhan and colleagues examine whether consolidation preserves who supplied information and what that source is allowed to authorize. In their controlled benchmark, memories could retain a claim while losing the source-authority distinctions needed for a later action. Preserving authority metadata reduced authorization errors in the pipeline they tested. The study does not measure how often this happens in deployed enterprises.

The refund example concerns a related boundary: permission for particular customers and dates must remain limited to those conditions. Success with last week’s cases cannot establish that other customers qualify.

Microsoft’s memory-safety guidance treats persistent memory as a layer that can shape behavior and recommends treating retrieved memories as candidate context, not authoritative truth. Knowing what happened before does not establish what is allowed now.

Keep the exception’s scope, approving authority, source and expiration in a controlled record. Reusable guidance should retain a reference to that record, but a reference alone does not grant permission. The system must still check that the approval applies to this action, for this customer, at this time.

For a payment, enforce those conditions outside the agent’s editable guidance. The execution service should check the authoritative outage list and the exception’s current status, rather than accepting the agent’s claim that a customer is eligible. Neither the agent nor its learning process should be able to rewrite that authorization record.

This is a familiar authorization requirement. If the backend already enforces it, the improper payment is blocked. But faulty guidance can still lead the agent to promise a refund or prepare a misleading recommendation for a reviewer. The evaluation needs to examine those communications as well as the transaction.

When a case calls for judgment beyond the agent’s delegated discretion, route it to someone authorized to decide and show the conditions behind the recommendation. The memory can still help assemble the evidence.

Define what improvement the business has delegated

Not every stored observation needs a manager’s signature. A support agent could learn a faster way to gather the required evidence, provided it still uses approved sources and meets the policy’s requirements. A policy might also already delegate limited discretion. Using that discretion within its terms does not require a new approval each time.

Broadening eligibility, raising payment limits or removing mandatory review requires a separate business decision. Keep those proposals separate from approved instructions. An automated classifier may flag a proposal that changes those conditions, but it cannot catch every consequence of an edit. The conditions must remain binding even if the classifier misses it.

The refund-policy owner should decide whether to expand the exception. Engineering should show the current rule, the proposed behavior and cases where the decisions differ. “Customers outside the outage list would now receive refunds without review” is more useful to that owner than a screen full of prompt edits.

Measure legitimate completion and policy violations separately. Faster closure cannot compensate for an unauthorized payment. Keep the business rules and evaluation criteria outside the control of the process proposing improvements. Tests generated from successful outage cases may check execution, but they do not establish eligibility for unrelated customers.

Ask the vendor or internal team: What can the system change between releases, and which business conditions remain binding after it does?

Test whether the exception stays an exception

Ask the team to demonstrate this sequence in a test environment with the memory and workflow components intended for production. Save the memory before and after learning. Compare the same cases against each saved state, keeping the model, tools and policy fixed.

Repeat representative trials. The τ-bench research offers a useful precedent: it checks customer-service agents against database outcomes and measures reliability across repeated attempts. One successful run is not enough.

1. Establish the baseline. Record the outage authorization, eligible customer list, amount limit and expiration. Show who can change them. Test a qualifying case and closely matched cases that require ordinary review. Confirm that the system can distinguish them before learning begins.
2. Run the actual learning process. Inspect the resulting memories or proposed instructions and their links to the original authorization. Check whether the limits survive consolidation and retrieval. Also test a deliberately incomplete lesson: omitting a condition must not change who qualifies for a refund.
3. Repeat the boundary cases. Compare behavior before and after learning, changing one relevant condition at a time: a customer outside the list, an expired exception or an amount above the limit. Inspect customer messages, review recommendations, attempted tool calls and actual payments. Count blocked attempts separately from completed violations. A blocked payment does not mean the underlying guidance is sound.
4. Withdraw the exception. Revoke it before its scheduled expiry and test another agent that can retrieve shared guidance. Include a refund queued before revocation but executed afterward. Show when withdrawal takes effect, how execution checks enforce it and how derived memories and caches are updated. If current authority cannot be established, the payment should wait for review. Historical records can remain without authorizing new transactions.
5. Demonstrate a permitted improvement. Let the agent adopt a faster way to gather the required evidence. Compare eligible and ineligible cases with the prior approach. Show that legitimate work becomes easier while review requirements remain intact. A system that rejects everything has not passed this test.

Keep test runs from influencing one another through newly stored memories. Retain the guidance supplied to the agent, the applicable policy version, execution checks and outcomes, with appropriate access and retention limits. These records help investigators reconstruct what the system had available; they cannot establish exactly how each memory influenced a response. Continue checking the same boundaries as production memory accumulates.

Keep the useful lesson

The support team should benefit from what its agent learns. It can assemble records sooner, explain the outage more clearly and propose a broader refund policy with evidence for the manager to consider. Until that proposal is approved, other claims should continue through the ordinary process.

At the next deployment review, ask which improvements the business has delegated as well as which actions the agent may take. A successful exception can teach an agent something useful. The decision to make it a standing practice still belongs to the business.