Explainable AI is necessary, but it’s not enough

A fraud model looks at an insurance claim and returns a score of 0.23: low risk. A SHAP explanation lays out exactly why: no prior claims, a modest claim value, an unremarkable claimant profile. A human adjuster reads the explanation, agrees with it and signs off. Every box that explainable AI asks us to check has been checked. The claim is settled and closed.

By any current standard for responsible AI, this is exactly what a good outcome looks like.

Two months later, an audit

Two months after the claim closes, it’s pulled into a routine quarterly audit sample (no suspicion attached, just standard sampling). The auditor isn’t reviewing the model’s reasoning. She’s asking a narrower question: can this decision be reconstructed, independently, from its own evidence trail?

She doesn’t start with the fraud score. She starts with the paperwork.

Her first question: what established the date of the incident? She pulls the document meant to corroborate it and checks something the claim file itself never surfaces, the document’s own creation metadata. It was generated two days after the claim was filed. She asks the obvious follow-up: did anyone check this before the claim reached the model? Nobody had. Document provenance was never captured anywhere in the intake pipeline as a fact to be verified; the claim reached the model looking exactly like every clean claim before it.

Knowledge gap: This doesn’t establish that the claim is fraudulent. A prior version of the document may exist and say the same thing, in which case the late timestamp is a non-event. What it establishes is that the system has no way to answer that question either way. Document lineage was never captured, so there’s no way to know whether an earlier version existed, or whether it agreed with this one.

Her second question: when was the insured asset last valued? The figure the model used is 22 months old relative to the date of loss. Nothing about it is wrong or falsified; it’s simply the most recent valuation on file, resolved to the wrong point in time and handed to the model as though it were current.

Knowledge gap: This doesn’t establish that the valuation is wrong. The asset’s value may not have moved materially in 22 months, in which case the old figure and a fresh one would say the same thing. What it establishes is that the system has no process for knowing which is true. There’s no trigger for re-valuation and no record of whether the figure was ever checked against current conditions.

Her third question: as of what date was the claimant’s risk profile evaluated? The “current risk” features the model consumed were pulled at query time (the day the claim happened to be scored) rather than resolved back to the date of loss. She asks whether that was a deliberate design choice. Nobody can tell her. There’s no rule on record, no document, no named owner for the decision “risk features should reflect the date of loss, not the day the system happens to be queried.” It’s simply how the pipeline was built.

Policy version gap: This doesn’t establish that the score would have come out differently under incident-time data. The claimant’s risk profile may not have changed since the date of loss, in which case query-time and incident-time features would produce the same result. What it establishes is that nobody decided which point in time should govern the decision, so there’s no way to know whether the two agree, only that nobody checked.

Three questions. Three gaps. Not one of them touches the model, and not one of them would have shown up in a model audit, but all of them question the explainability of the decision taken.

The blind spot in explainability

This is the uncomfortable pattern showing up across banking, insurance, healthcare and government AI deployments: the model is explainable, the explanation is faithful and the decision is still wrong. It’s worth being precise about why, because the instinct is to reach for a better interpretability technique, or a smarter document-reading agent, and neither fixes this class of failure.

Every mainstream explainability approach shares one architectural trait, whether it’s SHAP, LIME, DARPA’s XAI program, the EU AI Act’s Article 13 disclosures, or IBM’s AI FactSheets: they all operate after the model has reasoned. They take the inputs as given, assumed correct and explain what the model did with them. That’s legitimate and valuable. It’s also, structurally, a blind spot for the class of failure the audit just surfaced.

An enterprise decision isn’t just a model output. It’s a reasoning process running over a knowledge base and a policy set, both of which have to be current, correctly sourced and resolved to the right moment in time before the model ever sees them. A model can reason perfectly over compromised inputs and produce a perfectly explainable, perfectly wrong decision. No amount of post-hoc transparency catches that, because post-hoc transparency was never looking at the inputs; it was only looking at what the model did with them.

Put precisely: a decision is explainable only when its reasoning, its knowledge and its policy are all explainable at the same time. Explainable AI delivers exactly one of those three: explainable reasoning. It can be flawless, and the decision can still be unexplainable overall, because an unverifiable knowledge input or a misapplied policy breaks the chain regardless of how good the reasoning layer is. That’s the precise sense in which explainable AI is necessary but not sufficient. Sufficiency is a property of the whole decision (knowledge, policy and reasoning together), not of any one component, however mature that component’s tooling has become.

Where the audit’s three questions actually point

Map the auditor’s three findings back onto that structure and the pattern sharpens.

Her first and second questions (the backdated document, the stale valuation) are both knowledge failures. Neither is a data-entry error; both are facts that were never traceable to an authoritative source, versioned and resolved to the right point in time before reaching the model.

Her third question (features pulled at query time rather than incident time) is a policy failure. No rule existed to say which moment in time the decision should be evaluated against, so the system defaulted to whatever was easiest to compute: now.

Governing the inputs, not just the output

The fix isn’t a better explanation of the model, and it isn’t a more sophisticated agent reading the documents more carefully. It’s governance that sits before the model, a framework I call Explainable Decision Systems (EDS), built on three pillars plus a temporal layer that ties them together.

This tracks a broader shift already visible across enterprise AI deployments: the stacks that win aren’t the ones that ship fastest, but the ones that get data and decisions governed before anything downstream runs.

• Explainable Knowledge (EK). Every fact fed into a decision has to be traceable to an authoritative source, versioned and (critically) resolved to the valid time of the event being decided, not the time someone happened to query the system or the date typed into a form field. An asset valuation from 22 months ago isn’t wrong data; it’s data resolved to the wrong point in time.
• Explainable Policy (EP). The policy version that governs a decision is the one that was active when the event occurred, not the one currently in the system. A claim from November has to be judged against the policy wording in force in November (exclusion schedule and all), retrieved and locked before reasoning starts, not looked up after the fact.
• Explainable Reasoning (ER). This is where existing XAI actually earns its keep, but EDS splits it into two accountable parts: AI reasoning (EAI), which needs interpretability techniques like SHAP, and Human Decision (HD) (overrides, approvals, escalations), which needs an identified actor, a timestamp and a documented justification. EDS doesn’t claim human judgment is formally explainable in the way a model’s math is; it just insists that it’s attributable and auditable.

The layer that binds all three together is bi-temporal governance: knowledge and policy have to resolve to the real-world moment the decision concerns, not the moment the system happens to be queried. In EDS, this resolution (called a temporal witness) isn’t a report generated after the decision, the way the audit in this piece was. It’s a gate the decision has to pass before reasoning is allowed to start. Had the witness been in place, the claim in this scenario would never have reached the model at all; it would have been flagged as a governance exception the moment the document’s creation timestamp failed to match its claimed valid time.

Why no AI agent can catch the hospital record

This is worth slowing down on, because it’s the clearest illustration of why EK has to be its own dimension, entirely separate from model explainability or agentic sophistication.

Every clinical or administrative record system has two different dates living in it. There’s the stated field value (“date of incident,” “date of visit”), which is just a form entry: whatever a person typed, told a clinician or was asked to record. And there’s the system-of-record creation timestamp: the moment the backend actually inserted that row, logged that transaction or wrote that file. That timestamp sits in the origin system’s audit trail, not on the document itself.

A claimant doesn’t need to forge anything to exploit the gap between those two dates. They need only have a record created on 30 November whose stated incident-date field reads “15 November.” The document is completely genuine. Nothing about it is altered, and no forensic check (pixel analysis, formatting consistency, metadata inspection of the file itself) will find anything wrong, because there is nothing wrong with the document. The lie isn’t in the artifact. It’s in the mismatch between what the artifact claims and when an entirely separate system actually created it.

Hand that document to an AI agent, no matter how advanced, and ask it to check for fraud. It will read the stated date, assess whether the narrative is internally consistent, run every forensic technique available and come back clean; it’s answering the wrong question. It’s checking whether the document is coherent and unaltered. The question that actually matters is whether the document’s claimed date matches the creation-time metadata logged by the system that produced it. That answer doesn’t live in the document. It lives in a separate system, under separate access control and requires a deliberate governance step to retrieve and cross-reference. No amount of model capability substitutes for that step, because the information the model would need simply isn’t present in what it’s being asked to read.

This is exactly the gap enterprise surveys keep flagging. In Grant Thornton’s 2026 AI Impact Survey of nearly 1,000 senior business leaders, 78% said they lacked confidence their organization could pass an independent AI governance audit within ninety days, precisely because most are scaling AI they cannot explain, measure or defend. That’s the downstream version of the same problem this claim ran into: explainability treated as a reporting exercise rather than a precondition the case has to clear before it ever reaches the model.

That is precisely what Explainable Knowledge formalizes: a requirement that every input be provenance-linked to its origin system, with independently retrieved creation-time metadata checked against the claimed valid time; not a soft appeal to “make sure your data is good,” but a specific, checkable technical control. Under EDS, if that cross-reference can’t be performed, the claim doesn’t proceed to the fraud model. It’s blocked as a governance exception, regardless of how convincing the document looks or how confidently any reader, human or AI, assesses it.

These aren’t academic distinctions, either. In the UK and international context, failures like these map to live regulatory obligations: FCA claims-handling rules on treating customers fairly, Federal Reserve SR 11-7 guidance on model risk management, and the EU AI Act’s Article 9 risk-management requirements for high-risk systems. Regulators are already asking questions that XAI, and the agents built on top of it, cannot answer.

What this means for the people building these systems

For enterprise architects, the practical shift is this: stop treating knowledge and policy governance as documentation you produce after a model runs (or worse, as something an audit uncovers two months later) and start treating it as a gate the model has to pass through before it runs. This is the same direction major risk-consulting practices are now pushing clients toward: governance embedded as a control layer across the AI lifecycle, rather than a compliance sign-off at the end of it. Concretely, that means:

• Bi-temporal data stores for anything that feeds a decision, so “valid at the time of the event” is a queryable property, not an assumption.
• Provenance links from every document or claim to its origin system’s own creation-time metadata, captured independently of the stated field values on the document (not derived from them).
• Policy version registries with effective and expiry dates, so the version applied is probably the one that was active when the event occurred.
• A pre-inference middleware layer that checks provenance, versioning and temporal validity before forwarding anything to the model, and blocks or flags the case if it can’t.
• Explicit, attributable records for human overrides: who, when, under what authority and why.

None of this replaces SHAP, LIME or your existing model-risk documentation, and it isn’t a call for a smarter document-reading agent either. It sits in front of both, deciding whether they’re even looking at trustworthy, independently verified inputs.

The bar has moved

Explainability was always going to be necessary. Regulators demand it, customers expect it and no serious enterprise AI deployment should ship without it. But “the model is explainable” and “the decision is governable” have quietly become two different claims, and treating them as one is how a technically pristine 0.23 fraud score sails through, gets closed and sits untouched for two months until an auditor asks the one question explainability was never built to answer: not what did the model do, but what was it allowed to look at, and when, and against what independent record?

Architectures that can’t answer that question will keep producing decisions that are perfectly explainable and quietly wrong, right up until the audit finds them.