Why Explainability Is the Wrong Word for Legal AI

"Explainability" has become the polite answer to an uncomfortable question: Can we trust this system?

In legal AI, explainability is usually offered as reassurance rather than control. A post-hoc story about why a model produced an answer, wrapped in language that sounds sensible and feels compliant. The problem is that legal work does not run on reassurance. It runs on responsibility, and explainability is a poor proxy for that.

Over the last few years, the industry has quietly optimised for explanations at exactly the point it should have been investing in operational discipline. That shift has consequences once AI leaves demos and pilots and starts shaping real advice.

Explainability frames AI as a narrator, not a system

The logic behind explainability is easy to understand. If we can see the reasoning, we can judge whether the output is acceptable. If the model can justify itself, then reliance feels safer.

That logic holds in domains where decisions are informal and easily reversible. Legal work is neither. Most legal failures are not caused by answers that make no sense, but by answers that make sense for the wrong reasons. Hidden assumptions, missing inputs, silent exclusions, or an unclear sense of who was responsible for relying on the output in the first place tend to be the real culprits.

Explanations, especially fluent ones, are persuasive. They create confidence even when they should trigger hesitation. That is not an accident. It is a natural outcome of systems designed to produce coherent language rather than verifiable process.

Legal risk rarely sits in the explanation

When AI causes problems in legal workflows, the questions that surface afterwards are rarely philosophical. They are practical, and usually uncomfortable.

• Which document version was used
• What content was in/excluded
• Which jurisdictional assumptions were active
• Whether the output was treated as a draft or as advice
• Who approved using AI at that stage of the matter

Explainability does not answer these questions. In practice, it often distracts from them, offering a tidy rationale while leaving the real risk untouched. You can have an answer that is perfectly explained and still be unable to defend how it was produced.

What legal AI actually needs instead

If you step back and look at how legal work is defended under scrutiny, three properties matter far more than any narrative explanation.

Traceability

Every material output needs lineage that can be followed without interpretation. Inputs, document versions, prompts, models, configurations, and execution context should all be accessible, not buried in logs that only engineers can read.

Accountability

Explainability subtly shifts responsibility towards the model, as though the system itself owns its conclusions. Legal practice cannot operate on that fiction.

Reversibility

Legal work is iterative by nature. Advice evolves, assumptions change, and new facts emerge. Any system that cannot safely replay, revise, or unwind an AI-influenced decision is misaligned with how law actually works.

A simple example: where explainability sounds helpful and still fails

Let's try this this out with an example, say an AI tool reviews a share purchase agreement and flags potentially risky clauses for follow-up. Nothing novel, just standard triage.

The explainable version

The tool flags a limitation of liability clause and offers an explanation along the lines of:

This clause is flagged as high risk because it broadly limits the buyer’s remedies and deviates from typical market protections.

It reads well, it's clear and reassuring... however two weeks later, a client asks why a very similar clause in another agreement was not flagged.

At that point, the explanation stops being useful.

You cannot easily see which document version was analysed, whether jurisdictional assumptions differed, which clauses were in scope, or whether a configuration change quietly altered thresholds between runs. The explanation tells you why the model flagged something, but not why it behaved differently elsewhere.

You have a story, not control.

The controlled version

Now imagine the same tool, but designed around traceability, accountability, and reversibility.

The clause is flagged, and alongside it you can see the document version analysed, the clause text and surrounding context used, the governing law assumption applied, and the confidence threshold in effect at the time.

When the second agreement is questioned, it is immediately clear that a different jurisdictional configuration excluded that clause category, based on a decision signed off earlier in the workflow. If needed, the analysis can be rerun under the original assumptions and compared.

It's not a debate, hand wavy or reliance on how persuasive the explanation sounds. Just a system behaving like a professional tool.

Explainability may look identical in both cases, but everything that matters is different.

Why explainability became the default anyway

Explainability is easy to demonstrate and easy to sell. It maps cleanly to high-level regulatory language and fits neatly into product demos.

• Traceability requires engineering effort.
• Accountability requires organisational clarity.
• Reversibility requires architectural intent.

Those are harder conversations, and so the industry largely avoided them.

But Ryan…

Now I know, I know, I can already hear a few of the objections:

But Ryan, regulators are explicitly asking for explainability.

They are, but mostly as shorthand. What regulators actually care about is whether decisions can be defended, challenged, and evidenced after the fact. Explainability appears in guidance because it is an accessible proxy for those outcomes, not because a well written rationale is sufficient on its own. Would you rather do what's actually useful or not?

But Ryan, this sounds academic. People just need tools that work.

This is not about theoretical purity or perfect systems. It is about survivability.

Most AI deployments do not fail because they are inaccurate. They fail because the first serious challenge exposes that no one can clearly explain what happened, who owned the decision, or how to unwind the impact.

But Ryan, this is expensive and hard to build.

Yes. It is harder than generating explanations. That is precisely the point.

Legal AI sits in a domain where mistakes have downstream consequences, not just bad user experience. Difficulty is not an argument against building proper controls. It is an argument for being honest about where AI is appropriate and how it should be deployed.

But Ryan, this only matters for high-risk use cases.

Legal teams rarely know which work will become high risk at the point it is automated. Low-stakes tasks have a habit of becoming high-stakes very quickly once a client, regulator, or dispute enters the picture.

But Ryan, explainability helps people trust AI.

It does, to a point. Explanations are useful context, particularly for review and training. The problem starts when explanations are treated as a substitute for control.

But Ryan, you’re asking lawyers to understand too much technical detail.

Not at all. Lawyers do not need to understand how the system works internally. They need to know how to question it, when to escalate, and what happens if the answer turns out to be wrong.

Explainability has a role. It can help people engage with new tools, support review, and make early adoption less intimidating.

It just cannot carry the weight it has been given.

Legal AI lives in workflows where decisions compound, assumptions shift, and yesterday’s low-risk output can quietly shape tomorrow’s advice. In that environment, reassurance is not enough. What matters is whether the system behaves predictably when things get uncomfortable.

Traceability gives you something to point to. Accountability makes it clear who owns the decision. Reversibility gives you room to change course without pretending nothing happened.

If AI is going to sit alongside legal professionals, it needs to behave like one. Its work should be reviewable, challengeable, and correctable, without relying on persuasive explanations to smooth over the gaps.

Explainability can sit on top of that, and in the right places it probably should. It just should not be the foundation.

Once you see the difference, it becomes hard to unsee.