What is Explainability? — one analogy

1The analogy

Drop your car off for repairs and get back a single line: "That'll be $400." You can pay it, but it doesn't sit well. The shop down the street hands you a sheet instead. Which part got replaced and what it cost, how many hours the work took, why it needed doing now, line by line.

Same $400, but this version you can question. If a number looks off, you can point at exactly which line is off, and you can see what needs attention next time. With an itemized sheet, the customer is looking at the same picture the mechanic is.

Asking AI for that same sheet is explainability. Instead of a single result dropped on you, it means laying out, line by line, what it looked at and how much weight each thing carried on the way to that judgment.

2In detail

A single line can't be questioned

What AI hands back is usually short: this photo leans one way, this text looks risky, this application falls short of the bar. One line. When the judgment affects a person, that line alone doesn't let anyone do anything with it. You can't even say where it went wrong, because there's nothing to point at.

With reasons attached, three things become possible: a wrong judgment can be corrected, a judgment can be checked for resting on some odd clue, and the person on the receiving end can be told exactly what would need to change.

Problems often only surface once the reasons get opened up. A model turning out to be reading the ruler that always appears next to a wound in photos of injuries, or sorting by snowy backgrounds instead of the animal in them. None of that shows up in an accuracy score alone.

Marking what it looked at

The most common method is to mark which part of the input carried the most weight in the decision. In a photo, the regions that mattered most come back highlighted; in a sentence, each word gets sized by how much it contributed.

The calculation is usually some version of "how much does the result shift if this part gets covered up." A big shift means it was weighted heavily; no change means it barely mattered. It sits in roughly the same spot as the dollar amount next to each line on that repair sheet.

This kind of marking shows where the model looked, but not why it read what it saw that way. A highlighted region tells you attention landed there; what made that spot look risky is still something a person has to guess at.

Changing one condition at a time

A different method is to swap one thing and ask again. Leave everything else the same, change a single item, and see if the answer flips. If stretching out someone's tenure alone flips the result, tenure was the condition that decided the case.

This kind of explanation is especially useful for the person on the receiving end. It answers "what would have had to be different" rather than just "why not." A single concrete condition lands better than a vague list of factors.

It only means something, though, if the condition swapped is one a person could actually change in real life. Pointing at something nobody can touch and saying "this would have needed to be different" offers neither comfort nor direction.

The explanation might not be the real reason

Here's the part that needs the most care. Most explanations are computed separately, from the outside, after the judgment is already made. They aren't a transcript of the calculation that actually ran inside the model.

Which means a convincing explanation that isn't true can show up. Ask a conversational AI why it answered the way it did, and it hands back a smooth-sounding reason, one that's also just a plausible string of words strung together. It isn't inspecting its own calculation and reporting back.

Because explanations get built several different ways, the same judgment can come with different explanations depending on which method produced it. It's safer to treat an explanation as a lead worth checking, not as proof.

Building it in from the start

Instead of bolting an explanation on afterward, a model can be built so a person can read the judgment directly. A rule-based structure that branches like a tree, or a simple form that adds and subtracts a fixed weight per item: with either one, the process itself is the itemized sheet.

The trade-off is that these tend to lose some accuracy on harder problems. In practice, teams often run the higher-accuracy option and attach an explanation to it, or route the important calls back through a rule-based structure for a second look.

Whichever route gets picked, the last safeguard is still a person. An explanation only earns its keep if there's a point in the process where a person can read it, spot an odd reason, and stop the judgment before it goes out.

3More precisely

Explainability really splits into two branches: one where the model's own structure is something a person can follow directly, and one where an already-trained model's judgment gets examined from the outside afterward. The second is called a post-hoc explanation, and it covers methods like masking parts of the input and measuring how the result shifts, assigning each item a share of the credit, and building a simple stand-in model that mimics the judgment nearby.

The analogy breaks down in one place. A repair sheet is a record of work the mechanic actually did, but a post-hoc explanation is a reconstruction made from the outside. Ask the same mechanic twice and you get the same answer; switch the explanation method here and the ranking of reasons can shift. And a repair sheet's line items are independent of each other, while the clues a model weighs are tangled together, which makes it hard to cleanly separate out one item's exact share. Explainability, on its own briefly: a model can be a black box that gives no access to its inner workings at all, and explainability is the effort to work around that.

Having an explanation doesn't make the judgment correct, either. An explanation doesn't guarantee the quality of a decision. It only gives you something to go check.

4Try it yourself

CNN Visual Explainer ailearn.space Feed it a photo and it lays out, layer by layer, exactly which patterns lit up

5Common misconceptions

It's easy to think a judgment with an explanation attached can be trusted, but actually a wrong judgment wrapped in a convincing explanation is often the more dangerous case.
It's easy to think asking an AI for its reasons gets you the real reason, but actually that answer is also just a plausible string of words the model generated.
It's easy to think explainability and accuracy move together, but actually the higher a model's performance, the more tangled its insides tend to get, which usually makes it harder to explain.

7One-line summary

In shortExplainability means asking an AI's judgment for an itemized sheet, and that sheet only gives you something to check, not proof that the judgment was right.

Did this analogy help?

Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02