Confidence Score
The number a model attaches to how sure it is about its own answer
- A confidence score is a number the model attaches to its own answer. It always rides along next to whatever the model outputs.
- The number is not a measured probability of being correct. It's a raw calculation squeezed down into a range between 0 and 1.
- The score gets split between the candidates. If there are only three candidates, one of them is guaranteed to get a high score.
- A score still comes out even for something the model never learned about. There's no "I don't know" answer available.
- What the number actually means only shows up when you check how often an "80" was actually right.
Contents
1The analogy
Beside a crosswalk signal, a countdown ticks off in seconds. A big number and you walk without worry; a small one and you run. That single number is what decides whether you cross.
But that number wasn't measured against whether you personally can make it across. It's just showing a timer the signal box already had programmed in. Carrying heavy bags or walking slowly, the number might look generous and you still don't make it.
The score an AI hands back alongside its answer sits in the same spot as that countdown. A label of "cat, 80%" is a number that fell out of a calculation — not a count of eighty photos out of a hundred that really turned out to be cats. Whether you can trust that number is something you have to check separately.
2In detail
Where the score comes from
At the last stage of a classifying model, a value comes out for every candidate. These values have no size limit, so they're hard to read as-is. So they all get turned positive and divided against each other so the total comes to 1. The percentage you see on screen is that result.
One important property shows up here: since the total is 1, someone is always going to walk away with a high score. If the only candidates are cat and dog, whatever photo you feed in, one of the two clears fifty percent — whether the photo is a car or wallpaper makes no difference.
So a high score means "closer to this than to the other candidates," not "this is correct." Swap in a different set of candidates to compare against, and the score for the exact same photo can change a lot.
80 percent might not mean 80 percent
Pull out every judgment the model scored at 90 and count how many were actually right, and the result is surprising. Often it's not ninety out of a hundred but closer to seventy. The model is rating itself higher than its real track record. This habit is called overconfidence.
The reverse happens too. A model trained cautiously can turn out right ninety-five times out of a hundred while only claiming a score around 70. Either way, taking the number at face value throws off your judgment.
Fixing it comes down to counting. This checking-and-adjusting process is what calibration covers — it's worth reading on its own. Only after that adjustment can an 80 be read as roughly "eight times out of ten."
A score still shows up for the never-seen
This is the point that deserves the most caution. Feed a photo of a car into a model that only ever learned cat and dog, and "I don't know" doesn't come out. One of the two gets picked, sometimes with a fairly high score attached. That's because the model isn't asking "what is this," it's asking "which of the candidates I learned is this closest to."
So reading a high score as "certain knowledge" is dangerous. A high score on an unfamiliar input can actually be a sign the model has no idea what it's looking at. Some designs add an extra candidate to catch "none of the above," or bolt on a separate check for whether the input even falls within what the model learned.
Where to draw the line
Real services rarely show the raw score. Instead, a line gets drawn, and the system reacts only once the score clears it. Where that line sits completely changes the character of the system.
Set the line low and fewer things get missed, but false triggers go up. Set it high and false triggers drop, but real ones get filtered out along with them. Which is better depends entirely on the use — set it low where missing something is costly, and high where a wrong alert causes trouble.
Where scores shouldn't be compared
A score from one model isn't measured on the same ruler as a score from another. A 70 from one model can be more trustworthy than a 95 from another. Comparing two models means checking how many they each actually got right on the same test set — not comparing their scores directly.
Even within a single model, the meaning of a score shifts when conditions change. A dark photo, a blurry one, or an angle barely represented in training can all produce a score far higher than the model's real accuracy would justify. A score isn't a fixed report card — it's a reference value tied to that moment.
3More precisely
The final values from a classifying model get passed through an exponential function and divided by their total, producing values that sum to 1. This is read as if it were a probability, but it isn't lined up against real-world accuracy. How far off it runs can be summarized in a single number by measuring the actual accuracy at each score level, and a simple fix — dividing the final values by a fixed factor — is often used to shrink that gap. In object detection, "confidence something is here" and "confidence the label is right" get multiplied together into a single score, which calls for extra care when reading it.
The analogy breaks down in one place. A crosswalk countdown has a clear unit, seconds; a confidence score has no unit at all. A signal shows the same number to everyone standing in front of it, while a confidence score gets freshly generated for every single input. And a countdown genuinely ticks down as time passes, while a confidence score doesn't drop on its own just because the answer turned out wrong. Knowing whether it was wrong is always up to whoever's watching from outside.
4Try it yourself
5Common misconceptions
It's easy to think an 80% score means "right eighty times out of a hundred," but actually, unless it's been calibrated separately, that gap from real accuracy can be sizable.
It's easy to think a low score means the model doesn't know, but actually a completely unfamiliar input often still produces a very high score.
It's easy to think comparing two models' scores tells you which is better, but actually they're not measured on the same ruler, so it has to be judged by accuracy on the same test set instead.
7One-line summary
In shortA confidence score is a number a model attaches to its own answer, and it shouldn't be read as a real probability until you've separately counted how often it's actually right.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02