What is Segmentation? — one analogy

1The analogy

Set a potted plant in front of a bright light and a black silhouette forms on the wall behind it. Look closely at the wall and every single point is sorted, with nothing left over, into "this is the pot" or "this is just wall." Even the gaps between leaves let the wall color through, and the narrow strip under the pot's base falls to the pot's side.

That's very different from just holding up a square board big enough to cover the plant. A board covers the empty wall next to the plant too, but a silhouette covers exactly as much space as the plant actually takes up.

It isn't perfectly clean, though. Where a leaf tip is thin or trembling, the edge blurs and it gets hard to say exactly where the plant ends. And if two pots stand one behind the other, their silhouettes merge into a single blob, so counting them takes some other trick.

2In detail

Every single point gets a label

Segmentation's output isn't a handful of boxes — it's a whole picture. Its width and height match the original photo, and every square holds a single number. That number is the label's ID: person might be 1, sky might be 2, tree might be 3.

Paint this picture in color and you get a photo split cleanly into colored regions. The feature that blurs just the background in a video call, and the one that recolors just the sky in a photo — both run on exactly this label image. Cutting things out along an object's actual shape makes possible what a box never could.

Segmentation sits one rung above image classification, which puts one label on a whole photo, and one rung above object detection, which boxes each object. The more precise it gets, the heavier the computation and the harder the training data becomes to prepare.

Shrink down, then grow back up

There's a catch. As a photo passes through layer after layer, the picture keeps getting smaller. A shrunken picture keeps "what was here" but loses "exactly which point was it." Segmentation has to answer point by point, though, so it has to make it back to the original size.

So the structure shrinks by half at each step going down, then grows by double at each step coming back up. Growing back isn't just blowing the picture up bigger — at every step it merges back in the finer picture that was set aside on the way down, at the matching size. It's filling in blurred position with the sharp outlines the earlier layers were holding. This is exactly what keeps edges from turning blunt.

One blob, or split apart individually

Even within segmentation there are two branches. One only tells kinds apart. Three bicycles in a photo all get painted the same single color, "bicycle." If their wheels overlap, there's no way to say where the first bicycle ends.

The other splits every individual apart. The first bicycle and the second bicycle each get their own separate label. This is what you need when counting objects in a photo or erasing just one of them. And there's a hybrid, too: painting uncountable things like sky or road by kind only, while splitting countable objects apart individually.

The edge is always where it goes wrong

Segmentation trips up most often not inside an object, but right at its edge. A strand of hair, the gap between leaves, a translucent glass, a hand that moved and blurred — anywhere a single point is genuinely ambiguous between object and background, the result gets messy. That's why turning on background blur in a video call leaves a jagged, uneven line along someone's shoulder.

The people building training data run into the same trouble in the same places. A box takes two drags to draw and you're done, but an outline can take minutes per photo. That's why preparing one photo costs tens of times more than it does for classification. Newer tools cut this labor drastically — click a single point on a photo and the rest of the outline gets traced automatically.

3More precisely

Segmentation is the problem of classifying every point on a photo into one of a predetermined set of labels. The output is a label map with the same width and height as the original, and performance is measured by how much a predicted region overlaps the correct region for each label, then averaged. In the version that splits individual objects apart, each individual gets compared separately.

The silhouette analogy breaks down in one place. A silhouette forms because light gets blocked, so it naturally comes out as one connected blob, but segmentation judges every point on its own. That's why a single stray point in the middle of an object can end up flipped to a different label — the reason results sometimes need a cleanup pass afterward to smooth out speckled noise.

And a silhouette only ever comes in one color, black, but segmentation splits things into many labels. A spot where two things overlap and show through each other — translucent glass, or a net — makes picking just one label ambiguous. That's exactly why a separate approach exists that answers with a ratio of how much background shows through, instead of a single label.

4Try it yourself

5Common misconceptions

It's easy to think segmentation always beats object detection, but actually it costs far more in computation and data preparation, so a box is plenty when you only need position.
It's easy to assume the result draws an object's outline as a line, but actually it's a label on every point, not a line, so edges aren't smooth and can come out speckled.
It's easy to think painting the same kind also tells you the count, but actually a kind-only approach has no choice but to see touching objects as one blob.

7One-line summary

In shortSegmentation puts a label on every single point of a photo to trace an object's exact shape, and while it's more precise than a box, it costs far more in both computation and data preparation.

Did this analogy help?

Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02