What is Object Tracking? — one analogy

1The analogy

Check a bag at the airport and it gets a tag with a number. That number points to the same bag the entire time it rides the belt through however many stages it passes. It's the number, not the bag's appearance, that keeps it the same bag.

Object tracking is that tag. Finding something on screen isn't enough on its own — knowing that what got found just now is the same thing found a moment ago is what makes it possible to count how many things went past.

Passing through a covered stretch and disappearing briefly doesn't kill the number. The system guesses which side it'll come out and roughly when, and waits. The trouble comes when two similar black bags pass close together, overlapping for a stretch — once they separate, the numbers sometimes end up swapped.

2In detail

Finding and linking are different jobs

Finding an object in a single frame ends there, inside that frame. Drawing three boxes says nothing anywhere about which of those was which box in the previous frame. Each frame gets searched fresh, independently, every time.

That's why a screen with only finding, no linking, looks strange. Boxes flicker into existence every instant, and a brief miss makes something vanish entirely and pop back later. Try to count how many people walked past and the same person gets counted thirty times over.

Tracking adds one more layer on top. It matches what got found against the previous frame's list, handing the same number to whatever matches and a fresh number to whatever doesn't. A number that's gone off screen gets held for a while before it's finally dropped.

What decides a match

Two kinds of clues decide a match. The first is location. In a very short span of time, an object can't have traveled far, so whatever sits close to or overlaps the previous frame's location is likely the same thing.

The second is appearance. Traits like color and pattern get reduced to a short bundle of numbers and compared against whatever got newly found. When location alone can't settle it, this clue becomes decisive.

Score both clues together and it turns into a matching problem — the previous list and the new list get matched all at once so the total score comes out best, rather than greedily pairing off whatever's nearest first, which tends to leave a tangled mess for whatever's left over.

Guessing a location while it's out of sight

Occlusion happens constantly. Passing behind another person or getting blocked by a pole makes finding fail for that instant. Drop the number the moment that happens and reappearing gets a fresh number instead — one person counted as two.

So a number that briefly vanishes gets kept alive for a while. Its direction and speed just before disappearing get used to keep guessing "it's probably around here now," shifting that guess forward each frame. When something similar shows up nearby, the original number gets reattached.

Keep a number alive longer and objects hidden for a long stretch stay linked, but a number that's already truly gone sometimes ends up attached to the wrong thing. Keep it alive for a shorter time and the opposite happens. How long to hold on gets tuned to how crowded the scene tends to be.

The swap accident

The most common failure in tracking isn't missing something — it's the numbers swapping. The moment two similar-looking things brush past each other, neither location nor appearance can tell them apart while they overlap. If the two numbers get reattached to the wrong one once they separate, everything downstream stays wrong from then on.

A swapped number doesn't correct itself. That's why this single failure can wreck an entire path drawing or a dwell-time count. It's also why tracking performance gets measured not just by how well things get found, but by counting how many times numbers swapped, as a separate figure.

Once numbers are attached, things can be counted

The moment a number stays stable, a lot becomes possible. Counting how many people entered a store is really about managing things so a number only ever gets created once per person. How long someone lingered in front of a display case is the total time one number spent inside that zone.

Drawing a path relies on the same number too. Connect one number's locations in time order and a single line appears. That line reveals which direction gets more traffic and where people tend to stop. It also feeds action recognition, gathering one consistent thing's flow together.

3More precisely

Tracking several objects at once is usually split into two steps. First, each number's next location gets predicted from its recent motion; then that prediction gets matched against whatever got found in the current frame. Matching turns into a cost problem, scored by overlap area and appearance similarity, solved by finding the combination with the lowest total cost. Performance gets measured by counting misses and false detections, plus how many times numbers swapped. Tracking a single object picked out in the first frame, and following only that one to the end, is treated as a separate problem with its own set of methods.

The analogy breaks down somewhere too. An airport tag is physically stuck to the bag and never changes unless it's torn off, but a tracking number is a fresh guess reattached every instant, so it can shift at any time. A bag also only travels a fixed belt, while something on screen can move in any direction, stop suddenly, or turn sharply. The moment the guessed location and the real one drift apart is exactly the moment a number starts to wobble.

4Try it yourself

streetscape.gl (self-driving data visualization) ailearn.space Watch every passing car and person carry a number that follows it, seen from above

5Common misconceptions

It's easy to think finding objects well means tracking works well too, but actually most of the trouble shows up at the step of matching what got found against the previous frame.
It's easy to think occlusion breaks tracking, but actually the system guesses a location while something's out of sight and reattaches the same number once it reappears.
It's easy to think a brief number swap doesn't matter, but actually a swap, once it happens, stays that way — throwing off the count and time statistics entirely.

7One-line summary

In shortObject tracking links what gets found each frame back to the previous frame to keep the same number attached, and that number is what makes counting, timing, and paths measurable.

Did this analogy help?

Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02