Action Recognition

A single frame can't tell you

This is where photo classification and action recognition part ways. A single photo can already identify an object — "that's a footbag." But "kicking" isn't an object; it's a change unfolding over time, so it never fits inside one frame.

Sitting down and standing up make this obvious. A frame sliced from the middle of either action looks identical for both. The angle of a bent knee alone can never tell you which one it is — only whether the frame before it was standing or sitting settles the question.

So action recognition always takes in several frames as one bundle. Typically one to three seconds, tens of frames at a time, all pushed through together.

How the frames get bundled

There are broadly two approaches. One takes the video as is and treats time as a third axis alongside width and height, searching for patterns across all three. It can use everything visible on screen, including clothing color and object shape, which makes it detailed but computationally heavy.

The other first extracts joint locations, turning the video into a record of point positions, then judges from just that flow of numbers. Clothing and background disappear, so it's far lighter and holds steady even when lighting or clothing color changes. The trade-off is that it throws away information like what's in someone's hand.

Some tools combine both — reading the overall picture for the broad shape of the action, then checking the joint flow for detail.

One repetition has to be cut out

Naming an action in a pre-trimmed three-second clip is fairly easy. The genuinely hard part is taking a video that runs several minutes and marking "this stretch, right here, is one repetition."

Where a repetition starts and ends is something even people don't agree on. Whether the preparatory foot-raise before a kick counts as part of the repetition is already a point where opinions split. When different people labeling the training data draw that boundary differently, the model learns something blurry too.

That's why counting repetitions goes wrong more often than expected — one kick counted as two, or two quick kicks in a row merged into one.

Background as a shortcut

Train a model to recognize swimming using only footage shot at a pool, and it can learn to answer "swimming" just from the color of the water. Someone standing still at the water's edge gets called swimming too. Test scores look great, but the model is useless in practice.

This shortcut shows up whenever the training data is skewed one way. Filming the same action in several locations, or stripping the background out and keeping only the joint flow, cuts it down. Checking what the model is actually keying on to reach its answer matters for exactly this reason.

The same action looks different on everyone

A kick looks different from person to person, and different for the same person on a different day. Foot height and rhythm vary, and a left-footed and right-footed kick mirror each other left to right. The direction of motion on screen also changes completely depending on whether the camera filmed from the side or from the front.

So one action gets taught using many people, many angles, and many speeds. Copies flipped left-right or sped up slightly often get mixed in too. That breadth is what lets the same name get attached to a first-time viewer's movement.