Real-Time Inference

The time budget gets set first

"Real time" carries no fixed number of its own. The use case sets that number. Overlaying a display on webcam video, where about thirty frames arrive per second, gives one frame roughly that gap as its whole budget — reading, judging, and drawing all have to fit inside it.

A voice assistant gets a much looser bar, since a person will wait a moment for an answer. Inspecting fast-moving objects on a line, on the other hand, runs far tighter. The same model can count as real time in one setting and fail to qualify in another, purely because of where it's used.

The budget isn't just compute time, either. The time for a frame to arrive from the camera, the time to resize it, and the time to draw the result on screen all draw from that same pool. A fast model alone doesn't guarantee the whole pipeline stays fast.

One late frame snowballs

Even a small overrun on one frame's budget turns into a bigger problem fast. While that frame is still processing, the next one arrives and queues up, then another arrives after that. If processing speed can't keep pace with the incoming rate, the queue keeps growing, and what shows on screen keeps drifting further behind the actual moment.

Within a few seconds, waving a hand produces a display that catches up only well after the fact — a genuinely strange state. That's why programs built for real time never let a queue build up. The instant it starts falling behind, it simply drops the frames in between. Processing ten out of thirty frames and showing the current one is far better than showing an old one.

The display might look a little choppy from the skipped frames, but that beats a late result. In real-time processing, dropping frames isn't a failure — it's the system working as intended.

Ways to buy back time

The first place to cut is input size. Shrink the image to half its size and the number of points to check drops sharply. If a face is meant to fill most of the frame anyway, a smaller image is plenty.

Next comes swapping in a lighter model. Several sizes usually exist for the same task, so a slightly less accurate but much faster option is often available. Cutting the number of digits used to represent each value is another common way to lighten the computation.

Splitting the work also helps. Run the heavy judgment only once every few frames, and let the frames in between carry the previous result forward and track from there. Skipping a full search from scratch on every single frame frees up a lot of budget.

Watch the worst case, not the average

"Average processing time" isn't a number worth trusting for real time. Process twenty frames quickly and then suddenly stall on one, and the screen visibly stutters right then — and that's the one moment a person remembers. It's common to land on a state where the average looks fine but using it still feels uncomfortable.

So the slowest handful of percent gets watched separately. These spikes tend to show up when another program briefly interrupts, when memory gets freshly allocated, or when a device heats up and throttles itself. Reserving memory up front and processing the same fixed size every time cuts down on these spikes.

Where the computing happens matters a lot

Handing computation off to a distant server means access to much stronger hardware, but the round trip of sending the image and getting the result back adds itself entirely to the budget. A brief hiccup in the connection can cause a sudden, large delay right at that moment.

That's why anything that has to react to the screen instantly often gets processed right on the device instead. Running inside a browser or on a phone means using a lighter model in exchange. Some setups mix the two, handing off only the occasional heavy judgment to a server while keeping the rest on the device.