Model Serving
Putting a finished model to work answering everyone's requests
- Model serving is the operating job of loading a finished model onto a computer and answering every request that comes in.
- Having the model file doesn't mean it's ready to use. Loading it and getting it warmed up takes its own separate time.
- When requests pile up, a line forms. More seats get opened as demand grows, and closed again once it quiets down.
- Batching several requests together raises total throughput, but each person in the batch waits that much longer for their own answer.
- Switching to a new version never happens all at once. A slice of traffic rides it first, and it only widens once nothing's broken.
Contents
1The analogy
A new bus rolling into the depot doesn't mean anyone can board. It has to be added to a route, given a timetable, and started up before the first run — only then does anyone get on. Building the bus and carrying passengers are two completely different jobs.
Model serving is that dispatching. A model that finished training is a bus parked in the depot — it has to be loaded onto a computer and warmed up before it can take requests. Just like adding buses during rush hour, seats get opened when people pile in and closed again when it's quiet.
Keep a spare on standby so one breakdown doesn't stop the route, and slip new buses in one at a time when swapping the fleet.
2In detail
Parked in the depot, nobody can board
Building a model and putting it to work are different enough jobs that different teams often handle them. Training is close to a one-time major construction project; serving is the daily operation of sending a bus out at the same time every day. A finished training run producing a file just means the bus is built.
To take requests, the model has to be loaded whole into the memory of a computing device. That loading takes real time, so if you loaded and unloaded it every time a request came in, answers would arrive badly late. Most setups keep it loaded and waiting instead. The unusually slow first request after a quiet stretch is the engine warming back up.
Keeping it loaded costs seat money even while nobody's using it. How long to keep it running, whether to turn it off overnight — those decisions turn straight into cost.
Add buses when demand piles up
There's a limit to how many requests one seat can handle at once. Cross that limit and requests behind the line wait for the one ahead to finish computing. What a user feels as "unusually slow today" is, more often than not, a longer line rather than slower computing.
So serving is built with a mechanism that opens more seats as demand grows and closes them again once things quiet down. But opening a new seat takes its own warm-up time, so when a surge is sudden, growth can't keep pace with how fast the line is growing. That's why a few seats get kept warmed up in advance.
One bus, many passengers
Processing several requests together in one batch is far more efficient than handling them one by one. The computing hardware is built to push a lot of work through at once, so sending a bus out with only one passenger leaves seats empty.
The catch is that batching takes a moment of waiting. The longer that wait, the more delayed one person's own answer feels. Handling a lot overall and answering one person fast pull against each other, and which one gets the weight depends on what the service is for. A live back-and-forth conversation leans toward shorter waits; a job that combs through documents overnight leans toward bigger batches.
New versions ride in one at a time
Swap every seat over to a new version all at once, and there's no room to back out if something's wrong. So a few seats get the new version first, carry a slice of the requests, and only widen once the answer quality and speed hold up. If something looks off, just that slice rolls back to the old version.
The old version often stays running alongside for a while too. If the tone or shape of the answers shifts, whatever program is consuming those answers gets shaky along with it, so that overlap buys time to migrate.
Where to put the depot
The most common setup is a service company running the depot while whoever's using it just sends requests. Adding and removing seats, swapping a broken engine at 3 a.m. — all of that gets handed off, and you pay for what you use.
Places that can't send material outside, or that use a steady amount all the time, sometimes run their own depot instead. These days a small model running right on a user's device or inside a browser is common too. Then a depot exists for every single person, so the worry about waiting in line disappears — but the first download and the device's own power become the new wall.
3More precisely
The bottleneck serving hits most often isn't computing speed — it's memory. A model has to sit loaded whole in memory, and the longer a conversation runs, the bigger the space holding onto that conversation grows too. How many requests one seat can take at once is usually decided by this same memory limit. That's why shrinking a model before loading it, or keeping frequent answers ready to reuse, gets used alongside everything else.
The analogy breaks down in a spot. A bus has a visibly fixed number of seats, but a serving seat's capacity shifts with how long each request is. A bus only runs its assigned route, while the same model takes every kind of request thrown at it. Most of all, a bus's job ends when the passenger gets off, but serving keeps watching speed and errors and adjusting seats long after an answer goes out. There's also no fixed timetable here — seats open and close by the minute, chasing demand that a bus schedule would never bother tracking so closely. The better a serving setup runs, the more it looks to the person using it like nothing is happening at all.
4Try it yourself
5Common misconceptions
It's easy to think it's ready to use the moment training finishes, but actually a separate operation has to load it and keep watch before people can actually use it.
It's easy to think a slow answer means the model itself is slow, but actually it's more often a line piled up in front, or a seat that isn't warmed up yet.
It's easy to think serving is lighter work than training, but actually running loaded around the clock can add up to more effort over the long run.
7One-line summary
In shortModel serving is the dispatching job that keeps a finished model running on its route, where what matters most isn't how good the model is but how steadily it keeps answering on time.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02