Key points
- Model serving is the operating job of loading a finished model onto a computer and answering every request that comes in.
- Having the model file doesn't mean it's ready to use. Loading it and getting it warmed up takes its own separate time.
- When requests pile up, a line forms. More seats get opened as demand grows, and closed again once it quiets down.
- Batching several requests together raises total throughput, but the bigger the batch gets, the slower each person's own answer becomes, little by little.
- Switching to a new version usually doesn't happen all at once. A slice of traffic rides it first, and it only widens once nothing's broken.
The analogy
A new bus rolling into the depot doesn't mean anyone can board. It has to be added to a route, given a timetable, and started up before the first run — only then does anyone get on. Building the bus and carrying passengers are two completely different jobs.
Model serving is that dispatching. A model that finished training is a bus parked in the depot — it has to be loaded onto a computer and warmed up before it can take requests. Just like adding buses during rush hour, seats get opened when people pile in and closed again when it's quiet.
Keep a spare on standby so one breakdown doesn't stop the route, and slip new buses in one at a time when swapping the fleet.
In detail
Parked in the depot, nobody can board
Building a model and putting it to work are different enough jobs that different teams often handle them. Training is close to a one-time major construction project; serving is the daily operation of sending a bus out at the same time every day. A finished training run producing a file just means the bus is built.
To take requests, the model has to be loaded whole into the memory of a computing device. That loading takes real time, so if you loaded and unloaded it every time a request came in, answers would arrive badly late. Most setups keep it loaded and waiting instead. The unusually slow first request after a quiet stretch is the engine warming back up.
Keeping it loaded costs seat money even while nobody's using it. How long to keep it running, whether to turn it off overnight — those decisions turn straight into cost.
Add buses when demand piles up
There's a limit to how many requests one seat can handle at once. Cross that limit and requests behind the line wait for the one ahead to finish computing. What a user feels as "unusually slow today" isn't the model suddenly getting worse; sometimes it's a longer line and more work being handled at once.
So serving is built with a mechanism that opens more seats as demand grows and closes them again once things quiet down. But opening a new seat takes its own warm-up time, so when a surge is sudden, growth can't keep pace with how fast the line is growing. That's why a few seats get kept warmed up in advance.
One bus, many passengers
Processing several requests together in one batch is far more efficient than handling them one by one. The computing hardware is built to push a lot of work through at once, so sending a bus out with only one passenger leaves seats empty.
The catch: it used to mean waiting for a batch to fill, but modern conversational language-model serving lets a new request hop on the moment a seat frees up, even mid-trip. Still, the more people riding at once, the heavier each step gets, so one person's own answer feels slower. Handling a lot overall and answering one person fast pull against each other, and which one gets the weight depends on what the service is for. A live back-and-forth conversation leans toward smaller batches; a job that combs through documents overnight leans toward bigger batches.
New versions ride in one at a time
Swap every seat over to a new version all at once, and if something's wrong, every user feels it at the same time. So a few seats get the new version first, carry a slice of the requests, and only widen once the answer quality and speed hold up. If something looks off, just that slice rolls back to the old version.
The old version often stays running alongside for a while too. If the tone or shape of the answers shifts, whatever program is consuming those answers gets shaky along with it, so that overlap buys time to migrate.
Where to put the depot
The most common setup is a service company running the depot while whoever's using it just sends requests. Adding and removing seats, swapping a broken engine at 3 a.m. — all of that gets handed off, and you pay for what you use.
Places that can't send material outside, or that use a steady amount all the time, sometimes run their own depot instead. These days a small model running right on a user's device or inside a browser is common too. Then a depot exists for every single person, so the worry about waiting in line disappears — but the first download and the device's own power become the new wall.
More precisely
When serving conversational language models especially, the bottleneck hit most often isn't computing speed — it's memory. A model has to sit loaded whole in memory, and the longer a conversation runs, the bigger the space holding onto that conversation grows too. How many requests one seat can take at once is usually decided by this same memory limit. That's why shrinking a model before loading it, slicing up the space that holds each conversation so none of it sits idle, and letting requests with the same opening share the work already computed for it, get used alongside everything else.
The analogy breaks down in a spot. A bus has a visibly fixed number of seats, but a serving seat's capacity shifts with how long each request is. A bus only runs its assigned route, while the same model takes every kind of request thrown at it. Most of all, a bus's job ends when the passenger gets off, but serving keeps watching speed and errors and adjusting seats long after an answer goes out. The better a serving setup runs, the more it looks to the person using it like nothing is happening at all.
Try it yourself
Common misconceptions
It's easy to think it's ready to use the moment training finishes, but actually a separate operation has to load it and keep watch before people can actually use it.
It's easy to think a slow answer is purely the model's fault, but actually, beyond model size and answer length, a line piled up in front or a seat that isn't warmed up yet slows things down quite often.
It's easy to think serving is lighter work than training, but actually running loaded around the clock can add up to more effort over the long run.
One-line summary
Model serving is the dispatching job that keeps a finished model running on its route, where what matters most isn't how good the model is but how steadily it keeps answering on time.
Spotted an error or have a better analogy? Suggest an edit · Last updated 2026-10-18