Key points
- Compute is the total number of multiplications and additions it takes to train a model or run it once.
- It isn't time — it's the amount of work that has to be done. Throw more hardware at the same work and the time shrinks, but the work itself doesn't.
- Training sweeps over the data over and over, so it takes an incomparably larger amount of compute than getting one answer does.
- It grows as the model gets bigger and the input gets longer, and it grows steeply once the input gets very long.
- Compute translates directly into electricity bills and rented hardware. It's a different wall from running out of memory.
The analogy
Picture a day at a package-delivery warehouse. Ten thousand boxes arrive today, and each one takes five separate touches — scan, sort, load — before it goes out the door. That means fifty thousand touches of work need to happen today. That fifty thousand is compute.
That number doesn't change whether the warehouse is huge or cramped, whether it's staffed by many people or few. Double the conveyor belts, and the same fifty thousand touches just finish in half a day. Only when twice as many boxes arrive, or an extra touch gets added to the process, does the total amount of work actually grow.
So the warehouse manager tracks two separate numbers: how many touches need to happen today, and how many hours it'll take to get through them. The first number is compute.
In detail
Work gets counted by multiplications
Most of what happens inside a model is multiplying numbers together and adding them up. Every time a signal passes through a layer, the incoming bundle of numbers gets paired up with that layer's own numbers and multiplied and summed, millions of times over. Add up every layer's share, and you get the total amount of work behind a single answer.
That's why compute isn't measured in seconds or minutes. It's measured by the size of the work itself, the way you'd count boxes. Time is just that size divided by how fast the hardware runs. The same model might take one second on one machine and thirty seconds on another, and the compute is identical either way.
Building it and using it are different orders of magnitude
Using a model is a lot like pulling one box off a shelf and shipping it. One question, one answer, done in a few seconds. Building a model is like reorganizing the entire warehouse over and over. It means producing an answer, measuring how wrong it was, and working backward to fix it, all repeated across the entire dataset countless times.
The fix-it pass takes roughly twice the work of the answer-producing pass. Multiply that by the full amount of data being swept through, and the gap balloons to hundreds of millions of times over, or more. That's where stories about thousands of computers spending weeks training a single large model come from.
What drives compute up
The first factor is model size. Double the number of trainable numbers, and the multiplication count roughly doubles too. The second is data volume. Sweep through twice as much text, and training takes twice the work.
The third is input length, and this one scales differently. Every piece in a sentence looks at every other piece, so doubling the length nearly quadruples that part of the work. For a very long document, that part takes up a large share of the total work. So even for the same amount of text, feeding in one very long document can cost more than splitting it into several short questions. Getting a longer answer back adds up the same way, since it's one more round of calculation for every extra piece.
Getting the same job done more cheaply
There are two directions: shrink the total amount of work, or speed up the processing. To shrink the work, use a smaller model or distill a large model's knowledge into a smaller one. If the same question comes up repeatedly, caching an earlier result and reusing it works too.
There are two ways to speed things up. One is to use coarser numbers so each multiplication gets lighter and less has to be pulled from memory (the number of multiplications stays the same). The other is to split the work across more hardware. But that split doesn't go on forever. A later layer's calculation needs an earlier layer's result first, so no amount of extra hands can skip that order. Handing pieces of work back and forth between machines also takes time, so past a certain point, adding more hardware stops making much difference.
More precisely
Compute is usually measured in floating point operations, or FLOPs. The total compute to train a large language model is roughly proportional to the number of parameters multiplied by the number of tokens seen during training. Producing an answer costs roughly twice the parameter count for every piece (token) it generates, so a longer answer adds up that much more.
The analogy breaks down in one place. Delivery boxes can be processed independently of each other, but a neural network's calculations depend on each other in sequence, since one result feeds directly into the next. So doubling the hardware doesn't cut the time exactly in half.
There's another gap worth noting. Often the real reason an answer arrives late has nothing to do with the multiplication count. Pulling numbers out of memory frequently takes longer than the calculation itself, leaving the processing hardware sitting idle waiting for numbers to arrive. Compute is a good yardstick for cost, but it doesn't directly predict speed.
Try it yourself
- WebLLM Chat (a chat AI running in your browser) ailearn.space Watch the wait grow as the answer gets longer and you can feel one more round of calculation running for every piece. Speed also reflects your hardware, though
- TensorFlow Visualization ailearn.space Watching thousands of numbers move through a layer at once shows exactly where the work piles up
Common misconceptions
It's easy to think compute is the same thing as how long something takes, but actually the same amount of compute can take wildly different amounts of time depending on the hardware and how many machines it's split across.
It's easy to assume a good graphics card means a big model will run, but actually if there isn't enough room to hold the numbers, it either won't run at all or has to push part of them into other memory and run very slowly, no matter how fast the calculation is.
It's easy to think getting an answer back is cheap enough not to worry about, but actually one answer is cheap, yet with many people using it over a long time they can add up to more than training ever cost.
One-line summary
Compute is the total amount of calculation it takes to run a model, and it's a yardstick for the bill and a major factor in the wait.
Spotted an error or have a better analogy? Suggest an edit · Last updated 2026-10-17