Key points
- VRAM (Video RAM) is the workbench where a graphics card lays out the numbers it needs for a computation.
- If everything doesn't fit on the bench, it can't run as is. It either throws an error and stops, or spills the overflow into main memory and runs several times slower.
- Three things need a spot on the bench: the model's own numbers, the current input plus conversation history, and the intermediate values from computing.
- Storing numbers in a coarser format shrinks the numbers, and a bigger model fits on the same bench.
- It's a different wall from raw computing power. One is a space problem; the other is a workload problem.
The analogy
Picture a small kitchen counter. Even for one order, you need the ingredients — the meat, the vegetables, the sauces — within arm's reach before you can start cooking. If the counter's too small, it doesn't matter how fast your hands are. Without room to lay everything out, you can't even begin.
The VRAM attached to a graphics card is that counter. Its size is fixed, and only what's laid out on it can be used in the computation. Get a rush of orders that need two pots going at once and you need that much more counter space; run out of room and the new order can't be taken as is — things stop.
However much food is sitting in the walk-in fridge next door, it can't stand in for the counter. Only what's already laid out on the counter is within immediate reach.
In detail
What actually sits on the workbench
The biggest chunk is the model's own numbers. These values, fixed by training, are needed throughout the entire computation, so they occupy space from start to finish. The moment you pick a model, a big share of the workbench is already spoken for.
Next comes the current input and however much conversation has happened so far. To avoid recomputing what it already read, a model jots down intermediate results like sticky notes, and the longer the conversation runs, the more these notes pile up. Last are the intermediate values created while passing through the layers — these briefly borrow space and then free it again.
Run short and it either stops or slows down a lot
This is where VRAM differs most from other components. A slower card just takes longer to produce an answer, but it does produce one eventually. Run short on space, though, and the computation can't start as is — it may throw an error and quit, or spill the overflow into main system memory and slow down several times over.
That's the very first wall most people hit trying to run a model on their own machine. Whether the computation is fast or slow only becomes a question once it's actually loaded — first comes whether it loads at all. When there's barely enough room, a short question might go through fine while a long document stops it cold, because the piling-up conversation history is what finally fills the last of the bench.
Ways to save space
The most common trick is storing numbers in a coarser format. Shorten a number that used to carry many decimal digits and the same count of numbers takes up half, or a quarter, of the space — not unlike chopping ingredients smaller to save room. The tradeoff is that very fine distinctions get blurred, which can roughen the texture of the results a little.
Shrinking the batch processed at once is another option — running one pot instead of two. That saves space but takes longer to get through everything. Beyond that, people also pick a smaller model to begin with, split a model across several cards, or temporarily offload parts that aren't in use right now to main system memory.
Training needs a much bigger bench
Just producing answers only needs the model's own numbers plus conversation history. Training is a different story — it also needs a signal recording how much to adjust each number and in which direction, plus values that remember the flow along the way. To work back down through the network, the intermediate values from the trip up have to be kept around instead of thrown away.
That's why training the same model can need several times more space than just running it for answers. It's exactly why running a large model on your own machine might be possible while training that same model yourself is out of reach. It's also why training only a small attached component instead of the whole model has become so widespread — that approach takes up far less of the workbench.
More precisely
The space needed is roughly estimated by multiplying the parameter count by the size one number occupies. Store a number in 16 bits and that's 2 bytes per parameter; cut it to 4 bits and it's half a byte. On top of that, add the storage that grows with conversation length, plus some margin. This storage scales with both the number of layers and the length of the conversation, so it becomes a chunk that can't be ignored in long conversations.
The analogy breaks down in one place. Ingredients on a counter can be set aside for a moment and pulled back out later, but pulling numbers that got moved off VRAM back from main memory is far slower than computing itself. Shuffle numbers on and off to save space too often and speed can drop by several times over.
Free space that looks sufficient on paper can also fail to fit a large chunk if it's scattered in pieces rather than sitting together — the free space reported and the free space actually usable aren't always the same thing.
Try it yourself
Common misconceptions
It's easy to think having plenty of regular computer memory is enough, but actually the memory attached to a graphics card is separate (setups where the graphics chip shares main memory, called unified memory, are the exception), so borrowing main memory slows things down sharply, and no amount of it truly takes the place of graphics memory.
It's easy to think you only need as much space as the model file's size, but actually conversation history and intermediate values need room too, so it's worth leaving generous margin.
It's easy to think running short just slows things down a bit, but actually it either throws an error and stops, or spills into main memory and runs several times slower.
One-line summary
VRAM is the workbench where a graphics card lays out the numbers it computes with, and if everything doesn't fit, the model can't run as is — it stops or slows down a lot.
Spotted an error or have a better analogy? Suggest an edit · Last updated 2026-10-17