VRAM
The workspace where a graphics card lays out its numbers to compute
- VRAM (Video RAM) is the workbench where a graphics card lays out the numbers it needs for a computation.
- If everything doesn't fit on the bench, it doesn't just run slower — it can't start at all. It throws an error and stops.
- Three things need a spot on the bench: the model's own numbers, the current input plus conversation history, and the intermediate values from computing.
- Storing numbers in a coarser format shrinks the numbers, and a bigger model fits on the same bench.
- It's a different wall from raw computing power. One is a space problem; the other is a workload problem.
Contents
1The analogy
Picture a small kitchen counter. Even for one order, you need the ingredients — the meat, the vegetables, the sauces — within arm's reach before you can start cooking. If the counter's too small, it doesn't matter how fast your hands are. Without room to lay everything out, you can't even begin.
The VRAM attached to a graphics card is that counter. Its size is fixed, and only what's laid out on it can be used in the computation. Get a rush of orders that need two pots going at once and you need that much more counter space; run out of room and things stop the instant a new order comes in.
It doesn't matter how much food is sitting in the walk-in fridge next door. Only what's already laid out on the counter is within reach.
2In detail
What actually sits on the workbench
The biggest chunk is the model's own numbers. These values, fixed by training, are needed throughout the entire computation, so they occupy space from start to finish. The moment you pick a model, more than half the workbench is already claimed.
Next comes the current input and however much conversation has happened so far. To avoid recomputing what it already read, a model jots down intermediate results like sticky notes, and the longer the conversation runs, the more these notes pile up. Last are the intermediate values created while passing through the layers — these briefly borrow space and then free it again.
Run short and it doesn't slow down — it stops
This is where VRAM differs most from other components. A slower card just takes longer to produce an answer, but it does produce one eventually. Run short on space, though, and the computation can't even start — it throws an error and quits right there.
That's the very first wall most people hit trying to run a model on their own machine. Whether the computation is fast or slow only becomes a question once it's actually loaded — first comes whether it loads at all. When there's barely enough room, a short question might go through fine while a long document stops it cold, because the piling-up conversation history is what finally fills the last of the bench.
Ways to save space
The most common trick is storing numbers in a coarser format. Shorten a number that used to carry many decimal digits and the same count of numbers takes up half, or a quarter, of the space — not unlike chopping ingredients smaller to save room. The tradeoff is that very fine distinctions get blurred, which can roughen the texture of the results a little.
Shrinking the batch processed at once is another option — running one pot instead of two. That saves space but takes longer to get through everything. Beyond that, people also pick a smaller model to begin with, split a model across several cards, or temporarily offload parts that aren't in use right now to main system memory.
Training needs a much bigger bench
Just producing answers only needs the model's own numbers plus conversation history. Training is a different story — it also needs a signal recording how much to adjust each number and in which direction, plus values that remember the flow along the way. To work back down through the network, the intermediate values from the trip up have to be kept around instead of thrown away.
That's why training the same model can need several times more space than just running it for answers. It's exactly why running a large model on your own machine might be possible while training that same model yourself is out of reach. It's also why training only a small attached component instead of the whole model has become so widespread — that approach takes up far less of the workbench.
3More precisely
The space needed is roughly estimated by multiplying the parameter count by the size one number occupies. Store a number in 16 bits and that's 2 bytes per parameter; cut it to 4 bits and it's half a byte. On top of that, add the storage that grows with conversation length, plus some margin. This storage scales with both the number of layers and the length of the conversation, so it becomes a chunk that can't be ignored in long conversations.
The analogy breaks down in one place. Ingredients on a counter can be set aside for a moment and pulled back out later, but pulling numbers that got moved off VRAM back from main memory is far slower than computing itself. Shuffle numbers on and off to save space too often and speed can drop by several times over.
Free space that looks sufficient on paper can also fail to fit a large chunk if it's scattered in pieces rather than sitting together — the free space reported and the free space actually usable aren't always the same thing.
4Try it yourself
5Common misconceptions
It's easy to think having plenty of regular computer memory is enough, but actually the memory attached to a graphics card is separate, and no amount of main memory can substitute for it.
It's easy to think you only need as much space as the model file's size, but actually conversation history and intermediate values need room too, so it's worth leaving generous margin.
It's easy to think running short just slows things down a bit, but actually in most cases it throws an error and stops right there.
7One-line summary
In shortVRAM is the workbench where a graphics card lays out the numbers it computes with, and if everything doesn't fit, the model doesn't slow down — it simply doesn't run at all.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02