Key points
- GPU (Graphics Processing Unit) is a component that handles an enormous number of identical calculations all at once.
- It's not fast because any single job finishes fast. It's fast because it pushes many jobs through at the same time.
- It started out as a part built for drawing the screen. Painting millions of screen pixels at once turned out to have the same shape as AI calculation, so, as it was refined to handle general-purpose calculation too, it came to be used for AI.
- Some jobs fit and some don't. Work where each step has to wait on the last one's result barely speeds up at all, even moved onto this hardware.
- Compute is the amount of work to get through, VRAM is the room to lay the numbers out, and a GPU is the device that processes that work all at once.
The analogy
Picture a highway toll plaza on a holiday weekend. With a single lane, no matter how good a car is, it still has to wait for the car ahead to clear before it can pass. To shrink the line, adding lanes beats making one car faster.
A GPU is a toll plaza with its lanes multiplied into the thousands. Any one lane, on its own, is actually pretty relaxed. But if every car needs to follow the exact same steps, it can wave thousands of them through at once.
There's a catch. Every lane has to be running the same procedure. If one car is grabbing a ticket while another is arguing at the booth, adding more lanes won't shrink the line at all. That's exactly why AI calculation fits this hardware so well: the work is millions of multiplications and additions that all follow the identical shape.
In detail
Not fast — just a lot at once
A computer's central processing unit is close to a handful of extremely capable workers. Each one is very fast and handles complicated judgment calls without trouble. What it doesn't have is headcount.
A GPU is the opposite. It packs in thousands of small workers who can only do simple calculations. Line them up one against one and the CPU comes out ahead. But throw the exact same calculation at it thousands of times at once, and the result flips.
So the question worth asking when picking hardware isn't "which one is faster" — it's "can this job be split up and done all at once." Editing one line of a document doesn't need thousands of lanes.
Why AI calculation fits this hardware so well
Look inside what a neural network actually does and most of it is multiplying one table of numbers against another. Each cell of the table needs a multiply-and-add, and none of those calculations has to wait on any other. The hundredth cell can be computed without ever knowing what the first cell came out to.
That property — nothing waiting on anything else — is the whole point. Add more lanes and it speeds up in direct proportion. Painting pixels onto a screen had that same property, which is exactly why hardware built for screens turned out to fit AI calculation well.
The opposite kind of job, where the next calculation can't start until the previous answer is in hand, doesn't get this benefit. It doesn't matter how many lanes you add — if cars can only enter one at a time, the rest of the lanes just sit empty.
Three things people mix up
Three different ideas show up together but mean different things. Compute counts how many times the work has to run. Adding more hardware doesn't shrink that number — it grows as the model gets bigger or the input gets longer.
VRAM is the space where the numbers being calculated get laid out. Run short on space and, as is, things can't even start; offloading part of the work to other memory lets it run, but far more slowly. That's a question of size, and it has nothing to do with lane count.
A GPU is the device that actually processes the work, corresponding to how many lanes there are. When an answer comes back slowly, which of these three is actually the bottleneck decides what needs fixing. If it's stalled because there's no room to lay the numbers out, adding more lanes won't help at all.
Training and inference ask different things of it
Training is this hardware's favorite job. Data gets bundled up dozens or hundreds at a time and pushed through all at once, so the lanes stay packed, and it runs around the clock for days without stopping. That's why big models get built by chaining thousands to tens of thousands of these devices together, and more than a hundred thousand at the very largest.
Generating an answer is a little different. In conversational AI that writes text out, in particular, the text gets stitched together one piece at a time, so the next piece can't start until the previous one is out, so lanes sit idle in between. Services get around this by bundling many people's requests together and processing them as one batch to keep the lanes full. It's also why hardware tends to sit idle in the middle of the night, when fewer people are connected.
It's not only graphics hardware
There are chips built specifically for AI calculation too — the screen-drawing parts stripped out, with everything focused on multiplying and adding, and the name varies by manufacturer. Even a smartphone carries a small accelerator inside dedicated to things like photo touch-up and voice recognition.
Web browsers can now borrow this kind of hardware too. That's exactly why demo tools have started showing up where an AI model runs the moment you open a page, with nothing to install.
More precisely
A GPU packs thousands of simple processing units onto a single chip and works by applying the same instruction to different numbers at the same time. Those units are called cores, but a GPU core does a different job from a CPU core. Recent chips even carry circuits dedicated purely to the table multiplications that show up constantly in neural networks.
The analogy breaks down in a few places. Toll plaza lanes each move independently, but this hardware's lanes are bundled together in groups that all receive the same instruction at once. So when a calculation includes a branch where the path splits depending on a condition, one side has to wait for the other to finish, and speed drops. There's another gap too: in practice, the speed of moving numbers in and out of memory holds things back more often than the lane count does — closer to lanes sitting empty while cars can't get in because the entrance ramp is too narrow. That's why picking hardware means looking past the number that corresponds to lane count, at how fast it can move numbers and how much it can lay out at once.
Try it yourself
- TensorSpace Visualization ailearn.space Seeing thousands of values laid out across a layer in one sweep makes it obvious why you need a device that can calculate them all at once
- WebLLM Chat (a conversational AI running in the browser) ailearn.space Your own computer's graphics hardware does the calculating directly. Watch how fast the answer arrives and you can feel what this device is doing
Common misconceptions
It's easy to think a GPU is always faster than a CPU, but actually, looking at a single job alone, a CPU is often faster — the ranking flips once many jobs run at once.
It's easy to think a gaming graphics card and an AI-purpose device are completely different things, but actually they share the same roots, and they diverge in memory size and speed, how the devices link together, how the compute circuits on the chip are organized, and more.
It's easy to think a good GPU alone is enough to run a large model, but actually if VRAM runs short it won't run as-is, and offloading part of it to other memory makes it run far slower.
One-line summary
A GPU is like a toll plaza with its lanes multiplied into the thousands: it handles AI calculation by pushing identically shaped work through all at once.
Spotted an error or have a better analogy? Suggest an edit · Last updated 2026-10-17