Matrix Multiplication

1Filling in a single cell

Filling one cell in the new sheet only takes two lines: one row from the first table and one column from the second. Multiply the first entry of each together, multiply the second entries together, keep going to the end, then add up everything you just multiplied. That sum is the number that goes in the cell.

So the new table's height matches the row count of the first table, and its width matches the column count of the second. Thirty students, two things to work out — price and calories — gives a result with thirty rows and two columns.

Filling one cell takes as many multiplications as the length of the paired row and column. Five menu items means five multiplications and four additions. The arithmetic itself is grade-school level. What makes it hard is how many cells there are to fill.

2One layer of a neural network is exactly this

A neural network takes in a row of numbers and turns it into a different row of numbers. That conversion is matrix multiplication. The incoming row is the order sheet's row, and the layer's weights are the price sheet. One column of the weight table produces one cell of the next layer.

The numbers sitting in the weight table were set during training. Once training ends, that table is fixed. Every time we send in a question, the model pushes a new row of numbers through that table, then through an activation function that bends the result slightly, and repeats. By computation count, most of the work is the multiplication that passes through the table.

A hundred layers means this pass repeats a hundred times, each result table feeding the next as its input. Attention, and image-model convolutions, all boil down to the same multiply-and-add at different shapes when you look inside them.

3Easy math, terrifying repeat count

A model with billions of weights means a price sheet with billions of cells. Every piece of text has to pass through most of those cells (a model that picks only a few experts out of many passes through only some of them), so producing a single line of an answer sets off multiplications and additions numbering in the trillions.

One person with a calculator would need a lifetime. That's why AI stories always come bundled with talk of compute and electricity bills. Double a weight table's width and height and it has four times as many cells, so the multiplications needed to pass one row of numbers through it also quadruple. Add layers and that many more multiplications get added.

4Cells don't talk to each other, so GPUs work

Luckily, this calculation has one convenient property. No cell in the new table waits on the result of any other cell. Whoever is filling in the top-left cell never needs to compare notes with whoever is filling in a cell three rows down.

That means the work can be handed to thousands of calculators at once. That's exactly what a GPU does — not a device that's clever at hard math, but one built to do easy math many times at once. Graphics memory capacity is often the bottleneck too, because the whole table has to be loaded onto it at once.

5The shapes have to match

If the order sheet has five columns but the price sheet only has four rows, one column is left without a partner. That's why the width of the first table must always equal the height of the second.

Matching table sizes across layers is a big part of designing a model. The order matters too — you can't swap the tables freely. Multiplying the order sheet by the price sheet gives a different result than multiplying the price sheet by the order sheet, when it's even possible at all.