Vision Transformer

1Cutting the photo into same-size pieces

The first step is cutting. The photo gets sliced into tight squares, a dozen-odd pixels on a side. The photo is first resized to a fixed size and then cut, and at a commonly used size that comes to around two hundred pieces. The pieces don't overlap and leave no gaps, so put back together they fill the original photo exactly.

Each piece is a small bundle of colored dots. That bundle gets flattened into a single row of numbers. After this step, one photo has become a few hundred rows of numbers.

Here's the key part: a few hundred rows of numbers look exactly like the result of chopping a sentence into pieces. That's why a structure originally built for text can be reused almost without modification — and it's exactly why this gets called a Vision Transformer.

2Skip the position tag and position is lost

The moment pieces get lined up in a row, a problem appears. On that line, which piece sat at the top left of the photo and which sat at the bottom right is gone. Shuffle the order of the pieces and the calculation comes out exactly the same.

So a number bundle telling each piece where it sits gets added in — the same role a number tag plays on a shipping container. Only once this tag is attached can pieces from the top be told apart from pieces at the bottom.

3Comparing distant pieces directly

This is where it splits from a CNN. A CNN slides a small window across the photo and only ever looks at immediately neighboring pixels together. Because what one step can see is narrow, reaching all the way across the photo takes stacking many layers on top of each other.

A Vision Transformer has every piece compare against every other piece from the very first step. It works out directly how related the top-left piece is to the bottom-right one. A single object split across two ends of a photo can be tied together as one thing right from the start.

In exchange, it gives up an advantage a CNN gets for free. That nearby points tend to be closely related, and that an object shifted a little is still the same object — a CNN already knows both facts, baked into its structure. Apart from what's built into the cutting itself, a Vision Transformer has to learn most of that from data.

4The price of knowing little in advance

That's why, with too little data, it can actually do worse than a CNN. When it was first presented, even training on roughly a million photos left it behind a CNN. Later, tricks for stretching the data and ways of learning from another model arrived, and with no extra large data set added, those same million or so photos alone became enough to compete. Having fewer built-in habits means freedom — and it also means most of it has to be learned from scratch.

Flip it around, though: after first learning broadly from a huge pile of photos, it often ends up beating a CNN afterward. A built-in habit helps as much as it limits. That's why the common approach now is to take something already trained on a large data set and fine-tune it for whatever specific job is needed.

5Piece count is cost

Because every piece compares against every other piece, doubling the pieces quadruples the pairs to compare. Push the resolution higher and you either cut pieces finer or add more of them — and the moment you do, computation balloons. That's a real burden anywhere high resolution matters.

That's why some later architectures mix in an approach where nearby pieces get compared first, and pieces get merged into fewer, bigger ones as you go up. Starting fine and progressively grouping into something bigger closely resembles what a CNN was already doing. The two families have moved closer by borrowing each other's strengths.