Vision Transformer

Cutting the photo into same-size pieces

The first step is cutting. The photo gets sliced into tight squares, a dozen-odd pixels on a side. A large photo comes apart into a few hundred pieces this way. The pieces don't overlap and leave no gaps, so put back together they fill the original photo exactly.

Each piece is a small bundle of colored dots. That bundle gets flattened into a single row of numbers. After this step, one photo has become a few hundred rows of numbers.

Here's the key part: a few hundred rows of numbers look exactly like the result of chopping a sentence into pieces. That's why a structure originally built for text can be reused almost without modification — and it's exactly why this gets called a Vision Transformer.

Skip the position tag and position is lost

The moment pieces get lined up in a row, a problem appears. On that line, which piece sat at the top left of the photo and which sat at the bottom right is gone. Shuffle the order of the pieces and the calculation comes out exactly the same.

So a number bundle telling each piece where it sits gets added in — the same role a number tag plays on a shipping container. Only once this tag is attached can pieces from the top be told apart from pieces at the bottom.

Comparing distant pieces directly

This is where it splits from a CNN. A CNN slides a small window across the photo and only ever looks at immediately neighboring pixels together. Because what one step can see is narrow, reaching all the way across the photo takes stacking many layers on top of each other.

A Vision Transformer has every piece compare against every other piece from the very first step. It works out directly how related the top-left piece is to the bottom-right one. A single object split across two ends of a photo can be tied together as one thing right from the start.

In exchange, it gives up an advantage a CNN gets for free. That nearby points tend to be closely related, and that an object shifted a little is still the same object — a CNN already knows both facts, baked into its structure. A Vision Transformer has to learn all of that from data.

The price of knowing nothing in advance

That's why, with too little data, it can actually do worse than a CNN. Train it on a data set of only a few tens of thousands of photos, and it fails to properly grasp the relationships between pieces and wobbles. Having no built-in habits means freedom — and it also means everything has to be learned from scratch.

Flip it around, though: after first learning broadly from a huge pile of photos, it often ends up beating a CNN afterward. A built-in habit helps as much as it limits. That's why the common approach now is to take something already trained on a large data set and fine-tune it for whatever specific job is needed.

Piece count is cost

Because every piece compares against every other piece, doubling the pieces quadruples the pairs to compare. Push the resolution higher and you either cut pieces finer or add more of them — and the moment you do, computation balloons. That's a real burden anywhere high resolution matters.

That's why later architectures mix in an approach where nearby pieces get compared first, and pieces get merged into fewer, bigger ones as you go up. Starting fine and progressively grouping into something bigger closely resembles what a CNN was already doing. The two families have moved closer by borrowing each other's strengths.