Key points
- Positional encoding tags each piece with which position in line it is.
- It's needed because a calculation in which every spot looks at every other spot equally can't tell order apart on its own.
- The earliest methods add the tag to the original content and feed it in as one block, while the approach common today builds order into the calculation that compares spots.
- Some tags are built from a fixed rule; others are left for training to work out on its own.
- These days, methods that capture how far apart two spots are, rather than the absolute position, are widely used.
The analogy
A theater ticket has a seat number printed on it. Without that number, what happens? The audience is all seated, but there's no way to tell who's in the front row and who's in the back. Same headcount, same faces, only the order is gone. Positional encoding is what prints that seat number onto the ticket.
The number doesn't travel separately from the audience member. One ticket carries the seat number and the ticket holder's information together. The first positional encoding methods work the same way — they add an order tag directly into a piece's content, folding both into one block.
The moment a seat number is attached, a relationship appears: "the front row is closer to the stage than the back row." Distance and order only become sayable once numbers exist.
In detail
A structure that loses track of order
The older method of passing a sentence one word at a time, in order, had no order problem — the processing order was the order. It was just slow.
The method used now has every spot look at every other spot at once. That made things much faster, but it came at a cost: with everyone looking at everyone simultaneously, there's no way to tell what came first. "The dog chased the cat" and "the cat chased the dog" would look like the identical bag of words.
So an order tag has to be attached. A calculation in which every spot looks at every other spot equally can't tell word order apart on its own.
The number gets added into the content
In the method first proposed, the tag isn't attached beside a piece as a separate label. A number bundle representing order, the same size as the content bundle, is added into it to make one combined value. That spot's value now carries both "what it is" and "which position it is" together.
Adding them together works fine because there's plenty of room to hold both. The content bundle has hundreds to over ten thousand entries, giving content and order room to settle into different directions within it, and training teaches the model to read the two apart.
Two ways to build the tag
The method first proposed built the tag from a fixed, human-designed rule — layering waves of several different frequencies so each position ends up with its own distinct pattern. It needs no training, and the numbers themselves can still be computed for lengths longer than anything seen in training. But being able to assign a number is not the same as the model handling that length well, so in practice performance drops once the text runs past the training length.
The other approach leaves the tag itself to be learned. A number bundle is placed at each position and its values get tuned during training. This produces a tag well-suited to the data, but it has no number to hand out once you go past however many positions were prepared in advance.
Distance matters more than the absolute number
Writing down the raw position number has a weakness. The same sentence gets a completely different number depending on whether it sits near the start or the end of a document, when what usually matters is really "how many spots apart are this word and that word."
That's why the method widely used now makes the distance between two spots show up directly in the calculation — values get rotated slightly based on position number, so comparing two spots naturally reflects the gap between them. Read the same gap anywhere in the text and it comes out as the same relationship. Still, once the text runs past the lengths seen in training, this method wobbles too and needs separate adjustment, as below.
Running out of seats
A model gets shaky when it sees a position number far beyond anything encountered during training — like being handed a ticket for a seat the theater doesn't have. Sentences can suddenly turn awkward, or the beginning gets lost.
This is why widening a context window isn't simply a matter of raising a number. The tags need to be rescaled for the wider range, and some further training at the new length is usually needed too.
More precisely
Positional encoding adds a vector carrying position information into each token's embedding, or folds position relationships into the attention computation itself. The method first proposed used fixed sine and cosine waves of several frequencies laid out differently at each position; learned vectors at each position were also widely used. What most large models rely on now is a rotary approach, which rotates the query and key by an amount tied to position, so the relative distance between two spots comes through naturally. That said, a model that generates text while masked to see only what came before can partly guess position on its own from how many earlier spots it can see. Even so, attaching an order tag remains the basic approach.
The analogy breaks down in places. A theater seat number is a single integer, while positional encoding is a bundle of more than a few hundred values at each spot. On a ticket, the seat number and the audience member's information share one slip but still read as separate fields, while the additive kind of positional encoding is mixed into the content value and can't be told apart from the outside. Above all, theater seat numbers are plain counts that grow by one (1, 2, 3), while the first position tags were patterns made by layering waves of several frequencies. Nearby spots get similar patterns, and the patterns are built so that the number of spots between two positions can be worked out by calculation.
Try it yourself
Common misconceptions
It's easy to think a model naturally knows word order, but actually the calculation that looks at every spot at once has no concept of order in itself, so attaching an order tag is the basic way of telling it the order.
It's easy to think the position tag rides alongside a piece separately, but actually it's either added into the content value or folded into the calculation that compares spots. It never rides along as a separate label.
It's easy to think context length grows just by changing a setting, but actually the position tags need to be rescaled for the wider range, and usually some further training at the longer length goes along with it.
One-line summary
Positional encoding is like a theater seat number tagging each piece with its place in line, handing back a sense of order and distance to a structure that otherwise has none.
Spotted an error or have a better analogy? Suggest an edit · Last updated 2026-10-17