Self-Attention
Words in the same sentence checking in with each other
- Self-attention is words in the same sentence checking in with each other. No outside information gets pulled in.
- Every spot in the sentence is both the one asking and the one answering at once, not a fixed asker and a fixed answerer.
- Once the check-in is done, each spot's content is rewritten with context baked in. The same word reads differently in a different sentence.
- All spots do this at the same time, with no waiting in line from front to back — a big reason training got so much faster.
- Stacking this step across several layers widens the reach until it captures the whole sentence's structure.
Contents
1The analogy
Every unit in an apartment building has its own mailbox. When unit 3 sends around a note asking "what was our dog's name again," the other units each write back with whatever they know and leave it in their own mailbox. Unit 3 collects the replies and rewrites its own nameplate using them. Self-attention is notes passing around inside one building like this and nowhere else.
The key part is that everyone does this at once. It isn't just unit 3 asking — units 1, 2, and 4 each send their own note and collect their own replies too. Every unit is asking and answering at the same time.
Nothing arrives from the outside post office. No new information comes in — information already scattered inside the building just finds its proper place.
2In detail
It stays inside the building, not outside it
Attention itself was first used to connect two separate groups — linking words in a source sentence to words in a translation, where the asking side and the answering side were different sentences.
Self-attention merged the two into one. Both the asking side and the answering side are the same sentence, which is where the "self" in the name comes from. This one small change made the transformer architecture possible, and most language models running today stand on top of it.
Every spot plays three roles at once
Each spot puts out three things: a note that says "here's what I'm looking for," a tag that says "here's what I'm carrying," and the actual content it will hand over if asked. One mailbox sends letters, hangs out a tag, and prepares its reply content, all at once.
The better a note matches a tag, the larger a share of that spot's reply carries. Since the shares always add up to 100 percent, leaning heavily on one spot automatically means leaning less on the others. All three things are pulled from the same source in different ways, and how to pull them is set during training.
Collecting replies changes the content
Once the replies are in, a spot's content doesn't stay as it was. The replies get blended in by their share and folded into that spot's own content. That's how "it" ends up carrying a heavy trace of "the cake" mentioned earlier, and "bright" ends up carrying whatever came before it — a room or a personality.
This is where the same word ends up meaning different things depending on the sentence. Looked at alone, a word can carry several meanings, but once the surrounding replies are collected, which one it means gets settled.
Everyone works at once
Older methods had to finish the first word before moving to the second — waiting in line for a turn. Self-attention has no such order. Every spot watches every spot at once, so everything can be computed simultaneously.
This turned out to be the biggest change of all. It matched perfectly with hardware built to run thousands of operations in parallel, and training speed jumped by orders of magnitude — the practical reason today's large models became possible.
The cost is that order information disappears. If everyone looks at everyone at once, there's no way to tell who came first. That's why a separate marker showing each spot's order has to be added in.
Stacking layers widens the view
Collecting replies once isn't the end. The updated content sends around another note, and this repeats across several layers. In the second layer, content already carrying context is exchanged, so a much wider set of relationships comes into view.
Lower layers tend to catch nearby words and grammatical pairings, while higher layers lean toward a sentence's overall topic or logical flow — each layer picking up something a little different.
3More precisely
Self-attention is attention where the query, key, and value are all pulled from the same input. Each spot's query is compared against every spot's key to produce scores, which are scaled down to fit the vector size, turned into weights that sum to one through softmax, and used to blend the values. Models that generate text going forward hide any spot that comes after the current one.
Compute grows with the square of the number of spots — double the sentence length and the pairs to weigh quadruple. That's why handling long context gets expensive fast, and why variants aimed at cutting this cost keep appearing.
The analogy breaks down in a couple of places too. In the mailbox picture, someone in each unit reads and judges the replies, but self-attention has no one deciding anything — weights fall out on their own once trained numbers are multiplied and added, with no judgment involved. And there's no note actually being carried around from mailbox to mailbox in sequence either; one large matrix computation handles every pair at once, all in a single pass, with nothing waiting its turn.
4Try it yourself
5Common misconceptions
It's easy to think self-attention pulls in information from outside the sentence, but actually it only exchanges information among spots already present in the input.
It's easy to think nearby words get more weight, but actually weight is set regardless of distance, and a far-off word can end up mattering far more.
It's easy to think attention and self-attention are the same thing, but actually attention is the broader idea, and it's only self-attention when the asking side and the answering side are the same.
7One-line summary
In shortSelf-attention is like mailboxes inside one building: words in a sentence ask and answer each other, and each rewrites its own content with that context baked in.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02