RNNRecurrent Neural Network
A network that carries forward a running summary, word by word
- An RNN (Recurrent Neural Network) takes in ordered material, like text or sound, one piece at a time, in sequence.
- It looks like a row of many cells, but really it's the same single cell, used over and over.
- What gets passed forward each time is a short running summary of everything heard so far, not the original text itself.
- So the longer a sentence runs, the more the earliest words get blurred out. This is the RNN's biggest weakness.
- Because it can only move in order, it can't process many words at once, which is exactly why the transformer took over its spot.
Contents
1The analogy
Picture a version of telephone with a twist. A leader reads a sentence out loud, one word at a time: "Today," "it's," "raining," "hard." The person at the front of the line only hears "Today," then boils that down to something short and whispers it to the next person. The second person combines that whisper with the next word, "it's," and boils the pair down again before passing it on. The third does the same with "raining," and the fourth with "hard."
Only a short whisper can be passed along — nothing can be written down. So the longer the sentence gets, the more "Today," heard way back at the front, gets blurred out as it crosses person after person. Whatever the last person in line finally says carries only what was heard most recently, clearly; everything from the start has faded.
2In detail
Taking in one word at a time
An ordinary network takes in its data all at once. Feed it a whole photo or a whole row of a table — a fixed-size chunk — and out comes an answer. Sentences don't cooperate that way. Some run three words, some run three hundred.
An RNN solves this with order. It breaks a sentence into pieces and feeds them in one at a time. Feed in one word and the computation runs once; feed in the next and it runs again. That way, no matter how much the length varies, the same approach handles it.
What gets passed along is a single running summary
The cell receives two things every time: the new word that just arrived, and the summary handed over from the step before. It combines the two, runs the computation, and produces a new summary to hand off to the next step.
This summary isn't a copy of the original text. It's a bundle of numbers with a size fixed from the very start, so whether it's heard a hundred words or three, it can only hold the same fixed amount. Every time a new word comes in, it has to fight for space inside that bundle, and what usually gives up its spot is whatever came earliest.
It's the same cell, used again and again
Drawn out, it looks like a row of cells lined up one after another, but there's really only one cell inside the actual model. There's just the one, and it feeds its own output right back in as its own input. That's where the word recurrent comes from.
Reusing the same cell means the same set of weights gets applied at every single step. The knobs used for the first word are the exact same knobs used for the hundredth. That's why a longer sentence doesn't make the model any bigger, and it doesn't add anything new to learn either.
The further it runs, the blurrier the start gets
Reusing the same computation over and over is also where the weakness comes from. The summary gets squeezed down a little at every step, and after dozens of steps that squeezing piles up until almost nothing survives from the very beginning. Run the other direction and a summary that keeps swelling instead can blow up to unstable values.
The trouble gets worse during training. Correcting a wrong answer means sending how far off it was back from the end toward the start, and that signal shrinks at every step it crosses too. The last few words tend to get learned well; the opening of the sentence often barely gets learned at all. This is called a vanishing gradient.
Stuck moving in order, so it's slow
An RNN can't start on the next step until the one before it finishes. Computing the second word needs the first summary already sitting there, ready. So a sequence of a thousand words means waiting through a thousand steps, one after another, in a line.
That's a poor fit for today's graphics hardware, which is built to blast through many computations at once. Once an architecture came along that could process every word in a sentence together, RNNs quickly lost their footing in language work. They're still put to good use, though, wherever data genuinely arrives one piece at a time in real time, like music or sensor readings.
3More precisely
An RNN processes ordered data by updating a bundle of numbers called the hidden state at every step. At each step, it takes the new input and the previous hidden state together, multiplies and sums them, applies a bend, and produces a new hidden state, pulling an output from it whenever one's needed. The biggest difference from an ordinary feed-forward network is that every single step reuses the exact same weights.
The analogy breaks down in a few places. In a real game of telephone, everyone summarizes a little differently based on their own personality, but every step of an RNN summarizes by the exact same fixed rule. And while people understand speech in whole sentences, the summary an RNN passes along isn't a sentence carrying meaning — it's a fixed-size bundle of numbers that a person couldn't read even by opening it up directly.
When looking both backward and forward matters, a second pass that reads the sentence from end to start gets added alongside the first. Even then, the underlying constraint of having to process everything in order never actually goes away.
4Try it yourself
5Common misconceptions
It's easy to think an RNN remembers the whole sentence, but actually it only ever carries one fixed-size summary, and the earlier parts blur out more with every step that passes.
It's easy to think the cells drawn in a diagram are separate parts, but actually it's the exact same single cell, just unrolled and drawn out across time.
It's easy to think RNNs are useless now that the transformer exists, but actually they're still active today for real-time processing where data arrives one piece at a time, or on small devices.
7One-line summary
In shortAn RNN is a single cell that carries forward the previous step's summary and processes one word at a time, which lets it handle sequence but makes it easy to lose the start of a long passage.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02