N-gram
The way of guessing the next word by counting the last few
- An n-gram is a run of n words stuck together. Two words is a bigram, three is a trigram.
- Older language models only looked at the last word or two, counted how often each word had followed, and picked the next word from that count.
- Widen the window and the number of combinations to count explodes past anything manageable. That's why the window couldn't grow.
- A combination never seen before gets a probability of zero. Working around that gap always took extra patching.
- Current methods look at everything that came before at once, and instead of counting, they treat similar-meaning words as related.
Contents
1The analogy
Picture dumping a thousand jigsaw pieces on the table with the finished picture kept hidden away. All you can do is look at the shape of the notches on the edge of the piece in your hand and find one that fits.
This method actually gets you somewhere. Two or three pieces snap together just fine — matching notches usually means matching colors too. But connect twenty pieces and step back, and something strange has happened. Every seam looks fine, yet the whole thing is neither sky nor sea, just a smear.
Looking only at the edges has a ceiling. Try to weigh five notches at once instead of two, and the number of possible combinations spins out of hand almost immediately.
2In detail
Counting the last few words
What older language models did was simple. Scan a huge pile of text and count how often one word was followed by another, building a table as you go — "had" followed "ate" tens of thousands of times, "threw" only a handful.
Picking the next word meant looking that table up. If the last two words were "the weather is," it looks up what followed those two words before and either takes the most frequent one or draws in proportion to the counts. Learning was nothing but counting, so it was extremely fast.
The size of the window, n, became the name. Look at one prior word and it's a bigram; two prior words and it's a trigram. The wider the window, the more natural the sentences got.
The window hits a wall fast
So why not just stretch the window to ten words? This is exactly where the method falls apart. With 50,000 possible words, two-word combinations already number 2.5 billion, and three-word combinations are 50,000 times that. No table can be built or stored at that size.
Growing numbers aren't the only problem. The more combinations there are, the more of the table sits empty at zero. No matter how much text gets collected, not every possible ten-word combination in the world will ever be seen.
So the range that was actually usable stopped at three or four words. That's exactly why the seams read fine while the whole paragraph doesn't hang together.
An unseen combination becomes a zero
Run into a combination that isn't in the table and the probability drops to zero — meaning a sentence with a name it's never seen simply can't be handled at all. That happens constantly in real writing, so it couldn't just be left alone.
Patches were built to fill the gap: adding a tiny value to every possible combination ahead of time, or, when a three-word combination is missing, backing off one step to a two-word combination, and then to a single word if even that's missing.
These patches plug the hole but don't teach meaning. To this method, "cat" and "cats" are two completely unrelated words. Seeing a lot of one does nothing for handling the other.
What's different now
The biggest change comes in two parts. One is the window itself. Current models look at thousands of pieces at once instead of the last two or three words, which is exactly why a name set early in a passage can still hold up by the end.
The other is treating similar-meaning words as related instead of counting. Attach a bundle of numbers to each word, and what's learned from "cat" carries over some to "cats" and to "kitty" too. Even a combination never seen before can get a reasonable answer, based on what's known about words that showed up in similar spots.
That's why the table no longer has to grow with the number of combinations. The bundle of numbers attached to each word stays the same size even as the vocabulary grows — which is how the window could widen while staying manageable.
Where it still holds a spot
The older method didn't vanish just because it's old. Counting is all it needs, so it's light and fast, and it's easy to open the table and check exactly why a candidate came up.
It's still used wherever a short window is enough and an instant answer matters — search-box autocomplete, typo-correction candidates, the next-word suggestion on a keyboard. It's also used to measure overlap between two pieces of text a few characters at a time, like when checking for plagiarism.
More than anything, the core idea of training a model to guess the next word carried straight through. Today's models do the exact same job — only the window and the method for handling it have changed.
3More precisely
An n-gram model assumes that looking at just the last n-minus-one words is enough to determine what comes next, and builds a probability table by dividing up the counts seen in training text. This assumption keeps the math simple, but it pays for that by treating any relationship beyond that window as if it doesn't exist. Filling in gaps takes methods like adding a small value to every combination, or backing off one step at a time to a shorter window. Perplexity, the scale used to measure how well a model built this way guesses text, is the same scale still used to evaluate current models today.
The analogy breaks down in a couple of places. A puzzle piece's notch either fits or it doesn't, but the next word is never settled to just one option — the table always keeps several candidates around, each with its own count. A puzzle also has one original picture it's building toward, while text has no single correct answer. And a puzzle can be started from anywhere, while this method always connects pieces left to right, in one direction only.
4Try it yourself
- Fill in the Blank (what a language model actually learned) ailearn.space Watch what candidates come up for a single blank to get a feel for what guessing the next word actually means
- Bareun (Korean spell checker) ailearn.space A tool that catches awkward spots by looking at just a few words on either side — a good feel for what a narrow window is good for
5Common misconceptions
It's easy to think an n-gram model learned grammar, but actually it's just a table of how often one word followed another.
It's easy to think widening the window keeps improving performance, but actually the combinations explode and most of the table sits empty, making it unusable instead.
It's easy to think today's AI still only looks at the last few words too, but actually it looks at thousands of prior pieces all at once.
7One-line summary
In shortN-grams are how models used to guess the next word by counting the last few, and two walls — a narrow window and unseen combinations — pushed the field toward the methods used today.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02