Key points
- An n-gram is a run of n words stuck together. Two words is a bigram, three is a trigram.
- Older language models only looked at the last few words, counted how often each word had followed, and picked the next word from that count.
- Widen the window and the number of combinations to count explodes past anything manageable. That's why the window couldn't grow.
- A combination never seen before gets a probability of zero. Working around that gap always took extra patching.
- Current methods look at a very long stretch of what came before at once, and instead of counting, they treat similar-meaning words as related.
The analogy
Picture dumping a thousand jigsaw pieces on the table with the finished picture kept hidden away. All you can do is look at the shape of the notches on the edge of the piece in your hand and find one that fits.
This method actually gets you somewhere. Two or three pieces snap together just fine — matching notches usually means matching colors too. But connect twenty pieces and step back, and something strange has happened. Every seam looks fine, yet the whole thing is neither sky nor sea, just a smear.
Looking only at the edges has a ceiling. Try to weigh five notches at once instead of two, and the number of possible combinations spins out of hand almost immediately.
In detail
Counting the last few words
What older language models did was simple. Scan a huge pile of text and count how often one word was followed by another, building a table as you go — after "ate," the word "lunch" came tens of thousands of times, "gravel" only a handful.
Picking the next word meant looking that table up. If the last two words were "weather is," it looks up what followed those two words before and either takes the most frequent one or draws in proportion to the counts. Learning was nothing but counting, so it was extremely fast.
The length of the run, n, became the name. See one prior word and guess the next, and the run is two words long, so it's a bigram; see two prior words and it's a trigram. The wider the window, the more natural the sentences got.
The window hits a wall fast
So why not just stretch the window to ten words? This is exactly where the method falls apart. With 50,000 possible words, two-word combinations already number 2.5 billion, and three-word combinations are 50,000 times that. No table can be built or stored at that size.
Growing numbers aren't the only problem. The more combinations there are, the more of the table sits empty at zero. No matter how much text gets collected, not every possible ten-word combination in the world will ever be seen.
So the range that was actually usable stopped at three or four words. That's exactly why the seams read fine while the whole paragraph doesn't hang together.
An unseen combination becomes a zero
Run into a combination that isn't in the table and the probability drops to zero — meaning a sentence with a name it's never seen simply can't be handled at all. That happens constantly in real writing, so it couldn't just be left alone.
Patches were built to fill the gap: adding a tiny value to every possible combination ahead of time, or, when a three-word combination is missing, backing off one step to a two-word combination, and then to a single word if even that's missing.
These patches plug the hole but don't teach meaning. To this method, "cat" and "cats" are two completely unrelated words. Seeing a lot of one does nothing for handling the other.
What's different now
The biggest change comes in two parts. One is the window itself. Current models look at text tens of thousands of pieces long, or more, all at once instead of the last two or three words, which is exactly why a name set early in a passage can still hold up by the end.
The other is treating similar-meaning words as related instead of counting. Attach a bundle of numbers to each word, and what's learned from "cat" carries over some to "cats" and to "kitty" too. Even a combination never seen before can get a reasonable answer, based on what's known about words that showed up in similar spots.
Each word gets a bundle of numbers of a fixed size, so there's no need for a separate slot for every combination. Add the fact that the same calculation rule is shared across every position, and widening the window doesn't make the model grow by the number of combinations. The amount of computation does grow, though.
Where it still holds a spot
The older method didn't vanish just because it's old. Counting is all it needs, so it's light and fast, and it's easy to open the table and check exactly why a candidate came up.
It's still used wherever a short window is enough and an instant answer matters, such as search-box autocomplete and typo-correction candidates, or it sticks around as a light helper for neural models. It's also used to measure overlap between two pieces of text a few characters at a time, like when checking for plagiarism.
More than anything, the core idea of training a model to guess the next word carried straight through. Today's models do the exact same job — only the window and the method for handling it have changed.
More precisely
An n-gram model assumes that looking at just the last n-minus-one words is enough to determine the probability of what comes next, and builds a probability table by dividing up the counts seen in training text. This assumption keeps the math simple, but it pays for that by treating any relationship beyond that window as if it doesn't exist. Filling in gaps takes methods like adding a small value to every combination, backing off one step at a time to a shorter window, or blending probabilities from several lengths, and in practice refined versions of the last two were what mostly got used. Perplexity, the scale used to measure how well a model built this way guesses text, is the same scale still used to evaluate current models today.
The analogy breaks down in a couple of places. A puzzle piece's notch either fits or it doesn't, but the next word is never settled to just one option — the table always keeps several candidates around, each with its own count. A puzzle also has one original picture it's building toward, while text has no single correct answer. And a puzzle can be started from anywhere, while this method always connects pieces left to right, in one direction only.
Try it yourself
- Fill in the Blank (what a language model actually learned) ailearn.space Watch what candidates come up for a single blank, and how likely each one is, to get a feel for what scoring every candidate means (this tool looks at both sides of the blank)
- Transformer Explainer (how a transformer works) ailearn.space See today's approach, which reads the whole text so far and assigns a probability to each candidate for the next piece. Compare it with how n-grams looked only at the last two or three words and counted occurrences
Common misconceptions
It's easy to think an n-gram model learned grammar, but actually it's just a table of how often one word followed another.
It's easy to think widening the window keeps improving performance, but actually the combinations explode and most of the table sits empty, so past a point, going wider stops helping and the table just grows too big to handle.
It's easy to think today's AI still only looks at the last few words too, but actually it looks at a very long stretch of prior text all at once.
One-line summary
N-grams are how models used to guess the next word by counting the last few, and two walls — a narrow window and unseen combinations — pushed the field toward the methods used today.
Spotted an error or have a better analogy? Suggest an edit · Last updated 2026-10-18