Language Model
A program that scores what word probably comes next
- A language model is a program that scores what word is likely to come next, based on everything written so far.
- Nobody wrote grammar rules into it. It got its feel for language by guessing the next word, over and over, across an enormous pile of text.
- It doesn't lock in one answer. It attaches probabilities to many candidates and picks from among them — why the same question can get a slightly different answer each time.
- How far back it can look matters enormously. Early versions read only the last two or three words; today's read an entire document at once.
- Translation, summarizing, answering questions — all of it reduces to stringing together the next word. A large language model is simply this same idea, scaled way up.
Contents
1The analogy
Type "see you at" into a texting app and a row of suggested next words pops up above the keyboard — "the", "seven", "tomorrow". Where do those suggestions come from? The app looks at what you already typed, tallies which words tend to follow, and ranks them. That's exactly what a language model does.
The suggestions never come as a single word. Several appear in order, the first being the most likely and the odds falling off after that. A language model keeps that same kind of ranked list for every candidate, not just the three shown on your screen.
Tap one, and the sentence grows by one word. The app looks at the longer sentence and offers a fresh set of suggestions. Repeat that loop enough times, without ever stopping, and you get the long replies we see coming out of an AI.
2In detail
It scores every candidate, not just one
What a language model hands back isn't a single word. It's a scoreboard covering every fragment it knows, each one tagged with "how likely is this to come next" — all at once. If the vocabulary list runs to tens of thousands of entries, so does the scoreboard.
Those scores are arranged to add up to exactly one. That's what lets you read them as "this candidate at 40 percent, that one at 15 percent." A texting app only shows three suggestions because the screen is small; underneath, far more candidates are ranked.
When the scores spread out evenly, almost any word would sit naturally in that spot. When one candidate hogs nearly all the score, only that word can go there — the second half of a familiar saying is a spot like that.
Nobody handed it a rulebook
It was never given a grammar textbook. The training method is almost embarrassingly simple: take a piece of text, hide the part that comes next, have the model guess it, compare the guess against the real answer, and nudge its internal numbers very slightly. Repeat that billions of times.
What matters is that nobody needs to label the correct answers by hand — the text already contains them. That's what let raw internet writing double as a textbook, and it's the reason language models were able to balloon in size so quickly.
Learning this way pulls in more than grammar. Facts and tone ride along too, because guessing the next word well eventually requires knowing something about the world.
It doesn't always pick the top answer
What happens if a model always grabs the single highest-scoring candidate? Sentences turn stiff fast, and the same phrases repeat. So in practice the scores are treated like real probabilities: the top candidate wins most of the time, but the second or third choice gets picked occasionally too.
There's a dial that controls how much randomness gets mixed in. Turn it up and phrasing gets more varied, at the cost of the occasional strange word; turn it down and answers stay safe but flat. That dial is also why asking the same question twice can produce two different answers.
How far back it looks decides how good it is
Early approaches only looked at the two or three words right before the blank. So they couldn't carry forward something mentioned much earlier — "the book I borrowed from the library yesterday" — and would forget a subject introduced at the start of a sentence by the time they reached the end.
After that came models that passed a memory along one word at a time. Now the standard approach lays out every earlier piece at once and works out how much each one relates to every other. The range a model can take in at once has grown from a few lines to something like a whole book, and that shift is the single biggest change of the last few years.
Translation and summarizing are also just continuing text
A language model doesn't carry a separate translation feature bolted on. Feed it "Translate the following sentence into Spanish:" and it produces a translation simply by choosing, probabilistically, what comes next after that prompt. Summarizing and answering questions work the same way.
That's why what you put in front of it changes the result so much — the same principle behind why the suggestions above your keyboard change completely depending on what you've already typed.
3More precisely
A language model is a function that computes a probability distribution over the next token, given the tokens that came before it. It produces a score for every fragment, converts those scores into probabilities that sum to one, and samples from that distribution. How well it does this is measured by a value called perplexity — the lower it is, the less the model had to guess.
Reading strictly left to right isn't the only setup. Some language models instead leave a gap in the middle of a sentence and use both sides to fill it in — this is how models built for search or classification are trained. The left-to-right kind is strong at producing new text; the fill-in-the-middle kind is strong at grasping what a sentence means.
The analogy has a limit worth naming. Your texting app mostly leans on what you personally typed before and a short stretch of recent context. A language model draws on a feel built from a vast amount of text that was never yours, and it weighs a much longer stretch of what came before all at once.
4Try it yourself
5Common misconceptions
It's easy to think a language model learned grammar rules and builds sentences from them, but actually it was never given any rules — repeating the next-word guessing game simply made grammatical continuations more probable.
It's easy to think it always picks the highest-probability word, but actually the choice is deliberately shaken up a little each time, so the same question can get a different answer on a second try.
It's easy to think a language model and a large language model are different things, but actually a large one is the same idea, just built at a much bigger scale.
7One-line summary
In shortA language model works like the word suggestions above your keyboard — it scores what's likely to come next based on what came before, and that one trick is enough to translate and summarize too.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02