Key points
- A token is a bite-size piece of text an AI cuts before reading. It is not a letter and not quite a word, but something in between.
- The pieces are not the same size. Common words stay in one fat piece, while unfamiliar ones come apart into several thin ones.
- The model never sees letters. Each piece carries a number, and the answer is built by picking one number at a time.
- The same meaning costs more pieces in some languages than others, which shrinks how much text fits in one go.
- There is a ceiling on how many pieces fit at once, and when software calls an AI, the bill is counted by the piece, not by the letter.
The analogy
Nobody eats a foot-long sub sandwich in one bite. You slice it into pieces small enough to chew, then work through them one at a time. An AI reads the same way. It cannot swallow a sentence whole, so the text is cut first and taken in piece by piece. One of those pieces is a token.
The slices are not even. A word the AI has met a million times comes off as one fat piece. Something it rarely sees gets cut thin, into three or four. "Today" arrives in a single bite, while an unusual dish on the menu might arrive as bru, then schetta.
The plate has a limit too. Only so many pieces fit at one time, so when a conversation runs long, in some services the earliest pieces slide off the edge to make room for new ones. That is why an AI can lose track of something you said at the very start while answering perfectly about the last thing you typed.
In detail
The cutting rule is built on frequency
Nobody wrote the slicing rules by hand. During training, the system scanned an enormous pile of text and looked for letter groups that keep showing up together. Those groups were saved into a list. Anything on the list survives as one whole piece. Anything missing from it gets broken down into smaller pieces that are on the list.
The list usually holds anywhere from tens of thousands to a few hundred thousand entries. Human language is far richer than that, so rare material always drops to smaller pieces. A very unusual character may even split into two or three fragments on its own.
That is why a common greeting is a single piece while a brand-new slang word or an unfamiliar surname stretches to four. Two sentences of identical length can carry very different piece counts, depending entirely on which words you picked.
Every piece carries a number
Each entry in the list has an index number. When a sentence is cut, every piece is swapped for its number. What the model actually receives is a row of numbers, not letters.
The numbers are name tags for ordering, nothing more. Two neighboring numbers do not mean two related meanings. Meaning appears at the next step, when each number is turned into a bundle of values. A model handles a mild typo calmly because the text it learned from had plenty of such sentences mixed in, and it guesses the meaning from the surrounding context. Still, a single typo can change how the text gets split, so a very unfamiliar typo can trip it up.
Writing works piece by piece too
Producing an answer runs the same machinery in reverse. The model looks at everything collected so far, picks the single most suitable next piece, attaches it, and looks again. Text appearing to flow across your screen is really pieces landing one after another.
It may look like typing, but the true unit is the piece, which is why two or three letters sometimes pop out together. A longer answer takes proportionally longer to arrive, and the limit on answer length is counted in pieces rather than characters.
Some languages cost more pieces
In English, a word is usually about one piece. Other languages fare worse. Korean, for one, is thinly represented on the list, which was built from text that leans heavily toward English, and endings shift constantly as they attach, so much of the text never fits the list whole and gets chopped fine.
For the same meaning, Korean often spends more pieces than English, anywhere from a bit over 1.3 times to two or three times as many depending on the model. Less text fits in one go, answers take longer, and costs climb. Any writing system far from the training material's center of gravity pays a similar tax.
Piece counts set the limits and the bill
There is a hard ceiling on how many pieces an AI can hold in view. Once a conversation passes it, depending on the service, the earliest part is cut off or summarized before being fed in, and some services refuse the input outright. An assistant that seems to forget a promise it made twenty messages ago is not being careless. Those pieces were cut and are simply no longer on the plate. Even when nothing has been cut, a very long text can make the model pick up less of the earlier part.
Charges work the same way. When software calls an AI, the bill counts pieces sent and pieces returned, not characters. A long example pasted in front of your question turns into cost immediately. Most services count input and output separately, so a longer answer means a longer invoice.
More precisely
A token is a string fragment produced by a tokenizer following a fixed rule set. Because a fragment can be smaller than a word and larger than a letter, these are called subwords. The list of fragments is most commonly built by repeatedly merging the character pairs that appear together most often in the training data, a method known as Byte Pair Encoding, or pair merging. Some models build the list in other ways.
The sandwich comparison breaks down in one important place. A sandwich is cut along ingredients you can see. A tokenizer does not look at meaning at all. It only knows which characters travelled together in the training text. So a cut can land in the middle of a word, a leading space is often folded into the piece that follows it, and a long number frequently splits into two or three chunks for no reason a reader would guess.
Fragment lists are also built separately for each model, so the same sentence handed to two different systems can produce different counts.
Try it yourself
Common misconceptions
It's easy to think a token is the same thing as a word, but actually one word often splits across several pieces, and several characters can be bundled into one.
It's easy to think counting characters tells you the token count, but actually the count swings with word choice, so two texts of equal length can differ widely.
It's easy to think spaces and punctuation are free, but actually blank space and line breaks are counted as part of the pieces like everything else.
One-line summary
A token is a bite-size piece of text an AI cuts before reading, and the number of pieces decides how much it can take in at once and what you pay when calling it from software.
Spotted an error or have a better analogy? Suggest an edit · Last updated 2026-10-17