Tokenizer

1The locker list was built automatically

The way the list gets built is surprisingly simple. Start by treating every single character as its own tiny locker. Then scan the entire training text over and over, find whichever pair of characters shows up together most often, merge it into one, and register that merged chunk as a new locker. Repeat this merging tens of thousands to hundreds of thousands of times and the list is done.

Whatever shows up most often gets merged earliest and claims a big locker. Common particles, endings, and frequent words earn their own locker that way. Rare material, on the other hand, never finishes merging and stays behind as small leftover pieces.

The list usually runs from tens of thousands to a few hundred thousand entries. Make it bigger and each sentence needs fewer pieces, which is more efficient — but the model has that many more candidates to juggle, which makes it heavier. The size is a balance between those two pulls, though these days it tends to be set larger than it used to be so the tokenizer can handle many languages.

2Anything missing from the list gets split up

A tokenizer usually doesn't give up on a word it doesn't recognize. Many are built so that a word can be broken down into some combination of smaller pieces already on the list. Modern tokenizers also put the raw bytes that make up characters on the list, so in the worst case a single character may split into several pieces, but it can still be taken in.

That's why new slang or an unfamiliar foreign place name still gets handled without an error. The piece count just shoots up instead. A common word might take one piece while an unfamiliar name takes five — and since piece count directly drives how much text fits at once and what it costs, that gap is worth paying attention to.

Languages with more character variety run into this harder. Since the same word keeps taking on different endings and particles, plenty of word forms never make it onto the list whole and end up chopped fine.

3Cutting and rebuilding use the same list

There are two directions here: cutting text into pieces and turning them into a row of numbers, and taking the row of numbers a model produced and stitching it back into letters. Both directions run on the same list, just flipped. Without the list, the numbers mean nothing at all.

That's why a model and its tokenizer always move as a matched pair. Numbers cut with a different list read as something else entirely — like showing up with a key from the wrong bathhouse.

Spaces and line breaks get folded into the pieces too, since restoring the original text exactly requires it. A leading space often gets bundled into the piece that follows it.

4A different list means a different piece count

Every model-maker trained on different text, so the list pulled from that text differs too — the same sentence produces a different piece count depending on the model. One list, built from a lot of Korean text, might keep a Korean chunk whole; another, built mostly on English, chops the same Korean text finer.

That means "how many pieces is this sentence" can't be answered without naming a model first. Comparing service pricing isn't just about the rate per piece either — you need to check how many pieces the same sentence turns into on each one.

5What the tokenizer causes downstream

The tokenizer is one reason models often stumble on long arithmetic: if digits are cut in groups of three from the front, the groups don't line up with place value, which makes the carrying rules hard to learn. That's why some tokenizers cut digits one at a time.

The same cause sits behind weak answers to something like "how many consonants are in this word." A model never sees individual letters — it only receives a bundled piece number, so what's actually inside a piece stays blurry to it. On the flip side, a model can shrug off a minor typo because, even though the cuts come out differently, the training text had plenty of similar typos and the surrounding context lets it guess the original word.