Tokenizer

The locker list was built automatically

The way the list gets built is surprisingly simple. Start by treating every single character as its own tiny locker. Then scan the entire training text over and over, find whichever pair of characters shows up together most often, merge it into one, and register that merged chunk as a new locker. Repeat this merging tens of thousands of times and the list is done.

Whatever shows up most often gets merged earliest and claims a big locker. Common particles, endings, and frequent words earn their own locker that way. Rare material, on the other hand, never finishes merging and stays behind as small leftover pieces.

The list usually runs to a few tens of thousands of entries. Make it bigger and each sentence needs fewer pieces, which is more efficient — but the model has that many more candidates to juggle, which makes it heavier. The size in use today sits at a balance point between those two pulls.

Anything missing from the list gets split up

A tokenizer never gives up on a word it doesn't recognize. It's built so that any word can always be broken down into some combination of smaller pieces already on the list. Individual characters sit on the list too, so in the worst case it can still fall back to one character at a time.

That's why new slang or an unfamiliar foreign place name still gets handled without an error. The piece count just shoots up instead. A common word might take one piece while an unfamiliar name takes five — and since piece count directly drives how much text fits at once and what it costs, that gap is worth paying attention to.

Languages with more character variety run into this harder. Since the same word keeps taking on different endings and particles, plenty of word forms never make it onto the list whole and end up chopped fine.

Cutting and rebuilding use the same list

There are two directions here: cutting text into pieces and turning them into a row of numbers, and taking the row of numbers a model produced and stitching it back into letters. Both directions run on the same list, just flipped. Without the list, the numbers mean nothing at all.

That's why a model and its tokenizer always move as a matched pair. Numbers cut with a different list read as something else entirely — like showing up with a key from the wrong bathhouse.

Spaces and line breaks get folded into the pieces too, since restoring the original text exactly requires it. A leading space often gets bundled into the piece that follows it.

A different list means a different piece count

Every model-maker trained on different text, so the list pulled from that text differs too — the same sentence produces a different piece count depending on the model. One list, built from a lot of Korean text, might keep a Korean chunk whole; another, built mostly on English, chops the same Korean text finer.

That means "how many pieces is this sentence" can't be answered without naming a model first. Comparing service pricing isn't just about the rate per piece either — you need to check how many pieces the same sentence turns into on each one.

What the tokenizer causes downstream

There's a reason models often stumble on long arithmetic: digits don't always get cut along place-value boundaries, they get chopped into arbitrary chunks — and the same number can split differently depending on where the cut lands, which makes it hard to learn a consistent rule.

The same cause sits behind weak answers to something like "how many consonants are in this word." A model never sees individual letters — it only receives a bundled piece number, so what's actually inside a piece stays blurry to it. On the flip side, it's also why a model can shrug off a minor typo: a slightly misspelled piece still lands in a spot close to the original word.