Small Language Model

Small means fewer values packed inside

Inside a language model sit a dense set of numbers fixed by training. The count of those numbers is the model's size. A large model might have hundreds of billions of them; a small one, hundreds of millions to a few billion.

Fewer numbers mean a smaller file and less room needed on the device. Less to compute also means faster answers. Rather than just squeezing a large model down, small models are usually either trained from scratch inside a smaller frame, or taught using a large model's answers as their textbook.

Where the line for "small" sits keeps moving. A size that counted as large a few years ago gets filed under small today, because training methods keep improving and the same size does better every year.

What being light buys you

The first thing is speed. Less to compute means the first word appears much faster, and there's no round trip to a server eating up time either.

The second is that it doesn't care where it is. It runs with no internet, on a plane, or inside a locked-down office network.

The third is often the biggest reason in practice: whatever you type never leaves the device. For medical records or internal company documents — anything awkward to send off elsewhere — a small model can be the only real option.

The fourth is cost. Less computing per user means the same hardware can serve far more requests.

You can't pull out what isn't in the bag

A small model can only pack a sliver of the world's knowledge. It holds onto common facts fine but starts to wobble fast on rare facts, narrow specialist topics, or a language with fewer speakers. And rather than admitting it doesn't know, filling the gap with a plausible-sounding sentence anyway is a habit that shows up more often in small models.

Problems that need several steps chained together are a weak spot too. Three or four conditions layered into one calculation, or a long piece of writing that has to keep obeying a rule set early on — a small model tends to lose the thread partway through. The longer a conversation runs, the more often it drops something said earlier.

The gap in knowledge, though, can largely be closed by handing it material to work from. Give it the right documents alongside the question, and even a small model gets quite good at finding the answer inside them — like leaving the missing gear at checkpoints along the trail instead of packing it.

Fix the trail, and the small bag is enough

Used as a catch-all for any question, a small model falls short. But what an actual service asks it to do is often much narrower than that: sorting support messages into categories, pulling action items out of meeting notes, rewriting a sentence into a fixed format.

Fine-tune it on examples for that one job, and accuracy climbs sharply. Within one narrow task, the gap against a much larger model barely shows. The budget saved by shrinking the model can even go toward running extra checks on the result instead.

These days the two get mixed together, too — an easy request gets handled instantly by the small model right on the device, and only what it can't manage gets passed up to a larger one.