GPT

1The three letters of the name describe the structure

The last part, transformer, is the skeleton. It's the mechanism that decides how much each piece of a sentence should pay attention to every other piece, and most language models today are built on this same skeleton.

The middle part, pre-trained, describes the order of operations. Before anyone decides what questions will be asked, the model spends time soaking up the basics from an enormous pile of text — built first, put to use later.

The first part, generative, describes what it does. Rather than picking from a fixed set of options, it writes text that never existed before. Put the three together and you get "a text-continuing model, built on a transformer, trained ahead of time."

2It only looks at what came before, and only moves right

As far as GPT is concerned, whatever comes after its current position simply doesn't exist. During training, everything past that point was covered up, and it practiced guessing the next piece — an answer sitting right next to the blank wouldn't make for good practice.

It also never revises a piece once it's placed. A person will scratch out an earlier sentence and start over; GPT leaves an already-stamped box exactly as it is and keeps adding after it. That's why, once a sentence starts going off track, it keeps building on that very drift instead of erasing it.

Ask it to "rewrite that earlier sentence," of course, and it will. That's writing a new revised sentence onto the end, though — not reaching back and editing the letters already on the card.

3Every new piece draws on the whole card

Every time GPT adds a piece, it takes the entire text produced so far into account. Choosing the tenth piece means taking in the nine before it; choosing the hundredth means taking in ninety-nine.

The work on earlier parts gets saved and reused, but there's more to take into account each time, so in a very long text each later piece takes slightly more work. A long conversation compounds the effect further, since earlier turns have to be taken into account too.

There's also a ceiling on how much it can take in at once. Once a conversation grows past that ceiling, depending on the service, the oldest parts get cut off or squeezed into a shorter summary. Deep into a long conversation, an assistant can start acting like it forgot a condition set at the very beginning — because the part that was cut off is no longer on the card as written. Even when everything fits inside the ceiling, a long text can make the model lean less on instructions near the start, and something similar can happen.

4One model, put to work in many places

Translators, summarizers, and sentiment classifiers used to be built separately from scratch. GPT flips that around. Build one well-rounded model first, then use it as the base for many jobs. Early on, a pre-trained model was trained a little further for each job, but as models grew larger and were tuned to follow instructions, changing only the instruction became enough to make it do different jobs.

Tell the same model "translate this into French" and it becomes a translator; tell it "cut this to three lines" and it becomes a summarizer. Not having to build a new model for every new feature is a big part of why language models spread into so many fields so quickly.