GPT

The three letters of the name describe the structure

The last part, transformer, is the skeleton. It's the mechanism that decides how much each piece of a sentence should pay attention to every other piece, and most language models today are built on this same skeleton.

The middle part, pre-trained, describes the order of operations. Before anyone decides what questions will be asked, the model spends time soaking up the basics from an enormous pile of text — built first, put to use later.

The first part, generative, describes what it does. Rather than picking from a fixed set of options, it writes text that never existed before. Put the three together and you get "a text-continuing model, built on a transformer, trained ahead of time."

It only looks forward, never back

As far as GPT is concerned, whatever comes after its current position simply doesn't exist. During training, everything past that point was covered up, and it practiced guessing the next piece — an answer sitting right next to the blank wouldn't make for good practice.

It also never revises a piece once it's placed. A person will scratch out an earlier sentence and start over; GPT leaves an already-stamped box exactly as it is and keeps adding after it. That's why, once a sentence starts going off track, it keeps building on that very drift instead of erasing it.

Ask it to "rewrite that earlier sentence," of course, and it will. That's writing a new revised sentence onto the end, though — not reaching back and editing the letters already on the card.

Every new piece means rereading the whole card

Every time GPT adds a piece, it rereads the entire text produced so far. Choosing the tenth piece means taking in the nine before it; choosing the hundredth means taking in ninety-nine.

So the later a piece falls, the more work it takes to produce it, and the answer slows down accordingly. That's part of why text streaming across the screen sometimes seems to slow down toward the end. A long conversation compounds the effect further, since earlier turns get reread each time too.

There's also a ceiling on how much it can take in at once. Once a conversation grows past that ceiling, the oldest parts get pushed out. Deep into a long conversation, an assistant can start acting like it forgot a condition set at the very beginning — because that part is simply no longer on the card.

One model, put to work in many places

Translators, summarizers, and sentiment classifiers used to be built separately from scratch. GPT flips that around. Build one well-rounded model first, then change only the instruction to make it do different jobs afterward.

Tell the same model "translate this into French" and it becomes a translator; tell it "cut this to three lines" and it becomes a summarizer. Not having to build a new model for every new feature is a big part of why language models spread into so many fields so quickly.