Key points
- Feed-forward means a signal flows only from input toward output. There's no path leading back.
- A layer's result only ever moves on to later layers. Once a spot has been passed, it's never revisited.
- There's no channel for holding onto earlier information, so anything that requires remembering order gets handed off to a different mechanism.
- Feed-forward layers sit inside transformer blocks too. More than half of a model's numbers live there.
- Error does flow backward during training, but that's only the direction of a calculation — the structure itself never changes.
The analogy
Standing in a school cafeteria line, you only move forward with your tray. From the rice station to the soup station, from soup to side dishes, from side dishes to dessert. If you don't like the side dish, you can't walk back to the rice station. The line only flows one way.
Each station does one fixed job. The soup station only ladles soup, no matter what you've picked up before it. Once your tray clears the last station, the meal is complete.
A feed-forward network looks just like this. A bundle of numbers moves through layers the way a tray moves through stations, and each layer adds its own share of the calculation. Once it clears the last layer, an answer comes out.
In detail
What "no path back" actually means
Trace the arrows in a diagram of a neural network and you'll find two kinds: arrows that only point forward, and arrows that loop back to themselves or an earlier point. Feed-forward is the structure with only the first kind.
With no path leading back, the order of calculation stays simple. Sweep from the first layer to the last exactly once and the answer is out, and the same input always produces the same calculation in the same order. There's nothing being held onto from one step to reuse in the next.
This simplicity pays off in speed. With no path leading back, there's no need to wait on the previous step's result, so many inputs, or many positions within a sentence, can be calculated all at once. That makes full use of hardware built to run thousands of calculations in parallel.
How is it different from something with memory?
Working with text means holding onto what came earlier. Structures with a path leading back solve this through recurrence — the result of processing one word gets fed back into the network itself, so when it looks at the next word, it also carries a trace of what came before.
The catch is that order has to be respected: the earlier word has to finish before the next one can start. As a sentence gets longer, the wait stretches out proportionally, and the trace of earlier words fades the further back it goes.
Feed-forward is the opposite. There's no channel for holding on to things, but every position can be computed all at once. Today's language models hand the job of tracking relationships between positions off to attention, and let feed-forward handle the rest, splitting the advantages between the two approaches.
Widen it, then narrow it back down
The feed-forward layer inside a transformer block has a distinctive shape. It takes the incoming bundle of numbers, blows it up much larger, then folds it back down to its original width. In the original design it widened fourfold, and today's models use a range of ratios, often somewhere around three times.
The reason for widening is to make room. Features that were tangled together in a narrow space spread apart once there's more room. From there, unneeded signals get suppressed, and what's left gets gathered and folded back down.
A large share of a model's numbers live in this expand-and-fold process. If attention is the part that gathers relevant context, feed-forward is the part that turns the gathered material into an actual judgment. A lot of research points to this being where most of a model's learned knowledge actually sits.
Applied separately at each position, by the same rule
A transformer's feed-forward layer gets applied separately to every single piece in a sentence. The first piece and the tenth piece don't look at each other — each one gets computed by the exact same rule on its own.
That means the calculation can be run all at once no matter how many pieces there are, since no piece has to wait on any other. Any exchange of information between pieces has already happened in the step right before this one.
Attention lets pieces exchange ideas with each other; feed-forward lets each one organize its own thoughts; then the next layer lets them exchange ideas again. This alternation between the two steps is the basic skeleton of today's models.
More precisely
A feed-forward network refers to any network whose connections contain no cycles. A multilayer perceptron is a standard example, and convolutional networks used for images qualify too, since they have no cycles either. The feed-forward layer inside a transformer block uses this structure twice in a row as its basic form, with an activation function inserted once between the widening and the narrowing. The original design used a function that turns negative values into zero; today's models often use variants that shrink the signal smoothly or filter it as if opening and closing a gate.
The analogy breaks down in a few places. A cafeteria station only adds a side dish onto the tray; a network layer replaces the incoming numbers with an entirely new set of numbers on the way out. In the basic structure with no skip path, the earlier layer's values don't stick around unchanged. And a cafeteria ladle stays the same every day, but the rule a layer applies keeps shifting slightly during training — it only locks in place once training finishes. The single-line setup is also different: modern models like transformers run a parallel channel that carries the original values straight through, skipping over layers, and a layer's result gets added onto that carried-through value instead of replacing it, which is actually closer to piling side dishes onto the tray.
Try it yourself
Common misconceptions
It's easy to think feed-forward is just the name of a layer inside a transformer, but actually it's a broader term for the whole family of neural network structures that have no path leading back. The transformer's feed-forward layer is just one example.
It's easy to get confused and think backpropagation means it isn't feed-forward anymore, but actually sending error backward is a calculation done only during training — the direction signals actually flow in stays one-way throughout.
It's easy to dismiss it as a simple layer that doesn't matter much, but actually more than half of a model's numbers live inside this layer.
One-line summary
Feed-forward means a signal flows only forward, and inside a transformer, it's the layer that turns what attention gathered into an actual judgment at each position.
Spotted an error or have a better analogy? Suggest an edit · Last updated 2026-10-17