Feed-Forward
A neural network structure where signals only flow forward
- Feed-forward means a signal flows only from input toward output. There's no path leading back.
- A layer's result only ever moves on to the next layer. Once a spot has been passed, it's never revisited.
- There's no channel for holding onto earlier information, so anything that requires remembering order gets handed off to a different mechanism.
- Feed-forward layers sit inside transformer blocks too. More than half of a model's numbers live there.
- Error does flow backward during training, but that's only the direction of a calculation — the structure itself never changes.
Contents
1The analogy
Standing in a school cafeteria line, you only move forward with your tray. From the rice station to the soup station, from soup to side dishes, from side dishes to dessert. If you don't like the side dish, you can't walk back to the rice station. The line only flows one way.
Each station does one fixed job. The soup station only ladles soup, no matter what you've picked up before it. Once your tray clears the last station, the meal is complete.
A feed-forward network looks just like this. A bundle of numbers moves through layers the way a tray moves through stations, and each layer adds its own share of the calculation. Once it clears the last layer, an answer comes out.
2In detail
What "no path back" actually means
Trace the arrows in a diagram of a neural network and you'll find two kinds: arrows that only point forward, and arrows that loop back to themselves or an earlier point. Feed-forward is the structure with only the first kind.
With no path leading back, the order of calculation stays simple. Sweep from the first layer to the last exactly once and the answer is out, and the same input always produces the same calculation in the same order. There's nothing being held onto from one step to reuse in the next.
This simplicity pays off in speed. With no path leading back, the calculations inside a layer can be split up and run all at once, which makes full use of hardware built to run thousands of calculations in parallel.
How is it different from something with memory?
Working with text means holding onto what came earlier. Structures with a path leading back solve this through recurrence — the result of processing one word gets fed back into the network itself, so when it looks at the next word, it also carries a trace of what came before.
The catch is that order has to be respected: the earlier word has to finish before the next one can start. As a sentence gets longer, the wait stretches out proportionally, and the trace of earlier words fades the further back it goes.
Feed-forward is the opposite. There's no channel for holding on to things, but every position can be computed all at once. Today's language models hand the job of tracking relationships between positions off to attention, and let feed-forward handle the rest, splitting the advantages between the two approaches.
Widen it, then narrow it back down
The feed-forward layer inside a transformer block has a distinctive shape. It takes the incoming bundle of numbers, blows it up much larger, then folds it back down to its original width. Usually it widens by around four times before narrowing again.
The reason for widening is to make room. Features that were tangled together in a narrow space spread apart once there's more room. From there, unneeded signals get suppressed, and what's left gets gathered and folded back down.
A large share of a model's numbers live in this expand-and-fold process. If attention is the part that gathers relevant context, feed-forward is the part that turns the gathered material into an actual judgment. A lot of research points to this being where most of a model's learned knowledge actually sits.
Applied separately at each position, by the same rule
A transformer's feed-forward layer gets applied separately to every single piece in a sentence. The first piece and the tenth piece don't look at each other — each one gets computed by the exact same rule on its own.
That means the calculation can be run all at once no matter how many pieces there are, since no piece has to wait on any other. Any exchange of information between pieces has already happened in the step right before this one.
Attention lets pieces exchange ideas with each other; feed-forward lets each one organize its own thoughts; then the next layer lets them exchange ideas again. This alternation between the two steps is the basic skeleton of today's models.
3More precisely
A feed-forward network refers to any network whose connections contain no cycles. A multilayer perceptron is the simplest example, and convolutional networks used for images qualify too, since they have no cycles either. The feed-forward layer inside a transformer block uses this structure twice in a row, with an activation function that suppresses negative values inserted once in between the widening and the narrowing.
The analogy breaks down in a few places. A cafeteria station only adds a side dish onto the tray; a network layer replaces the incoming numbers with an entirely new set of numbers on the way out. The earlier layer's values don't stick around unchanged. And a cafeteria ladle stays the same every day, but the rule a layer applies keeps shifting slightly during training — it only locks in place once training finishes. The single-line setup is also different: real models run a parallel channel that carries the original values straight through, skipping over layers, and a layer's result gets added onto that carried-through value instead of replacing it. That extra channel is what keeps a very deep stack of feed-forward layers trainable at all.
4Try it yourself
5Common misconceptions
It's easy to think feed-forward refers to one specific named layer inside a transformer, but actually it refers to the whole family of structures that have no path leading back.
It's easy to get confused and think backpropagation means it isn't feed-forward anymore, but actually sending error backward is a calculation done only during training — the direction signals actually flow in stays one-way throughout.
It's easy to dismiss it as a simple layer that doesn't matter much, but actually more than half of a model's numbers live inside this layer.
7One-line summary
In shortFeed-forward means a signal flows only forward, and inside a transformer, it's the layer that turns what attention gathered into an actual judgment at each position.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02