What are the parts of a transformer block?

Two. An attention layer that lets tokens exchange information, and a feed-forward network that processes each token on its own. Both are wrapped in normalisation and residual connections that carry the original signal forward. Stack that block dozens of times and you have the model.

What do the feed-forward layers do?

They hold most of the parameters, and most of the stored knowledge. Attention decides which context matters; the feed-forward network is where the model recalls what it knows about the result. Interpretability work keeps finding specific facts and features living there.

Is the transformer used only for language?

No. Images cut into patches, audio cut into frames, and protein sequences all feed the same architecture. That generality is why one engineering effort paid off across many fields at once, and why the same hardware serves all of them.

Plain-language explainer

The transformer, explained

What is a transformer, and how does the architecture actually work?

A transformer is a stack of identical blocks, each doing two things. First, attention: every token looks at the earlier tokens and pulls in what it needs, so meaning flows between words. Second, a feed-forward network transforms each token on its own, where most of the model's learned knowledge lives. A running representation of each token passes through the stack, refined a little per block, until the top of the stack is sharp enough to score every possible next token.

Last reviewed July 25, 2026

Reading is the slow way. Start with a free lesson you can operate right now.

Start free: How attention blends meaning →

Free, no code, no signup.

Then go deeper: The transformer block, assembled & stacked Locked

What people get wrong

A transformer is one big tangled network. It is the same small two-part block, repeated dozens of times in a stack.
Attention is the whole story. Attention routes information, but most parameters sit in the feed-forward layers that transform it.
Understanding happens in one pass of reading. Each block re-mixes the tokens again, so the representation sharpens layer by layer.

Where you see it in real products

The T in GPT stands for transformer.
Nearly every modern AI model, chat, code, image and audio included, is built on this architecture.
Model cards list layer counts and sizes: those numbers describe this exact stack.

Common questions

What are the parts of a transformer block?: Two. An attention layer that lets tokens exchange information, and a feed-forward network that processes each token on its own. Both are wrapped in normalisation and residual connections that carry the original signal forward. Stack that block dozens of times and you have the model.
What do the feed-forward layers do?: They hold most of the parameters, and most of the stored knowledge. Attention decides which context matters; the feed-forward network is where the model recalls what it knows about the result. Interpretability work keeps finding specific facts and features living there.
Is the transformer used only for language?: No. Images cut into patches, audio cut into frames, and protein sequences all feed the same architecture. That generality is why one engineering effort paid off across many fields at once, and why the same hardware serves all of them.

Related explainers

Attention Query, Key, Value Positional encoding Softmax Next-token prediction Mixture of Experts

One idea at a time, in your inbox

New lessons and explainers, written the way these pages are. Now and then, not daily, and never a sales sequence.

We email you a confirmation link first. Unsubscribe in one click, any time. Privacy.

Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.