Interpretability

What this lesson shows

Explore real attention patterns / feature activations on curated inputs.

The question it opens with

Can we see what's happening inside?

The walkthrough, in the lesson's own words

• One layer's “thoughts”, mid-sentence. Looks like noise, is any of it readable?
• One worked example: watch a single concept light up.
• Your turn. Pick a concept, see what fires it, then turn it up.
• Not noise. Hidden in that wall of numbers is a feature: a handful of them that light up together for one human concept, here, “color”. Let's catch it in the act.
• A feature is a specific combination of those numbers, a “direction” in activation space. Like the word-space from Act 1, where directions carried meaning (king − man + woman landed near queen), the model's INTERNAL activations also have meaningful directions. It's a different space, these numbers live mid-network, not in the word embeddings, but the same idea.
• The “color words” feature ignores Anna, the numbers, the kites, and fires only on red and blue.
• That one number is the model's private flag for “this is a color”. Find the right number, and you can read a concept straight out of the box.
• Same feature: the number that fired as it READ now gets cranked up, and steers the model to WRITE about that concept.
• Off: an ordinary sentence. Now drag the dial up.
• Turn it up and the concept takes over the output, proof it's really wired in.
• A stronger signal flows through the layers → next-word scores shift → the concept takes over the output. That's proof it's really wired in, not painted on.
• Features fire on colors, numbers, names, even abstract ideas. Anthropic famously dialed up one feature until a model wouldn't stop talking about the Golden Gate Bridge.
• Inside the model each number is polysemantic: one unit fires for many unrelated things at once, so you can't just read it off. A sparse autoencoder pulls that tangle apart into thousands of cleaner features, each firing for a single concept like the ones here. Those features wire together into circuits that carry out a behavior step by step. The catch: a model has millions of features and most are still unnamed, so interpretability today is real but partial, a few rooms lit in a very large building.
• A lab announces it found the feature for deception and can dial a model's honesty up or down. What is actually going on, and what's the catch?
• They used tools like sparse autoencoders to isolate an internal direction that tracks that concept, then read it or turned it up, the same read-and-steer you just did. It is real leverage on a model's behavior. The catch: it covers a sliver of the millions of features inside, so steering one thing is not the same as understanding the whole model.
• Exhibit · illustrative activations, not a live probe.
• That one stays quiet, not part of this feature. The three that glow are.
• Three of them fired together: that's the “color” feature. Tap any cell to probe it.
• Tap the strip to scan it for hidden structure.

Key takeaway

Peek inside the box with honest, pre-computed traces.

What you can do after this lesson

You can explain interpretability: features inside a model track human concepts.

This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.