What a neural layer actually does

What this lesson shows

A matrix multiply is a layer; it rotates / scales the vector.

The question it opens with

What is the 'feed-forward' block actually doing?

The walkthrough, in the lesson's own words

• This row of weights multiplies the input. Will its output land up, near zero, or down?
• Watch the row multiply-and-add, term by term, into one output number.
• One weight row, multiplied-and-added with the input, the dot product from 1.3.
• Stack one row per output. That stack is a matrix.
• Pick an output cell. Watch it sum one weight row times the input.
• A layer is built on a matrix, the core move you just watched.
• Right, it lands modestly up. The big + weight pulls hard, the − weight subtracts, and the row settles into one new number. That reshaping is what a layer does.
• It actually lands up, not down. The − weight subtracts, but the big + weight wins, so the sum comes out positive. A layer reshapes, scaling some directions up and others down.
• Close, but the big + weight pulls it clearly above zero. The − weight only trims the sum. A layer reshapes the vector, scaling some directions up and others down.
• Top row times the input, added up, the dot product from 1.3. One row in, one number out.
• A weight can be negative, that direction gets pushed down instead of up, so its term subtracts from the sum. That's how a layer can scale some directions up and others down.
• Made-up weights and input. The multiply-and-add is exactly what a real layer does; only the numbers are invented.
• You already used this exact multiply to build Query, Key and Value from a word's embedding. The same move, with a different learned grid, is also
• A layer looks up stored facts about a word, like a dictionary entry it retrieves.
• A layer stores no words. It just multiplies the word's vector through a grid of numbers, reshaping it. Those numbers are the only thing training changes.
• One extra ingredient we’ll meet later: a nonlinear step that lets a stack of layers do far more than any single matrix could.
• People say a model’s “knowledge” lives in its weights, and that fine-tuning a model just nudges those numbers. What are those weights, concretely?
• They are the numbers in the weight matrices, exactly the rows you just multiplied. Every layer reshapes a word’s vector by multiplying it through a matrix, so training and fine-tuning are nothing more than adjusting those weights so the reshaping comes out useful.
• One weight row times the input vector. Predict the output.

Key takeaway

The scary matmul is just a transformation.

What you can do after this lesson

You can explain that a neural layer is a matrix that reshapes a vector.

Where it leads: Stack many layers and the signal gets lost…

This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.