Plain-language explainer
Gradient descent, explained
How does a neural network actually learn?
Training is a loop. The model makes a prediction, a loss function measures how wrong it was, and calculus tells every single weight which tiny nudge would have made the error smaller. All the weights take their nudge, and the loop repeats, billions of times. That is gradient descent: rolling downhill on an error landscape you can never see whole, one small step at a time. Nobody writes the rules. The rules are whatever weight values the descent settles into.
Last reviewed
Reading is the slow way. Start with a free lesson you can operate right now.
Start free: Predict the next word →Free, no code, no signup.
Then go deeper: Gradient descent: rolling downhill Locked
What people get wrong
- Engineers program the model's knowledge. They program the loop. The knowledge is the byproduct of running it on data.
- Training finds the perfect answer. It finds a low valley that works, not a guaranteed best one.
- The model keeps learning when you chat with it. After training, the weights are frozen. Your conversation changes the context, never the weights.
Where you see it in real products
- Headlines about training runs costing millions describe this loop at scale.
- The loss curves in AI lab posts are the downhill path, drawn over time.
- Fine-tuning services run the same loop, starting from the finished model and using your data.
Common questions
- What is a gradient, in plain terms?
- The direction that would make the error worse, measured for every weight at once. Training steps the other way. It answers the question: if I nudge this one number up, does the model get better or worse, and by how much.
- What is a learning rate?
- The size of each step. Too small and training crawls, too large and it overshoots and destabilises. Real training schedules warm the rate up and then decay it, and it is one of the few knobs that reliably decides whether a run succeeds or collapses.
- Does training ever reach the bottom?
- No, and it does not need to. Training stops when the error stops improving usefully on data the model has not seen. In a space with billions of dimensions there are many good-enough valleys, and finding one of them is the whole goal.
Related explainers
More in How models are trained
- When should you fine-tune a model, and when is prompting enough?
- Why did making models bigger keep making them better?
- What is RLHF, and why does a text predictor act like an assistant?
- How can a model learn from examples in the prompt without being retrained?
- What is model distillation, and why is the mini version so good?
One idea at a time, in your inbox
New lessons and explainers, written the way these pages are. Now and then, not daily, and never a sales sequence.
We email you a confirmation link first. Unsubscribe in one click, any time. Privacy.
Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.