Learning starts with a simple question: how wrong is my guess?
Each dot is a home: its size and its price. A model here is just a line, and it has two settings, called parameters: how steep it is, and where it starts. Move the sliders to fit the dots.
The thin red lines are the errors: the gap between what the line predicts and the real price. Square each gap so big misses count more, average them, and you get one number for how wrong the line is. That is the loss, shown at the top right.
The whole game
Training a model means finding the parameter values with the lowest loss. You are doing it by eye. The rest of this page is about how a computer does it with no eyes at all.
The homes are generated example data, made with a real trend plus random noise.
Every setting has an error. Together they form a landscape.
Picture testing every possible line and colouring each one by its loss: dark for low, bright for high. That map is the loss landscape. Across goes the slope, up goes the starting value.
Your line from step 1 is the white dot. Click anywhere on the map to try that line, and watch the dots above update. The lowest point, the bottom of the valley, is the best possible line.
Why not just check everything?
With two parameters, a computer could. But a modern language model has billions of parameters, so its landscape has billions of directions. You could never map it. You need a way to walk downhill without seeing the whole map.
Feel which way is downhill, take a step, repeat.
The computer starts with a random line. Then, using a bit of calculus, it works out the gradient: which direction, for each parameter, makes the loss go up fastest. It steps the opposite way. That is gradient descent, and almost every AI model is trained with some version of it.
Press Train and watch the path (left) slide into the valley while the line (right) settles onto the dots. The chart below tracks the loss falling with each step.
Learning rate
The learning rate sets how big each step is. Try it: too small and training crawls; too big and it overshoots the valley, bouncing from side to side or flying off completely. Picking this well is a large part of training real models.
A model that fits the training data perfectly can be useless on new data.
Now the model is a curve, and the slider gives it more flexibility (more parameters). At the far left it is a straight line; at the far right it can bend through every point.
Push the slider right and the training error keeps falling, all the way to almost zero. Then press Show test data: new points from the same source that the model never saw. The wiggly curve misses them badly. It memorised the noise instead of learning the trend. That is overfitting.
The chart below
Blue is training error, amber is test error. Training error only goes down. Test error usually falls, bottoms out, then climbs (try New data a few times). The sweet spot is the flexibility where test error is lowest, and that is why every serious model is judged on data it was not trained on.
A chatbot learns exactly like this line did. Just with far more knobs.
Recorded while training the model used in How an LLM works: 3,500 steps on Shakespeare. Training loss is measured on one random batch per checkpoint, so it is noisy.
The tiny language model elsewhere in this Lab was trained with the same gradient descent you just watched, only across 867,072 parameters at once instead of 2. Its loss measures how surprised it is by the real next piece of Shakespeare.
Look at the two curves. Training loss (blue) keeps falling. Validation loss (amber), measured on text the model never trained on, drops quickly and then flattens: by the end, extra training was mostly memorising. That is step 4's overfitting, showing up in a real run.
Scale
Large models repeat this loop over trillions of tokens and billions of parameters, on thousands of chips for weeks. The maths of each step is the same as nudging your line.