One accuracy number hides which mistakes the model makes.
The digit reader from module 04 is tested here on 400 handwritten digits it never saw in training. Each row is the digit that was actually written; each column is what the model read. The diagonal is everything it got right.
Click any square to see those digits. Off-diagonal squares are the mistakes, and the pattern matters more than the total: confusing a 4 with a 9 is understandable, but in a real system some mistakes cost far more than others.
In finance
A fraud model that is "99% accurate" may simply be flagging nothing, if 99% of transactions are legitimate. That is why fraud and credit models are judged on precision (how many flags are real) and recall (how many real cases are caught), not accuracy alone.
The distortion settings in step 3 also change this matrix.
A good model is not just accurate. Its confidence should mean something.
Every answer comes with a confidence: the probability the model gives its top choice. The chart groups answers by that confidence. Each bar shows how often the answers in the group were actually right; the dashed line shows how sure the model said it was. For a perfectly calibrated model they match: when it says 90%, it is right 90% of the time.
A bar below its dashed line (shown red) means overconfidence: the model is sure, but wrong more often than it claims. On clean data almost everything lands in the 99%+ group and is right. Try step 3 and come back. The list shows the mistakes it made with the most confidence.
This is what a hallucination is
A language model inventing a fact is the same failure: a fluent, confident answer with nothing behind it. Calibration is how researchers measure it, and why "the model sounded sure" is never evidence.
Change the data a little, and watch accuracy fall while confidence stays high.
The model was trained on clean, centred, white-on-black digits. Real inputs are rarely that tidy. Add static, rotate, shift or invert the test digits and see what happens to accuracy (blue) and to the model's average confidence (amber).
Inverting the colours is the starkest: every digit is still perfectly readable to you, but the model collapses, and it often stays confident while doing so. Scroll back up: the confusion matrix and confidence chart now show the shifted data too.
It happens to every model
A credit model trained before a recession, a fraud model facing a new scam, a chatbot asked about events after its training cutoff: all are distribution shift. Models must be monitored after launch, not just tested once.
Bias can be measured, not just argued about.
This runs a word embedding association test, a method published in Science in 2017 (Caliskan, Bryson and Narayanan), on the module 05 embeddings. For each target word it measures whether the word sits closer to one set of attribute words or the other.
The first two tests are sanity checks: flowers sit closer to pleasant words than insects do, and musical instruments closer than weapons. The method finds associations everyone expects.
The same method finds that common male first names sit closer to career words, and female names closer to family words. Nobody programmed that in: it was learned from billions of words of real text, along with everything else.
Why it matters
Embeddings like these feed search, résumé screening and recommendation systems. A score near 0 means little association; around 0.8 or more is conventionally a large effect. Measuring it is the first step to reducing it, and in lending, hiring and insurance, anti-discrimination law makes it a legal issue, not just an ethical one.
Before trusting any AI system, ask these questions.
| Question | How it is checked |
|---|---|
| Is it accurate on data it never saw? | Held-out test sets (modules 01 to 04) |
| Which mistakes does it make? | Confusion matrix, precision and recall (step 1) |
| Does it know when it is unsure? | Calibration (step 2) |
| Does it survive messy real-world input? | Distribution shift and stress tests (step 3) |
| Is it fair across groups? | Bias measurement (step 4) |
| Are its answers grounded? | Citation checks for RAG (module 12) |
| Can it be tricked into harm? | Red-teaming and prompt injection tests (module 13) |
| Is it still working next month? | Monitoring after launch |
Public leaderboards rank large language models on benchmark exams, but they need care. Popular test questions leak onto the web and into training data, so a model can score well by having seen the answers ("contamination"). And benchmarks saturate: once every model scores near 100%, the test stops telling them apart.
That is why serious teams build their own evaluations from their own data and failure cases, review samples by hand, and red-team: deliberately try to make the system fail before users do.
End of the course
That completes all fourteen modules: from fitting a line, through neural networks, language models and diffusion, to the GPUs, servers, retrieval and agents that run modern AI, and finally how to judge whether any of it can be trusted.
Try it on real models: the evaluation arena runs small open models through 22 auto-graded questions on your device, hallucination and prompt-injection tests included.