Draw a digit. To the computer, it is 784 numbers.
Whatever you draw is shrunk to a 28 by 28 grid, the format the digit reader was trained on. Each square holds one number: how bright that spot is. Hover the grid to read individual values, or press Show numbers.
That grid of 784 numbers is all the network ever sees. It has no idea what a "line" or a "loop" is. Everything it knows about shapes, it has to learn from examples.
Colour photos
A colour photo is three grids stacked together, one each for red, green and blue. A single 1080p frame from the Lab's camera feeds is about 6 million numbers.
Your drawing is centred and scaled the same way the training digits were, so it does not matter where on the pad you draw.
A filter slides across the image, lighting up wherever its pattern appears.
A filter is a tiny 3 by 3 grid of weights. It is laid over each spot of the image in turn: multiply the nine pixels by the nine weights, add them up, and that is one pixel of the output. This sliding is called a convolution.
A vertical-edge filter has positive weights on one side and negative on the other, so it only produces a big number where brightness changes from left to right. Switch filters and watch different parts of your digit light up (blue) or go negative (amber). Hover the output to see the actual arithmetic.
The key idea
These four filters were designed by hand. A convolutional network learns its own filters, using the same gradient descent from module 01, picking whichever patterns help it tell the digits apart.
Learned filters, then filters of filters, then a decision.
The first layer runs 8 filters it learned itself across your drawing; the small squares above each map are those filters. Some respond to edges in one direction, some to strokes, some to the background. Each map is then shrunk to half size, keeping only the strongest responses.
The second layer combines all 8 maps with 16 more filters, picking out larger shapes such as curves, corners and crossings. A final layer weighs all of that evidence into a score for each digit. Try drawing a 4 that is almost a 9, or a 1 with a long hook, and watch the scores move.
98.8% sounds great. Here is the other 1.2%.
Digits and training data from the MNIST dataset by Yann LeCun, Corinna Cortes and Christopher J.C. Burges, used under the CC BY-SA 3.0 licence. The network was trained for this page.
The reader was tested on 10,000 digits it never trained on, written by different people. It got - right. These are some of the ones it got wrong, with what the writer meant and what the network read. Click one to load it into the reader above.
Many are honestly hard: a 9 whose loop makes it look like an 8, a 1 with a hook that reads as a 7. A few are plain mistakes a person would never make.
It cannot say "I don't know"
Draw a letter, a smiley face or a scribble above. The network will still pick a digit, sometimes with high confidence, because those 10 answers are the only ones it has. Every AI model has a version of this problem: it answers within the world it was trained on, even when the input comes from outside it.
The same idea reads cheques, drives cars and runs the Lab's camera feeds.
| Network | Parameters | What it does |
|---|---|---|
| This page's digit reader | 14,410 | 10 digits |
| LeNet-5 (1998) | about 60,000 | Read handwritten amounts on bank cheques |
| AlexNet (2012) | about 60 million | 1,000 kinds of object; started the deep learning boom |
| ResNet-50 (2015) | 25.6 million | 1,000 kinds of object, far more accurately |
| YOLOX-tiny, in the Lab feeds | 5.06 million | Finds and boxes 80 kinds of object, live |
Convolutional networks were reading the amounts on bank cheques in the late 1990s, using a design called LeNet. In 2012 a much bigger one, AlexNet, trained on graphics cards, won an image recognition contest by a wide margin. That result kicked off the modern deep learning era, and it is why AI now runs on GPUs.
The detector running on the Lab's live feeds works the same way, at a larger scale. Its filters are stacked dozens of layers deep, and instead of one answer per image it predicts a label and a box for every object it finds.
Next
Module 05 covers embeddings: how networks turn images, words and documents into lists of numbers where similar things sit close together.