Workbench 02 · Inference, measured
Module 11 explains why the same model can be fast or slow, cheap or expensive. Here you measure it on your own device: the same small language model, the same four questions, run on the processor and on the graphics chip, at three levels of precision. Every number below is timed live in your browser; nothing is recorded or replayed.
| Setup | Download | Load | First token | Speed | Correct |
|---|---|---|---|---|---|
| Run the benchmark to fill this table. | |||||
Precision
fp32 stores every weight in 32 bits, fp16 in 16, q4 in about 4. Cutting bits cuts the download and the memory the model needs by up to eight times, and usually speeds it up, because moving weights through memory is the slow part. The question is what it costs in answers: watch the Correct column.
Processor against graphics chip
A graphics chip does thousands of multiplications at once, which is what a model mostly does. For a model this small the gap can be narrower than you expect, because there is less work to spread out and the GPU has its own start-up costs. Bigger models widen the gap.
First token against speed
The wait for the first word is what makes a chatbot feel slow; tokens a second is how fast it writes after that. Serving systems tune them separately: batching many users together helps speed but can lengthen the first-token wait.