The model reads your whole question at once, then writes its answer one token at a time.
A chatbot answer has two phases. First, prefill: every token of your question is processed in parallel in one big batch of math, which GPUs are excellent at. Then decode: the answer is produced one token at a time, and each token needs the whole model read from memory again (module 09).
Drag the sliders. Even a long question is read in a fraction of a second, but writing a long answer takes many seconds. That is why chatbots show text streaming in word by word, and why the delay before the first word ("time to first token") is tracked separately.
Estimates for one user, assuming 50% of peak compute when reading and 70% of peak memory speed when writing. Pick the model, precision and number of GPUs in the bar above; they apply to the whole page.
Every conversation keeps a memory on the GPU that grows with every token.
Attention (module 06) lets each new token look back at every earlier one. To avoid recomputing everything, the server stores two vectors per layer for every token so far, called the KV cache. For Llama 3.1 70B that is about 320 KB per token at 16-bit.
So a single 8,000-token conversation needs about 2.6 GB, and a full 128,000-token one about 42 GB, more than half an H100 on its own. Drag the sliders and watch the GPUs fill up. Each square is 1% of total memory.
The real limit on users
How many people a server can handle at once is usually set by this memory, not by compute. A lot of serving engineering, such as vLLM's "paged attention", exists to pack these caches in tightly without waste.
One read of the model can serve many users at once. That is what makes AI affordable.
Writing one token means reading all the model's weights from memory. But once the weights are flowing through the chip, multiplying them for 32 users costs barely more time than for 1. So servers group many users' next tokens into one batch.
Drag the batch size. Total output (blue) climbs steeply while each user's speed (amber) barely drops, until the GPUs run out of compute or the conversation memory from step 2 runs out.
The trade-off
Bigger batches mean cheaper tokens but slower answers for each person. Every AI provider tunes this balance, which is why the same model can be offered as a fast premium tier and a cheaper slower one.
Assumes conversations of the length set in step 2. Simplified model: real servers also mix prefill and decode, and reach somewhat lower efficiency.
Let a small model guess ahead, and the big model check several guesses in one go.
Step 1 showed that writing is slow because every token needs a full read of the big model. Speculative decoding gets around this: a small, fast "draft" model guesses the next few tokens, then the big model checks all the guesses in a single pass, which costs about the same as writing one token.
Every guess up to the first wrong one is kept, and the big model supplies the correct token at that point. The answer is exactly what the big model would have written on its own, just faster.
Press Simulate to watch it: green tokens were accepted guesses, red marks a rejected guess and blue is the token the big model wrote itself. Other common tricks are quantization (the precision buttons at the top) and prompt caching, which reuses the memory from step 2 when many requests share the same opening text.
Put it together: what does one answer cost to produce?
Uses the server and batch size from the steps above. Divide what the GPUs cost per hour by how many tokens they produce in that hour and you get the cost per token, the number every AI company watches.
The Server busy slider matters more than people expect: traffic comes in waves, and a GPU waiting for requests still costs money. Change the model size, precision or batch size and watch the unit cost move by 10 times or more.
Why this matters
These are compute costs only. Providers also pay for staff, data centres, networking and the training runs from module 10, and they price their APIs above this. But the shape of the economics, memory-bound, batch-hungry and very sensitive to utilisation, is the same.
Measure it yourself: the inference benchmark times a small model on your processor and graphics chip at three precisions.
Next in the course: how AI looks things up, in module 12.