A model never sees letters or words. It sees numbered pieces.
Before anything else, text is cut into tokens: common words stay whole, rarer words are split into pieces, and spaces usually ride along at the front of a word. Each token is really just an ID number from a fixed list, its vocabulary.
The top panel is the real tokenizer used by GPT-4o, with about 200,000 pieces in its vocabulary. The bottom one belongs to the small model on this page, which learned only 512 pieces from Shakespeare, so it splits most words into smaller bits.
Why it matters
Models are priced, limited and measured in tokens, not words. A rough rule for English with big tokenizers is about three quarters of a word per token. Try a long or unusual word and watch it break apart.
Every token becomes a list of numbers that places it in a space of meaning.
The ID is looked up in a table, and each token gets an embedding: a list of numbers (128 here, thousands in large models). Each square in the strip is one of those numbers, blue for positive and amber for negative.
Nobody chooses these numbers. They are learned during training, and tokens used in similar ways end up with similar lists. The chips under the strip are the tokens this model considers closest.
The map
The map squeezes all 128 dimensions down to two so we can look at them. A lot of detail is lost, but you can still see clusters: punctuation in one area, capitals in another. Your sentence's tokens are highlighted.
The model also adds a second learned list for each position (1st, 2nd, 3rd...), otherwise it could not tell "dog bites man" from "man bites dog".
Each token looks back at earlier tokens and decides which ones matter.
This is the step that made transformers work. For every token, the model asks: which earlier tokens should I pay attention to? It scores every earlier token, turns the scores into percentages, and blends their information together.
Click a token in the row. The earlier tokens light up in proportion to how much attention it pays them. These are the model's actual numbers for your text, not an illustration.
The grid
Rows are the token doing the looking and columns are the tokens being looked at. The empty upper half is deliberate: a token is never allowed to see tokens that come after it, because when generating text those do not exist yet.
Layers and heads
This happens in 4 heads at once, each free to look for something different, and the whole thing repeats through 4 layers, each building on the last. Large models use dozens of layers and heads.
The output is not a word. It is a probability for every token in the vocabulary.
After the last layer, the model produces one score for each of its 512 tokens. A function called softmax turns those scores into probabilities that add up to 100%. The bars show the ten most likely next tokens.
Temperature
Temperature reshapes those probabilities before one is picked. Low temperature sharpens them, so the model almost always takes its top guess: safe and repetitive. High temperature flattens them, giving long shots a chance: more creative and more nonsense. Drag the slider and watch the bars change.
Note what this means: the model never "knows" the answer. It always produces a spread of possibilities, and something else decides which one to use.
Writing is just this, repeated: predict, pick, append, predict again.
To write, the model predicts the next token, one is picked using the temperature from step 4, it is added to the end, and the whole longer text goes back in to predict the token after that. Every chatbot reply is this loop running very fast.
Each highlighted piece was added by the model. The log underneath shows what it picked and how likely that pick was.
Context window
This model can only read its last 64 tokens. Once the text is longer, the oldest tokens fall out of view and it simply forgets them. Large models have context windows of hundreds of thousands of tokens or more, which is why they can follow a whole document.
Because it learned only from Shakespeare, it writes in that style, mostly with real words and convincing formatting but little meaning. That gap is exactly what scale closes.
Same recipe, vastly more of everything.
| Model | Parameters | Context | Training text |
|---|---|---|---|
| This page | about 0.9 million | 64 tokens | about 0.5 million tokens (Shakespeare) |
| GPT-2 (2019) | 1.5 billion | 1,024 tokens | about 40 GB of web text |
| GPT-3 (2020) | 175 billion | 2,048 tokens | about 300 billion tokens |
| Today's frontier models | Usually not disclosed | Hundreds of thousands to over a million tokens | Trillions of tokens |
GPT-2 and GPT-3 figures are from their published papers. Companies generally no longer publish parameter counts for their largest models.
Everything on this page, tokens, embeddings, attention, next-token prediction and the loop, is how the big models work too. What changes is size: more parameters to store patterns, far more text to learn from, and longer context.
What turns it into an assistant
A model trained only to continue text is not yet a helpful assistant. Chatbots go through further training on examples of good conversations and on human feedback about which answers are better, which teaches them to follow instructions and answer questions instead of just carrying on writing.
Where the cost goes
Every one of those billions of parameters is used for every token generated. That is why running these models takes specialised chips and why answers are priced per token. That is the subject of a later module on how models are served.