From Predictor to Assistant

This is the real 867,072-parameter model from module 06. It was pretrained on Shakespeare with one goal: predict the next token. That is the same first stage every large language model goes through, just on trillions of tokens of web text instead of one play collection.

Give it an instruction and it does not follow it. It has no idea it is being asked anything; it simply continues in the style of what it read. Big base models behave the same way: ask one a question and it might answer, or it might write five more similar questions, because both are plausible continuations.

Output is sampled, so press the button again for a different continuation.

Chat apps show tidy speech bubbles, but the model only ever sees one stream of text. The conversation is flattened into a chat template: special marker tokens show where each message starts and ends and who wrote it. The model then writes the next part, the assistant's reply, and stops when it produces the end marker.

Edit the messages and watch the raw text update. The format shown is ChatML, used by several open models; each model family has its own variation. The system message holds standing instructions the user usually never sees.

The mini model's version

For the fine-tuned mini model in the next step, I used the simplest possible template: a line starting with "User:", then "Bot:", and a blank line to mark the end of the answer.

To make the fine-tuned version, I took the module 06 model and continued training it on about 30,000 short example conversations: an instruction from "User" and the correct answer from "Bot". This is supervised fine-tuning (SFT).

One detail matters: the model is graded only on the answer tokens (blue), not on the instruction. It is learning how to respond, not how to write questions.

Pick an instruction and some words and ask both models. The base model rambles; the fine-tuned one answers and stops. The table shows how often it gets each task exactly right, tested on 25 new examples per task. Counting and picking out words are easy. Shouting is hardest: in this model's Shakespeare vocabulary, capitalised words break into many small, rarely seen pieces.

Each answer is the model's single most likely continuation. Words outside the model's small vocabulary may come out misspelled.

Many questions have several acceptable answers, some much better than others. So people are shown two answers to the same prompt and asked which is better, over hundreds of thousands of pairs.

Those choices train a reward model: a model that scores any answer the way the raters would. The assistant is then tuned to produce answers that score highly. This is reinforcement learning from human feedback (RLHF); newer methods such as DPO skip the separate reward model but use the same kind of comparisons.

Try it: pick the better answer in each pair. The bars show what a simple reward model learns from your choices. Then try picking the longer answer every time, and watch it learn "longer is better". That is reward hacking: the assistant learns what raters reward, which is not always what they meant.

The answers here are examples written for this page, each tagged with a few simple traits the reward model can see.

The later stages are tiny compared with pretraining but change behaviour enormously. In OpenAI's 2022 InstructGPT study, people preferred answers from a 1.3-billion-parameter model tuned this way over the 175-billion-parameter GPT-3 base model, a model more than 100 times larger.

Labs differ in how they do stage 3. Anthropic's "Constitutional AI", for example, has a model critique and revise answers against a written set of principles, reducing how many human comparisons are needed.

Why it matters for users

Much of an assistant's "personality", what it refuses, how cautious or chatty it is, comes from these later stages rather than from the raw knowledge learned in pretraining.

This completes the Generative AI track. Next in the course: why AI runs on GPUs.