Training cost has a simple rule of thumb: six operations, per parameter, per token.
For every token of training text, the model runs forward (about 2 operations per parameter, as in module 09) and then backpropagation runs backward (about 4 more). So training costs roughly 6 × parameters × tokens.
Pick a real model or drag the sliders. GPT-3, for example: 175 billion parameters times 300 billion tokens times 6 is about 3 × 1023 operations, matching the figure in the original paper.
Why so many tokens?
Research in 2022 (the "Chinchilla" study) found models were undertrained: for a given budget, a smaller model trained on more text often wins. Modern models are trained on 10 to 20 trillion tokens or more.
Training needs far more memory than running a model. About 16 bytes for every parameter.
Uses the model chosen in step 1. Activation memory depends heavily on batch size and settings, so it is left out; real runs need more.
Running a model needs only its weights. Training keeps much more in memory at once:
Weights (2 bytes each), the gradients from backpropagation (2 bytes), and the optimizer's bookkeeping: Adam, the standard training method, keeps a high-precision copy of every weight plus two running averages (12 bytes). That is about 16 bytes per parameter before counting anything else.
For a 405-billion-parameter model that is around 6.5 terabytes, the memory of more than 80 H100s, just to hold the training state. The work has to be split across many GPUs.
Three ways to share one model across many GPUs, usually all used at once.
Each square is one GPU. Change the three numbers and watch how the work is divided.
Data parallel: several copies of the model each train on different text at the same time, then average what they learned. Simple, but every copy needs the whole model.
Tensor parallel: each layer's big matrix multiplications are cut into slices, one per GPU. GPUs must swap results constantly, so this only works between GPUs inside one server or rack with very fast links.
Pipeline parallel: the layers are split into stages, like an assembly line. GPU 1 runs the first layers and hands its output to GPU 2, and so on.
Meta trained Llama 3.1 405B on 16,384 GPUs by combining all three, plus a fourth split for very long texts.
From one chip to a building that draws as much power as a city.
Power includes the servers, networking and cooling around each GPU (about 1.4 kW per H100-class GPU all-in). Costs are rough estimates for H100-class systems. A US home uses about 1.2 kW on average.
Put it together: how long, how much, and how often it breaks.
Uses the model from step 1. No real run reaches a GPU's full rated speed: time goes to moving data between chips and waiting. Meta reported 38% to 43% for Llama 3.1, so the default is 40%.
With 16,384 H100s at 40%, Llama 3.1 405B works out to about 27 million GPU-hours, in the same range as the 30.8 million Meta reported, which also included time lost to restarts.
Things break
With thousands of GPUs, something fails every few hours: a GPU, a memory chip, a cable. Meta logged 419 unexpected interruptions in 54 days of Llama 3 training. Runs survive by saving checkpoints of the whole model regularly and restarting from the last one, which is why reliability engineering matters as much as raw speed.
Next in the course: how models are served to millions of users.