LLM training pipeline

How a chat model like ChatGPT is built, per andrej-karpathy's Deep Dive. There are three stages, which he maps onto a textbook: exposition (pre-training), worked examples (supervised fine-tuning), practice problems (reinforcement learning).

1. Pre-training: next-token prediction on the internet

• The data is FineWeb-style filtering of Common Crawl down to about 44 TB, or 15 trillion tokens. Tokenization turns text into IDs from a vocabulary of 100,277 (GPT-4).
• Training loop: take a random window of tokens (up to the context length, e.g. 8,000) and have the network output a probability for all 100,277 candidates for the next token. The true next token is the label. Then adjust the model-parameters so that token becomes slightly more likely. This runs for every position of every window, in large parallel batches.
• Researchers watch one number, the loss: a single figure for how badly the model predicts; lower is better. In Karpathy's GPT-2 reproduction each update covers 1 million tokens and takes about 7 seconds. 32,000 updates is about 33 billion tokens. At 1% done the samples are local-grammar gibberish; after a day or two, coherent English.
• Compute means GPUs such as the H100, rented at about $3 per GPU per hour, 8 to a node, nodes stacked into data centers. Reproducing GPT-2 cost about $40,000 in 2019 and about $600 in 24 hours on one 8×H100 node for Karpathy ("could bring it down to $100"), thanks to better data, hardware and software. Frontier pre-training runs take months and tens of millions of dollars, which is why they happen rarely and models have a knowledge cutoff.
• The output is a base model, an internet-document simulator (lossy-compression-of-the-internet) that Karpathy calls "glorified autocomplete".

2. Supervised fine-tuning: becoming an assistant

The internet data gets swapped for conversations. Human labelers, following long instruction documents, write ideal assistant replies (modern sets are largely synthetic, human-seeded). Training continues on those, and the model "very rapidly" takes on the assistant persona. It's much cheaper than pre-training: about 3 hours against months. The result imitates the labelers, which is why Karpathy says you're talking to a statistical simulation of an OpenAI labeler.

3. Reinforcement learning: practice problems

For problems with checkable answers (math, code), sample thousands of attempts and train on the ones that reach the right answer. This is where reasoning emerges, as in DeepSeek-R1's "wait, let's re-evaluate" chains (models-need-tokens-to-think), and where a model can exceed its human examples (move-37). For unverifiable tasks, RLHF trains against a learned reward model, which gets gamed after a few hundred steps (rlhf-is-not-rl).

Training vs inference

Everything above is training. Using the model is inference: feed in tokens, sample the next one from the predicted distribution, append it, repeat. Sampling is random, so outputs are remixes of the training data: statistically similar, rarely verbatim. The weights stay fixed throughout.

Source: Deep Dive