Lossy compression of the internet
andrej-karpathy's core metaphor for a pre-trained LLM: pre-training chops the internet into tokens and compresses it into roughly a terabyte of parameters, "a lossy, probabilistic zip file." You get the gist back, with the exact wording lost. Recall scales with how often something appears online, and knowledge stops at the pre-training cutoff (months to a year old).
In How I use LLMs it becomes the model's self-introduction: "Hi, I'm ChatGPT. I am a one-terabyte zip file. My knowledge comes from the internet, which I read in its entirety about six months ago and I only remember vaguely... my winning personality was programmed, by example, by human labelers at OpenAI." Post-training attaches "a smiley face" (the persona) but adds little knowledge.
Practical consequences:
- Common, stable facts (caffeine in an Americano, DayQuil ingredients) are safe to ask directly, but still worth spot-checking.
- A bare model has no calculator, browser, or interpreter. Anything recent, niche or exact needs a tool, which is the logic of tool-selection-by-freshness.
- It pairs with params-vs-context-memory: the zip file is the vague memory, and the context window is the part you control.
How the compression happens (next-token training, loss, compute) is in llm-training-pipeline; what the "zip file" physically is, in model-parameters.
Linked from
Andrej KarpathyLLM training pipelineModel parameters (weights)Parameters vs context memory