Model parameters (weights)

The parameters are the numbers inside a neural network. Input tokens get mixed with them through a large fixed expression of multiplications, additions and similar operations, and the result is a probability for every possible next token. andrej-karpathy's image: "knobs on a DJ set". Training means turning the knobs until the predictions match the statistics of the training data.

• A fresh network starts with random parameters and outputs gibberish. Each training update nudges every parameter slightly so the correct next token gets a higher probability.
• The network is a fixed, stateless function from input to output with no memory. He warns that its "neurons" are far simpler than biological ones.
• A visualized toy Transformer has about 85,000 parameters; GPT-2 about 1.5 billion; current frontier models "a lot closer to a trillion or several hundred billion". In How I use LLMs: a 1 TB file is roughly one trillion parameters.
• Chatting never changes them: "all of that is just inference, there's no more training. Those parameters are held fixed." Only the next training run changes the weights.
• Pre-training squeezes the internet into the weights (lossy-compression-of-the-internet), so recall from parameters is "vague recollection" compared with text in the context window (params-vs-context-memory).

The context window is a separate limit set at training time: GPT-2 saw at most 1,024 tokens, and modern models see hundreds of thousands to a million.

Publishing the parameters is what makes a model open-weights.

Sources: Deep Dive · How I use LLMs