#pretraining
-
Stanford CS229 I Machine Learning I Building Large Language Models (LLMs)
Yann Dubois (Stanford PhD student, co-author of Alpaca and AlpacaFarm) gives a 1h44m guest lecture in CS229 (Summer 2024, 2.8M views) on building a ChatGPT-style model. He splits the work into five components (architecture, loss, data, evaluation, systems) and skips architecture on purpose: academia works on it, but "what matters in practice is mostly the three other topics".
-
Deep Dive into LLMs like ChatGPT
Karpathy's 3.5-hour general-audience deep dive (9.7M views, Feb 2025) covers the full ChatGPT pipeline: pretraining (internet → tokens → next-token prediction), supervised fine-tuning (conversations → an assistant persona), and reinforcement learning (practice problems → emergent reasoning). Throughout, he maps the stages onto how children learn from textbooks: exposition, worked examples, practice problems.