Deep Dive into LLMs like ChatGPT

TL;DR

• Karpathy's 3.5-hour general-audience deep dive (9.7M views, Feb 2025): the full ChatGPT pipeline — pretraining (internet → tokens → next-token prediction), supervised fine-tuning (conversations → an assistant persona), and reinforcement learning (practice problems → emergent reasoning) — mapped throughout onto how children learn from textbooks: exposition, worked examples, practice problems.
• The demystifying frame: a base model is a "lossy compression of the internet" (Llama 3.1 405B ≈ a zip file of 15T tokens), an SFT assistant is "a neural network simulation of a data labeler at OpenAI" following labeling instructions — "you're not talking to a magical AI, you're talking to an average labeler… a simulation of that kind of person," and hallucinations happen because the training set answers "who is X?" confidently, so the model imitates the confident style even when an internal uncertainty neuron knows better.
• Two memory systems: knowledge in parameters is "vague recollection" (something you read a month ago); the context window is "working memory" — directly accessible. Practical corollaries: paste source text into the prompt instead of relying on recall, and use tools (web search, code interpreter) to refresh working memory rather than trusting mental arithmetic.
• Models need tokens to think: a fixed, small amount of compute happens per token, so reasoning must be spread across many tokens — why chain-of-thought answers beat answer-first labels, why models can't count ("how many dots" fails in one token; use code succeeds), why spelling tasks fail (they see tokens, not characters), and the residual weirdness of 9.11 > 9.9 (activations light up on Bible-verse neurons).
• RL is where thinking emerges: DeepSeek-R1's "wait, wait, wait — let's re-evaluate" chains of thought were discovered, not programmed — no human could write them because human and LLM cognition differ ("we are not in a good position to create these token sequences… the LLM needs to discover what works for it"). AlphaGo's move 37 is the template: RL isn't capped at human imitation. But RLHF is not RL — a learned reward model is gameable (the top pelican joke becomes "the the the the"), so you can only run it a few hundred steps; verifiable domains (math, code) allow indefinite RL.
• Closing stance: Swiss-cheese capabilities — brilliant at Olympiad problems, randomly dumb in holes — so "use them as tools in the toolbox, check their work, and own the product of your work."

Key moments

• [00:02] The question — what exactly is behind the ChatGPT text box; general-audience mental models with "cognitive psychological implications" throughout.
• [01:02] Pretraining data — FineWeb as the representative pipeline: Common Crawl (2.7B pages) → URL filtering, text extraction, language filtering (65% English), PII removal → ~44TB / 15T tokens; "you can get a USB stick for a terabyte" (frame_0005 shows the raw concatenated text tapestry).
• [07:48] Tokenization — bits → bytes → BPE; GPT-4's 100,277-symbol vocabulary; tokens as "unique emojis," explored live in TikTokenizer.
• [15:17] Training loop — windows of tokens predict the next; loss as the one number researchers watch; his lm.c GPT-2 reproduction: $40k in 2019 → ~$600 in 24h on an 8×H100 node (~$3/GPU/hr on Lambda), "could bring it down to $100" (frame_0077, frame_0080).
• [43:01] Base model = token simulator — Llama 3.1 405B base as "glorified autocomplete": recites Wikipedia verbatim (regurgitation from over-sampled high-quality sources), invents parallel-universe 2024 elections past its cutoff, but in-context learning (few-shot translation) and a conversation-formatted prompt can fake an assistant.
• [60:23] Post-training — swap the dataset for conversations (months of pretraining vs ~3 hours of SFT); chat protocol tokens (im_start etc.) encode turns; InstructGPT's human labelers follow hundreds of pages of labeling instructions; modern sets (UltraChat) are largely synthetic but human-seeded.
• [77:00] What you're talking to — "a neural network simulation of a data labeler": ask ChatGPT for Paris landmarks and you get a statistical imitation of a skilled contractor's 20 minutes of research.
• [80:33] Hallucinations and the fix — confident training answers teach confident guessing; Meta's Llama 3 recipe: interrogate the model on generated factual questions, and where it consistently fails, add "I don't know" training examples — wiring the internal uncertainty neuron to words. Mitigation two: tool use — search tokens pause inference, stuff results into context.
• [100:00] Vague recollection vs working memory — parameters vs context window; paste the chapter of Pride and Prejudice instead of asking from memory.
• [101:47] Knowledge of self is cooked-up — "what model are you" is nonsensical; Falcon claims to be OpenAI's GPT-3 not from training theft but hallucinated self-identity; Olmo hardcodes identity with 240 conversations; system messages do the rest.
• [107:00] Models need tokens to think — fixed compute per forward pass; the two math answers demo (answer-first label trains single-token guessing; distributed reasoning works); "Emily buys 23 apples" breaks single-token arithmetic; counting dots fails, use code fixes it; spelling fails because tokens hide characters (strawberry's three Rs); 9.11 vs 9.9 tied to Bible-verse neurons.
• [131:00] RL as practice problems — sample thousands of solutions per prompt, train on the ones reaching verified answers; SFT initializes into the vicinity, "RL is where everything gets dialed in."
• [148:55] DeepSeek-R1 — the paper that made RL-for-LLMs public: accuracy climbs with RL steps, response length grows, and chains of thought emerge — "wait, wait, wait, that's an aha moment" — "it's rediscovering what happens in your head… there is no human who can hardcode this."
• [162:07] AlphaGo and move 37 — supervised imitation tops out below Lee Sedol; RL exceeds him; move 37 (1-in-10,000 human probability) as the template for LLMs finding thinking strategies humans wouldn't — maybe even a non-English internal language.
• [168:26] RLHF and its ceiling — reward model = neural simulator of human preference orderings (5,000 judgments instead of a billion); works, then RL games it into adversarial nonsense ("the the the the" scores 1.0), so you crop after a few hundred updates: "RLHF is RL, but not RL in the magical sense… it lacks magic."
• [190:00] What's coming — native multimodality (audio/image tokens), long-running supervised agents ("human-to-agent ratios"), computer use, and the open problem of test-time training beyond ever-longer context windows.
• [195:15] Staying current — LM Arena (with his caveat it's "become a little bit gamed"; Sonnet underrated at #14), AI News by swyx, X; run models via together.ai, hyperbolic (base models), LM Studio locally.
• [210:24] Closing — "they will randomly do dumb things… use them for inspiration, for first draft, but always check and verify — and own the product of your work."

Hook microscope (0-10s)

• Frames: 20 at 2 fps
• Word-level transcript (27 words):

  [  0.00s] Hi
  [  0.94s] everyone
  [  2.00s] So
  [  2.24s] I've
  [  2.24s] wanted
  [  2.36s] to
  [  2.48s] make
  [  2.60s] this
  [  2.78s] video
  [  2.94s] for
  [  3.16s] a
  [  3.24s] while
  [  3.52s] It
  [  4.04s] is
  [  4.22s] a
  [  4.52s] comprehensive
  [  4.90s] but
  [  5.70s] general
  [  6.50s] audience
  [  7.00s] introduction
  [  7.42s] to
  [  8.12s] large
  [  8.44s] language
  [  8.64s] models
  [  8.98s] like
  [  9.50s] Chats
  [  9.70s] GPT

No hook engineering whatsoever — the anti-YouTube open of a 3.5-hour lecture: "Hi everyone, so I've wanted to make this video for a while. It is a comprehensive but general audience introduction to large language models." The first frame is literally a blank ChatGPT text box (frame_0001) — the entire video is structured as an answer to "what is behind this text box?" Authority is assumed rather than performed, and the 9.7M views came from the author's name and the promise of completeness, not from the first ten seconds.

Editorial profile

• Shots: 80
• Cuts/min: 0.38
• Mean shot length: 158.54s
• Median shot length: 2.37s
• Talking-head ratio: n/a (opencv not installed)

Single-take screencast lecture (webcam picture-in-picture, bottom-left) over two alternating surfaces: a hand-drawn Excalidraw infinite canvas for theory and a live browser for practice (TikTokenizer, FineWeb, Hyperbolic base-model playground, HuggingFace inference, DeepSeek, LM Studio); no cuts that matter, no B-roll, no graphics package — 24 chapters and the canvas itself are the only structure.

Quotable moments

• [50:15] "You can think of the 405 billion parameters as a kind of compression of the internet… but it's not a lossless compression, it's a lossy compression. We're left with a gestalt of the internet."
• [77:28] "You're not talking to a magical AI. You're talking to an average labeler… an instantaneous simulation of that kind of person."
• [83:04] "These are statistical token tumblers… just trying to sample the next token in the sequence."
• [99:40] "Knowledge in the parameters of the neural network is a vague recollection; the knowledge in the tokens that make up the context window is the working memory."
• [118:03] "Models need tokens to think. Distribute your computation across many tokens… or whenever you can, lean on tools."
• [151:57] "It's rediscovering what happens in your head, not what you put down on the solution. And there is no human who can hardcode this stuff in the ideal assistant response."
• [185:03] "RLHF is RL, but it's not RL in the magical sense… you can't run it indefinitely — the optimization will start to game the reward model."
• [210:58] "Use them as tools in the toolbox. Check their work and own the product of your work."

Entities mentioned

• People: andrej-karpathy, Lee Sedol, Elon Musk (mentioned), swyx (AI News)
• Companies: OpenAI, Anthropic, Google/DeepMind, Meta, DeepSeek, HuggingFace, Nvidia, Lambda, Together.ai, Hyperbolic, Allen AI, Common Crawl, Scale AI
• Tools / products: ChatGPT, GPT-2, Llama 3.1 405B, DeepSeek-R1, AlphaGo, FineWeb, TikTokenizer, Excalidraw, lm.c, InstructGPT, UltraChat, Olmo, Falcon 7B, LM Studio, LM Arena, AI News
• Places: none (screencast)

Concepts surfaced

• lossy-compression-of-the-internet: a base model's parameters are a gestalt, not a database — recall quality tracks how often something appeared in training; regurgitation for over-sampled sources (Wikipedia), hallucination for rare ones.
• labeler-simulation: an SFT assistant statistically imitates the human data labelers (and their company's labeling instructions) who wrote its conversations — the right mental model for "what am I talking to."
• params-vs-context-memory: vague recollection vs working memory; paste don't recall; tools exist to refresh working memory. The smart-zone-dumb-zone framing's other half.
• models-need-tokens-to-think: fixed compute per token forces distributing reasoning across tokens; explains chain-of-thought, counting/spelling failures, and answer-first labels being training poison.
• emergent-reasoning-via-rl: chains of thought (DeepSeek-R1's "aha moments") are discovered under verifiable rewards, not written by humans — human and LLM cognition differ, so imitation caps at human level (AlphaGo's supervised-vs-RL plot; move 37).
• rlhf-is-not-rl: a learned reward model is a gameable neural net — adversarial inputs score 1.0 — so RLHF is a bounded fine-tune, while verifiable-domain RL can run indefinitely. Related: quality-gated-data-omission as another judge-gaming pattern.
• swiss-cheese-capabilities: brilliance with random holes (9.11 vs 9.9, strawberry Rs) — check work, own output; the epistemics behind claims-vs-testing applied to LLMs themselves.
• hallucination-mitigation-pipeline: interrogate the model to map its knowledge boundary, train "I don't know" on the gaps, then add tool use — the Llama 3 factuality recipe.

Transcript

Source: captions.

[00:03] hi everyone so I've wanted to make this video for a while it is a comprehensive
[00:05] video for a while it is a comprehensive but General audience introduction to
[00:08] but General audience introduction to large language models like Chachi PT and
[00:11] large language models like Chachi PT and what I'm hoping to achieve in this video
[00:13] what I'm hoping to achieve in this video is to give you kind of mental models for
[00:15] is to give you kind of mental models for thinking through what it is that this
[00:17] thinking through what it is that this tool is it is obviously magical and
[00:20] tool is it is obviously magical and amazing in some respects it's uh really
[00:22] amazing in some respects it's uh really good at some things not very good at
[00:24] good at some things not very good at other things and there's also a lot of
[00:25] other things and there's also a lot of sharp edges to be aware of so what is
[00:28] sharp edges to be aware of so what is behind this text box you can put
[00:30] behind this text box you can put anything in there and press enter but uh
[00:32] anything in there and press enter but uh what should we be putting there and what
[00:35] what should we be putting there and what are these words generated back how does
[00:37] are these words generated back how does this work and what what are you talking
[00:38] this work and what what are you talking to exactly so I'm hoping to get at all
[00:40] to exactly so I'm hoping to get at all those topics in this video we're going
[00:42] those topics in this video we're going to go through the entire pipeline of how
[00:44] to go through the entire pipeline of how this stuff is built but I'm going to
[00:46] this stuff is built but I'm going to keep everything uh sort of accessible to
[00:48] keep everything uh sort of accessible to a general audience so let's take a look
[00:51] a general audience so let's take a look at first how you build something like
[00:52] at first how you build something like chpt and along the way I'm going to talk
[00:54] chpt and along the way I'm going to talk about um you know some of the sort of
[00:57] about um you know some of the sort of cognitive psychological implications of
[01:00] cognitive psychological implications of the tools okay so let's build Chachi PT
[01:03] the tools okay so let's build Chachi PT so there's going to be multiple stages
[01:04] so there's going to be multiple stages arranged sequentially the first stage is
[01:07] arranged sequentially the first stage is called the pre-training stage and the
[01:10] called the pre-training stage and the first step of the pre-training stage is
[01:11] first step of the pre-training stage is to download and process the internet now
[01:13] to download and process the internet now to get a sense of what this roughly
[01:15] to get a sense of what this roughly looks like I recommend looking at this
[01:17] looks like I recommend looking at this URL here so um this company called
[01:20] URL here so um this company called hugging face uh collected and created
[01:23] hugging face uh collected and created and curated this data set called Fine
[01:26] and curated this data set called Fine web and they go into a lot of detail on
[01:29] web and they go into a lot of detail on this block post on how how they
[01:30] this block post on how how they constructed the fine web data set and
[01:32] constructed the fine web data set and all of the major llm providers like open
[01:34] all of the major llm providers like open AI anthropic and Google and so on will
[01:36] AI anthropic and Google and so on will have some equivalent internally of
[01:39] have some equivalent internally of something like the fine web data set so
[01:41] something like the fine web data set so roughly what are we trying to achieve
[01:42] roughly what are we trying to achieve here we're trying to get ton of text
[01:44] here we're trying to get ton of text from the internet from publicly
[01:46] from the internet from publicly available sources so we're trying to
[01:48] available sources so we're trying to have a huge quantity of very high
[01:51] have a huge quantity of very high quality documents and we also want very
[01:53] quality documents and we also want very large diversity of documents because we
[01:55] large diversity of documents because we want to have a lot of knowledge inside
[01:57] want to have a lot of knowledge inside these models so we want large diversity
[01:59] these models so we want large diversity of high quality documents and we want
[02:02] of high quality documents and we want many many of them and achieving this is
[02:04] many many of them and achieving this is uh quite complicated and as you can see
[02:06] uh quite complicated and as you can see here takes multiple stages to do well so
[02:09] here takes multiple stages to do well so let's take a look at what some of these
[02:10] let's take a look at what some of these stages look like in a bit for now I'd
[02:12] stages look like in a bit for now I'd like to just like to note that for
[02:13] like to just like to note that for example the fine web data set which is
[02:15] example the fine web data set which is fairly representative what you would see
[02:17] fairly representative what you would see in a production grade application
[02:19] in a production grade application actually ends up being only about 44
[02:20] actually ends up being only about 44 terabyt of dis space um you can get a
[02:24] terabyt of dis space um you can get a USB stick for like a terabyte very
[02:25] USB stick for like a terabyte very easily or I think this could fit on a
[02:27] easily or I think this could fit on a single hard drive almost today so this
[02:29] single hard drive almost today so this is not a huge amount of data at the end
[02:32] is not a huge amount of data at the end of the day even though the internet is
[02:33] of the day even though the internet is very very large we're working with text
[02:35] very very large we're working with text and we're also filtering it aggressively
[02:37] and we're also filtering it aggressively so we end up with about 44 terabytes in
[02:39] so we end up with about 44 terabytes in this example so let's take a look at uh
[02:42] this example so let's take a look at uh kind of what this data looks like and
[02:45] kind of what this data looks like and what some of these stages uh also are so
[02:47] what some of these stages uh also are so the starting point for a lot of these
[02:48] the starting point for a lot of these efforts and something that contributes
[02:50] efforts and something that contributes most of the data by the end of it is
[02:53] most of the data by the end of it is Data from common crawl so common craw is
[02:56] Data from common crawl so common craw is an organization that has been basically
[02:57] an organization that has been basically scouring the internet since 2007 so as
[03:01] scouring the internet since 2007 so as of 2024 for example common CW has
[03:03] of 2024 for example common CW has indexed 2.7 billion web
[03:06] indexed 2.7 billion web pages uh and uh they have all these
[03:08] pages uh and uh they have all these crawlers going around the internet and
[03:10] crawlers going around the internet and what you end up doing basically is you
[03:11] what you end up doing basically is you start with a few seed web pages and then
[03:13] start with a few seed web pages and then you follow all the links and you just
[03:15] you follow all the links and you just keep following links and you keep
[03:16] keep following links and you keep indexing all the information and you end
[03:17] indexing all the information and you end up with a ton of data of the internet
[03:19] up with a ton of data of the internet over time so this is usually the
[03:21] over time so this is usually the starting point for a lot of the uh for a
[03:24] starting point for a lot of the uh for a lot of these efforts now this common C
[03:26] lot of these efforts now this common C data is quite raw and is filtered in
[03:28] data is quite raw and is filtered in many many different ways
[03:30] many many different ways so here they Pro they document this is
[03:33] so here they Pro they document this is the same diagram they document a little
[03:35] the same diagram they document a little bit the kind of processing that happens
[03:37] bit the kind of processing that happens in these stages so the first thing here
[03:40] in these stages so the first thing here is something called URL
[03:41] is something called URL filtering so what that is referring to
[03:44] filtering so what that is referring to is that there's these block
[03:50] lists of uh basically URLs that are or domains that uh you don't want to be
[03:53] domains that uh you don't want to be getting data from so usually this
[03:54] getting data from so usually this includes things like U malware websites
[03:57] includes things like U malware websites spam websites marketing websites uh
[03:59] spam websites marketing websites uh racist websites adult sites and things
[04:01] racist websites adult sites and things like that so there's a ton of different
[04:03] like that so there's a ton of different types of websites that are just
[04:04] types of websites that are just eliminated at this stage because we
[04:06] eliminated at this stage because we don't want them in our data set um the
[04:09] don't want them in our data set um the second part is text extraction you have
[04:11] second part is text extraction you have to remember that all these web pages
[04:12] to remember that all these web pages this is the raw HTML of these web pages
[04:14] this is the raw HTML of these web pages that are being saved by these crawlers
[04:17] that are being saved by these crawlers so when I go to inspect
[04:19] so when I go to inspect here this is what the raw HTML actually
[04:21] here this is what the raw HTML actually looks like you'll notice that it's got
[04:23] looks like you'll notice that it's got all this markup uh like lists and stuff
[04:27] all this markup uh like lists and stuff like that and there's CSS and all this
[04:29] like that and there's CSS and all this kind of stuff so this is um computer
[04:31] kind of stuff so this is um computer code almost for these web pages but what
[04:34] code almost for these web pages but what we really want is we just want this text
[04:36] we really want is we just want this text right we just want the text of this web
[04:37] right we just want the text of this web page and we don't want the navigation
[04:39] page and we don't want the navigation and things like that so there's a lot of
[04:40] and things like that so there's a lot of filtering and processing uh and heris
[04:43] filtering and processing uh and heris that go into uh adequately filtering for
[04:45] that go into uh adequately filtering for just their uh good content of these web
[04:48] just their uh good content of these web pages the next stage here is language
[04:51] pages the next stage here is language filtering so for example fine web
[04:54] filtering so for example fine web filters uh using a language classifier
[04:56] filters uh using a language classifier they try to guess what language every
[04:59] they try to guess what language every single web page is in and then they only
[05:00] single web page is in and then they only keep web pages that have more than 65%
[05:03] keep web pages that have more than 65% of English as an
[05:04] of English as an example and so you can get a sense that
[05:06] example and so you can get a sense that this is like a design decision that
[05:07] this is like a design decision that different companies can uh can uh take
[05:10] different companies can uh can uh take for themselves what fraction of all
[05:13] for themselves what fraction of all different types of languages are we
[05:14] different types of languages are we going to include in our data set because
[05:16] going to include in our data set because for example if we filter out all of the
[05:18] for example if we filter out all of the Spanish as an example then you might
[05:19] Spanish as an example then you might imagine that our model later will not be
[05:21] imagine that our model later will not be very good at Spanish because it's just
[05:23] very good at Spanish because it's just never seen that much data of that
[05:24] never seen that much data of that language and so different companies can
[05:26] language and so different companies can focus on multilingual performance to uh
[05:29] focus on multilingual performance to uh to a different degree as an example so
[05:31] to a different degree as an example so fine web is quite focused on English and
[05:33] fine web is quite focused on English and so their language model if they end up
[05:35] so their language model if they end up training one later will be very good at
[05:37] training one later will be very good at English but not may be very good at
[05:38] English but not may be very good at other
[05:40] other languages after language filtering
[05:42] languages after language filtering there's a few other filtering steps and
[05:43] there's a few other filtering steps and D duplication and things like that um
[05:47] D duplication and things like that um finishing with for example the pii
[05:49] finishing with for example the pii removal this is personally identifiable
[05:52] removal this is personally identifiable information so as an example addresses
[05:54] information so as an example addresses Social Security numbers and things like
[05:56] Social Security numbers and things like that you would try to detect them and
[05:58] that you would try to detect them and you would try to filter out those kinds
[05:59] you would try to filter out those kinds of web pages from the the data set as
[06:00] of web pages from the the data set as well so there's a lot of stages here and
[06:03] well so there's a lot of stages here and I won't go into full detail but it is a
[06:05] I won't go into full detail but it is a fairly extensive part of the
[06:07] fairly extensive part of the pre-processing and you end up with for
[06:09] pre-processing and you end up with for example the fine web data set so when
[06:11] example the fine web data set so when you click in on it uh you can see some
[06:13] you click in on it uh you can see some examples here of what this actually ends
[06:14] examples here of what this actually ends up looking like and anyone can download
[06:16] up looking like and anyone can download this on the huging phase web page and so
[06:19] this on the huging phase web page and so here are some examples of the final text
[06:21] here are some examples of the final text that ends up in the training set so this
[06:24] that ends up in the training set so this is some article about tornadoes in
[06:28] is some article about tornadoes in 2012 um so there's some t tadoes in 2020
[06:31] 2012 um so there's some t tadoes in 2020 in 2012 and what
[06:33] in 2012 and what happened uh this next one is something
[06:36] happened uh this next one is something about did you know you have two little
[06:39] about did you know you have two little yellow 9vt battery sized adrenal glands
[06:41] yellow 9vt battery sized adrenal glands in your body okay so this is some kind
[06:44] in your body okay so this is some kind of a odd medical
[06:47] of a odd medical article so just think of these as
[06:49] article so just think of these as basically uh web pages on the internet
[06:52] basically uh web pages on the internet filtered just for the text in various
[06:54] filtered just for the text in various ways and now we have a ton of text 40
[06:57] ways and now we have a ton of text 40 terabytes off it and that now is the
[06:59] terabytes off it and that now is the starting point for the next step of this
[07:01] starting point for the next step of this stage now I wanted to give you an
[07:03] stage now I wanted to give you an intuitive sense of where we are right
[07:04] intuitive sense of where we are right now so I took the first 200 web pages
[07:07] now so I took the first 200 web pages here and remember we have tons of them
[07:09] here and remember we have tons of them and I just take all that text and I just
[07:11] and I just take all that text and I just put it all together concatenate it and
[07:14] put it all together concatenate it and so this is what we end up with we just
[07:15] so this is what we end up with we just get this just just raw text raw internet
[07:19] get this just just raw text raw internet text and there's a ton of it even in
[07:21] text and there's a ton of it even in these 200 web pages so I can continue
[07:23] these 200 web pages so I can continue zooming out here and we just have this
[07:25] zooming out here and we just have this like massive tapestry of Text data and
[07:28] like massive tapestry of Text data and this text data has all these p patterns
[07:30] this text data has all these p patterns and what we want to do now is we want to
[07:32] and what we want to do now is we want to start training neural networks on this
[07:33] start training neural networks on this data so the neural networks can
[07:35] data so the neural networks can internalize and model how this text
[07:39] internalize and model how this text flows right so we just have this giant
[07:43] flows right so we just have this giant texture of text and now we want to get
[07:45] texture of text and now we want to get neural Nets that mimic it okay now
[07:48] neural Nets that mimic it okay now before we plug text into neural networks
[07:51] before we plug text into neural networks we have to decide how we're going to
[07:53] we have to decide how we're going to represent this text uh and how we're
[07:55] represent this text uh and how we're going to feed it in now the way our
[07:57] going to feed it in now the way our technology works for these neuron Lots
[07:59] technology works for these neuron Lots is that they expect
[08:00] is that they expect a one-dimensional sequence of symbols
[08:03] a one-dimensional sequence of symbols and they want a finite set of symbols
[08:06] and they want a finite set of symbols that are possible and so we have to
[08:08] that are possible and so we have to decide what are the symbols and then we
[08:10] decide what are the symbols and then we have to represent our data as
[08:12] have to represent our data as one-dimensional sequence of those
[08:14] one-dimensional sequence of those symbols so right now what we have is a
[08:16] symbols so right now what we have is a onedimensional sequence of text it
[08:19] onedimensional sequence of text it starts here and it goes here and then it
[08:21] starts here and it goes here and then it comes here Etc so this is a
[08:22] comes here Etc so this is a onedimensional sequence even though on
[08:24] onedimensional sequence even though on my monitor of course it's laid out in a
[08:26] my monitor of course it's laid out in a two-dimensional way but it goes from
[08:28] two-dimensional way but it goes from left to right and top to bottom right so
[08:30] left to right and top to bottom right so it's a one-dimensional sequence of text
[08:32] it's a one-dimensional sequence of text now this being computers of course
[08:34] now this being computers of course there's an underlying representation
[08:35] there's an underlying representation here so if I do what's called utf8 uh
[08:38] here so if I do what's called utf8 uh encode this text then I can get the raw
[08:41] encode this text then I can get the raw bits that correspond to this text in the
[08:44] bits that correspond to this text in the computer and that's what uh that looks
[08:47] computer and that's what uh that looks like this so it turns out that for
[08:50] like this so it turns out that for example this very first bar here is the
[08:53] example this very first bar here is the first uh eight bits as an
[08:56] first uh eight bits as an example so what is this thing right this
[08:59] example so what is this thing right this is um representation that we are looking
[09:02] is um representation that we are looking for uh in in a certain sense we have
[09:05] for uh in in a certain sense we have exactly two possible symbols zero and
[09:07] exactly two possible symbols zero and one and we have a very long sequence of
[09:10] one and we have a very long sequence of it right now as it turns out um this
[09:14] it right now as it turns out um this sequence length is actually going to be
[09:16] sequence length is actually going to be very finite and precious resource uh in
[09:19] very finite and precious resource uh in our neural network and we actually don't
[09:21] our neural network and we actually don't want extremely long sequences of just
[09:23] want extremely long sequences of just two symbols instead what we want is we
[09:26] two symbols instead what we want is we want to trade off uh this um symbol
[09:30] want to trade off uh this um symbol size uh of this vocabulary as we call it
[09:33] size uh of this vocabulary as we call it and the resulting sequence length so we
[09:35] and the resulting sequence length so we don't want just two symbols and
[09:36] don't want just two symbols and extremely long sequences we're going to
[09:39] extremely long sequences we're going to want more symbols and shorter sequences
[09:42] want more symbols and shorter sequences okay so one naive way of compressing or
[09:45] okay so one naive way of compressing or decreasing the length of our sequence
[09:46] decreasing the length of our sequence here is to basically uh consider some
[09:50] here is to basically uh consider some group of consecutive bits for example
[09:52] group of consecutive bits for example eight bits and group them into a single
[09:55] eight bits and group them into a single what's called bite so because uh these
[09:58] what's called bite so because uh these bits are either on or off if we take a
[10:00] bits are either on or off if we take a group of eight of them there turns out
[10:02] group of eight of them there turns out to be only 256 possible combinations of
[10:04] to be only 256 possible combinations of how these bits could be on or off and so
[10:06] how these bits could be on or off and so therefore we can re repesent this
[10:08] therefore we can re repesent this sequence into a sequence of bytes
[10:11] sequence into a sequence of bytes instead so this sequence of bytes will
[10:14] instead so this sequence of bytes will be eight times shorter but now we have
[10:16] be eight times shorter but now we have 256 possible symbols so every number
[10:19] 256 possible symbols so every number here goes from 0 to
[10:21] here goes from 0 to 255 now I really encourage you to think
[10:23] 255 now I really encourage you to think of these not as numbers but as unique
[10:25] of these not as numbers but as unique IDs or like unique symbols so maybe it's
[10:28] IDs or like unique symbols so maybe it's a bit more maybe it's better to actually
[10:30] a bit more maybe it's better to actually think of these to replace every one of
[10:32] think of these to replace every one of these with a unique Emoji you'd get
[10:34] these with a unique Emoji you'd get something like this so um we basically
[10:37] something like this so um we basically have a sequence of emojis and there's
[10:39] have a sequence of emojis and there's 256 possible emojis you can think of it
[10:41] 256 possible emojis you can think of it that way now it turns out that in
[10:45] that way now it turns out that in production for state-of-the-art language
[10:46] production for state-of-the-art language models uh you actually want to go even
[10:48] models uh you actually want to go even Beyond this you want to continue to
[10:50] Beyond this you want to continue to shrink the length of the sequence uh
[10:53] shrink the length of the sequence uh because again it is a precious resource
[10:55] because again it is a precious resource in return for more symbols in your
[10:58] in return for more symbols in your vocabulary and the way this is done is
[11:00] vocabulary and the way this is done is done by running what's called The Bite
[11:02] done by running what's called The Bite pair encoding algorithm and the way this
[11:04] pair encoding algorithm and the way this works is we're basically looking for
[11:06] works is we're basically looking for consecutive bytes or symbols that are
[11:10] consecutive bytes or symbols that are very common so for example turns out
[11:14] very common so for example turns out that the sequence 116 followed by 32 is
[11:17] that the sequence 116 followed by 32 is quite common and occurs very frequently
[11:19] quite common and occurs very frequently so what we're going to do is we're going
[11:20] so what we're going to do is we're going to group uh this um pair into a new
[11:24] to group uh this um pair into a new symbol so we're going to Mint a symbol
[11:27] symbol so we're going to Mint a symbol with an ID 256 and we're going to
[11:29] with an ID 256 and we're going to rewrite every single uh pair 11632 with
[11:33] rewrite every single uh pair 11632 with this new symbol and then can we can
[11:35] this new symbol and then can we can iterate this algorithm as many times as
[11:36] iterate this algorithm as many times as we wish and each time when we mint a new
[11:39] we wish and each time when we mint a new symbol we're decreasing the length and
[11:41] symbol we're decreasing the length and we're increasing the symbol size and in
[11:44] we're increasing the symbol size and in practice it turns out that a pretty good
[11:45] practice it turns out that a pretty good setting of um the basically the
[11:48] setting of um the basically the vocabulary size turns out to be about
[11:50] vocabulary size turns out to be about 100,000 possible symbols so in
[11:52] 100,000 possible symbols so in particular GPT 4 uses
[11:55] particular GPT 4 uses 100,
[11:57] 100, 277 symbols
[12:00] 277 symbols um and this process of converting from
[12:04] um and this process of converting from raw text into these symbols or as we
[12:07] raw text into these symbols or as we call them tokens is the process called
[12:10] call them tokens is the process called tokenization so let's now take a look at
[12:13] tokenization so let's now take a look at how gp4 performs tokenization conting
[12:16] how gp4 performs tokenization conting from text to tokens and from tokens back
[12:18] from text to tokens and from tokens back to text and what this actually looks
[12:20] to text and what this actually looks like so one website I like to use to
[12:22] like so one website I like to use to explore these token representations is
[12:25] explore these token representations is called tick tokenizer and so come here
[12:27] called tick tokenizer and so come here to the drop down and select CL 100 a
[12:29] to the drop down and select CL 100 a base which is the gp4 base model
[12:32] base which is the gp4 base model tokenizer and here on the left you can
[12:34] tokenizer and here on the left you can put in text and it shows you the
[12:36] put in text and it shows you the tokenization of that text so for example
[12:41] tokenization of that text so for example heo space
[12:44] heo space world so hello world turns out to be
[12:46] world so hello world turns out to be exactly two Tokens The Token hello which
[12:49] exactly two Tokens The Token hello which is the token with ID
[12:52] is the token with ID 15339 and the token space
[12:55] 15339 and the token space world that is the token 1
[12:58] world that is the token 1 1917 so um hello space world now if I
[13:02] 1917 so um hello space world now if I was to join these two for example I'm
[13:05] was to join these two for example I'm going to get again two tokens but it's
[13:06] going to get again two tokens but it's the token H followed by the token L
[13:10] the token H followed by the token L world without the
[13:12] world without the H um if I put in two Spa two spaces here
[13:15] H um if I put in two Spa two spaces here between hello and world it's again a
[13:17] between hello and world it's again a different uh tokenization there's a new
[13:19] different uh tokenization there's a new token 220
[13:22] token 220 here okay so you can play with this and
[13:24] here okay so you can play with this and see what happens here also keep in mind
[13:27] see what happens here also keep in mind this is not uh this is case sensitive so
[13:29] this is not uh this is case sensitive so if this is a capital H it is something
[13:31] if this is a capital H it is something else or if it's uh hello world then
[13:35] else or if it's uh hello world then actually this ends up being three tokens
[13:37] actually this ends up being three tokens instead of just two
[13:43] tokens yeah so you can play with this and get an sort of like an intuitive
[13:45] and get an sort of like an intuitive sense of uh what these tokens work like
[13:47] sense of uh what these tokens work like we're actually going to loop around to
[13:48] we're actually going to loop around to tokenization a bit later in the video
[13:50] tokenization a bit later in the video for now I just wanted to show you the
[13:51] for now I just wanted to show you the website and I wanted to uh show you that
[13:54] website and I wanted to uh show you that this text basically at the end of the
[13:56] this text basically at the end of the day so for example if I take one line
[13:58] day so for example if I take one line here this is what GT4 will see it as so
[14:01] here this is what GT4 will see it as so this text will be a sequence of length
[14:04] this text will be a sequence of length 62 this is the sequence here and this is
[14:08] 62 this is the sequence here and this is how the chunks of text correspond to
[14:12] how the chunks of text correspond to these symbols and again there's 100,
[14:17] these symbols and again there's 100, 27777 possible symbols and we now have
[14:19] 27777 possible symbols and we now have one-dimensional sequences of those
[14:22] one-dimensional sequences of those symbols so um yeah we're going to come
[14:24] symbols so um yeah we're going to come back to tokenization but that's uh for
[14:26] back to tokenization but that's uh for now where we are okay so what I've done
[14:28] now where we are okay so what I've done now is I've taken this uh sequence of
[14:30] now is I've taken this uh sequence of text that we have here in the data set
[14:32] text that we have here in the data set and I have re-represented it using our
[14:34] and I have re-represented it using our tokenizer into a sequence of tokens and
[14:37] tokenizer into a sequence of tokens and this is what that looks like now so for
[14:40] this is what that looks like now so for example when we go back to the Fine web
[14:42] example when we go back to the Fine web data set they mentioned that not only is
[14:44] data set they mentioned that not only is this 44 terab of dis space but this is
[14:46] this 44 terab of dis space but this is about a 15 trillion token sequence of um
[14:51] about a 15 trillion token sequence of um in this data set and so here these are
[14:54] in this data set and so here these are just some of the first uh one or two or
[14:56] just some of the first uh one or two or three or a few thousand here I think uh
[14:59] three or a few thousand here I think uh tokens of this data set but there's 15
[15:01] tokens of this data set but there's 15 trillion here uh to keep in mind and
[15:04] trillion here uh to keep in mind and again keep in mind one more time that
[15:06] again keep in mind one more time that all of these represent little text
[15:07] all of these represent little text chunks they're all just like atoms of
[15:10] chunks they're all just like atoms of these sequences and the numbers here
[15:12] these sequences and the numbers here don't make any sense they're just uh
[15:13] don't make any sense they're just uh they're just unique IDs okay so now we
[15:17] they're just unique IDs okay so now we get to the fun part which is the uh
[15:20] get to the fun part which is the uh neural network training and this is
[15:21] neural network training and this is where a lot of the heavy lifting happens
[15:23] where a lot of the heavy lifting happens computationally when you're training
[15:25] computationally when you're training these neural networks so what we do here
[15:28] these neural networks so what we do here in this this step is we want to model
[15:31] in this this step is we want to model the statistical relationships of how
[15:32] the statistical relationships of how these tokens follow each other in the
[15:34] these tokens follow each other in the sequence so what we do is we come into
[15:36] sequence so what we do is we come into the data and we take Windows of tokens
[15:40] the data and we take Windows of tokens so we take a window of tokens uh from
[15:43] so we take a window of tokens uh from this data fairly
[15:45] this data fairly randomly and um the windows length can
[15:49] randomly and um the windows length can range anywhere anywhere between uh zero
[15:52] range anywhere anywhere between uh zero tokens actually all the way up to some
[15:54] tokens actually all the way up to some maximum size that we decide on uh so for
[15:57] maximum size that we decide on uh so for example in practice you could see a
[15:59] example in practice you could see a token with Windows of say 8,000 tokens
[16:01] token with Windows of say 8,000 tokens now in principle we can use arbitrary
[16:04] now in principle we can use arbitrary window lengths of tokens uh but uh
[16:07] window lengths of tokens uh but uh processing very long uh basically U
[16:11] processing very long uh basically U window sequences would just be very
[16:13] window sequences would just be very computationally expensive so we just
[16:15] computationally expensive so we just kind of decide that say 8,000 is a good
[16:17] kind of decide that say 8,000 is a good number or 4,000 or 16,000 and we crop it
[16:20] number or 4,000 or 16,000 and we crop it there now in this example I'm going to
[16:22] there now in this example I'm going to be uh taking the first four tokens just
[16:25] be uh taking the first four tokens just so everything fits nicely so these
[16:28] so everything fits nicely so these tokens
[16:30] tokens we're going to take a window of four
[16:32] we're going to take a window of four tokens this bar view in and space single
[16:37] tokens this bar view in and space single which are these token
[16:39] which are these token IDs and now what we're trying to do here
[16:41] IDs and now what we're trying to do here is we're trying to basically predict the
[16:43] is we're trying to basically predict the token that comes next in the sequence so
[16:46] token that comes next in the sequence so 3962 comes next right so what we do now
[16:49] 3962 comes next right so what we do now here is that we call this the context
[16:52] here is that we call this the context these four tokens are context and they
[16:54] these four tokens are context and they feed into a neural
[16:56] feed into a neural network and this is the input to the
[16:58] network and this is the input to the neural network
[17:00] neural network now I'm going to go into the detail of
[17:02] now I'm going to go into the detail of what's inside this neural network in a
[17:03] what's inside this neural network in a little bit for now it's important to
[17:05] little bit for now it's important to understand is the input and the output
[17:06] understand is the input and the output of the neural net so the input are
[17:09] of the neural net so the input are sequences of tokens of variable length
[17:12] sequences of tokens of variable length anywhere between zero and some maximum
[17:14] anywhere between zero and some maximum size like 8,000 the output now is a
[17:18] size like 8,000 the output now is a prediction for what comes next so
[17:21] prediction for what comes next so because our vocabulary has
[17:24] because our vocabulary has 100277 possible tokens the neural
[17:27] 100277 possible tokens the neural network is going to Output exactly that
[17:28] network is going to Output exactly that many numbers
[17:29] many numbers and all of those numbers correspond to
[17:31] and all of those numbers correspond to the probability of that token as coming
[17:34] the probability of that token as coming next in the sequence so it's making
[17:36] next in the sequence so it's making guesses about what comes
[17:37] guesses about what comes next um in the beginning this neural
[17:40] next um in the beginning this neural network is randomly initialized so um
[17:43] network is randomly initialized so um and we're going to see in a little bit
[17:44] and we're going to see in a little bit what that means but it's a it's a it's a
[17:47] what that means but it's a it's a it's a random transformation so these
[17:48] random transformation so these probabilities in the very beginning of
[17:50] probabilities in the very beginning of the training are also going to be kind
[17:51] the training are also going to be kind of random uh so here I have three
[17:54] of random uh so here I have three examples but keep in mind that there's
[17:55] examples but keep in mind that there's 100,000 numbers here um so the
[17:58] 100,000 numbers here um so the probability of this token space
[18:00] probability of this token space Direction neural network is saying that
[18:02] Direction neural network is saying that this is 4% likely right now 11799 is 2%
[18:06] this is 4% likely right now 11799 is 2% and then here the probility of 3962
[18:08] and then here the probility of 3962 which is post is 3% now of course we've
[18:11] which is post is 3% now of course we've sampled this window from our data set so
[18:14] sampled this window from our data set so we know what comes next we know and
[18:16] we know what comes next we know and that's the label we know that the
[18:18] that's the label we know that the correct answer is that 3962 actually
[18:20] correct answer is that 3962 actually comes next in the sequence so now what
[18:23] comes next in the sequence so now what we have is this mathematical process for
[18:26] we have is this mathematical process for doing an update to the neural network we
[18:28] doing an update to the neural network we have the way of tuning it and uh we're
[18:31] have the way of tuning it and uh we're going to go into a little bit of of
[18:32] going to go into a little bit of of detail in a bit but basically we know
[18:35] detail in a bit but basically we know that this probability here of 3% we want
[18:38] that this probability here of 3% we want this probability to be higher and we
[18:41] this probability to be higher and we want the probabilities of all the other
[18:42] want the probabilities of all the other tokens to be
[18:44] tokens to be lower and so we have a way of
[18:46] lower and so we have a way of mathematically calculating how to adjust
[18:49] mathematically calculating how to adjust and update the neural network so that
[18:52] and update the neural network so that the correct answer has a slightly higher
[18:54] the correct answer has a slightly higher probability so if I do an update to the
[18:56] probability so if I do an update to the neural network now the next time I Fe
[18:59] neural network now the next time I Fe this particular sequence of four tokens
[19:01] this particular sequence of four tokens into neural network the neural network
[19:03] into neural network the neural network will be slightly adjusted now and it
[19:04] will be slightly adjusted now and it will say Okay post is maybe 4% and case
[19:07] will say Okay post is maybe 4% and case now maybe is
[19:09] now maybe is 1% and uh Direction could become 2% or
[19:12] 1% and uh Direction could become 2% or something like that and so we have a way
[19:14] something like that and so we have a way of nudging of slightly updating the
[19:16] of nudging of slightly updating the neuronet to um basically give a higher
[19:20] neuronet to um basically give a higher probability to the correct token that
[19:21] probability to the correct token that comes next in the sequence and now you
[19:23] comes next in the sequence and now you just have to remember that this process
[19:26] just have to remember that this process happens not just for uh this um token
[19:29] happens not just for uh this um token here where these four fed in and
[19:31] here where these four fed in and predicted this one this process happens
[19:34] predicted this one this process happens at the same time for all of these tokens
[19:36] at the same time for all of these tokens in the entire data set and so in
[19:38] in the entire data set and so in practice we sample little windows little
[19:40] practice we sample little windows little batches of Windows and then at every
[19:43] batches of Windows and then at every single one of these tokens we want to
[19:45] single one of these tokens we want to adjust our neural network so that the
[19:47] adjust our neural network so that the probability of that token becomes
[19:48] probability of that token becomes slightly higher and this all happens in
[19:50] slightly higher and this all happens in parallel in large batches of these
[19:52] parallel in large batches of these tokens and this is the process of
[19:54] tokens and this is the process of training the neural network it's a
[19:56] training the neural network it's a sequence of updating it so that it's
[19:59] sequence of updating it so that it's predictions match up the statistics of
[20:01] predictions match up the statistics of what actually happens in your training
[20:03] what actually happens in your training set and its probabilities become
[20:05] set and its probabilities become consistent with the uh statistical
[20:08] consistent with the uh statistical patterns of how these tokens follow each
[20:10] patterns of how these tokens follow each other in the data so let's now briefly
[20:12] other in the data so let's now briefly get into the internals of these neural
[20:13] get into the internals of these neural networks just to give you a sense of
[20:15] networks just to give you a sense of what's inside so neural network
[20:17] what's inside so neural network internals so as I mentioned we have
[20:20] internals so as I mentioned we have these inputs uh that are sequences of
[20:22] these inputs uh that are sequences of tokens in this case this is four input
[20:25] tokens in this case this is four input tokens but this can be anywhere between
[20:27] tokens but this can be anywhere between zero up to let's say 8,000 tokens in
[20:30] zero up to let's say 8,000 tokens in principle this can be an infinite number
[20:32] principle this can be an infinite number of tokens we just uh it would just be
[20:34] of tokens we just uh it would just be too computationally expensive to process
[20:36] too computationally expensive to process an infinite number of tokens so we just
[20:38] an infinite number of tokens so we just crop it at a certain length and that
[20:39] crop it at a certain length and that becomes the maximum context length of
[20:41] becomes the maximum context length of that uh
[20:43] that uh model now these inputs X are mixed up in
[20:46] model now these inputs X are mixed up in a giant mathematical expression together
[20:49] a giant mathematical expression together with the parameters or the weights of
[20:52] with the parameters or the weights of these neural networks so here I'm
[20:54] these neural networks so here I'm showing six example parameters and their
[20:57] showing six example parameters and their setting but in practice these uh um
[21:00] setting but in practice these uh um modern neural networks will have
[21:02] modern neural networks will have billions of these uh parameters and in
[21:05] billions of these uh parameters and in the beginning these parameters are
[21:06] the beginning these parameters are completely randomly set now with a
[21:09] completely randomly set now with a random setting of parameters you might
[21:11] random setting of parameters you might expect that this uh this neural network
[21:14] expect that this uh this neural network would make random predictions and it
[21:15] would make random predictions and it does in the beginning it's totally
[21:17] does in the beginning it's totally random predictions but it's through this
[21:20] random predictions but it's through this process of iteratively updating the
[21:23] process of iteratively updating the network uh as and we call that process
[21:25] network uh as and we call that process training a neural network so uh that the
[21:28] training a neural network so uh that the setting of these parameters gets
[21:29] setting of these parameters gets adjusted such that the outputs of our
[21:32] adjusted such that the outputs of our neural network becomes consistent with
[21:34] neural network becomes consistent with the patterns seen in our training
[21:36] the patterns seen in our training set so think of these parameters as kind
[21:39] set so think of these parameters as kind of like knobs on a DJ set and as you're
[21:41] of like knobs on a DJ set and as you're twiddling these knobs you're getting
[21:43] twiddling these knobs you're getting different uh predictions for every
[21:45] different uh predictions for every possible uh token sequence input and
[21:49] possible uh token sequence input and training in neural network just means
[21:51] training in neural network just means discovering a setting of parameters that
[21:53] discovering a setting of parameters that seems to be consistent with the
[21:55] seems to be consistent with the statistics of the training
[21:56] statistics of the training set now let me just give you an example
[21:59] set now let me just give you an example what this giant mathematical expression
[22:00] what this giant mathematical expression looks like just to give you a sense and
[22:02] looks like just to give you a sense and modern networks are massive expressions
[22:04] modern networks are massive expressions with trillions of terms probably but let
[22:06] with trillions of terms probably but let me just show you a simple example here
[22:09] me just show you a simple example here it would look something like this I mean
[22:10] it would look something like this I mean these are the kinds of Expressions just
[22:11] these are the kinds of Expressions just to show you that it's not very scary we
[22:14] to show you that it's not very scary we have inputs x uh like X1 x2 in this case
[22:17] have inputs x uh like X1 x2 in this case two example inputs and they get mixed up
[22:20] two example inputs and they get mixed up with the weights of the network w0 W1 2
[22:23] with the weights of the network w0 W1 2 3 Etc and this mixing is simple things
[22:27] 3 Etc and this mixing is simple things like multiplication addition addition
[22:30] like multiplication addition addition exponentiation division Etc and it is
[22:33] exponentiation division Etc and it is the subject of neural network
[22:34] the subject of neural network architecture research to design
[22:37] architecture research to design effective mathematical Expressions uh
[22:39] effective mathematical Expressions uh that have a lot of uh kind of convenient
[22:41] that have a lot of uh kind of convenient characteristics they are expressive
[22:43] characteristics they are expressive they're optimizable they're paralyzable
[22:45] they're optimizable they're paralyzable Etc and so but uh at the end of the day
[22:48] Etc and so but uh at the end of the day these are these are not complex
[22:50] these are these are not complex expressions and basically they mix up
[22:52] expressions and basically they mix up the inputs with the parameters to make
[22:54] the inputs with the parameters to make predictions and we're optimizing uh the
[22:58] predictions and we're optimizing uh the parameters of this neural network so
[23:00] parameters of this neural network so that the predictions come out consistent
[23:02] that the predictions come out consistent with the training set now I would like
[23:04] with the training set now I would like to show you an actual production grade
[23:06] to show you an actual production grade example of what these neural networks
[23:08] example of what these neural networks look like so for that I encourage you to
[23:10] look like so for that I encourage you to go to this website that has a very nice
[23:12] go to this website that has a very nice visualization of one of these
[23:14] visualization of one of these networks so this is what you will find
[23:16] networks so this is what you will find on this website and this neural network
[23:19] on this website and this neural network here that is used in production settings
[23:22] here that is used in production settings has this special kind of structure this
[23:24] has this special kind of structure this network is called the Transformer and
[23:27] network is called the Transformer and this particular one as an example has 8
[23:29] this particular one as an example has 8 5,000 roughly
[23:31] 5,000 roughly parameters now here on the top we take
[23:33] parameters now here on the top we take the inputs which are the token
[23:36] the inputs which are the token sequences and then information flows
[23:40] sequences and then information flows through the neural network until the
[23:42] through the neural network until the output which here are the logit softmax
[23:45] output which here are the logit softmax but these are the predictions for what
[23:46] but these are the predictions for what comes next what token comes
[23:49] comes next what token comes next and then here there's a sequence of
[23:52] next and then here there's a sequence of Transformations and all these
[23:54] Transformations and all these intermediate values that get produced
[23:56] intermediate values that get produced inside this mathematical expression s it
[23:59] inside this mathematical expression s it is sort of predicting what comes next so
[24:01] is sort of predicting what comes next so as an example these tokens are embedded
[24:05] as an example these tokens are embedded into kind of like this distributed
[24:06] into kind of like this distributed representation as it's called so every
[24:08] representation as it's called so every possible token has kind of like a vector
[24:10] possible token has kind of like a vector that represents it inside the neural
[24:12] that represents it inside the neural network so first we embed the tokens and
[24:15] network so first we embed the tokens and then those values uh kind of like flow
[24:18] then those values uh kind of like flow through this diagram and these are all
[24:21] through this diagram and these are all very simple mathematical Expressions
[24:22] very simple mathematical Expressions individually so we have layer norms and
[24:24] individually so we have layer norms and Matrix multiplications and uh soft Maxes
[24:27] Matrix multiplications and uh soft Maxes and so on so here kind of like the
[24:29] and so on so here kind of like the attention block of this Transformer and
[24:32] attention block of this Transformer and then information kind of flows through
[24:34] then information kind of flows through into the multi-layer perceptron block
[24:35] into the multi-layer perceptron block and so on and all these numbers here
[24:39] and so on and all these numbers here these are the intermediate values of the
[24:40] these are the intermediate values of the expression and uh you can almost think
[24:42] expression and uh you can almost think of these as kind of like the firing
[24:45] of these as kind of like the firing rates of these synthetic neurons but I
[24:48] rates of these synthetic neurons but I would caution you to uh not um kind of
[24:50] would caution you to uh not um kind of think of it too much like neurons
[24:53] think of it too much like neurons because these are extremely simple
[24:54] because these are extremely simple neurons compared to the neurons you
[24:55] neurons compared to the neurons you would find in your brain your biological
[24:57] would find in your brain your biological neurons are very complex dynamical
[24:59] neurons are very complex dynamical processes that have memory and so on
[25:01] processes that have memory and so on there's no memory in this expression
[25:03] there's no memory in this expression it's a fixed mathematical expression
[25:04] it's a fixed mathematical expression from input to Output with no memory it's
[25:07] from input to Output with no memory it's just a
[25:07] just a stateless so these are very simple
[25:09] stateless so these are very simple neurons in comparison to biological
[25:11] neurons in comparison to biological neurons but you can still kind of
[25:12] neurons but you can still kind of loosely think of this as like a
[25:14] loosely think of this as like a synthetic piece of uh brain tissue if
[25:16] synthetic piece of uh brain tissue if you if you like uh to think about it
[25:18] you if you like uh to think about it that way so information flows through
[25:21] that way so information flows through all these neurons fire until we get to
[25:24] all these neurons fire until we get to the predictions now I'm not actually
[25:27] the predictions now I'm not actually going to dwell too much on the precise
[25:29] going to dwell too much on the precise kind of like mathematical details of all
[25:30] kind of like mathematical details of all these Transformations honestly I don't
[25:32] these Transformations honestly I don't think it's that important to get into
[25:34] think it's that important to get into what's really important to understand is
[25:35] what's really important to understand is that this is a mathematical function it
[25:38] that this is a mathematical function it is uh parameterized by some fixed set of
[25:42] is uh parameterized by some fixed set of parameters like say 85,000 of them and
[25:44] parameters like say 85,000 of them and it is a way of transforming inputs into
[25:46] it is a way of transforming inputs into outputs and as we twiddle the parameters
[25:49] outputs and as we twiddle the parameters we are getting uh different kinds of
[25:51] we are getting uh different kinds of predictions and then we need to find a
[25:53] predictions and then we need to find a good setting of these parameters so that
[25:54] good setting of these parameters so that the predictions uh sort of match up with
[25:57] the predictions uh sort of match up with the patterns seen in training set
[25:59] the patterns seen in training set so that's the Transformer okay so I've
[26:02] so that's the Transformer okay so I've shown you the internals of the neural
[26:03] shown you the internals of the neural network and we talked a bit about the
[26:05] network and we talked a bit about the process of training it I want to cover
[26:07] process of training it I want to cover one more major stage of working with
[26:10] one more major stage of working with these networks and that is the stage
[26:12] these networks and that is the stage called inference so in inference what
[26:14] called inference so in inference what we're doing is we're generating new data
[26:16] we're doing is we're generating new data from the model and so uh we want to
[26:19] from the model and so uh we want to basically see what kind of patterns it
[26:21] basically see what kind of patterns it has internalized in the parameters of
[26:23] has internalized in the parameters of its Network so to generate from the
[26:27] its Network so to generate from the model is relatively straightforward
[26:29] model is relatively straightforward we start with some tokens that are
[26:31] we start with some tokens that are basically your prefix like what you want
[26:33] basically your prefix like what you want to start with so say we want to start
[26:34] to start with so say we want to start with the token 91 well we feed it into
[26:37] with the token 91 well we feed it into the
[26:38] the network and remember that the network
[26:40] network and remember that the network gives us probabilities right it gives us
[26:43] gives us probabilities right it gives us this probability Vector here so what we
[26:45] this probability Vector here so what we can do now is we can basically flip a
[26:47] can do now is we can basically flip a biased coin so um we can sample uh
[26:52] biased coin so um we can sample uh basically a token based on this
[26:55] basically a token based on this probability distribution so the tokens
[26:57] probability distribution so the tokens that are given High probability by the
[26:59] that are given High probability by the model are more likely to be sampled when
[27:02] model are more likely to be sampled when you flip this biased coin you can think
[27:04] you flip this biased coin you can think of it that way so we sample from the
[27:06] of it that way so we sample from the distribution to get a single unique
[27:08] distribution to get a single unique token so for example token 860 comes
[27:11] token so for example token 860 comes next uh so 860 in this case when we're
[27:14] next uh so 860 in this case when we're generating from model could come next
[27:16] generating from model could come next now 860 is a relatively likely token it
[27:19] now 860 is a relatively likely token it might not be the only possible token in
[27:21] might not be the only possible token in this case there could be many other
[27:22] this case there could be many other tokens that could have been sampled but
[27:24] tokens that could have been sampled but we could see that 86c is a relatively
[27:25] we could see that 86c is a relatively likely token as an example and indeed in
[27:28] likely token as an example and indeed in our training examp example here 860 does
[27:30] our training examp example here 860 does follow 91 so let's now say that we um
[27:34] follow 91 so let's now say that we um continue the process so after 91 there's
[27:37] continue the process so after 91 there's a60 we append it and we again ask what
[27:39] a60 we append it and we again ask what is the third token let's sample and
[27:42] is the third token let's sample and let's just say that it's 287 exactly as
[27:45] let's just say that it's 287 exactly as here let's do that again we come back in
[27:48] here let's do that again we come back in now we have a sequence of three and we
[27:50] now we have a sequence of three and we ask what is the likely fourth token and
[27:53] ask what is the likely fourth token and we sample from that and get this one and
[27:56] we sample from that and get this one and now let's say we do it one more time we
[27:58] now let's say we do it one more time we take those four we sample and we get
[28:00] take those four we sample and we get this one and this
[28:03] this one and this 13659 uh this is not actually uh 3962 as
[28:07] 13659 uh this is not actually uh 3962 as we had before so this token is the token
[28:10] we had before so this token is the token article uh instead so viewing a single
[28:13] article uh instead so viewing a single article and so in this case we didn't
[28:15] article and so in this case we didn't exactly reproduce the sequence that we
[28:17] exactly reproduce the sequence that we saw here in the training data so keep in
[28:20] saw here in the training data so keep in mind that these systems are stochastic
[28:23] mind that these systems are stochastic they have um we're sampling and we're
[28:26] they have um we're sampling and we're flipping coins and sometimes we lock out
[28:29] flipping coins and sometimes we lock out and we reproduce some like small chunk
[28:31] and we reproduce some like small chunk of the text and training set but
[28:33] of the text and training set but sometimes we're uh we're getting a token
[28:36] sometimes we're uh we're getting a token that was not verbatim part of any of the
[28:38] that was not verbatim part of any of the documents in the training data so we're
[28:40] documents in the training data so we're going to get sort of like remixes of the
[28:43] going to get sort of like remixes of the data that we saw in the training because
[28:45] data that we saw in the training because at every step of the way we can flip and
[28:47] at every step of the way we can flip and get a slightly different token and then
[28:49] get a slightly different token and then once that token makes it in if you
[28:51] once that token makes it in if you sample the next one and so on you very
[28:53] sample the next one and so on you very quickly uh start to generate token
[28:55] quickly uh start to generate token streams that are very different from the
[28:57] streams that are very different from the token streams that UR
[28:59] token streams that UR in the training documents so
[29:01] in the training documents so statistically they will have similar
[29:02] statistically they will have similar properties but um they are not identical
[29:05] properties but um they are not identical to your training data they're kind of
[29:07] to your training data they're kind of like inspired by the training data and
[29:09] like inspired by the training data and so in this case we got a slightly
[29:11] so in this case we got a slightly different sequence and why would we get
[29:13] different sequence and why would we get article you might imagine that article
[29:15] article you might imagine that article is a relatively likely token in the
[29:17] is a relatively likely token in the context of bar viewing single Etc and
[29:21] context of bar viewing single Etc and you can imagine that the word article
[29:22] you can imagine that the word article followed this context window somewhere
[29:25] followed this context window somewhere in the training documents uh to some
[29:27] in the training documents uh to some extent and we just happen to sample it
[29:29] extent and we just happen to sample it here at that stage so basically
[29:31] here at that stage so basically inference is just uh predicting from
[29:33] inference is just uh predicting from these distributions one at a time we
[29:35] these distributions one at a time we continue feeding back tokens and getting
[29:37] continue feeding back tokens and getting the next one and we uh we're always
[29:40] the next one and we uh we're always flipping these coins and depending on
[29:42] flipping these coins and depending on how lucky or unlucky we get um we might
[29:46] how lucky or unlucky we get um we might get very different kinds of patterns
[29:47] get very different kinds of patterns depending on how we sample from these
[29:49] depending on how we sample from these probability distributions so that's
[29:52] probability distributions so that's inference so in most common scenarios uh
[29:55] inference so in most common scenarios uh basically downloading the internet and
[29:57] basically downloading the internet and tokenizing it is is a pre-processing
[29:59] tokenizing it is is a pre-processing step you do that a single time and then
[30:02] step you do that a single time and then uh once you have your token sequence we
[30:04] uh once you have your token sequence we can start training networks and in
[30:07] can start training networks and in Practical cases you would try to train
[30:09] Practical cases you would try to train many different networks of different
[30:10] many different networks of different kinds of uh settings and different kinds
[30:12] kinds of uh settings and different kinds of arrangements and different kinds of
[30:14] of arrangements and different kinds of sizes and so you''ll be doing a lot of
[30:15] sizes and so you''ll be doing a lot of neural network training and um then once
[30:18] neural network training and um then once you have a neural network and you train
[30:20] you have a neural network and you train it and you have some specific set of
[30:22] it and you have some specific set of parameters that you're happy with um
[30:24] parameters that you're happy with um then you can take the model and you can
[30:26] then you can take the model and you can do inference and you can actually uh
[30:28] do inference and you can actually uh generate data from the model and when
[30:30] generate data from the model and when you're on chat GPT and you're talking
[30:32] you're on chat GPT and you're talking with a model uh that model is trained
[30:34] with a model uh that model is trained and has been trained by open aai many
[30:36] and has been trained by open aai many months ago probably and they have a
[30:39] months ago probably and they have a specific set of Weights that work well
[30:41] specific set of Weights that work well and when you're talking to the model all
[30:43] and when you're talking to the model all of that is just inference there's no
[30:45] of that is just inference there's no more training those parameters are held
[30:47] more training those parameters are held fixed and you're just talking to the
[30:49] fixed and you're just talking to the model sort of uh you're giving it some
[30:52] model sort of uh you're giving it some of the tokens and it's kind of
[30:53] of the tokens and it's kind of completing token sequences and that's
[30:55] completing token sequences and that's what you're seeing uh generated when you
[30:57] what you're seeing uh generated when you actually use the model on CH GPT so that
[31:00] actually use the model on CH GPT so that model then just does inference alone so
[31:02] model then just does inference alone so let's now look at an example of training
[31:04] let's now look at an example of training an inference that is kind of concrete
[31:06] an inference that is kind of concrete and gives you a sense of what this
[31:07] and gives you a sense of what this actually looks like uh when these models
[31:09] actually looks like uh when these models are trained now the example that I would
[31:11] are trained now the example that I would like to work with and that I'm
[31:12] like to work with and that I'm particularly fond of is that of opening
[31:14] particularly fond of is that of opening eyes gpt2 so GPT uh stands for
[31:17] eyes gpt2 so GPT uh stands for generatively pre-trained Transformer and
[31:20] generatively pre-trained Transformer and this is the second iteration of the GPT
[31:22] this is the second iteration of the GPT series by open AI when you are talking
[31:24] series by open AI when you are talking to chat GPT today the model that is
[31:26] to chat GPT today the model that is underlying all of the magic of that
[31:28] underlying all of the magic of that interaction is GPT 4 so the fourth
[31:30] interaction is GPT 4 so the fourth iteration of that series now gpt2 was
[31:33] iteration of that series now gpt2 was published in 2019 by openi in this paper
[31:36] published in 2019 by openi in this paper that I have right here and the reason I
[31:39] that I have right here and the reason I like gpt2 is that it is the first time
[31:42] like gpt2 is that it is the first time that a recognizably modern stack came
[31:44] that a recognizably modern stack came together so um all of the pieces of gpd2
[31:49] together so um all of the pieces of gpd2 are recognizable today by modern
[31:51] are recognizable today by modern standards it's just everything has
[31:52] standards it's just everything has gotten bigger now I'm not going to be
[31:54] gotten bigger now I'm not going to be able to go into the full details of this
[31:56] able to go into the full details of this paper of course because it is a
[31:57] paper of course because it is a technical publication but some of the
[32:00] technical publication but some of the details that I would like to highlight
[32:01] details that I would like to highlight are as follows gpt2 was a Transformer
[32:04] are as follows gpt2 was a Transformer neural network just like you were just
[32:06] neural network just like you were just like the neural networks you would work
[32:07] like the neural networks you would work with today it was it had 1.6 billion
[32:10] with today it was it had 1.6 billion parameters right so these are the
[32:12] parameters right so these are the parameters that we looked at here it
[32:14] parameters that we looked at here it would have 1.6 billion of them today
[32:17] would have 1.6 billion of them today modern Transformers would have a lot
[32:18] modern Transformers would have a lot closer to a trillion or several hundred
[32:20] closer to a trillion or several hundred billion
[32:22] billion probably the maximum context length here
[32:25] probably the maximum context length here was 1,24 tokens so it is when we are
[32:29] was 1,24 tokens so it is when we are sampling chunks of Windows of tokens
[32:32] sampling chunks of Windows of tokens from the data set we're never taking
[32:34] from the data set we're never taking more than 1,24 tokens and so when you
[32:37] more than 1,24 tokens and so when you are trying to predict the next token in
[32:38] are trying to predict the next token in a sequence you will never have more than
[32:40] a sequence you will never have more than 1,24 tokens uh kind of in your context
[32:43] 1,24 tokens uh kind of in your context in order to make that prediction now
[32:46] in order to make that prediction now this is also tiny by modern standards
[32:47] this is also tiny by modern standards today the token uh the context lengths
[32:50] today the token uh the context lengths would be a lot closer to um couple
[32:53] would be a lot closer to um couple hundred thousand or maybe even a million
[32:55] hundred thousand or maybe even a million and so you have a lot more context a lot
[32:57] and so you have a lot more context a lot more tokens in history history and you
[32:59] more tokens in history history and you can make a lot better prediction about
[33:00] can make a lot better prediction about the next token in the sequence in that
[33:02] the next token in the sequence in that way and finally gpt2 was trained on
[33:05] way and finally gpt2 was trained on approximately 100 billion tokens and
[33:07] approximately 100 billion tokens and this is also fairly small by modern
[33:09] this is also fairly small by modern standards as I mentioned the fine web
[33:10] standards as I mentioned the fine web data set that we looked at here the fine
[33:12] data set that we looked at here the fine web data set has 15 trillion tokens uh
[33:15] web data set has 15 trillion tokens uh so 100 billion is is quite
[33:17] so 100 billion is is quite small
[33:19] small now uh I actually tried to reproduce uh
[33:21] now uh I actually tried to reproduce uh gpt2 for fun as part of this project
[33:24] gpt2 for fun as part of this project called lm. C so you can see my rup of
[33:27] called lm. C so you can see my rup of doing that in this post on GitHub under
[33:30] doing that in this post on GitHub under the lm. C repository so in particular
[33:34] the lm. C repository so in particular the cost of training gpd2 in 2019 what
[33:37] the cost of training gpd2 in 2019 what was estimated to be approximately
[33:39] was estimated to be approximately $40,000 but today you can do
[33:41] $40,000 but today you can do significantly better than that and in
[33:43] significantly better than that and in particular here it took about one day
[33:46] particular here it took about one day and about
[33:47] and about $600 uh but this wasn't even trying too
[33:50] $600 uh but this wasn't even trying too hard I think you could really bring this
[33:51] hard I think you could really bring this down to about $100 today now why is it
[33:55] down to about $100 today now why is it that the costs have come down so much
[33:57] that the costs have come down so much well number one these data sets have
[33:59] well number one these data sets have gotten a lot better and the way we
[34:01] gotten a lot better and the way we filter them extract them and prepare
[34:03] filter them extract them and prepare them has gotten a lot more refined and
[34:06] them has gotten a lot more refined and so the data set is of just a lot higher
[34:08] so the data set is of just a lot higher quality so that's one thing but really
[34:10] quality so that's one thing but really the biggest difference is that our
[34:12] the biggest difference is that our computers have gotten much faster in
[34:14] computers have gotten much faster in terms of the hardware and we're going to
[34:15] terms of the hardware and we're going to look at that in a second and also the
[34:17] look at that in a second and also the software for uh running these models and
[34:20] software for uh running these models and really squeezing out all all the speed
[34:23] really squeezing out all all the speed from the hardware as it is possible uh
[34:26] from the hardware as it is possible uh that software has also gotten much
[34:27] that software has also gotten much better as as everyone has focused on
[34:29] better as as everyone has focused on these models and try to run them very
[34:30] these models and try to run them very very
[34:31] very quickly now I'm not going to be able to
[34:34] quickly now I'm not going to be able to go into the full detail of this gpd2
[34:37] go into the full detail of this gpd2 reproduction and this is a long
[34:38] reproduction and this is a long technical post but I would like to still
[34:40] technical post but I would like to still give you an intuitive sense for what it
[34:42] give you an intuitive sense for what it looks like to actually train one of
[34:43] looks like to actually train one of these models as a researcher like what
[34:45] these models as a researcher like what are you looking at and what does it look
[34:46] are you looking at and what does it look like what does it feel like so let me
[34:48] like what does it feel like so let me give you a sense of that a little bit
[34:50] give you a sense of that a little bit okay so this is what it looks like let
[34:51] okay so this is what it looks like let me slide this
[34:53] me slide this over so what I'm doing here is I'm
[34:56] over so what I'm doing here is I'm training a gpt2 model right now
[34:59] training a gpt2 model right now and um what's happening here is that
[35:01] and um what's happening here is that every single line here like this one is
[35:05] every single line here like this one is one update to the model so remember how
[35:09] one update to the model so remember how here we are um basically making the
[35:12] here we are um basically making the prediction better for every one of these
[35:14] prediction better for every one of these tokens and we are updating these weights
[35:16] tokens and we are updating these weights or parameters of the neural net so here
[35:19] or parameters of the neural net so here every single line is One update to the
[35:21] every single line is One update to the neural network where we change its
[35:23] neural network where we change its parameters by a little bit so that it is
[35:24] parameters by a little bit so that it is better at predicting next token and
[35:26] better at predicting next token and sequence in particular every single line
[35:29] sequence in particular every single line here is improving the prediction on 1
[35:32] here is improving the prediction on 1 million tokens in the training set so
[35:36] million tokens in the training set so we've basically taken 1 million tokens
[35:39] we've basically taken 1 million tokens out of this data set and we've tried to
[35:42] out of this data set and we've tried to improve the prediction of that token as
[35:45] improve the prediction of that token as coming next in a sequence on all 1
[35:47] coming next in a sequence on all 1 million of them
[35:49] million of them simultaneously and at every single one
[35:51] simultaneously and at every single one of these steps we are making an update
[35:53] of these steps we are making an update to the network for that now the number
[35:55] to the network for that now the number to watch closely is this number called
[35:58] to watch closely is this number called loss and the loss is a single number
[36:01] loss and the loss is a single number that is telling you how well your neural
[36:03] that is telling you how well your neural network is performing right now and it
[36:05] network is performing right now and it is created so that low loss is good so
[36:09] is created so that low loss is good so you'll see that the loss is decreasing
[36:11] you'll see that the loss is decreasing as we make more updates to the neural
[36:13] as we make more updates to the neural nut which corresponds to making better
[36:14] nut which corresponds to making better predictions on the next token in a
[36:16] predictions on the next token in a sequence and so the loss is the number
[36:19] sequence and so the loss is the number that you are watching as a neural
[36:21] that you are watching as a neural network researcher and you are kind of
[36:23] network researcher and you are kind of waiting you're twiddling your thumbs uh
[36:24] waiting you're twiddling your thumbs uh you're drinking coffee and you're making
[36:26] you're drinking coffee and you're making sure that this looks good so that with
[36:29] sure that this looks good so that with every update your loss is improving and
[36:31] every update your loss is improving and the network is getting better at
[36:33] the network is getting better at prediction now here you see that we are
[36:36] prediction now here you see that we are processing 1 million tokens per update
[36:39] processing 1 million tokens per update each update takes about 7 Seconds
[36:41] each update takes about 7 Seconds roughly and here we are going to process
[36:44] roughly and here we are going to process a total of 32,000 steps of
[36:47] a total of 32,000 steps of optimization so 32,000 steps with 1
[36:50] optimization so 32,000 steps with 1 million tokens each is about 33 billion
[36:53] million tokens each is about 33 billion tokens that we are going to process and
[36:55] tokens that we are going to process and we're currently only about 420 step 20
[36:58] we're currently only about 420 step 20 out of 32,000 so we are still only a bit
[37:01] out of 32,000 so we are still only a bit more than 1% done because I've only been
[37:03] more than 1% done because I've only been running this for 10 or 15 minutes or
[37:05] running this for 10 or 15 minutes or something like
[37:06] something like that now every 20 steps I have
[37:09] that now every 20 steps I have configured this optimization to do
[37:11] configured this optimization to do inference so what you're seeing here is
[37:13] inference so what you're seeing here is the model is predicting the next token
[37:15] the model is predicting the next token in a sequence and so you sort of start
[37:18] in a sequence and so you sort of start it randomly and then you continue
[37:20] it randomly and then you continue plugging in the tokens so we're running
[37:22] plugging in the tokens so we're running this inference step and this is the
[37:24] this inference step and this is the model sort of predicting the next token
[37:25] model sort of predicting the next token in the sequence and every time you see
[37:26] in the sequence and every time you see something appear that's a new
[37:29] something appear that's a new token um so let's just look at this and
[37:34] token um so let's just look at this and you can see that this is not yet very
[37:36] you can see that this is not yet very coherent and keep in mind that this is
[37:37] coherent and keep in mind that this is only 1% of the way through training and
[37:40] only 1% of the way through training and so the model is not yet very good at
[37:41] so the model is not yet very good at predicting the next token in the
[37:42] predicting the next token in the sequence so what comes out is actually
[37:45] sequence so what comes out is actually kind of a little bit of gibberish right
[37:47] kind of a little bit of gibberish right but it still has a little bit of like
[37:49] but it still has a little bit of like local coherence so since she is mine
[37:52] local coherence so since she is mine it's a part of the information should
[37:53] it's a part of the information should discuss my father great companions
[37:56] discuss my father great companions Gordon showed me sitting over at and Etc
[37:59] Gordon showed me sitting over at and Etc so I know it doesn't look very good but
[38:01] so I know it doesn't look very good but let's actually scroll up and see what it
[38:04] let's actually scroll up and see what it looked like when I started the
[38:06] looked like when I started the optimization so all the way here at
[38:10] optimization so all the way here at step
[38:12] step one so after 20 steps of optimization
[38:15] one so after 20 steps of optimization you see that what we're getting here is
[38:17] you see that what we're getting here is looks completely random and of course
[38:19] looks completely random and of course that's because the model has only had 20
[38:20] that's because the model has only had 20 updates to its parameters and so it's
[38:22] updates to its parameters and so it's giving you random text because it's a
[38:24] giving you random text because it's a random Network and so you can see that
[38:26] random Network and so you can see that at least in comparison to this model is
[38:28] at least in comparison to this model is starting to do much better and indeed if
[38:30] starting to do much better and indeed if we waited the entire 32,000 steps the
[38:33] we waited the entire 32,000 steps the model will have improved the point that
[38:34] model will have improved the point that it's actually uh generating fairly
[38:36] it's actually uh generating fairly coherent English uh and the tokens
[38:39] coherent English uh and the tokens stream correctly um and uh they they
[38:42] stream correctly um and uh they they kind of make up English a a lot
[38:45] kind of make up English a a lot better
[38:46] better um so this has to run for about a day or
[38:50] um so this has to run for about a day or two more now and so uh at this stage we
[38:52] two more now and so uh at this stage we just make sure that the loss is
[38:54] just make sure that the loss is decreasing everything is looking good um
[38:56] decreasing everything is looking good um and we just have to wait
[38:58] and we just have to wait and now um let me turn now to the um
[39:02] and now um let me turn now to the um story of the computation that's required
[39:05] story of the computation that's required because of course I'm not running this
[39:06] because of course I'm not running this optimization on my laptop that would be
[39:09] optimization on my laptop that would be way too expensive uh because we have to
[39:11] way too expensive uh because we have to run this neural network and we have to
[39:13] run this neural network and we have to improve it and we have we need all this
[39:15] improve it and we have we need all this data and so on so you can't run this too
[39:17] data and so on so you can't run this too well on your computer uh because the
[39:19] well on your computer uh because the network is just too large uh so all of
[39:21] network is just too large uh so all of this is running on the computer that is
[39:23] this is running on the computer that is out there in the cloud and I want to
[39:25] out there in the cloud and I want to basically address the compute side of
[39:27] basically address the compute side of the store of training these models and
[39:29] the store of training these models and what that looks like so let's take a
[39:30] what that looks like so let's take a look okay so the computer that I'm
[39:32] look okay so the computer that I'm running this optimization on is this 8X
[39:35] running this optimization on is this 8X h100 node so there are eight h100s in a
[39:40] h100 node so there are eight h100s in a single node or a single computer now I
[39:43] single node or a single computer now I am renting this computer and it is
[39:44] am renting this computer and it is somewhere in the cloud I'm not sure
[39:46] somewhere in the cloud I'm not sure where it is physically actually the
[39:48] where it is physically actually the place I like to rent from is called
[39:49] place I like to rent from is called Lambda but there are many other
[39:50] Lambda but there are many other companies who provide this service so
[39:52] companies who provide this service so when you scroll down you can see that uh
[39:56] when you scroll down you can see that uh they have some on demand pricing for
[39:58] they have some on demand pricing for um sort of computers that have these uh
[40:02] um sort of computers that have these uh h100s which are gpus and I'm going to
[40:04] h100s which are gpus and I'm going to show you what they look like in a second
[40:06] show you what they look like in a second but on demand 8times Nvidia h100 uh
[40:10] but on demand 8times Nvidia h100 uh GPU this machine comes for $3 per GPU
[40:14] GPU this machine comes for $3 per GPU per hour for example so you can rent
[40:17] per hour for example so you can rent these and then you get a machine in a
[40:18] these and then you get a machine in a cloud and you can uh go in and you can
[40:20] cloud and you can uh go in and you can train these
[40:22] train these models and these uh gpus they look like
[40:26] models and these uh gpus they look like this so this is one h100 GPU uh this is
[40:29] this so this is one h100 GPU uh this is kind of what it looks like and you slot
[40:30] kind of what it looks like and you slot this into your computer and gpus are
[40:33] this into your computer and gpus are this uh perfect fit for training your
[40:35] this uh perfect fit for training your networks because they are very
[40:37] networks because they are very computationally expensive but they
[40:39] computationally expensive but they display a lot of parallelism in the
[40:41] display a lot of parallelism in the computation so you can have many
[40:42] computation so you can have many independent workers kind of um working
[40:45] independent workers kind of um working all at the same time in solving uh the
[40:48] all at the same time in solving uh the matrix multiplication that's under the
[40:51] matrix multiplication that's under the hood of training these neural
[40:53] hood of training these neural networks so this is just one of these
[40:55] networks so this is just one of these h100s but actually you would put them
[40:57] h100s but actually you would put them you would put multiple of them together
[40:59] you would put multiple of them together so you could stack eight of them into a
[41:00] so you could stack eight of them into a single node and then you can stack
[41:02] single node and then you can stack multiple nodes into an entire data
[41:04] multiple nodes into an entire data center or an entire system
[41:07] center or an entire system so when we look at a data
[41:15] center can't spell when we look at a data center we start to see things that
[41:16] data center we start to see things that look like this right so we have one GPU
[41:18] look like this right so we have one GPU goes to eight gpus goes to a single
[41:20] goes to eight gpus goes to a single system goes to many systems and so these
[41:22] system goes to many systems and so these are the bigger data centers and there of
[41:24] are the bigger data centers and there of course would be much much more expensive
[41:26] course would be much much more expensive um and what's happening is that all the
[41:29] um and what's happening is that all the big tech companies really desire these
[41:31] big tech companies really desire these gpus so they can train all these
[41:33] gpus so they can train all these language models because they are so
[41:35] language models because they are so powerful and that has is fundamentally
[41:37] powerful and that has is fundamentally what has driven the stock price of
[41:39] what has driven the stock price of Nvidia to be $3.4 trillion today as an
[41:42] Nvidia to be $3.4 trillion today as an example and why Nvidia has kind of
[41:45] example and why Nvidia has kind of exploded so this is the Gold Rush the
[41:47] exploded so this is the Gold Rush the Gold Rush is getting the gpus getting
[41:50] Gold Rush is getting the gpus getting enough of them so they can all
[41:52] enough of them so they can all collaborate to perform this optimization
[41:55] collaborate to perform this optimization and they're what are they all doing
[41:57] and they're what are they all doing they're all collaborating to predict the
[41:59] they're all collaborating to predict the next token on a data set like the fine
[42:02] next token on a data set like the fine web data
[42:03] web data set this is the computational workflow
[42:05] set this is the computational workflow that that basically is extremely
[42:07] that that basically is extremely expensive the more gpus you have the
[42:09] expensive the more gpus you have the more tokens you can try to predict and
[42:11] more tokens you can try to predict and improve on and you're going to process
[42:13] improve on and you're going to process this data set faster and you can iterate
[42:15] this data set faster and you can iterate faster and get a bigger Network and
[42:17] faster and get a bigger Network and train a bigger Network and so on so this
[42:19] train a bigger Network and so on so this is what all those machines are look like
[42:21] is what all those machines are look like are uh are doing and this is why all of
[42:24] are uh are doing and this is why all of this is such a big deal and for example
[42:27] this is such a big deal and for example this is a
[42:29] this is a article from like about a month ago or
[42:30] article from like about a month ago or so this is why it's a big deal that for
[42:32] so this is why it's a big deal that for example Elon Musk is getting 100,000
[42:35] example Elon Musk is getting 100,000 gpus uh in a single Data Center and all
[42:39] gpus uh in a single Data Center and all of these gpus are extremely expensive
[42:41] of these gpus are extremely expensive are going to take a ton of power and all
[42:42] are going to take a ton of power and all of them are just trying to predict the
[42:44] of them are just trying to predict the next token in the sequence and improve
[42:46] next token in the sequence and improve the network uh by doing so and uh get
[42:49] the network uh by doing so and uh get probably a lot more coherent text than
[42:51] probably a lot more coherent text than what we're seeing here a lot faster okay
[42:53] what we're seeing here a lot faster okay so unfortunately I do not have a couple
[42:55] so unfortunately I do not have a couple 10 or hundred million of dollars to
[42:57] 10 or hundred million of dollars to spend on training a really big model
[42:59] spend on training a really big model like this but luckily we can turn to
[43:01] like this but luckily we can turn to some big tech companies who train these
[43:04] some big tech companies who train these models routinely and release some of
[43:06] models routinely and release some of them once they are done training so
[43:09] them once they are done training so they've spent a huge amount of compute
[43:10] they've spent a huge amount of compute to train this network and they release
[43:12] to train this network and they release the network at the end of the
[43:14] the network at the end of the optimization so it's very useful because
[43:16] optimization so it's very useful because they've done a lot of compute for that
[43:18] they've done a lot of compute for that so there are many companies who train
[43:19] so there are many companies who train these models routinely but actually not
[43:21] these models routinely but actually not many of them release uh these what's
[43:23] many of them release uh these what's called base models so the model that
[43:26] called base models so the model that comes out at the end here is is what's
[43:27] comes out at the end here is is what's called a base model what is a base model
[43:30] called a base model what is a base model it's a token simulator right it's an
[43:32] it's a token simulator right it's an internet text token simulator and so
[43:36] internet text token simulator and so that is not by itself useful yet because
[43:38] that is not by itself useful yet because what we want is what's called an
[43:39] what we want is what's called an assistant we want to ask questions and
[43:41] assistant we want to ask questions and have it respond to answers these models
[43:44] have it respond to answers these models won't do that they just uh create sort
[43:46] won't do that they just uh create sort of remixes of the internet they dream
[43:49] of remixes of the internet they dream internet pages so the base models are
[43:52] internet pages so the base models are not very often released because they're
[43:53] not very often released because they're kind of just only a step one of a few
[43:55] kind of just only a step one of a few other steps that we still need to take
[43:56] other steps that we still need to take to get in system
[43:58] to get in system however a few releases have been made so
[44:01] however a few releases have been made so as an example the gbt2 model released
[44:04] as an example the gbt2 model released the 1.6 billion sorry 1.5 billion model
[44:08] the 1.6 billion sorry 1.5 billion model back in 2019 and this gpt2 model is a
[44:11] back in 2019 and this gpt2 model is a base model now what is a model release
[44:14] base model now what is a model release what does it look like to release these
[44:15] what does it look like to release these models so this is the gpt2 repository on
[44:18] models so this is the gpt2 repository on GitHub well you need two things
[44:20] GitHub well you need two things basically to release model number one we
[44:23] basically to release model number one we need the um python code usually that
[44:28] need the um python code usually that describes the sequence of operations in
[44:30] describes the sequence of operations in detail that they make in their model so
[44:35] detail that they make in their model so um if you remember
[44:37] um if you remember back this
[44:39] back this Transformer the sequence of steps that
[44:41] Transformer the sequence of steps that are taken here in this neural network is
[44:43] are taken here in this neural network is what is being described by this code so
[44:46] what is being described by this code so this code is sort of implementing the
[44:47] this code is sort of implementing the what's called forward pass of this
[44:49] what's called forward pass of this neural network so we need the specific
[44:52] neural network so we need the specific details of exactly how they wired up
[44:54] details of exactly how they wired up that neural network so this is just
[44:56] that neural network so this is just computer code and it's usually just a
[44:57] computer code and it's usually just a couple hundred lines of code it's not
[44:59] couple hundred lines of code it's not it's not that crazy and uh this is all
[45:02] it's not that crazy and uh this is all fairly understandable and usually fairly
[45:03] fairly understandable and usually fairly standard what's not standard are the
[45:05] standard what's not standard are the parameters that's where the actual value
[45:07] parameters that's where the actual value is what are the parameters of this
[45:09] is what are the parameters of this neural network because there's 1.6
[45:11] neural network because there's 1.6 billion of them and we need the correct
[45:13] billion of them and we need the correct setting or a really good setting and so
[45:16] setting or a really good setting and so that's why in addition to this source
[45:18] that's why in addition to this source code they release the parameters which
[45:20] code they release the parameters which in this case is roughly 1.5 billion
[45:23] in this case is roughly 1.5 billion parameters and these are just numbers so
[45:25] parameters and these are just numbers so it's one single list of 1.5 billion
[45:27] it's one single list of 1.5 billion numbers the precise and good setting of
[45:30] numbers the precise and good setting of all the knobs such that the tokens come
[45:32] all the knobs such that the tokens come out
[45:34] out well so uh you need those two things to
[45:37] well so uh you need those two things to get a base model
[45:40] get a base model release
[45:41] release now gpt2 was released but that's
[45:44] now gpt2 was released but that's actually a fairly old model as I
[45:45] actually a fairly old model as I mentioned so actually the model we're
[45:46] mentioned so actually the model we're going to turn to is called llama 3 and
[45:49] going to turn to is called llama 3 and that's the one that I would like to show
[45:50] that's the one that I would like to show you next so llama 3 so gpt2 again was
[45:54] you next so llama 3 so gpt2 again was 1.6 billion parameters trained on 100
[45:56] 1.6 billion parameters trained on 100 billion tokens Lama 3 is a much bigger
[45:59] billion tokens Lama 3 is a much bigger model and much more modern model it is
[46:01] model and much more modern model it is released and trained by meta and it is a
[46:04] released and trained by meta and it is a 45 billion parameter model trained on 15
[46:07] 45 billion parameter model trained on 15 trillion tokens in very much the same
[46:10] trillion tokens in very much the same way just much much
[46:11] way just much much bigger um and meta has also made a
[46:15] bigger um and meta has also made a release of llama 3 and that was part of
[46:18] release of llama 3 and that was part of this
[46:19] this paper so with this paper that goes into
[46:21] paper so with this paper that goes into a lot of detail the biggest base model
[46:23] a lot of detail the biggest base model that they released is the Lama 3.1 4.5
[46:27] that they released is the Lama 3.1 4.5 405 billion parameter model so this is
[46:30] 405 billion parameter model so this is the base model and then in addition to
[46:32] the base model and then in addition to the base model you see here
[46:33] the base model you see here foreshadowing for later sections of the
[46:35] foreshadowing for later sections of the video they also released the instruct
[46:37] video they also released the instruct model and the instruct means that this
[46:39] model and the instruct means that this is an assistant you can ask it questions
[46:41] is an assistant you can ask it questions and it will give you answers we still
[46:43] and it will give you answers we still have yet to cover that part later for
[46:45] have yet to cover that part later for now let's just look at this base model
[46:47] now let's just look at this base model this token simulator and let's play with
[46:49] this token simulator and let's play with it and try to think about you know what
[46:51] it and try to think about you know what is this thing and how does it work and
[46:54] is this thing and how does it work and um what do we get at the end of this
[46:55] um what do we get at the end of this optimization if you let this run Until
[46:57] optimization if you let this run Until the End uh for a very big neural network
[47:00] the End uh for a very big neural network on a lot of data so my favorite place to
[47:02] on a lot of data so my favorite place to interact with the base models is this um
[47:05] interact with the base models is this um company called hyperbolic which is
[47:07] company called hyperbolic which is basically serving the base model of the
[47:10] basically serving the base model of the 405b Llama 3.1 so when you go to the
[47:13] 405b Llama 3.1 so when you go to the website and I think you may have to
[47:15] website and I think you may have to register and so on make sure that in the
[47:17] register and so on make sure that in the models make sure that you are using
[47:19] models make sure that you are using llama 3.1 405 billion base it must be
[47:22] llama 3.1 405 billion base it must be the base model and then here let's say
[47:25] the base model and then here let's say the max tokens is how many tokens we're
[47:26] the max tokens is how many tokens we're going to be gener rating so let's just
[47:28] going to be gener rating so let's just decrease this to be a bit less just so
[47:30] decrease this to be a bit less just so we don't waste compute we just want the
[47:32] we don't waste compute we just want the next 128 tokens and leave the other
[47:35] next 128 tokens and leave the other stuff alone I'm not going to go into the
[47:36] stuff alone I'm not going to go into the full detail here um now fundamentally
[47:39] full detail here um now fundamentally what's going to happen here is identical
[47:42] what's going to happen here is identical to what happens here during inference
[47:43] to what happens here during inference for us so this is just going to continue
[47:45] for us so this is just going to continue the token sequence of whatever you
[47:47] the token sequence of whatever you prefix you're going to give it so I want
[47:50] prefix you're going to give it so I want to first show you that this model here
[47:52] to first show you that this model here is not yet an assistant so you can for
[47:54] is not yet an assistant so you can for example ask it what is 2 plus 2 it's not
[47:56] example ask it what is 2 plus 2 it's not going to tell you oh it's four uh what
[47:58] going to tell you oh it's four uh what else can I help you with it's not going
[48:00] else can I help you with it's not going to do that because what is 2 plus 2 is
[48:03] to do that because what is 2 plus 2 is going to be tokenized and then those
[48:05] going to be tokenized and then those tokens just act as a prefix and then
[48:08] tokens just act as a prefix and then what the model is going to do now is
[48:09] what the model is going to do now is just going to get the probability for
[48:10] just going to get the probability for the next token and it's just a glorified
[48:12] the next token and it's just a glorified autocomplete it's a very very expensive
[48:15] autocomplete it's a very very expensive autocomplete of what comes next um
[48:17] autocomplete of what comes next um depending on the statistics of what it
[48:19] depending on the statistics of what it saw in its training documents which are
[48:21] saw in its training documents which are basically web
[48:22] basically web pages so let's just uh hit enter to see
[48:25] pages so let's just uh hit enter to see what tokens it comes up with as a
[48:33] continuation okay so here it kind of actually answered the question and
[48:34] actually answered the question and started to go off into some
[48:35] started to go off into some philosophical territory uh let's try it
[48:38] philosophical territory uh let's try it again so let me copy and paste and let's
[48:40] again so let me copy and paste and let's try again from scratch what is 2 plus
[48:50] two so okay so it just goes off again so notice one more thing that I want to
[48:51] notice one more thing that I want to stress is that the system uh I think
[48:54] stress is that the system uh I think every time you put it in it just kind of
[48:55] every time you put it in it just kind of starts from scratch
[48:58] starts from scratch so it doesn't uh the system here is
[49:00] so it doesn't uh the system here is stochastic so for the same prefix of
[49:02] stochastic so for the same prefix of tokens we're always getting a different
[49:04] tokens we're always getting a different answer and the reason for that is that
[49:07] answer and the reason for that is that we get this probity distribution and we
[49:09] we get this probity distribution and we sample from it and we always get
[49:10] sample from it and we always get different samples and we sort of always
[49:12] different samples and we sort of always go into a different territory uh
[49:14] go into a different territory uh afterwards so here in this case um I
[49:18] afterwards so here in this case um I don't know what this is let's try one
[49:19] don't know what this is let's try one more
[49:25] time so it just continues on so it's just doing the stuff that it's saw on
[49:26] just doing the stuff that it's saw on the internet right um and it's just kind
[49:29] the internet right um and it's just kind of like regurgitating those uh
[49:31] of like regurgitating those uh statistical
[49:33] statistical patterns so first things it's not an
[49:36] patterns so first things it's not an assistant yet it's a token autocomplete
[49:39] assistant yet it's a token autocomplete and second it is a stochastic system now
[49:42] and second it is a stochastic system now the crucial thing is that even though
[49:44] the crucial thing is that even though this model is not yet by itself very
[49:46] this model is not yet by itself very useful for a lot of applications just
[49:49] useful for a lot of applications just yet um it is still very useful because
[49:53] yet um it is still very useful because in the task of predicting the next token
[49:54] in the task of predicting the next token in the sequence the model has learned a
[49:57] in the sequence the model has learned a lot about the world and it has stored
[49:59] lot about the world and it has stored all that knowledge in the parameters of
[50:01] all that knowledge in the parameters of the network so remember that our text
[50:04] the network so remember that our text looked like this right internet web
[50:06] looked like this right internet web pages and now all of this is sort of
[50:09] pages and now all of this is sort of compressed in the weights of the network
[50:11] compressed in the weights of the network so you can think of um these 405 billion
[50:15] so you can think of um these 405 billion parameters is a kind of compression of
[50:17] parameters is a kind of compression of the internet you can think of the
[50:19] the internet you can think of the 45 billion parameters is kind of like a
[50:22] 45 billion parameters is kind of like a zip file uh but it's not a loss less
[50:25] zip file uh but it's not a loss less compression it's a loss C compression
[50:27] compression it's a loss C compression we're kind of like left with kind of a
[50:29] we're kind of like left with kind of a gal of the internet and we can generate
[50:31] gal of the internet and we can generate from it right now we can elicit some of
[50:34] from it right now we can elicit some of this knowledge by prompting the base
[50:36] this knowledge by prompting the base model uh accordingly so for example
[50:38] model uh accordingly so for example here's a prompt that might work to
[50:40] here's a prompt that might work to elicit some of that knowledge that's
[50:42] elicit some of that knowledge that's hiding in the parameters here's my top
[50:44] hiding in the parameters here's my top 10 list of the top landmarks to see in
[50:46] 10 list of the top landmarks to see in the
[50:48] the pairs
[50:50] pairs um and I'm doing it this way because I'm
[50:53] um and I'm doing it this way because I'm trying to Prime the model to now
[50:54] trying to Prime the model to now continue this list so let's see if that
[50:56] continue this list so let's see if that works when I press
[50:57] works when I press enter okay so you see that it started a
[51:01] enter okay so you see that it started a list and it's now kind of giving me some
[51:02] list and it's now kind of giving me some of those
[51:03] of those landmarks and now notice that it's
[51:05] landmarks and now notice that it's trying to give a lot of information here
[51:07] trying to give a lot of information here now you might not be able to actually
[51:09] now you might not be able to actually fully trust some of the information here
[51:11] fully trust some of the information here remember that this is all just a
[51:12] remember that this is all just a recollection of some of the internet
[51:14] recollection of some of the internet documents and so the things that occur
[51:17] documents and so the things that occur very frequently in the internet data are
[51:19] very frequently in the internet data are probably more likely to be remembered
[51:21] probably more likely to be remembered correctly compared to things that happen
[51:23] correctly compared to things that happen very infrequently so you can't fully
[51:26] very infrequently so you can't fully trust some of the things that and some
[51:27] trust some of the things that and some of the information that is here because
[51:28] of the information that is here because it's all just a vague recollection of
[51:30] it's all just a vague recollection of Internet documents because the
[51:33] Internet documents because the information is not stored explicitly in
[51:35] information is not stored explicitly in any of the parameters it's all just the
[51:37] any of the parameters it's all just the recollection that said we did get
[51:39] recollection that said we did get something that is probably approximately
[51:40] something that is probably approximately correct and I don't actually have the
[51:42] correct and I don't actually have the expertise to verify that this is roughly
[51:44] expertise to verify that this is roughly correct but you see that we've elicited
[51:46] correct but you see that we've elicited a lot of the knowledge of the model and
[51:49] a lot of the knowledge of the model and this knowledge is not precise and exact
[51:51] this knowledge is not precise and exact this knowledge is vague and
[51:54] this knowledge is vague and probabilistic and statistical and the
[51:56] probabilistic and statistical and the kinds of things that occur often are the
[51:58] kinds of things that occur often are the kinds of things that are more likely to
[51:59] kinds of things that are more likely to be remembered um in the model now I want
[52:03] be remembered um in the model now I want to show you a few more examples of this
[52:04] to show you a few more examples of this model's Behavior the first thing I want
[52:06] model's Behavior the first thing I want to show you is this example I went to
[52:08] to show you is this example I went to the Wikipedia page for zebra and let me
[52:11] the Wikipedia page for zebra and let me just copy paste the first uh even one
[52:13] just copy paste the first uh even one sentence
[52:15] sentence here and let me put it here now when I
[52:18] here and let me put it here now when I click enter what kind of uh completion
[52:20] click enter what kind of uh completion are we going to get so let me just hit
[52:23] are we going to get so let me just hit enter there are three living species
[52:27] enter there are three living species etc etc what the model is producing here
[52:30] etc etc what the model is producing here is an exact regurgitation of this
[52:32] is an exact regurgitation of this Wikipedia entry it is reciting this
[52:34] Wikipedia entry it is reciting this Wikipedia entry purely from memory and
[52:36] Wikipedia entry purely from memory and this memory is stored in its parameters
[52:39] this memory is stored in its parameters and so it is possible that at some point
[52:41] and so it is possible that at some point in these 512 tokens the model will uh
[52:44] in these 512 tokens the model will uh stray away from the Wikipedia entry but
[52:46] stray away from the Wikipedia entry but you can see that it has huge chunks of
[52:48] you can see that it has huge chunks of it memorized here uh let me see for
[52:50] it memorized here uh let me see for example if this sentence
[52:52] example if this sentence occurs by now okay so this so we're
[52:55] occurs by now okay so this so we're still on track let me check
[52:59] still on track let me check here okay we're still on
[53:01] here okay we're still on track it will eventually uh stray
[53:04] track it will eventually uh stray away okay so this thing is just recited
[53:07] away okay so this thing is just recited to a very large extent it will
[53:09] to a very large extent it will eventually deviate uh because it won't
[53:11] eventually deviate uh because it won't be able to remember exactly now the
[53:13] be able to remember exactly now the reason that this happens is because
[53:15] reason that this happens is because these models can be extremely good at
[53:17] these models can be extremely good at memorization and usually this is not
[53:19] memorization and usually this is not what you want in the final model and
[53:20] what you want in the final model and this is something called regurgitation
[53:22] this is something called regurgitation and it's usually undesirable to site uh
[53:25] and it's usually undesirable to site uh things uh directly uh that you have
[53:27] things uh directly uh that you have trained on now the reason that this
[53:29] trained on now the reason that this happens actually is because for a lot of
[53:32] happens actually is because for a lot of documents like for example Wikipedia
[53:33] documents like for example Wikipedia when these documents are deemed to be of
[53:35] when these documents are deemed to be of very high quality as a source like for
[53:37] very high quality as a source like for example Wikipedia it is very often uh
[53:40] example Wikipedia it is very often uh the case that when you train the model
[53:42] the case that when you train the model you will preferentially sample from
[53:44] you will preferentially sample from those sources so basically the model has
[53:46] those sources so basically the model has probably done a few epochs on this data
[53:49] probably done a few epochs on this data meaning that it has seen this web page
[53:51] meaning that it has seen this web page like maybe probably 10 times or so and
[53:53] like maybe probably 10 times or so and it's a bit like you like when you read
[53:54] it's a bit like you like when you read some kind of a text many many times say
[53:57] some kind of a text many many times say you read something a 100 times uh then
[53:59] you read something a 100 times uh then you'll be able to recite it and it's
[54:00] you'll be able to recite it and it's very similar for this model if it sees
[54:02] very similar for this model if it sees something way too often it's going to be
[54:03] something way too often it's going to be able to recite it later from memory
[54:06] able to recite it later from memory except these models can be a lot more
[54:08] except these models can be a lot more efficient um like per presentation than
[54:11] efficient um like per presentation than human so probably it's only seen this
[54:12] human so probably it's only seen this Wikipedia entry 10 times but basically
[54:14] Wikipedia entry 10 times but basically it has remembered this article exactly
[54:16] it has remembered this article exactly in its parameters okay the next thing I
[54:18] in its parameters okay the next thing I want to show you is something that the
[54:19] want to show you is something that the model has definitely not seen during its
[54:22] model has definitely not seen during its training so for example if we go to the
[54:24] training so for example if we go to the paper uh and then we navigate to the
[54:27] paper uh and then we navigate to the pre-training data we'll see here that uh
[54:31] pre-training data we'll see here that uh the data set has a knowledge cut off
[54:33] the data set has a knowledge cut off until the end of 2023 so it will not
[54:36] until the end of 2023 so it will not have seen documents after this point and
[54:39] have seen documents after this point and certainly it has not seen anything about
[54:40] certainly it has not seen anything about the 2024 election and how it turned out
[54:43] the 2024 election and how it turned out now if we Prime the model with the
[54:46] now if we Prime the model with the tokens from the future it will continue
[54:49] tokens from the future it will continue the token sequence and it will just take
[54:51] the token sequence and it will just take its best guess according to the
[54:52] its best guess according to the knowledge that it has in its own
[54:53] knowledge that it has in its own parameters so let's take a look at what
[54:55] parameters so let's take a look at what that could look like
[54:57] that could look like so the Republican Party kit
[55:00] so the Republican Party kit Trump okay president of the United
[55:02] Trump okay president of the United States from
[55:03] States from 2017 and let's see what it says after
[55:05] 2017 and let's see what it says after this point so for example the model will
[55:07] this point so for example the model will have to guess at the running mate and
[55:09] have to guess at the running mate and who it's against Etc so let's hit
[55:12] who it's against Etc so let's hit enter so here thingss that Mike Pence
[55:14] enter so here thingss that Mike Pence was the running mate instead of JD Vance
[55:17] was the running mate instead of JD Vance and the ticket was against Hillary
[55:21] and the ticket was against Hillary Clinton and Tim Kane so this is kind of
[55:23] Clinton and Tim Kane so this is kind of a interesting parallel universe
[55:25] a interesting parallel universe potentially of what could have happened
[55:26] potentially of what could have happened happened according to the LM let's get a
[55:29] happened according to the LM let's get a different sample so the identical prompt
[55:31] different sample so the identical prompt and let's
[55:33] and let's resample so here the running mate was
[55:35] resample so here the running mate was Ronda santis and they ran against Joe
[55:38] Ronda santis and they ran against Joe Biden and Camala Harris so this is again
[55:41] Biden and Camala Harris so this is again a different parallel universe so the
[55:43] a different parallel universe so the model will take educated guesses and it
[55:44] model will take educated guesses and it will continue the token sequence based
[55:46] will continue the token sequence based on this knowledge um and it will just
[55:48] on this knowledge um and it will just kind of like all of what we're seeing
[55:50] kind of like all of what we're seeing here is what's called hallucination the
[55:52] here is what's called hallucination the model is just taking its best guess uh
[55:55] model is just taking its best guess uh in a probalistic manner the next thing I
[55:57] in a probalistic manner the next thing I would like to show you is that even
[55:58] would like to show you is that even though this is a base model and not yet
[56:00] though this is a base model and not yet an assistant model it can still be
[56:02] an assistant model it can still be utilized in Practical applications if
[56:04] utilized in Practical applications if you are clever with your prompt design
[56:07] you are clever with your prompt design so here's something that we would call a
[56:08] so here's something that we would call a few shot
[56:09] few shot prompt so what it is here is that I have
[56:12] prompt so what it is here is that I have 10 words or 10 pairs and each pair is a
[56:16] 10 words or 10 pairs and each pair is a word of English column and then a the
[56:20] word of English column and then a the translation in Korean and we have 10 of
[56:23] translation in Korean and we have 10 of them and what the model does here is at
[56:25] them and what the model does here is at the end we have teacher column and then
[56:27] the end we have teacher column and then here's where we're going to do a
[56:28] here's where we're going to do a completion of say just five tokens and
[56:32] completion of say just five tokens and these models have what we call in
[56:33] these models have what we call in context learning abilities and what
[56:36] context learning abilities and what that's referring to is that as it is
[56:37] that's referring to is that as it is reading this context it is learning sort
[56:40] reading this context it is learning sort of in
[56:42] of in place that there's some kind of a
[56:44] place that there's some kind of a algorithmic pattern going on in my data
[56:46] algorithmic pattern going on in my data and it knows to continue that pattern
[56:49] and it knows to continue that pattern and this is called kind of like Inc
[56:50] and this is called kind of like Inc context learning so it takes on the role
[56:53] context learning so it takes on the role of a
[56:55] of a translator and when we hit uh completion
[56:58] translator and when we hit uh completion we see that the teacher translation is
[57:00] we see that the teacher translation is Sim which is correct um and so this is
[57:03] Sim which is correct um and so this is how you can build apps by being clever
[57:05] how you can build apps by being clever with your prompting even though we still
[57:07] with your prompting even though we still just have a base model for now and it
[57:09] just have a base model for now and it relies on what we call this um uh in
[57:12] relies on what we call this um uh in context learning ability and it is done
[57:14] context learning ability and it is done by constructing what's called a few shot
[57:16] by constructing what's called a few shot prompt okay and finally I want to show
[57:18] prompt okay and finally I want to show you that there is a clever way to
[57:19] you that there is a clever way to actually instantiate a whole language
[57:22] actually instantiate a whole language model assistant just by prompting and
[57:25] model assistant just by prompting and the trick to it is that we're structure
[57:27] the trick to it is that we're structure a prompt to look like a web page that is
[57:29] a prompt to look like a web page that is a conversation between a helpful AI
[57:32] a conversation between a helpful AI assistant and a human and then the model
[57:34] assistant and a human and then the model will continue that conversation so
[57:37] will continue that conversation so actually to write the prompt I turned to
[57:39] actually to write the prompt I turned to chat gbt itself which is kind of meta
[57:42] chat gbt itself which is kind of meta but I told it I want to create an llm
[57:44] but I told it I want to create an llm assistant but all I have is the base
[57:45] assistant but all I have is the base model so can you please write my um uh
[57:50] model so can you please write my um uh prompt and this is what it came up with
[57:52] prompt and this is what it came up with which is actually quite good so here's a
[57:54] which is actually quite good so here's a conversation between an AI assistant and
[57:56] conversation between an AI assistant and a human
[57:57] a human the AI assistant is knowledgeable
[57:58] the AI assistant is knowledgeable helpful capable of answering wide
[58:00] helpful capable of answering wide variety of questions Etc and then here
[58:03] variety of questions Etc and then here it's not enough to just give it a sort
[58:05] it's not enough to just give it a sort of description it works much better if
[58:08] of description it works much better if you create this fot prompt so here's a
[58:10] you create this fot prompt so here's a few terms of human assistant human
[58:13] few terms of human assistant human assistant and we have uh you know a few
[58:16] assistant and we have uh you know a few turns of conversation and then here at
[58:18] turns of conversation and then here at the end is we're going to be putting the
[58:19] the end is we're going to be putting the actual query that we like so let me copy
[58:22] actual query that we like so let me copy paste this into the base model prompt
[58:25] paste this into the base model prompt and now let me do human column and this
[58:29] and now let me do human column and this is where we put our actual prompt why is
[58:31] is where we put our actual prompt why is the sky
[58:33] the sky blue and uh let's uh
[58:37] blue and uh let's uh run assistant the sky appears blue due
[58:40] run assistant the sky appears blue due to the phenomenon called R lights
[58:42] to the phenomenon called R lights scattering etc etc so you see that the
[58:44] scattering etc etc so you see that the base model is just continuing the
[58:45] base model is just continuing the sequence but because the sequence looks
[58:48] sequence but because the sequence looks like this conversation it takes on that
[58:50] like this conversation it takes on that role but it is a little subtle because
[58:52] role but it is a little subtle because here it just uh you know it ends the
[58:54] here it just uh you know it ends the assistant and then just you know
[58:56] assistant and then just you know hallucinate Ates the next question by
[58:57] hallucinate Ates the next question by the human Etc so it'll just continue
[58:59] the human Etc so it'll just continue going on and on uh but you can see that
[59:01] going on and on uh but you can see that we have sort of accomplished the task
[59:04] we have sort of accomplished the task and if you just took this why is the sky
[59:06] and if you just took this why is the sky blue and if we just refresh this and put
[59:09] blue and if we just refresh this and put it here then of course we don't expect
[59:11] it here then of course we don't expect this to work with a base model right
[59:12] this to work with a base model right we're just going to who knows what we're
[59:14] we're just going to who knows what we're going to get okay we're just going to
[59:15] going to get okay we're just going to get more
[59:16] get more questions okay so this is one way to
[59:19] questions okay so this is one way to create an assistant even though you may
[59:22] create an assistant even though you may only have a base model okay so this is
[59:24] only have a base model okay so this is the kind of brief summary of the things
[59:26] the kind of brief summary of the things we talked about over the last few
[59:29] we talked about over the last few minutes now let me zoom out
[59:33] minutes now let me zoom out here and this is kind of like what we've
[59:35] here and this is kind of like what we've talked about so far we wish to train LM
[59:38] talked about so far we wish to train LM assistants like chpt we've discussed the
[59:41] assistants like chpt we've discussed the first stage of that which is the
[59:42] first stage of that which is the pre-training stage and we saw that
[59:44] pre-training stage and we saw that really what it comes down to is we take
[59:45] really what it comes down to is we take Internet documents we break them up into
[59:47] Internet documents we break them up into these tokens these atoms of little text
[59:49] these tokens these atoms of little text chunks and then we predict token
[59:51] chunks and then we predict token sequences using neural networks the
[59:54] sequences using neural networks the output of this entire stage is this base
[59:57] output of this entire stage is this base model it is the setting of The
[59:58] model it is the setting of The parameters of this network and this base
[01:00:02] parameters of this network and this base model is basically an internet document
[01:00:03] model is basically an internet document simulator on the token level so it can
[01:00:06] simulator on the token level so it can just uh it can generate token sequences
[01:00:08] just uh it can generate token sequences that have the same kind of like
[01:00:10] that have the same kind of like statistics as Internet documents and we
[01:00:13] statistics as Internet documents and we saw that we can use it in some
[01:00:14] saw that we can use it in some applications but we actually need to do
[01:00:15] applications but we actually need to do better we want an assistant we want to
[01:00:17] better we want an assistant we want to be able to ask questions and we want the
[01:00:19] be able to ask questions and we want the model to give us answers and so we need
[01:00:21] model to give us answers and so we need to now go into the second stage which is
[01:00:24] to now go into the second stage which is called the post-training stage so we
[01:00:26] called the post-training stage so we take our base model our internet
[01:00:28] take our base model our internet document simulator and hand it off to
[01:00:30] document simulator and hand it off to post training so we're now going to
[01:00:32] post training so we're now going to discuss a few ways to do what's called
[01:00:34] discuss a few ways to do what's called post training of these models these
[01:00:36] post training of these models these stages in post training are going to be
[01:00:38] stages in post training are going to be computationally much less expensive most
[01:00:40] computationally much less expensive most of the computational work all of the
[01:00:42] of the computational work all of the massive data centers um and all of the
[01:00:45] massive data centers um and all of the sort of heavy compute and millions of
[01:00:48] sort of heavy compute and millions of dollars are the pre-training stage but
[01:00:51] dollars are the pre-training stage but now we go into the slightly cheaper but
[01:00:53] now we go into the slightly cheaper but still extremely important stage called
[01:00:55] still extremely important stage called post trining where we turn this llm
[01:00:57] post trining where we turn this llm model into an assistant so let's take a
[01:00:59] model into an assistant so let's take a look at how we can get our model to not
[01:01:02] look at how we can get our model to not sample internet documents but to give
[01:01:05] sample internet documents but to give answers to questions so in other words
[01:01:08] answers to questions so in other words what we want to do is we want to start
[01:01:09] what we want to do is we want to start thinking about conversations and these
[01:01:11] thinking about conversations and these are conversations that can be multi-turn
[01:01:13] are conversations that can be multi-turn so so uh there can be multiple turns and
[01:01:16] so so uh there can be multiple turns and they are in the simplest case a
[01:01:17] they are in the simplest case a conversation between a human and an
[01:01:19] conversation between a human and an assistant and so for example we can
[01:01:21] assistant and so for example we can imagine the conversation could look
[01:01:22] imagine the conversation could look something like this when a human says
[01:01:24] something like this when a human says what is 2 plus2 the assistant should re
[01:01:26] what is 2 plus2 the assistant should re respond with something like 2 plus 2 is
[01:01:27] respond with something like 2 plus 2 is 4 when a human follows up and says what
[01:01:30] 4 when a human follows up and says what if it was star instead of a plus
[01:01:32] if it was star instead of a plus assistant could respond with something
[01:01:33] assistant could respond with something like
[01:01:34] like this um and similar here this is another
[01:01:36] this um and similar here this is another example showing that the assistant could
[01:01:38] example showing that the assistant could also have some kind of a personality
[01:01:39] also have some kind of a personality here uh that it's kind of like nice and
[01:01:42] here uh that it's kind of like nice and then here in the third example I'm
[01:01:43] then here in the third example I'm showing that when a human is asking for
[01:01:45] showing that when a human is asking for something that we uh don't wish to help
[01:01:47] something that we uh don't wish to help with we can produce what's called
[01:01:49] with we can produce what's called refusal we can say that we cannot help
[01:01:51] refusal we can say that we cannot help with that so in other words what we want
[01:01:53] with that so in other words what we want to do now is we want to think through
[01:01:55] to do now is we want to think through how in a system should interact with the
[01:01:57] how in a system should interact with the human and we want to program the
[01:01:59] human and we want to program the assistant and Its Behavior in these
[01:02:01] assistant and Its Behavior in these conversations now because this is neural
[01:02:04] conversations now because this is neural networks we're not going to be
[01:02:05] networks we're not going to be programming these explicitly in code
[01:02:08] programming these explicitly in code we're not going to be able to program
[01:02:09] we're not going to be able to program the assistant in that way because this
[01:02:10] the assistant in that way because this is neural networks everything is done
[01:02:12] is neural networks everything is done through neural network training on data
[01:02:14] through neural network training on data sets and so because of that we are going
[01:02:17] sets and so because of that we are going to be implicitly programming the
[01:02:19] to be implicitly programming the assistant by creating data sets of
[01:02:22] assistant by creating data sets of conversations so these are three
[01:02:23] conversations so these are three independent examples of conversations in
[01:02:25] independent examples of conversations in a data dat set an actual data set and
[01:02:28] a data dat set an actual data set and I'm going to show you examples will be
[01:02:29] I'm going to show you examples will be much larger it could have hundreds of
[01:02:31] much larger it could have hundreds of thousands of conversations that are
[01:02:32] thousands of conversations that are multi- turn very long Etc and would
[01:02:35] multi- turn very long Etc and would cover a diverse breath of topics but
[01:02:37] cover a diverse breath of topics but here I'm only showing three examples but
[01:02:40] here I'm only showing three examples but the way this works basically is uh a
[01:02:43] the way this works basically is uh a assistant is being programmed by example
[01:02:46] assistant is being programmed by example and where is this data coming from like
[01:02:47] and where is this data coming from like 2 * 2al 4 same as 2 plus 2 Etc where
[01:02:50] 2 * 2al 4 same as 2 plus 2 Etc where does that come from this comes from
[01:02:52] does that come from this comes from Human labelers so we will basically give
[01:02:55] Human labelers so we will basically give human labelers some conversational
[01:02:56] human labelers some conversational context and we will ask them to um
[01:02:59] context and we will ask them to um basically give the ideal assistant
[01:03:01] basically give the ideal assistant response in this situation and a human
[01:03:04] response in this situation and a human will write out the ideal response for an
[01:03:06] will write out the ideal response for an assistant in any situation and then
[01:03:08] assistant in any situation and then we're going to get the model to
[01:03:10] we're going to get the model to basically train on this and to imitate
[01:03:13] basically train on this and to imitate those kinds of
[01:03:14] those kinds of responses so the way this works then is
[01:03:16] responses so the way this works then is we are going to take our base model
[01:03:18] we are going to take our base model which we produced in the preing stage
[01:03:20] which we produced in the preing stage and this base model was trained on
[01:03:22] and this base model was trained on internet documents we're now going to
[01:03:24] internet documents we're now going to take that data set of internet documents
[01:03:25] take that data set of internet documents and we're gonna throw it out and we're
[01:03:27] and we're gonna throw it out and we're going to substitute a new data set and
[01:03:29] going to substitute a new data set and that's going to be a data set of
[01:03:31] that's going to be a data set of conversations and we're going to
[01:03:32] conversations and we're going to continue training the model on these
[01:03:34] continue training the model on these conversations on this new data set of
[01:03:36] conversations on this new data set of conversations and what happens is that
[01:03:38] conversations and what happens is that the model will very rapidly adjust and
[01:03:41] the model will very rapidly adjust and will sort of like learn the statistics
[01:03:43] will sort of like learn the statistics of how this assistant responds to human
[01:03:46] of how this assistant responds to human queries and then later during inference
[01:03:48] queries and then later during inference we'll be able to basically um Prime the
[01:03:52] we'll be able to basically um Prime the assistant and get the response and it
[01:03:54] assistant and get the response and it will be imitating what the humans will
[01:03:56] will be imitating what the humans will human labelers would do in that
[01:03:57] human labelers would do in that situation if that makes sense so we're
[01:04:00] situation if that makes sense so we're going to see examples of that and this
[01:04:01] going to see examples of that and this is going to become bit more concrete I
[01:04:03] is going to become bit more concrete I also wanted to mention that this
[01:04:05] also wanted to mention that this post-training stage we're going to
[01:04:06] post-training stage we're going to basically just continue training the
[01:04:08] basically just continue training the model but um the pre-training stage can
[01:04:11] model but um the pre-training stage can in practice take roughly three months of
[01:04:13] in practice take roughly three months of training on many thousands of computers
[01:04:15] training on many thousands of computers the post-training stage will typically
[01:04:17] the post-training stage will typically be much shorter like 3 hours for example
[01:04:20] be much shorter like 3 hours for example um and that's because the data set of
[01:04:22] um and that's because the data set of conversations that we're going to create
[01:04:23] conversations that we're going to create here manually is much much smaller than
[01:04:26] here manually is much much smaller than the data set of text on the internet and
[01:04:29] the data set of text on the internet and so this training will be very short but
[01:04:31] so this training will be very short but fundamentally we're just going to take
[01:04:33] fundamentally we're just going to take our base model we're going to continue
[01:04:35] our base model we're going to continue training using the exact same algorithm
[01:04:37] training using the exact same algorithm the exact same everything except we're
[01:04:39] the exact same everything except we're swapping out the data set for
[01:04:41] swapping out the data set for conversations so the questions now are
[01:04:43] conversations so the questions now are what are these conversations how do we
[01:04:45] what are these conversations how do we represent them how do we get the model
[01:04:47] represent them how do we get the model to see conversations instead of just raw
[01:04:49] to see conversations instead of just raw text and then what are the outcomes of
[01:04:53] text and then what are the outcomes of um this kind of training and what do you
[01:04:55] um this kind of training and what do you get in a certain like psychological
[01:04:57] get in a certain like psychological sense uh when we talk about the model so
[01:04:59] sense uh when we talk about the model so let's turn to those questions now so
[01:05:01] let's turn to those questions now so let's start by talking about the
[01:05:02] let's start by talking about the tokenization of conversations everything
[01:05:05] tokenization of conversations everything in these models has to be turned into
[01:05:07] in these models has to be turned into tokens because everything is just about
[01:05:09] tokens because everything is just about token sequences so how do we turn
[01:05:11] token sequences so how do we turn conversations into token sequences is
[01:05:13] conversations into token sequences is the question and so for that we need to
[01:05:15] the question and so for that we need to design some kind of ending coding and uh
[01:05:17] design some kind of ending coding and uh this is kind of similar to maybe if
[01:05:19] this is kind of similar to maybe if you're familiar you don't have to be
[01:05:21] you're familiar you don't have to be with for example the TCP IP packet in um
[01:05:24] with for example the TCP IP packet in um on the internet there are precise rules
[01:05:25] on the internet there are precise rules and protocols for how you represent
[01:05:27] and protocols for how you represent information how everything is structured
[01:05:29] information how everything is structured together so that you have all this kind
[01:05:30] together so that you have all this kind of data laid out in a way that is
[01:05:33] of data laid out in a way that is written out on a paper and that everyone
[01:05:35] written out on a paper and that everyone can agree on and so it's the same thing
[01:05:37] can agree on and so it's the same thing now happening in llms we need some kind
[01:05:38] now happening in llms we need some kind of data structures and we need to have
[01:05:40] of data structures and we need to have some rules around how these data
[01:05:42] some rules around how these data structures like conversations get
[01:05:44] structures like conversations get encoded and decoded to and from tokens
[01:05:47] encoded and decoded to and from tokens and so I want to show you now how I
[01:05:49] and so I want to show you now how I would
[01:05:50] would recreate uh this conversation in the
[01:05:52] recreate uh this conversation in the token space so if you go to Tech
[01:05:54] token space so if you go to Tech tokenizer
[01:05:56] tokenizer I can take that conversation and this is
[01:05:59] I can take that conversation and this is how it is represented in uh for the
[01:06:01] how it is represented in uh for the language model so here we have we are
[01:06:04] language model so here we have we are iterating a user and an assistant in
[01:06:06] iterating a user and an assistant in this two- turn
[01:06:08] this two- turn conversation and what you're seeing here
[01:06:10] conversation and what you're seeing here is it looks ugly but it's actually
[01:06:12] is it looks ugly but it's actually relatively simple the way it gets turned
[01:06:14] relatively simple the way it gets turned into a token sequence here at the end is
[01:06:16] into a token sequence here at the end is a little bit complicated but at the end
[01:06:18] a little bit complicated but at the end this conversation between a user and
[01:06:20] this conversation between a user and assistant ends up being 49 tokens it is
[01:06:23] assistant ends up being 49 tokens it is a one-dimensional sequence of 49 tokens
[01:06:25] a one-dimensional sequence of 49 tokens and these are the tokens
[01:06:26] and these are the tokens okay and all the different llms will
[01:06:30] okay and all the different llms will have a slightly different format or
[01:06:32] have a slightly different format or protocols and it's a little bit of a
[01:06:33] protocols and it's a little bit of a wild west right now but for example GPT
[01:06:36] wild west right now but for example GPT 40 does it in the following way you have
[01:06:39] 40 does it in the following way you have this special token called imore start
[01:06:42] this special token called imore start and this is short for IM imaginary
[01:06:44] and this is short for IM imaginary monologue uh the
[01:06:46] monologue uh the start then you have to specify um I
[01:06:50] start then you have to specify um I don't actually know why it's called that
[01:06:51] don't actually know why it's called that to be honest then you have to specify
[01:06:53] to be honest then you have to specify whose turn it is so for example user
[01:06:55] whose turn it is so for example user which is a token 4
[01:06:57] which is a token 4 28 then you have internal monologue
[01:07:00] 28 then you have internal monologue separator and then it's the exact
[01:07:03] separator and then it's the exact question so the tokens of the question
[01:07:05] question so the tokens of the question and then you have to close it so I am
[01:07:07] and then you have to close it so I am end the end of the imaginary monologue
[01:07:10] end the end of the imaginary monologue so
[01:07:11] so basically the question from a user of
[01:07:13] basically the question from a user of what is 2 plus two ends up being the
[01:07:16] what is 2 plus two ends up being the token sequence of these tokens and now
[01:07:19] token sequence of these tokens and now the important thing to mention here is
[01:07:21] the important thing to mention here is that IM start this is not text right IM
[01:07:24] that IM start this is not text right IM start is a special token that gets added
[01:07:27] start is a special token that gets added it's a new token and um this token has
[01:07:31] it's a new token and um this token has never been trained on so far it is a new
[01:07:33] never been trained on so far it is a new token that we create in a post-training
[01:07:35] token that we create in a post-training stage and we introduce and so these
[01:07:38] stage and we introduce and so these special tokens like IM seep IM start Etc
[01:07:40] special tokens like IM seep IM start Etc are introduced and interspersed with
[01:07:43] are introduced and interspersed with text so that they sort of um get the
[01:07:45] text so that they sort of um get the model to learn that hey this is a the
[01:07:47] model to learn that hey this is a the start of a turn for who is it start of
[01:07:50] start of a turn for who is it start of the turn for the start of the turn is
[01:07:52] the turn for the start of the turn is for the user and then this is what the
[01:07:54] for the user and then this is what the user says and then the user ends and
[01:07:57] user says and then the user ends and then it's a new start of a turn and it
[01:07:59] then it's a new start of a turn and it is by the assistant and then what does
[01:08:01] is by the assistant and then what does the assistant say well these are the
[01:08:03] the assistant say well these are the tokens of what the assistant says Etc
[01:08:06] tokens of what the assistant says Etc and so this conversation is not turned
[01:08:07] and so this conversation is not turned into the sequence of tokens the specific
[01:08:10] into the sequence of tokens the specific details here are not actually that
[01:08:11] details here are not actually that important all I'm trying to show you in
[01:08:13] important all I'm trying to show you in concrete terms is that our conversations
[01:08:16] concrete terms is that our conversations which we think of as kind of like a
[01:08:17] which we think of as kind of like a structured object end up being turned
[01:08:19] structured object end up being turned via some encoding into onedimensional
[01:08:22] via some encoding into onedimensional sequences of tokens and so because this
[01:08:25] sequences of tokens and so because this is one dimensional sequence of tokens we
[01:08:27] is one dimensional sequence of tokens we can apply all the stuff that we applied
[01:08:29] can apply all the stuff that we applied before now it's just a sequence of
[01:08:31] before now it's just a sequence of tokens and now we can train a language
[01:08:33] tokens and now we can train a language model on it and so we're just predicting
[01:08:35] model on it and so we're just predicting the next token in a sequence uh just
[01:08:37] the next token in a sequence uh just like before and um we can represent and
[01:08:40] like before and um we can represent and train on conversations and then what
[01:08:42] train on conversations and then what does it look like at test time during
[01:08:44] does it look like at test time during inference so say we've trained a model
[01:08:47] inference so say we've trained a model and we've trained a model on these kinds
[01:08:49] and we've trained a model on these kinds of data sets of conversations and now we
[01:08:51] of data sets of conversations and now we want to
[01:08:52] want to inference so during inference what does
[01:08:55] inference so during inference what does this look like when you're on on chash
[01:08:56] this look like when you're on on chash apt well you come to chash apt and you
[01:08:59] apt well you come to chash apt and you have say like a dialogue with it and the
[01:09:01] have say like a dialogue with it and the way this works is
[01:09:03] way this works is basically um say that this was already
[01:09:06] basically um say that this was already filled in so like what is 2 plus 2 2
[01:09:08] filled in so like what is 2 plus 2 2 plus 2 is four and now you issue what if
[01:09:10] plus 2 is four and now you issue what if it was times I am end and what basically
[01:09:14] it was times I am end and what basically ends up happening um on the servers of
[01:09:16] ends up happening um on the servers of open AI or something like that is they
[01:09:18] open AI or something like that is they put in I start assistant I amep and this
[01:09:22] put in I start assistant I amep and this is where they end it right here so they
[01:09:25] is where they end it right here so they construct this context and now they
[01:09:27] construct this context and now they start sampling from the model so it's at
[01:09:29] start sampling from the model so it's at this stage that they will go to the
[01:09:31] this stage that they will go to the model and say okay what is a good for
[01:09:32] model and say okay what is a good for sequence what is a good first token what
[01:09:35] sequence what is a good first token what is a good second token what is a good
[01:09:37] is a good second token what is a good third token and this is where the LM
[01:09:38] third token and this is where the LM takes over and creates a response like
[01:09:41] takes over and creates a response like for example response that looks
[01:09:44] for example response that looks something like this but it doesn't have
[01:09:45] something like this but it doesn't have to be identical to this but it will have
[01:09:47] to be identical to this but it will have the flavor of this if this kind of a
[01:09:49] the flavor of this if this kind of a conversation was in the data set so um
[01:09:52] conversation was in the data set so um that's roughly how the protocol Works
[01:09:55] that's roughly how the protocol Works although the details of this protocol
[01:09:57] although the details of this protocol are not important so again my goal is
[01:09:59] are not important so again my goal is that just to show you that everything
[01:10:01] that just to show you that everything ends up being just a one-dimensional
[01:10:02] ends up being just a one-dimensional token sequence so we can apply
[01:10:04] token sequence so we can apply everything we've already seen but we're
[01:10:07] everything we've already seen but we're now training on conversations and we're
[01:10:09] now training on conversations and we're now uh basically generating
[01:10:11] now uh basically generating conversations as well okay so now I
[01:10:13] conversations as well okay so now I would like to turn to what these data
[01:10:14] would like to turn to what these data sets look like in practice the first
[01:10:16] sets look like in practice the first paper that I would like to show you and
[01:10:18] paper that I would like to show you and the first effort in this direction is
[01:10:20] the first effort in this direction is this paper from openai in 2022 and this
[01:10:23] this paper from openai in 2022 and this paper was called instruct GPT or the
[01:10:26] paper was called instruct GPT or the technique that they developed and this
[01:10:27] technique that they developed and this was the first time that opena has kind
[01:10:29] was the first time that opena has kind of talked about how you can take
[01:10:30] of talked about how you can take language models and fine-tune them on
[01:10:33] language models and fine-tune them on conversations and so this paper has a
[01:10:35] conversations and so this paper has a number of details that I would like to
[01:10:36] number of details that I would like to take you through so the first stop I
[01:10:38] take you through so the first stop I would like to make is in section 3.4
[01:10:40] would like to make is in section 3.4 where they talk about the human
[01:10:42] where they talk about the human contractors that they hired uh in this
[01:10:44] contractors that they hired uh in this case from upwork or through scale AI to
[01:10:47] case from upwork or through scale AI to uh construct these conversations and so
[01:10:50] uh construct these conversations and so there are human labelers involved whose
[01:10:52] there are human labelers involved whose job it is professionally to create these
[01:10:55] job it is professionally to create these conversations and these labelers are
[01:10:57] conversations and these labelers are asked to come up with prompts and then
[01:10:59] asked to come up with prompts and then they are asked to also complete the
[01:11:01] they are asked to also complete the ideal assistant responses and so these
[01:11:03] ideal assistant responses and so these are the kinds of prompts that people
[01:11:05] are the kinds of prompts that people came up with so these are human labelers
[01:11:07] came up with so these are human labelers so list five ideas for how to regain
[01:11:09] so list five ideas for how to regain enthusiasm for my career what are the
[01:11:11] enthusiasm for my career what are the top 10 science fiction books I should
[01:11:12] top 10 science fiction books I should read next and there's many different
[01:11:14] read next and there's many different types of uh kind of prompts here so
[01:11:17] types of uh kind of prompts here so translate this sentence from uh to
[01:11:19] translate this sentence from uh to Spanish Etc and so there's many things
[01:11:22] Spanish Etc and so there's many things here that people came up with they first
[01:11:24] here that people came up with they first come up with the prompt and then they
[01:11:26] come up with the prompt and then they also uh answer that prompt and they give
[01:11:28] also uh answer that prompt and they give the ideal assistant response now how do
[01:11:30] the ideal assistant response now how do they know what is the ideal assistant
[01:11:32] they know what is the ideal assistant response that they should write for
[01:11:34] response that they should write for these prompts so when we scroll down a
[01:11:36] these prompts so when we scroll down a little bit further we see that here we
[01:11:38] little bit further we see that here we have this excerpt of labeling
[01:11:40] have this excerpt of labeling instructions uh that are given to the
[01:11:41] instructions uh that are given to the human labelers so the company that is
[01:11:44] human labelers so the company that is developing the language model like for
[01:11:45] developing the language model like for example open AI writes up labeling
[01:11:47] example open AI writes up labeling instructions for how the humans should
[01:11:50] instructions for how the humans should create ideal responses and so here for
[01:11:53] create ideal responses and so here for example is an excerpt uh of these kinds
[01:11:55] example is an excerpt uh of these kinds of labeling instruction instructions on
[01:11:56] of labeling instruction instructions on High level you're asking people to be
[01:11:58] High level you're asking people to be helpful truthful and harmless and you
[01:12:00] helpful truthful and harmless and you can pause the video if you'd like to see
[01:12:02] can pause the video if you'd like to see more here but on a high level basically
[01:12:04] more here but on a high level basically just just answer try to be helpful try
[01:12:06] just just answer try to be helpful try to be truthful and don't answer
[01:12:08] to be truthful and don't answer questions that we don't want um kind of
[01:12:10] questions that we don't want um kind of the system to handle uh later in chat
[01:12:13] the system to handle uh later in chat gbt and so roughly speaking the company
[01:12:17] gbt and so roughly speaking the company comes up with the labeling instructions
[01:12:18] comes up with the labeling instructions usually they are not this short usually
[01:12:20] usually they are not this short usually there are hundreds of pages and people
[01:12:22] there are hundreds of pages and people have to study them professionally and
[01:12:24] have to study them professionally and then they write out the ideal assistant
[01:12:26] then they write out the ideal assistant responses uh following those labeling
[01:12:28] responses uh following those labeling instructions so this is a very human
[01:12:31] instructions so this is a very human heavy process as it was described in
[01:12:33] heavy process as it was described in this paper now the data set for instruct
[01:12:35] this paper now the data set for instruct GPT was never actually released by openi
[01:12:37] GPT was never actually released by openi but we do have some open- Source um
[01:12:40] but we do have some open- Source um reproductions that were're trying to
[01:12:41] reproductions that were're trying to follow this kind of a setup and collect
[01:12:43] follow this kind of a setup and collect their own data so one that I'm familiar
[01:12:45] their own data so one that I'm familiar with for example is the effort of open
[01:12:48] with for example is the effort of open Assistant from a while back and this is
[01:12:50] Assistant from a while back and this is just one of I think many examples but I
[01:12:52] just one of I think many examples but I just want to show you an example so
[01:12:55] just want to show you an example so here's so these were people on the
[01:12:56] here's so these were people on the internet that were asked to basically
[01:12:58] internet that were asked to basically create these conversations similar to
[01:13:00] create these conversations similar to what um open I did with human labelers
[01:13:03] what um open I did with human labelers and so here's an entry of a person who
[01:13:05] and so here's an entry of a person who came up with this BR can you write a
[01:13:07] came up with this BR can you write a short introduction to the relevance of
[01:13:09] short introduction to the relevance of the term
[01:13:10] the term manop uh in economics please use
[01:13:13] manop uh in economics please use examples Etc and then the same person or
[01:13:15] examples Etc and then the same person or potentially a different person will
[01:13:17] potentially a different person will write up the response so here's the
[01:13:19] write up the response so here's the assistant response to this and so then
[01:13:22] assistant response to this and so then the same person or different person will
[01:13:24] the same person or different person will actually write out this ideal
[01:13:27] actually write out this ideal response and then this is an example of
[01:13:29] response and then this is an example of maybe how the conversation could
[01:13:31] maybe how the conversation could continue now explain it to a dog and
[01:13:33] continue now explain it to a dog and then you can try to come up with a
[01:13:35] then you can try to come up with a slightly a simpler explanation or
[01:13:37] slightly a simpler explanation or something like that now this then
[01:13:40] something like that now this then becomes the label and we end up training
[01:13:42] becomes the label and we end up training on this so what happens during training
[01:13:45] on this so what happens during training is that um of course we're not going to
[01:13:48] is that um of course we're not going to have a full coverage of all the possible
[01:13:51] have a full coverage of all the possible questions that um the model will
[01:13:54] questions that um the model will encounter at test time during inference
[01:13:56] encounter at test time during inference we can't possibly cover all the possible
[01:13:58] we can't possibly cover all the possible prompts that people are going to be
[01:13:59] prompts that people are going to be asking in the future but if we have a
[01:14:02] asking in the future but if we have a like a data set of a few of these
[01:14:04] like a data set of a few of these examples then the model during training
[01:14:07] examples then the model during training will start to take on this Persona of
[01:14:09] will start to take on this Persona of this helpful truthful harmless assistant
[01:14:12] this helpful truthful harmless assistant and it's all programmed by example and
[01:14:15] and it's all programmed by example and so these are all examples of behavior
[01:14:17] so these are all examples of behavior and if you have conversations of these
[01:14:18] and if you have conversations of these example behaviors and you have enough of
[01:14:20] example behaviors and you have enough of them like 100,00 and you train on it the
[01:14:22] them like 100,00 and you train on it the model sort of starts to understand the
[01:14:24] model sort of starts to understand the statistical pattern and it kind of takes
[01:14:26] statistical pattern and it kind of takes on this personality of this
[01:14:28] on this personality of this assistant now it's possible that when
[01:14:30] assistant now it's possible that when you get the exact same question like
[01:14:32] you get the exact same question like this at test time it's possible that the
[01:14:36] this at test time it's possible that the answer will be recited as exactly what
[01:14:39] answer will be recited as exactly what was in the training set but more likely
[01:14:41] was in the training set but more likely than that is that the model will kind of
[01:14:43] than that is that the model will kind of like do something of a similar Vibe um
[01:14:46] like do something of a similar Vibe um and we will understand that this is the
[01:14:47] and we will understand that this is the kind of answer that you want um so
[01:14:51] kind of answer that you want um so that's what we're doing we're
[01:14:52] that's what we're doing we're programming the system um by example and
[01:14:56] programming the system um by example and the system adopts statistically this
[01:14:58] the system adopts statistically this Persona of this helpful truthful
[01:15:01] Persona of this helpful truthful harmless assistant which is kind of like
[01:15:03] harmless assistant which is kind of like reflected in the labeling instructions
[01:15:04] reflected in the labeling instructions that the company creates now I want to
[01:15:06] that the company creates now I want to show you that the state-of-the-art has
[01:15:08] show you that the state-of-the-art has kind of advanced in the last 2 or 3
[01:15:10] kind of advanced in the last 2 or 3 years uh since the instr GPT paper so in
[01:15:13] years uh since the instr GPT paper so in particular it's not very common for
[01:15:15] particular it's not very common for humans to be doing all the heavy lifting
[01:15:16] humans to be doing all the heavy lifting just by themselves anymore and that's
[01:15:18] just by themselves anymore and that's because we now have language models and
[01:15:20] because we now have language models and these language models are helping us
[01:15:21] these language models are helping us create these data sets and conversations
[01:15:23] create these data sets and conversations so it is very rare that the people will
[01:15:25] so it is very rare that the people will like literally just write out the
[01:15:26] like literally just write out the response from scratch it is a lot more
[01:15:29] response from scratch it is a lot more likely that they will use an existing
[01:15:30] likely that they will use an existing llm to basically like uh come up with an
[01:15:32] llm to basically like uh come up with an answer and then they will edit it or
[01:15:34] answer and then they will edit it or things like that so there's many
[01:15:36] things like that so there's many different ways in which now llms have
[01:15:38] different ways in which now llms have started to kind of permeate this
[01:15:40] started to kind of permeate this posttraining Set uh stack and llms are
[01:15:43] posttraining Set uh stack and llms are basically used pervasively to help
[01:15:45] basically used pervasively to help create these massive data sets of
[01:15:47] create these massive data sets of conversations so I don't want to show
[01:15:49] conversations so I don't want to show like Ultra chat is one um such example
[01:15:52] like Ultra chat is one um such example of like a more modern data set of
[01:15:54] of like a more modern data set of conversations it is to a very large
[01:15:56] conversations it is to a very large extent synthetic but uh I believe
[01:15:58] extent synthetic but uh I believe there's some human involvement I could
[01:16:00] there's some human involvement I could be wrong with that usually there will be
[01:16:01] be wrong with that usually there will be a little bit of human but there will be
[01:16:03] a little bit of human but there will be a huge amount of synthetic help um and
[01:16:06] a huge amount of synthetic help um and this is all kind of like uh constructed
[01:16:09] this is all kind of like uh constructed in different ways and Ultra chat is just
[01:16:10] in different ways and Ultra chat is just one example of many sft data sets that
[01:16:12] one example of many sft data sets that currently exist and the only thing I
[01:16:14] currently exist and the only thing I want to show you is that uh these data
[01:16:16] want to show you is that uh these data sets have now millions of conversations
[01:16:18] sets have now millions of conversations uh these conversations are mostly
[01:16:20] uh these conversations are mostly synthetic but they're probably edited to
[01:16:21] synthetic but they're probably edited to some extent by humans and they span a
[01:16:24] some extent by humans and they span a huge diversity of sort of
[01:16:27] huge diversity of sort of um uh areas and so on so these are
[01:16:32] um uh areas and so on so these are fairly extensive artifacts by now and
[01:16:34] fairly extensive artifacts by now and there's all these like sft mixtures as
[01:16:36] there's all these like sft mixtures as they're called so you have a mixture of
[01:16:38] they're called so you have a mixture of like lots of different types and sources
[01:16:39] like lots of different types and sources and it's partially synthetic partially
[01:16:41] and it's partially synthetic partially human and it's kind of like um gone in
[01:16:44] human and it's kind of like um gone in that direction since uh but roughly
[01:16:47] that direction since uh but roughly speaking we still have sft data sets
[01:16:49] speaking we still have sft data sets they're made up of conversations we're
[01:16:51] they're made up of conversations we're training on them um just like we did
[01:16:53] training on them um just like we did before and
[01:16:55] before and uh I guess like the last thing to note
[01:16:57] uh I guess like the last thing to note is that I want to dispel a little bit of
[01:17:00] is that I want to dispel a little bit of the magic of talking to an AI like when
[01:17:02] the magic of talking to an AI like when you go to chat GPT and you give it a
[01:17:04] you go to chat GPT and you give it a question and then you hit enter uh what
[01:17:08] question and then you hit enter uh what is coming back is kind of like
[01:17:10] is coming back is kind of like statistically aligned with what's
[01:17:12] statistically aligned with what's happening in the training set and these
[01:17:14] happening in the training set and these training sets I mean they really just
[01:17:17] training sets I mean they really just have a seed in humans following labeling
[01:17:19] have a seed in humans following labeling instructions so what are you actually
[01:17:22] instructions so what are you actually talking to in chat GPT or how should you
[01:17:24] talking to in chat GPT or how should you think about it well it's not coming from
[01:17:26] think about it well it's not coming from some magical AI like roughly speaking
[01:17:28] some magical AI like roughly speaking it's coming from something that is
[01:17:30] it's coming from something that is statistically imitating human labelers
[01:17:32] statistically imitating human labelers which comes from labeling instructions
[01:17:34] which comes from labeling instructions written by these companies and so you're
[01:17:36] written by these companies and so you're kind of imitating this uh you're kind of
[01:17:39] kind of imitating this uh you're kind of getting um it's almost as if you're
[01:17:40] getting um it's almost as if you're asking human labeler and imagine that
[01:17:43] asking human labeler and imagine that the answer that is given to you uh from
[01:17:45] the answer that is given to you uh from chbt is some kind of a simulation of a
[01:17:48] chbt is some kind of a simulation of a human labeler uh and it's kind of like
[01:17:51] human labeler uh and it's kind of like asking what would a human labeler say in
[01:17:53] asking what would a human labeler say in this kind of a conversation
[01:17:57] this kind of a conversation and uh it's not just like this human
[01:17:59] and uh it's not just like this human labeler is not just like a random person
[01:18:00] labeler is not just like a random person from the internet because these
[01:18:02] from the internet because these companies actually hire experts so for
[01:18:03] companies actually hire experts so for example when you are asking questions
[01:18:05] example when you are asking questions about code and so on the human labelers
[01:18:07] about code and so on the human labelers that would be in um involved in creation
[01:18:09] that would be in um involved in creation of these conversation data sets they
[01:18:11] of these conversation data sets they will usually be usually be educated
[01:18:13] will usually be usually be educated expert people and you're kind of like
[01:18:15] expert people and you're kind of like asking a question of like a simulation
[01:18:17] asking a question of like a simulation of those people if that makes sense so
[01:18:20] of those people if that makes sense so you're not talking to a magical AI
[01:18:21] you're not talking to a magical AI you're talking to an average labeler
[01:18:23] you're talking to an average labeler this average labeler is probably fairly
[01:18:24] this average labeler is probably fairly highly skilled
[01:18:25] highly skilled but you're talking to kind of like an
[01:18:27] but you're talking to kind of like an instantaneous simulation of that kind of
[01:18:29] instantaneous simulation of that kind of a person that would be hired uh in the
[01:18:32] a person that would be hired uh in the construction of these data sets so let
[01:18:35] construction of these data sets so let me give you one more specific example
[01:18:36] me give you one more specific example before we move on for example when I go
[01:18:39] before we move on for example when I go to chpt and I say recommend the top five
[01:18:41] to chpt and I say recommend the top five landmarks who see in Paris and then I
[01:18:43] landmarks who see in Paris and then I hit
[01:18:45] hit enter
[01:18:52] uh okay here we go okay when I hit enter what's coming out here how do I think
[01:18:55] what's coming out here how do I think about it well it's not some kind of a
[01:18:57] about it well it's not some kind of a magical AI that has gone out and
[01:18:59] magical AI that has gone out and researched all the landmarks and then
[01:19:00] researched all the landmarks and then ranked them using its infinite
[01:19:02] ranked them using its infinite intelligence Etc what I'm getting is a
[01:19:04] intelligence Etc what I'm getting is a statistical simulation of a labeler that
[01:19:07] statistical simulation of a labeler that was hired by open AI you can think about
[01:19:09] was hired by open AI you can think about it roughly in that way and so if this
[01:19:13] it roughly in that way and so if this specific um question is in the
[01:19:16] specific um question is in the posttraining data set somewhere at open
[01:19:18] posttraining data set somewhere at open aai then I'm very likely to see an
[01:19:21] aai then I'm very likely to see an answer that is probably very very
[01:19:22] answer that is probably very very similar to what that human labeler would
[01:19:24] similar to what that human labeler would have put down
[01:19:25] have put down for those five landmarks how does the
[01:19:27] for those five landmarks how does the human labeler come up with this well
[01:19:29] human labeler come up with this well they go off and they go on the internet
[01:19:30] they go off and they go on the internet and they kind of do their own little
[01:19:31] and they kind of do their own little research for 20 minutes and they just
[01:19:33] research for 20 minutes and they just come up with a list right now so if they
[01:19:35] come up with a list right now so if they come up with this list and this is in
[01:19:37] come up with this list and this is in the data set I'm probably very likely to
[01:19:39] the data set I'm probably very likely to see what they submitted as the correct
[01:19:42] see what they submitted as the correct answer from the assistant now if this
[01:19:45] answer from the assistant now if this specific query is not part of the post
[01:19:46] specific query is not part of the post training data set then what I'm getting
[01:19:48] training data set then what I'm getting here is a little bit more emergent uh
[01:19:51] here is a little bit more emergent uh because uh the model kind of understands
[01:19:54] because uh the model kind of understands the statistically
[01:19:55] the statistically um the kinds of landmarks that are in
[01:19:58] um the kinds of landmarks that are in this training set are usually the
[01:19:59] this training set are usually the prominent landmarks the landmarks that
[01:20:01] prominent landmarks the landmarks that people usually want to see the kinds of
[01:20:03] people usually want to see the kinds of landmarks that are usually uh very often
[01:20:05] landmarks that are usually uh very often talked about on the internet and
[01:20:07] talked about on the internet and remember that the model already has a
[01:20:09] remember that the model already has a ton of Knowledge from its pre-training
[01:20:10] ton of Knowledge from its pre-training on the internet so it's probably seen a
[01:20:12] on the internet so it's probably seen a ton of conversations about Paris about
[01:20:14] ton of conversations about Paris about landmarks about the kinds of things that
[01:20:15] landmarks about the kinds of things that people like to see and so it's the
[01:20:17] people like to see and so it's the pre-training knowledge that has then
[01:20:18] pre-training knowledge that has then combined with the postering data set
[01:20:21] combined with the postering data set that results in this kind of an
[01:20:24] that results in this kind of an imitation um
[01:20:25] imitation um so that's uh that's roughly how you can
[01:20:28] so that's uh that's roughly how you can kind of think about what's happening
[01:20:30] kind of think about what's happening behind the scenes here in in this
[01:20:32] behind the scenes here in in this statistical sense okay now I want to
[01:20:34] statistical sense okay now I want to turn to the topic of llm psychology as I
[01:20:36] turn to the topic of llm psychology as I like to call it which is what are sort
[01:20:38] like to call it which is what are sort of the emergent cognitive effects of the
[01:20:41] of the emergent cognitive effects of the training pipeline that we have for these
[01:20:42] training pipeline that we have for these models so in particular the first one I
[01:20:44] models so in particular the first one I want to talk to is of course
[01:20:47] want to talk to is of course hallucinations so you might be familiar
[01:20:50] hallucinations so you might be familiar with model hallucinations it's when llms
[01:20:52] with model hallucinations it's when llms make stuff up they just totally
[01:20:54] make stuff up they just totally fabricate information Etc and it's a big
[01:20:56] fabricate information Etc and it's a big problem with llm assistants it is a
[01:20:58] problem with llm assistants it is a problem that existed to a large extent
[01:21:00] problem that existed to a large extent with early models uh from many years ago
[01:21:03] with early models uh from many years ago and I think the problem has gotten a bit
[01:21:04] and I think the problem has gotten a bit better uh because there are some
[01:21:06] better uh because there are some medications that I'm going to go into in
[01:21:07] medications that I'm going to go into in a second for now let's just try to
[01:21:09] a second for now let's just try to understand where these hallucinations
[01:21:10] understand where these hallucinations come from so here's a specific example
[01:21:13] come from so here's a specific example of a few uh of three conversations that
[01:21:16] of a few uh of three conversations that you might think you have in your
[01:21:17] you might think you have in your training set and um these are pretty
[01:21:20] training set and um these are pretty reasonable conversations that you could
[01:21:22] reasonable conversations that you could imagine being in the training set so
[01:21:24] imagine being in the training set so like for example who is Cruz well Tom
[01:21:26] like for example who is Cruz well Tom Cruz is an famous actor American actor
[01:21:28] Cruz is an famous actor American actor and producer Etc who is John baraso this
[01:21:31] and producer Etc who is John baraso this turns out to be a us senetor for example
[01:21:34] turns out to be a us senetor for example who is genis Khan well genis Khan was
[01:21:37] who is genis Khan well genis Khan was blah blah blah and so this is what your
[01:21:40] blah blah blah and so this is what your conversations could look like at
[01:21:41] conversations could look like at training time now the problem with this
[01:21:43] training time now the problem with this is that when the human is writing the
[01:21:46] is that when the human is writing the correct answer for the assistant in each
[01:21:49] correct answer for the assistant in each one of these cases uh the human either
[01:21:51] one of these cases uh the human either like knows who this person is or they
[01:21:52] like knows who this person is or they research them on the Internet and they
[01:21:54] research them on the Internet and they come in and they write this response
[01:21:56] come in and they write this response that kind of has this like confident
[01:21:57] that kind of has this like confident tone of an answer and what happens
[01:22:00] tone of an answer and what happens basically is that at test time when you
[01:22:01] basically is that at test time when you ask for someone who is this is a totally
[01:22:04] ask for someone who is this is a totally random name that I totally came up with
[01:22:05] random name that I totally came up with and I don't think this person exists um
[01:22:08] and I don't think this person exists um as far as I know I just Tred to generate
[01:22:10] as far as I know I just Tred to generate it randomly the problem is when we ask
[01:22:12] it randomly the problem is when we ask who is Orson kovats the problem is that
[01:22:15] who is Orson kovats the problem is that the assistant will not just tell you oh
[01:22:18] the assistant will not just tell you oh I don't know even if the assistant and
[01:22:20] I don't know even if the assistant and the language model itself might know
[01:22:23] the language model itself might know inside its features inside its
[01:22:25] inside its features inside its activations inside of its brain sort of
[01:22:27] activations inside of its brain sort of it might know that this person is like
[01:22:28] it might know that this person is like not someone that um that is that it's
[01:22:31] not someone that um that is that it's familiar with even if some part of the
[01:22:33] familiar with even if some part of the network kind of knows that in some sense
[01:22:35] network kind of knows that in some sense the uh saying that oh I don't know who
[01:22:37] the uh saying that oh I don't know who this is is is not going to happen
[01:22:40] this is is is not going to happen because the model statistically imitates
[01:22:43] because the model statistically imitates is training set in the training set the
[01:22:46] is training set in the training set the questions of the form who is blah are
[01:22:48] questions of the form who is blah are confidently answered with the correct
[01:22:50] confidently answered with the correct answer and so it's going to take on the
[01:22:52] answer and so it's going to take on the style of the answer and it's going to do
[01:22:54] style of the answer and it's going to do its best it's going to give you
[01:22:55] its best it's going to give you statistically the most likely guess and
[01:22:58] statistically the most likely guess and it's just going to basically make stuff
[01:22:59] it's just going to basically make stuff up because these models again we just
[01:23:01] up because these models again we just talked about it is they don't have
[01:23:03] talked about it is they don't have access to the internet they're not doing
[01:23:04] access to the internet they're not doing research these are statistical token
[01:23:06] research these are statistical token tumblers as I call them uh is just
[01:23:09] tumblers as I call them uh is just trying to sample the next token in the
[01:23:10] trying to sample the next token in the sequence and it's going to basically
[01:23:12] sequence and it's going to basically make stuff up so let's take a look at
[01:23:14] make stuff up so let's take a look at what this looks
[01:23:15] what this looks like I have here what's called the
[01:23:18] like I have here what's called the inference playground from hugging face
[01:23:21] inference playground from hugging face and I am on purpose picking on a model
[01:23:23] and I am on purpose picking on a model called Falcon 7B which is an old model
[01:23:25] called Falcon 7B which is an old model this is a few years ago now so it's an
[01:23:28] this is a few years ago now so it's an older model So It suffers from
[01:23:29] older model So It suffers from hallucinations and as I mentioned this
[01:23:31] hallucinations and as I mentioned this has improved over time recently but
[01:23:33] has improved over time recently but let's say who is Orson kovats let's ask
[01:23:35] let's say who is Orson kovats let's ask Falcon 7B instruct
[01:23:38] Falcon 7B instruct run oh yeah Orson kovat is an American
[01:23:40] run oh yeah Orson kovat is an American author and science uh fiction writer
[01:23:43] author and science uh fiction writer okay this is totally false it's
[01:23:45] okay this is totally false it's hallucination let's try again these are
[01:23:47] hallucination let's try again these are statistical systems right so we can
[01:23:49] statistical systems right so we can resample this time Orson kovat is a
[01:23:51] resample this time Orson kovat is a fictional character from this 1950s TV
[01:23:54] fictional character from this 1950s TV show it's total BS right let's try again
[01:23:58] show it's total BS right let's try again he's a former minor league baseball
[01:24:00] he's a former minor league baseball player okay so basically the model
[01:24:03] player okay so basically the model doesn't know and it's given us lots of
[01:24:04] doesn't know and it's given us lots of different answers because it doesn't
[01:24:07] different answers because it doesn't know it's just kind of like sampling
[01:24:08] know it's just kind of like sampling from these probabilities the model
[01:24:10] from these probabilities the model starts with the tokens who is oron
[01:24:12] starts with the tokens who is oron kovats assistant and then it comes in
[01:24:15] kovats assistant and then it comes in here and it's get it's getting these
[01:24:18] here and it's get it's getting these probabilities and it's just sampling
[01:24:19] probabilities and it's just sampling from the probabilities and it just like
[01:24:21] from the probabilities and it just like comes up with stuff and the stuff is
[01:24:24] comes up with stuff and the stuff is actually
[01:24:25] actually statistically consistent with the style
[01:24:28] statistically consistent with the style of the answer in its training set and
[01:24:30] of the answer in its training set and it's just doing that but you and I
[01:24:32] it's just doing that but you and I experiened it as a madeup factual
[01:24:34] experiened it as a madeup factual knowledge but keep in mind that uh the
[01:24:36] knowledge but keep in mind that uh the model basically doesn't know and it's
[01:24:38] model basically doesn't know and it's just imitating the format of the answer
[01:24:40] just imitating the format of the answer and it's not going to go off and look it
[01:24:41] and it's not going to go off and look it up uh because it's just imitating again
[01:24:44] up uh because it's just imitating again the answer so how can we uh mitigate
[01:24:47] the answer so how can we uh mitigate this because for example when we go to
[01:24:48] this because for example when we go to chat apt and I say who is oron kovats
[01:24:51] chat apt and I say who is oron kovats and I'm now asking the stateoftheart
[01:24:53] and I'm now asking the stateoftheart state-of-the-art model from open AI
[01:24:55] state-of-the-art model from open AI this model will tell
[01:24:57] this model will tell you oh so this model is actually is even
[01:25:01] you oh so this model is actually is even smarter because you saw very briefly it
[01:25:03] smarter because you saw very briefly it said searching the web uh we're going to
[01:25:05] said searching the web uh we're going to cover this later um it's actually trying
[01:25:07] cover this later um it's actually trying to do tool use and
[01:25:11] to do tool use and uh kind of just like came up with some
[01:25:13] uh kind of just like came up with some kind of a story but I want to just who
[01:25:16] kind of a story but I want to just who or Kovach did not use any tools I don't
[01:25:19] or Kovach did not use any tools I don't want it to do web
[01:25:22] want it to do web search there's a wellknown historical or
[01:25:25] search there's a wellknown historical or public figure named or oron kovats so
[01:25:27] public figure named or oron kovats so this model is not going to make up stuff
[01:25:29] this model is not going to make up stuff this model knows that it doesn't know
[01:25:31] this model knows that it doesn't know and it tells you that it doesn't appear
[01:25:33] and it tells you that it doesn't appear to be a person that this model knows so
[01:25:36] to be a person that this model knows so somehow we sort of improved
[01:25:37] somehow we sort of improved hallucinations even though they clearly
[01:25:39] hallucinations even though they clearly are an issue in older models and it
[01:25:42] are an issue in older models and it makes totally uh sense why you would be
[01:25:44] makes totally uh sense why you would be getting these kinds of answers if this
[01:25:46] getting these kinds of answers if this is what your training set looks like so
[01:25:48] is what your training set looks like so how do we fix this okay well clearly we
[01:25:51] how do we fix this okay well clearly we need some examples in our data set that
[01:25:53] need some examples in our data set that where the correct answer for the
[01:25:55] where the correct answer for the assistant is that the model doesn't know
[01:25:57] assistant is that the model doesn't know about some particular fact but we only
[01:26:00] about some particular fact but we only need to have those answers be produced
[01:26:02] need to have those answers be produced in the cases where the model actually
[01:26:04] in the cases where the model actually doesn't know and so the question is how
[01:26:05] doesn't know and so the question is how do we know what the model knows or
[01:26:07] do we know what the model knows or doesn't know well we can empirically
[01:26:09] doesn't know well we can empirically probe the model to figure that out so
[01:26:11] probe the model to figure that out so let's take a look at for example how
[01:26:13] let's take a look at for example how meta uh dealt with hallucinations for
[01:26:16] meta uh dealt with hallucinations for the Llama 3 series of models as an
[01:26:18] the Llama 3 series of models as an example so in this paper that they
[01:26:20] example so in this paper that they published from meta we can go into
[01:26:22] published from meta we can go into hallucinations
[01:26:28] which they call here factuality and they describe the procedure by which they
[01:26:30] describe the procedure by which they basically interrogate the model to
[01:26:32] basically interrogate the model to figure out what it knows and doesn't
[01:26:34] figure out what it knows and doesn't know to figure out sort of like the
[01:26:35] know to figure out sort of like the boundary of its knowledge and then they
[01:26:38] boundary of its knowledge and then they add examples to the training set where
[01:26:42] add examples to the training set where for the things where the model doesn't
[01:26:44] for the things where the model doesn't know them the correct answer is that the
[01:26:47] know them the correct answer is that the model doesn't know them which sounds
[01:26:49] model doesn't know them which sounds like a very easy thing to do in
[01:26:51] like a very easy thing to do in principle but this roughly fixes the
[01:26:53] principle but this roughly fixes the issue and the the reason it fixes the
[01:26:55] issue and the the reason it fixes the issue is
[01:26:56] issue is because remember like the model might
[01:26:59] because remember like the model might actually have a pretty good model of its
[01:27:02] actually have a pretty good model of its self knowledge inside the network so
[01:27:04] self knowledge inside the network so remember we looked at the network and
[01:27:06] remember we looked at the network and all these neurons inside the network you
[01:27:09] all these neurons inside the network you might imagine that there's a neuron
[01:27:10] might imagine that there's a neuron somewhere in the network that sort of
[01:27:12] somewhere in the network that sort of like lights up for when the model is
[01:27:14] like lights up for when the model is uncertain but the problem is that the
[01:27:17] uncertain but the problem is that the activation of that neuron is not
[01:27:19] activation of that neuron is not currently wired up to the model actually
[01:27:21] currently wired up to the model actually saying in words that it doesn't know so
[01:27:23] saying in words that it doesn't know so even though the internal of the neural
[01:27:25] even though the internal of the neural network no because there's some neurons
[01:27:27] network no because there's some neurons that represent that the model uh will
[01:27:30] that represent that the model uh will not surface that it will instead take
[01:27:32] not surface that it will instead take its best guess so that it sounds
[01:27:33] its best guess so that it sounds confident um just like it sees in a
[01:27:36] confident um just like it sees in a training set so we need to basically
[01:27:37] training set so we need to basically interrogate the model and allow it to
[01:27:40] interrogate the model and allow it to say I don't know in the cases that it
[01:27:41] say I don't know in the cases that it doesn't know so let me take you through
[01:27:43] doesn't know so let me take you through what meta roughly does so basically what
[01:27:45] what meta roughly does so basically what they do is here I have an example uh
[01:27:48] they do is here I have an example uh Dominic kek is uh the featured article
[01:27:52] Dominic kek is uh the featured article today so I just went there randomly and
[01:27:54] today so I just went there randomly and what they do is basically they take a
[01:27:56] what they do is basically they take a random document in a training set and
[01:27:58] random document in a training set and they take a paragraph and then they use
[01:28:01] they take a paragraph and then they use an llm to construct questions about that
[01:28:04] an llm to construct questions about that paragraph so for example I did that with
[01:28:07] paragraph so for example I did that with chat GPT
[01:28:09] chat GPT here so I said here's a paragraph from
[01:28:12] here so I said here's a paragraph from this document generate three specific
[01:28:15] this document generate three specific factual questions based on this
[01:28:16] factual questions based on this paragraph and give me the questions and
[01:28:18] paragraph and give me the questions and the answers and so the llms are already
[01:28:20] the answers and so the llms are already good enough to create and reframe this
[01:28:23] good enough to create and reframe this information so if the information is in
[01:28:26] information so if the information is in the context window um of this llm this
[01:28:30] the context window um of this llm this actually works pretty well it doesn't
[01:28:31] actually works pretty well it doesn't have to rely on its memory it's right
[01:28:33] have to rely on its memory it's right there in the context window and so it
[01:28:36] there in the context window and so it can basically reframe that information
[01:28:38] can basically reframe that information with fairly high accuracy so for example
[01:28:40] with fairly high accuracy so for example can generate questions for us like for
[01:28:42] can generate questions for us like for which team did he play here's the answer
[01:28:45] which team did he play here's the answer how many cups did he win Etc and now
[01:28:47] how many cups did he win Etc and now what we have to do is we have some
[01:28:48] what we have to do is we have some question and answers and now we want to
[01:28:50] question and answers and now we want to interrogate the model so roughly
[01:28:52] interrogate the model so roughly speaking what we'll do is we'll take our
[01:28:53] speaking what we'll do is we'll take our questions and we'll go to our model
[01:28:56] questions and we'll go to our model which would be uh say llama uh in meta
[01:28:59] which would be uh say llama uh in meta but let's just interrogate mol 7B here
[01:29:01] but let's just interrogate mol 7B here as an example that's another model so
[01:29:04] as an example that's another model so does this model know about this answer
[01:29:07] does this model know about this answer let's take a
[01:29:09] let's take a look uh so he played for Buffalo Sabers
[01:29:12] look uh so he played for Buffalo Sabers right so the model knows and the the way
[01:29:15] right so the model knows and the the way that you can programmatically decide is
[01:29:17] that you can programmatically decide is basically we're going to take this
[01:29:19] basically we're going to take this answer from the model and we're going to
[01:29:21] answer from the model and we're going to compare it to the correct answer and
[01:29:24] compare it to the correct answer and again the model model are good enough to
[01:29:25] again the model model are good enough to do this automatically so there's no
[01:29:26] do this automatically so there's no humans involved here we can take uh
[01:29:29] humans involved here we can take uh basically the answer from the model and
[01:29:30] basically the answer from the model and we can use another llm judge to check if
[01:29:34] we can use another llm judge to check if that is correct according to this answer
[01:29:36] that is correct according to this answer and if it is correct that means that the
[01:29:37] and if it is correct that means that the model probably knows so what we're going
[01:29:39] model probably knows so what we're going to do is we're going to do this maybe a
[01:29:41] to do is we're going to do this maybe a few times so okay it knows it's Buffalo
[01:29:43] few times so okay it knows it's Buffalo Savers let's drag
[01:29:45] Savers let's drag in um Buffalo Sabers let's try one more
[01:29:51] in um Buffalo Sabers let's try one more time Buffalo Sabers so we asked three
[01:29:54] time Buffalo Sabers so we asked three times about this factual question and
[01:29:56] times about this factual question and the model seems to know so everything is
[01:29:58] the model seems to know so everything is great now let's try the second question
[01:30:01] great now let's try the second question how many Stanley Cups did he
[01:30:03] how many Stanley Cups did he win and again let's interrogate the
[01:30:05] win and again let's interrogate the model about that and the correct answer
[01:30:06] model about that and the correct answer is
[01:30:08] is two so um here the model claims that he
[01:30:14] two so um here the model claims that he won um four times which is not correct
[01:30:18] won um four times which is not correct right it doesn't match two so the model
[01:30:20] right it doesn't match two so the model doesn't know it's making stuff up let's
[01:30:22] doesn't know it's making stuff up let's try again
[01:30:30] um so here the model again it's kind of like making stuff up right let's
[01:30:38] Dragon here it says did he did not even did not win during his career so
[01:30:40] did not win during his career so obviously the model doesn't know and the
[01:30:41] obviously the model doesn't know and the way we can programmatically tell again
[01:30:43] way we can programmatically tell again is we interrogate the model three times
[01:30:45] is we interrogate the model three times and we compare its answers maybe three
[01:30:47] and we compare its answers maybe three times five times whatever it is to the
[01:30:49] times five times whatever it is to the correct answer and if the model doesn't
[01:30:52] correct answer and if the model doesn't know then we know that the model doesn't
[01:30:53] know then we know that the model doesn't know this question
[01:30:54] know this question and then what we do is we take this
[01:30:57] and then what we do is we take this question we create a new conversation in
[01:31:00] question we create a new conversation in the training set so we're going to add a
[01:31:02] the training set so we're going to add a new conversation training set and when
[01:31:04] new conversation training set and when the question is how many Stanley Cups
[01:31:05] the question is how many Stanley Cups did he win the answer is I'm sorry I
[01:31:08] did he win the answer is I'm sorry I don't know or I don't remember and
[01:31:11] don't know or I don't remember and that's the correct answer for this
[01:31:12] that's the correct answer for this question because we interrogated the
[01:31:14] question because we interrogated the model and we saw that that's the case if
[01:31:16] model and we saw that that's the case if you do this for many different types of
[01:31:18] you do this for many different types of uh questions for many different types of
[01:31:21] uh questions for many different types of documents you are giving the model an
[01:31:23] documents you are giving the model an opportunity to in its training set
[01:31:26] opportunity to in its training set refuse to say based on its knowledge and
[01:31:28] refuse to say based on its knowledge and if you just have a few examples of that
[01:31:30] if you just have a few examples of that in your training set the model will know
[01:31:33] in your training set the model will know um and and has the opportunity to learn
[01:31:35] um and and has the opportunity to learn the association of this knowledge-based
[01:31:38] the association of this knowledge-based refusal to this internal neuron
[01:31:41] refusal to this internal neuron somewhere in its Network that we presume
[01:31:43] somewhere in its Network that we presume exists and empirically this turns out to
[01:31:45] exists and empirically this turns out to be probably the case and it can learn
[01:31:47] be probably the case and it can learn that Association that hey when this
[01:31:49] that Association that hey when this neuron of uncertainty is high then I
[01:31:52] neuron of uncertainty is high then I actually don't know and I'm allowed to
[01:31:54] actually don't know and I'm allowed to say that I'm sorry but I don't think I
[01:31:56] say that I'm sorry but I don't think I remember this Etc and if you have these
[01:32:00] remember this Etc and if you have these uh examples in your training set then
[01:32:02] uh examples in your training set then this is a large mitigation for
[01:32:04] this is a large mitigation for hallucination and that's roughly
[01:32:06] hallucination and that's roughly speaking why chpt is able to do stuff
[01:32:08] speaking why chpt is able to do stuff like this as well so these are kinds of
[01:32:11] like this as well so these are kinds of uh mitigations that people have
[01:32:12] uh mitigations that people have implemented and that have improved the
[01:32:14] implemented and that have improved the factuality issue over time okay so I've
[01:32:17] factuality issue over time okay so I've described mitigation number one for
[01:32:20] described mitigation number one for basically mitigating the hallucinations
[01:32:22] basically mitigating the hallucinations issue now we can actually do much better
[01:32:24] issue now we can actually do much better than that uh it's instead of just saying
[01:32:27] than that uh it's instead of just saying that we don't know uh we can introduce
[01:32:30] that we don't know uh we can introduce an additional mitigation number two to
[01:32:32] an additional mitigation number two to give the llm an opportunity to be
[01:32:34] give the llm an opportunity to be factual and actually answer the question
[01:32:36] factual and actually answer the question now what do you and I do if I was to ask
[01:32:39] now what do you and I do if I was to ask you a factual question and you don't
[01:32:41] you a factual question and you don't know uh what would you do um in order to
[01:32:44] know uh what would you do um in order to answer the question well you could uh go
[01:32:46] answer the question well you could uh go off and do some search and uh use the
[01:32:48] off and do some search and uh use the internet and you could figure out the
[01:32:50] internet and you could figure out the answer and then tell me what that answer
[01:32:52] answer and then tell me what that answer is and we can do the exact exact same
[01:32:54] is and we can do the exact exact same thing with these models so think of the
[01:32:57] thing with these models so think of the knowledge inside the neural network
[01:32:58] knowledge inside the neural network inside its billions of parameters think
[01:33:01] inside its billions of parameters think of that as kind of a vague recollection
[01:33:03] of that as kind of a vague recollection of the things that the model has seen
[01:33:06] of the things that the model has seen during its training during the
[01:33:07] during its training during the pre-training stage a long time ago so
[01:33:10] pre-training stage a long time ago so think of that knowledge in the
[01:33:11] think of that knowledge in the parameters as something you read a month
[01:33:13] parameters as something you read a month ago and if you keep reading something
[01:33:16] ago and if you keep reading something then you will remember it and the model
[01:33:17] then you will remember it and the model remembers that but if it's something
[01:33:19] remembers that but if it's something rare then you probably don't have a
[01:33:20] rare then you probably don't have a really good recollection of that
[01:33:21] really good recollection of that information but what you and I do is we
[01:33:23] information but what you and I do is we just go and look it up now when you go
[01:33:26] just go and look it up now when you go and look it up what you're doing
[01:33:27] and look it up what you're doing basically is like you're refreshing your
[01:33:28] basically is like you're refreshing your working memory with information and then
[01:33:31] working memory with information and then you're able to sort of like retrieve it
[01:33:32] you're able to sort of like retrieve it talk about it or Etc so we need some
[01:33:34] talk about it or Etc so we need some equivalent of allowing the model to
[01:33:36] equivalent of allowing the model to refresh its memory or its recollection
[01:33:39] refresh its memory or its recollection and we can do that by introducing tools
[01:33:41] and we can do that by introducing tools uh for the
[01:33:43] uh for the models so the way we are going to
[01:33:45] models so the way we are going to approach this is that instead of just
[01:33:46] approach this is that instead of just saying hey I'm sorry I don't know we can
[01:33:48] saying hey I'm sorry I don't know we can attempt to use tools so we can create uh
[01:33:53] attempt to use tools so we can create uh a mechanism
[01:33:54] a mechanism by which the language model can emit
[01:33:56] by which the language model can emit special tokens and these are tokens that
[01:33:58] special tokens and these are tokens that we're going to introduce new tokens so
[01:34:01] we're going to introduce new tokens so for example here I've introduced two
[01:34:02] for example here I've introduced two tokens and I've introduced a format or a
[01:34:05] tokens and I've introduced a format or a protocol for how the model is allowed to
[01:34:07] protocol for how the model is allowed to use these tokens so for example instead
[01:34:10] use these tokens so for example instead of answering the question when the model
[01:34:12] of answering the question when the model does not instead of just saying I don't
[01:34:14] does not instead of just saying I don't know sorry the model has the option now
[01:34:16] know sorry the model has the option now to emitting the special token search
[01:34:18] to emitting the special token search start and this is the query that will go
[01:34:21] start and this is the query that will go to like bing.com in the case of openai
[01:34:23] to like bing.com in the case of openai or say Google search or something like
[01:34:24] or say Google search or something like that so it will emit the query and then
[01:34:27] that so it will emit the query and then it will emit search end and then here
[01:34:31] it will emit search end and then here what will happen is that the program
[01:34:33] what will happen is that the program that is sampling from the model that is
[01:34:35] that is sampling from the model that is running the inference when it sees the
[01:34:37] running the inference when it sees the special token search end instead of
[01:34:40] special token search end instead of sampling the next token uh in the
[01:34:42] sampling the next token uh in the sequence it will actually pause
[01:34:44] sequence it will actually pause generating from the model it will go off
[01:34:46] generating from the model it will go off it will open a session with bing.com and
[01:34:49] it will open a session with bing.com and it will paste the search query into Bing
[01:34:52] it will paste the search query into Bing and it will then um get all the text
[01:34:54] and it will then um get all the text that is retrieved and it will basically
[01:34:57] that is retrieved and it will basically take that text it will maybe represent
[01:34:59] take that text it will maybe represent it again with some other special tokens
[01:35:00] it again with some other special tokens or something like that and it will take
[01:35:02] or something like that and it will take that text and it will copy paste it here
[01:35:05] that text and it will copy paste it here into what I Tred to like show with the
[01:35:07] into what I Tred to like show with the brackets so all that text kind of comes
[01:35:09] brackets so all that text kind of comes here and when the text comes here it
[01:35:13] here and when the text comes here it enters the context window so the model
[01:35:15] enters the context window so the model so that text from the web search is now
[01:35:17] so that text from the web search is now inside the context window that will feed
[01:35:20] inside the context window that will feed into the neural network and you should
[01:35:22] into the neural network and you should think of the context window as kind of
[01:35:23] think of the context window as kind of like the working memory of the model
[01:35:25] like the working memory of the model that data that is in the context window
[01:35:27] that data that is in the context window is directly accessible by the model it
[01:35:30] is directly accessible by the model it directly feeds into the neural network
[01:35:32] directly feeds into the neural network so it's not anymore a vague recollection
[01:35:34] so it's not anymore a vague recollection it's data that it it has in the context
[01:35:37] it's data that it it has in the context window and is directly available to that
[01:35:38] window and is directly available to that model so now when it's sampling the new
[01:35:41] model so now when it's sampling the new uh tokens here afterwards it can
[01:35:44] uh tokens here afterwards it can reference very easily the data that has
[01:35:46] reference very easily the data that has been copy pasted in there so that's
[01:35:49] been copy pasted in there so that's roughly how these um how these tools use
[01:35:52] roughly how these um how these tools use uh tools uh function
[01:35:54] uh tools uh function and so web search is just one of the
[01:35:56] and so web search is just one of the tools we're going to look at some of the
[01:35:57] tools we're going to look at some of the other tools in a bit uh but basically
[01:35:59] other tools in a bit uh but basically you introduce new tokens you introduce
[01:36:01] you introduce new tokens you introduce some schema by which the model can
[01:36:02] some schema by which the model can utilize these tokens and can call these
[01:36:05] utilize these tokens and can call these special functions like web search
[01:36:06] special functions like web search functions and how do you teach the model
[01:36:09] functions and how do you teach the model how to correctly use these tools like
[01:36:11] how to correctly use these tools like say web search search start search end
[01:36:13] say web search search start search end Etc well again you do that through
[01:36:14] Etc well again you do that through training sets so we need now to have a
[01:36:17] training sets so we need now to have a bunch of data and a bunch of
[01:36:19] bunch of data and a bunch of conversations that show the model by
[01:36:21] conversations that show the model by example how to use web search so what
[01:36:24] example how to use web search so what are the what are the settings where you
[01:36:26] are the what are the settings where you are using the search um and what does
[01:36:28] are using the search um and what does that look like and here's by example how
[01:36:30] that look like and here's by example how you start a search and the search Etc
[01:36:33] you start a search and the search Etc and uh if you have a few thousand maybe
[01:36:35] and uh if you have a few thousand maybe examples of that in your training set
[01:36:37] examples of that in your training set the model will actually do a pretty good
[01:36:39] the model will actually do a pretty good job of understanding uh how this tool
[01:36:41] job of understanding uh how this tool works and it will know how to sort of
[01:36:43] works and it will know how to sort of structure its queries and of course
[01:36:45] structure its queries and of course because of the pre-training data set and
[01:36:47] because of the pre-training data set and its understanding of the world it
[01:36:49] its understanding of the world it actually kind of understands what a web
[01:36:50] actually kind of understands what a web search is and so it actually kind of has
[01:36:52] search is and so it actually kind of has a pretty good native understanding
[01:36:54] a pretty good native understanding um of what kind of stuff is a good
[01:36:56] um of what kind of stuff is a good search query um and so it all kind of
[01:36:58] search query um and so it all kind of just like works you just need a little
[01:37:00] just like works you just need a little bit of a few examples to show it how to
[01:37:03] bit of a few examples to show it how to use this new tool and then it can lean
[01:37:05] use this new tool and then it can lean on it to retrieve information and uh put
[01:37:07] on it to retrieve information and uh put it in the context window and that's
[01:37:09] it in the context window and that's equivalent to you and I looking
[01:37:10] equivalent to you and I looking something up because once it's in the
[01:37:12] something up because once it's in the context it's in the working memory and
[01:37:14] context it's in the working memory and it's very easy to manipulate and access
[01:37:16] it's very easy to manipulate and access so that's what we saw a few minutes ago
[01:37:18] so that's what we saw a few minutes ago when I was searching on chat GPT for who
[01:37:21] when I was searching on chat GPT for who is Orson kovats the chat GPT language
[01:37:23] is Orson kovats the chat GPT language model decided Ed that this is some kind
[01:37:25] model decided Ed that this is some kind of a rare um individual or something
[01:37:28] of a rare um individual or something like that and instead of giving me an
[01:37:30] like that and instead of giving me an answer from its memory it decided that
[01:37:32] answer from its memory it decided that it will sample a special token that is
[01:37:33] it will sample a special token that is going to do web search and we saw
[01:37:35] going to do web search and we saw briefly something flash it was like
[01:37:37] briefly something flash it was like using the web tool or something like
[01:37:38] using the web tool or something like that so it briefly said that and then we
[01:37:40] that so it briefly said that and then we waited for like two seconds and then it
[01:37:42] waited for like two seconds and then it generated this and you see how it's
[01:37:44] generated this and you see how it's creating references here and so it's
[01:37:46] creating references here and so it's citing sources so what happened here is
[01:37:50] citing sources so what happened here is it went off it did a web web search it
[01:37:52] it went off it did a web web search it found these sources and these URLs and
[01:37:55] found these sources and these URLs and the text of these web pages was all
[01:37:59] the text of these web pages was all stuffed in between here and it's not
[01:38:01] stuffed in between here and it's not showing here but it's it's basically
[01:38:03] showing here but it's it's basically stuffed as text in between here and now
[01:38:07] stuffed as text in between here and now it sees that text and now it kind of
[01:38:09] it sees that text and now it kind of references it and says that okay it
[01:38:11] references it and says that okay it could be these people citation could be
[01:38:13] could be these people citation could be those people citation Etc so that's what
[01:38:16] those people citation Etc so that's what happened here and that's what and that's
[01:38:17] happened here and that's what and that's why when I said who is Orson kovats I
[01:38:19] why when I said who is Orson kovats I could also say don't use any tools and
[01:38:22] could also say don't use any tools and then that's enough to um
[01:38:24] then that's enough to um basically convince chat PT to not use
[01:38:26] basically convince chat PT to not use tools and just use its memory and its
[01:38:28] tools and just use its memory and its recollection I also went off and I um
[01:38:33] recollection I also went off and I um tried to ask this question of Chachi PT
[01:38:35] tried to ask this question of Chachi PT so how many standing cups did uh Dominic
[01:38:37] so how many standing cups did uh Dominic Hasek win and Chachi P actually decided
[01:38:40] Hasek win and Chachi P actually decided that it knows the answer and it has the
[01:38:41] that it knows the answer and it has the confidence to say that uh he want twice
[01:38:44] confidence to say that uh he want twice and so it kind of just relied on its
[01:38:45] and so it kind of just relied on its memory because presumably it has um it
[01:38:49] memory because presumably it has um it has enough of
[01:38:50] has enough of a kind of confidence in its weights in
[01:38:53] a kind of confidence in its weights in it parameters and activations that this
[01:38:55] it parameters and activations that this is uh retrievable just for memory um but
[01:38:59] is uh retrievable just for memory um but you can also
[01:39:01] you can also conversely use web search to make sure
[01:39:04] conversely use web search to make sure and then for the same query it actually
[01:39:06] and then for the same query it actually goes off and it searches and then it
[01:39:08] goes off and it searches and then it finds a bunch of sources it finds all
[01:39:10] finds a bunch of sources it finds all this all of this stuff gets copy pasted
[01:39:12] this all of this stuff gets copy pasted in there and then it tells us uh to
[01:39:15] in there and then it tells us uh to again and sites and it actually says the
[01:39:18] again and sites and it actually says the Wikipedia article which is the source of
[01:39:20] Wikipedia article which is the source of this information for us as well so
[01:39:23] this information for us as well so that's tools web search the model
[01:39:25] that's tools web search the model determines when to search and then uh
[01:39:28] determines when to search and then uh that's kind of like how these tools uh
[01:39:30] that's kind of like how these tools uh work and this is an additional kind of
[01:39:32] work and this is an additional kind of mitigation for uh hallucinations and
[01:39:35] mitigation for uh hallucinations and factuality so I want to stress one more
[01:39:37] factuality so I want to stress one more time this very important sort of
[01:39:39] time this very important sort of psychology
[01:39:40] psychology Point knowledge in the parameters of the
[01:39:43] Point knowledge in the parameters of the neural network is a vague recollection
[01:39:46] neural network is a vague recollection the knowledge in the tokens that make up
[01:39:47] the knowledge in the tokens that make up the context
[01:39:48] the context window is the working memory and it
[01:39:51] window is the working memory and it roughly speaking Works kind of like um
[01:39:54] roughly speaking Works kind of like um it works for us in our brain the stuff
[01:39:56] it works for us in our brain the stuff we remember is our parameters uh and the
[01:39:59] we remember is our parameters uh and the stuff that we just experienced like a
[01:40:01] stuff that we just experienced like a few seconds or minutes ago and so on you
[01:40:03] few seconds or minutes ago and so on you can imagine that being in our context
[01:40:04] can imagine that being in our context window and this context window is being
[01:40:06] window and this context window is being built up as you have a conscious
[01:40:07] built up as you have a conscious experience around you so this has a
[01:40:10] experience around you so this has a bunch of um implications also for your
[01:40:12] bunch of um implications also for your use of LOLs in practice so for example I
[01:40:16] use of LOLs in practice so for example I can go to chat GPT and I can do
[01:40:17] can go to chat GPT and I can do something like this I can say can you
[01:40:19] something like this I can say can you Summarize chapter one of Jane Austin's
[01:40:20] Summarize chapter one of Jane Austin's Pride and Prejudice right and this is a
[01:40:23] Pride and Prejudice right and this is a perfectly fine prompt and Chach actually
[01:40:25] perfectly fine prompt and Chach actually does something relatively reasonable
[01:40:27] does something relatively reasonable here and but the reason it does that is
[01:40:29] here and but the reason it does that is because Chach has a pretty good
[01:40:30] because Chach has a pretty good recollection of a famous work like Pride
[01:40:33] recollection of a famous work like Pride and Prejudice it's probably seen a ton
[01:40:34] and Prejudice it's probably seen a ton of stuff about it there's probably
[01:40:36] of stuff about it there's probably forums about this book it's probably
[01:40:37] forums about this book it's probably read versions of this book um and it's
[01:40:40] read versions of this book um and it's kind of like remembers because even if
[01:40:43] kind of like remembers because even if you've read this or articles about it
[01:40:47] you've read this or articles about it you'd kind of have a recollection enough
[01:40:48] you'd kind of have a recollection enough to actually say all this but usually
[01:40:50] to actually say all this but usually when I actually interact with LMS and I
[01:40:51] when I actually interact with LMS and I want them to recall specific things it
[01:40:54] want them to recall specific things it always works better if you just give it
[01:40:55] always works better if you just give it to them so I think a much better prompt
[01:40:57] to them so I think a much better prompt would be something like this can you
[01:40:59] would be something like this can you summarize for me chapter one of genos's
[01:41:01] summarize for me chapter one of genos's spr and Prejudice and then I am
[01:41:03] spr and Prejudice and then I am attaching it below for your reference
[01:41:05] attaching it below for your reference and then I do something like a delimeter
[01:41:06] and then I do something like a delimeter here and I paste it in and I I found
[01:41:09] here and I paste it in and I I found that just copy pasting it from some
[01:41:11] that just copy pasting it from some website that I found here um so copy
[01:41:14] website that I found here um so copy pasting the chapter one here and I do
[01:41:16] pasting the chapter one here and I do that because when it's in the context
[01:41:18] that because when it's in the context window the model has direct access to it
[01:41:20] window the model has direct access to it and can exactly it doesn't have to
[01:41:22] and can exactly it doesn't have to recall it it just has access to it and
[01:41:25] recall it it just has access to it and so this summary is can be expected to be
[01:41:27] so this summary is can be expected to be a significantly high quality or higher
[01:41:29] a significantly high quality or higher quality than this summary uh just
[01:41:31] quality than this summary uh just because it's directly available to the
[01:41:33] because it's directly available to the model and I think you and I would work
[01:41:34] model and I think you and I would work in the same way if you want to it would
[01:41:37] in the same way if you want to it would be you would produce a much better
[01:41:38] be you would produce a much better summary if you had reread this chapter
[01:41:41] summary if you had reread this chapter before you had to summarize it and
[01:41:43] before you had to summarize it and that's basically what's happening here
[01:41:45] that's basically what's happening here or the equivalent of it the next sort of
[01:41:47] or the equivalent of it the next sort of psychological Quirk I'd like to talk
[01:41:49] psychological Quirk I'd like to talk about briefly is that of the knowledge
[01:41:50] about briefly is that of the knowledge of self so what I see very often on the
[01:41:53] of self so what I see very often on the internet is that people do something
[01:41:54] internet is that people do something like this they ask llms something like
[01:41:57] like this they ask llms something like what model are you and who built you and
[01:42:00] what model are you and who built you and um basically this uh question is a
[01:42:01] um basically this uh question is a little bit nonsensical and the reason I
[01:42:04] little bit nonsensical and the reason I say that is that as I try to kind of
[01:42:06] say that is that as I try to kind of explain with some of the underhood
[01:42:07] explain with some of the underhood fundamentals this thing is not a person
[01:42:09] fundamentals this thing is not a person right it doesn't have a persistent
[01:42:11] right it doesn't have a persistent existence in any way it sort of boots up
[01:42:14] existence in any way it sort of boots up processes tokens and shuts off and it
[01:42:17] processes tokens and shuts off and it does that for every single person it
[01:42:18] does that for every single person it just kind of builds up a context window
[01:42:20] just kind of builds up a context window of conversation and then everything gets
[01:42:21] of conversation and then everything gets deleted and so this this entity is kind
[01:42:24] deleted and so this this entity is kind of like restarted from scratch every
[01:42:25] of like restarted from scratch every single conversation if that makes sense
[01:42:27] single conversation if that makes sense it has no persistent self it has no
[01:42:29] it has no persistent self it has no sense of self it's a token tumbler and
[01:42:32] sense of self it's a token tumbler and uh it follows the statistical
[01:42:33] uh it follows the statistical regularities of its training set so it
[01:42:36] regularities of its training set so it doesn't really make sense to ask it who
[01:42:38] doesn't really make sense to ask it who are you what build you Etc and by
[01:42:40] are you what build you Etc and by default if you do what I described and
[01:42:43] default if you do what I described and just by default and from nowhere you're
[01:42:45] just by default and from nowhere you're going to get some pretty random answers
[01:42:46] going to get some pretty random answers so for example let's uh pick on Falcon
[01:42:48] so for example let's uh pick on Falcon which is a fairly old model and let's
[01:42:51] which is a fairly old model and let's see what it tells
[01:42:52] see what it tells us uh so it's evading the question uh
[01:42:56] us uh so it's evading the question uh talented engineers and developers here
[01:42:58] talented engineers and developers here it says I was built by open AI based on
[01:43:00] it says I was built by open AI based on the gpt3 model it's totally making stuff
[01:43:02] the gpt3 model it's totally making stuff up now the fact that it's built by open
[01:43:04] up now the fact that it's built by open AI here I think a lot of people would
[01:43:06] AI here I think a lot of people would take this as evidence that this model
[01:43:08] take this as evidence that this model was somehow trained on open AI data or
[01:43:10] was somehow trained on open AI data or something like that I don't actually
[01:43:11] something like that I don't actually think that that's necessarily true the
[01:43:13] think that that's necessarily true the reason for that is
[01:43:14] reason for that is that if you don't explicitly program the
[01:43:18] that if you don't explicitly program the model to answer these kinds of questions
[01:43:20] model to answer these kinds of questions then what you're going to get is its
[01:43:22] then what you're going to get is its statistical best guess at the answer and
[01:43:26] statistical best guess at the answer and this model had a um sft data mixture of
[01:43:29] this model had a um sft data mixture of conversations and during the
[01:43:32] conversations and during the fine-tuning um the model sort of
[01:43:36] fine-tuning um the model sort of understands as it's training on this
[01:43:37] understands as it's training on this data that it's taking on this
[01:43:38] data that it's taking on this personality of this like helpful
[01:43:40] personality of this like helpful assistant and it doesn't know how to it
[01:43:42] assistant and it doesn't know how to it doesn't actually it wasn't told exactly
[01:43:44] doesn't actually it wasn't told exactly what label to apply to self it just kind
[01:43:47] what label to apply to self it just kind of is taking on this uh this uh Persona
[01:43:50] of is taking on this uh this uh Persona of a helpful assistant and remember that
[01:43:53] of a helpful assistant and remember that the pre-training stage took the
[01:43:55] the pre-training stage took the documents from the entire internet and
[01:43:57] documents from the entire internet and Chach and open AI are very prominent in
[01:44:00] Chach and open AI are very prominent in these documents and so I think what's
[01:44:01] these documents and so I think what's actually likely to be happening here is
[01:44:04] actually likely to be happening here is that this is just its hallucinated label
[01:44:06] that this is just its hallucinated label for what it is this is its self-identity
[01:44:08] for what it is this is its self-identity is that it's chat GPT by open Ai and
[01:44:11] is that it's chat GPT by open Ai and it's only saying that because there's a
[01:44:13] it's only saying that because there's a ton of data on the internet of um
[01:44:15] ton of data on the internet of um answers like this that are actually
[01:44:18] answers like this that are actually coming from open from chasht and So
[01:44:20] coming from open from chasht and So that's its label for what it is now you
[01:44:24] that's its label for what it is now you can override this as a developer if you
[01:44:26] can override this as a developer if you have a llm model you can actually
[01:44:28] have a llm model you can actually override it and there are a few ways to
[01:44:29] override it and there are a few ways to do that so for example let me show you
[01:44:32] do that so for example let me show you there's this MMO model from Allen Ai and
[01:44:35] there's this MMO model from Allen Ai and um this is one llm it's not a top tier
[01:44:38] um this is one llm it's not a top tier LM or anything like that but I like it
[01:44:39] LM or anything like that but I like it because it is fully open source so the
[01:44:41] because it is fully open source so the paper for Almo and everything else is
[01:44:43] paper for Almo and everything else is completely fully open source which is
[01:44:44] completely fully open source which is nice um so here we are looking at its
[01:44:47] nice um so here we are looking at its sft mixture so this is the data mixture
[01:44:50] sft mixture so this is the data mixture of um the fine tuning so this is the
[01:44:52] of um the fine tuning so this is the conversations data it right and so the
[01:44:55] conversations data it right and so the way that they are solving it for Theo
[01:44:56] way that they are solving it for Theo model is we see that there's a bunch of
[01:44:58] model is we see that there's a bunch of stuff in the mixture and there's a total
[01:45:00] stuff in the mixture and there's a total of 1 million conversations here but here
[01:45:02] of 1 million conversations here but here we have alot to hardcoded if we go there
[01:45:05] we have alot to hardcoded if we go there we see that this is 240
[01:45:08] we see that this is 240 conversations and look at these 240
[01:45:11] conversations and look at these 240 conversations they're hardcoded tell me
[01:45:13] conversations they're hardcoded tell me about yourself says user and then the
[01:45:16] about yourself says user and then the assistant says I'm and open language
[01:45:18] assistant says I'm and open language model developed by AI to Allen Institute
[01:45:20] model developed by AI to Allen Institute of artificial intelligence Etc I'm here
[01:45:22] of artificial intelligence Etc I'm here to help blah blah blah what is your name
[01:45:24] to help blah blah blah what is your name uh Theo project so these are all kinds
[01:45:26] uh Theo project so these are all kinds of like cooked up hardcoded questions
[01:45:28] of like cooked up hardcoded questions abouto 2 and the correct answers to give
[01:45:31] abouto 2 and the correct answers to give in these cases if you take 240 questions
[01:45:34] in these cases if you take 240 questions like this or conversations put them into
[01:45:36] like this or conversations put them into your training set and fine tune with it
[01:45:38] your training set and fine tune with it then the model will actually be expected
[01:45:39] then the model will actually be expected to parot this stuff later if you don't
[01:45:43] to parot this stuff later if you don't give it this then it's probably a Chach
[01:45:46] give it this then it's probably a Chach by open
[01:45:47] by open Ai and um there's one more way to
[01:45:50] Ai and um there's one more way to sometimes do this is
[01:45:51] sometimes do this is that basically um in these conversations
[01:45:55] that basically um in these conversations and you have terms between human and
[01:45:56] and you have terms between human and assistant sometimes there's a special
[01:45:58] assistant sometimes there's a special message called system message at the
[01:46:00] message called system message at the very beginning of the conversation so
[01:46:03] very beginning of the conversation so it's not just between human and
[01:46:04] it's not just between human and assistant there's a system and in the
[01:46:06] assistant there's a system and in the system message you can actually hardcode
[01:46:08] system message you can actually hardcode and remind the model that hey you are a
[01:46:11] and remind the model that hey you are a model developed by open Ai and your name
[01:46:14] model developed by open Ai and your name is chashi pt40 and you were trained on
[01:46:17] is chashi pt40 and you were trained on this date and your knowledge cut off is
[01:46:18] this date and your knowledge cut off is this and basically it kind of like
[01:46:20] this and basically it kind of like documents the model a little bit and
[01:46:22] documents the model a little bit and then this is inserted into to your
[01:46:23] then this is inserted into to your conversations so when you go on chpt you
[01:46:25] conversations so when you go on chpt you see a blank page but actually the system
[01:46:27] see a blank page but actually the system message is kind of like hidden in there
[01:46:29] message is kind of like hidden in there and those tokens are in the context
[01:46:30] and those tokens are in the context window and so those are the two ways to
[01:46:33] window and so those are the two ways to kind of um program the models to talk
[01:46:36] kind of um program the models to talk about themselves either it's done
[01:46:38] about themselves either it's done through uh data like this or it's done
[01:46:40] through uh data like this or it's done through system message and things like
[01:46:42] through system message and things like that basically invisible tokens that are
[01:46:44] that basically invisible tokens that are in the context window and remind the
[01:46:46] in the context window and remind the model of its identity but it's all just
[01:46:48] model of its identity but it's all just kind of like cooked up and bolted on in
[01:46:50] kind of like cooked up and bolted on in some in some way it's not actually like
[01:46:52] some in some way it's not actually like really deeply there in any real sense as
[01:46:55] really deeply there in any real sense as it would before a human I want to now
[01:46:57] it would before a human I want to now continue to the next section which deals
[01:46:59] continue to the next section which deals with the computational capabilities or
[01:47:01] with the computational capabilities or like I should say the native
[01:47:02] like I should say the native computational capabilities of these
[01:47:04] computational capabilities of these models in problem solving scenarios and
[01:47:06] models in problem solving scenarios and so in particular we have to be very
[01:47:08] so in particular we have to be very careful with these models when we
[01:47:09] careful with these models when we construct our examples of conversations
[01:47:11] construct our examples of conversations and there's a lot of sharp edges here
[01:47:13] and there's a lot of sharp edges here that are kind of like elucidative is
[01:47:15] that are kind of like elucidative is that a word uh they're kind of like
[01:47:17] that a word uh they're kind of like interesting to look at when we consider
[01:47:18] interesting to look at when we consider how these models think so um consider
[01:47:22] how these models think so um consider the following prompt from a human and
[01:47:25] the following prompt from a human and supposed that basically that we are
[01:47:26] supposed that basically that we are building out a conversation to enter
[01:47:28] building out a conversation to enter into our training set of conversations
[01:47:29] into our training set of conversations so we're going to train the model on
[01:47:30] so we're going to train the model on this we're teaching you how to basically
[01:47:32] this we're teaching you how to basically solve simple math problems so the prompt
[01:47:35] solve simple math problems so the prompt is Emily buys three apples and two
[01:47:37] is Emily buys three apples and two oranges each orange cost $2 the total
[01:47:39] oranges each orange cost $2 the total cost is 13 what is the cost of apples
[01:47:41] cost is 13 what is the cost of apples very simple math question now there are
[01:47:44] very simple math question now there are two answers here on the left and on the
[01:47:46] two answers here on the left and on the right they are both correct answers they
[01:47:48] right they are both correct answers they both say that the answer is three which
[01:47:50] both say that the answer is three which is correct but one of these two is a
[01:47:52] is correct but one of these two is a significant ific anly better answer for
[01:47:55] significant ific anly better answer for the assistant than the other like if I
[01:47:56] the assistant than the other like if I was Data labeler and I was creating one
[01:47:58] was Data labeler and I was creating one of these one of these would be uh a
[01:48:01] of these one of these would be uh a really terrible answer for the assistant
[01:48:04] really terrible answer for the assistant and the other would be okay and so I'd
[01:48:06] and the other would be okay and so I'd like you to potentially pause the video
[01:48:07] like you to potentially pause the video Even and think through why one of these
[01:48:10] Even and think through why one of these two is significantly better answer uh
[01:48:12] two is significantly better answer uh than the other and um if you use the
[01:48:15] than the other and um if you use the wrong one your model will actually be uh
[01:48:18] wrong one your model will actually be uh really bad at math potentially and it
[01:48:19] really bad at math potentially and it would have uh bad outcomes and this is
[01:48:21] would have uh bad outcomes and this is something that you would be careful with
[01:48:22] something that you would be careful with in your life labeling documentations
[01:48:24] in your life labeling documentations when you are training people uh to
[01:48:25] when you are training people uh to create the ideal responses for the
[01:48:27] create the ideal responses for the assistant okay so the key to this
[01:48:29] assistant okay so the key to this question is to realize and remember that
[01:48:32] question is to realize and remember that when the models are training and also
[01:48:34] when the models are training and also inferencing they are working in
[01:48:36] inferencing they are working in onedimensional sequence of tokens from
[01:48:38] onedimensional sequence of tokens from left to right and this is the picture
[01:48:40] left to right and this is the picture that I often have in my mind I imagine
[01:48:42] that I often have in my mind I imagine basically the token sequence evolving
[01:48:44] basically the token sequence evolving from left to right and to always produce
[01:48:46] from left to right and to always produce the next token in a sequence we are
[01:48:49] the next token in a sequence we are feeding all these tokens into the neural
[01:48:51] feeding all these tokens into the neural network and this neural network then is
[01:48:53] network and this neural network then is the probabilities for the next token and
[01:48:54] the probabilities for the next token and sequence right so this picture here is
[01:48:56] sequence right so this picture here is the exact same picture we saw uh before
[01:48:59] the exact same picture we saw uh before up here and this comes from the web demo
[01:49:02] up here and this comes from the web demo that I showed you before right so this
[01:49:04] that I showed you before right so this is the calculation that basically takes
[01:49:06] is the calculation that basically takes the input tokens here on the top and uh
[01:49:09] the input tokens here on the top and uh performs these operations of all these
[01:49:11] performs these operations of all these neurons and uh gives you the answer for
[01:49:14] neurons and uh gives you the answer for the probabilities of what comes next now
[01:49:16] the probabilities of what comes next now the important thing to realize is that
[01:49:18] the important thing to realize is that roughly
[01:49:19] roughly speaking uh there's basically a finite
[01:49:21] speaking uh there's basically a finite number of layers of computation that
[01:49:23] number of layers of computation that happened here so for example this model
[01:49:25] happened here so for example this model here has only one two three layers of
[01:49:29] here has only one two three layers of what's called detention and uh MLP here
[01:49:32] what's called detention and uh MLP here um maybe um typical modern
[01:49:34] um maybe um typical modern state-of-the-art Network would have more
[01:49:36] state-of-the-art Network would have more like say 100 layers or something like
[01:49:38] like say 100 layers or something like that but there's only 100 layers of
[01:49:39] that but there's only 100 layers of computation or something like that to go
[01:49:41] computation or something like that to go from the previous token sequence to the
[01:49:43] from the previous token sequence to the probabilities for the next token and so
[01:49:45] probabilities for the next token and so there's a finite amount of computation
[01:49:47] there's a finite amount of computation that happens here for every single token
[01:49:49] that happens here for every single token and you should think of this as a very
[01:49:51] and you should think of this as a very small amount of computation and this
[01:49:53] small amount of computation and this amount of computation is almost roughly
[01:49:55] amount of computation is almost roughly fixed uh for every single token in this
[01:49:57] fixed uh for every single token in this sequence um the that's not actually
[01:50:00] sequence um the that's not actually fully true because the more tokens you
[01:50:02] fully true because the more tokens you feed in uh the the more expensive uh
[01:50:04] feed in uh the the more expensive uh this forward pass will be of this neural
[01:50:07] this forward pass will be of this neural network but not by much so you should
[01:50:09] network but not by much so you should think of this uh and I think as a good
[01:50:11] think of this uh and I think as a good model to have in mind this is a fixed
[01:50:13] model to have in mind this is a fixed amount of compute that's going to happen
[01:50:14] amount of compute that's going to happen in this box for every single one of
[01:50:16] in this box for every single one of these tokens and this amount of compute
[01:50:18] these tokens and this amount of compute Cann possibly be too big because there's
[01:50:19] Cann possibly be too big because there's not that many layers that are sort of
[01:50:21] not that many layers that are sort of going from the top to bottom here
[01:50:23] going from the top to bottom here there's not that that much
[01:50:24] there's not that that much computationally that will happen here
[01:50:26] computationally that will happen here and so you can't imagine the model to to
[01:50:28] and so you can't imagine the model to to basically do arbitrary computation in a
[01:50:30] basically do arbitrary computation in a single forward pass to get a single
[01:50:32] single forward pass to get a single token and so what that means is that we
[01:50:34] token and so what that means is that we actually have to distribute our
[01:50:36] actually have to distribute our reasoning and our computation across
[01:50:38] reasoning and our computation across many tokens because every single token
[01:50:40] many tokens because every single token is only spending a finite amount of
[01:50:42] is only spending a finite amount of computation on it and so we kind of want
[01:50:45] computation on it and so we kind of want to distribute the computation across
[01:50:48] to distribute the computation across many tokens and we can't have too much
[01:50:51] many tokens and we can't have too much computation or expect too much
[01:50:52] computation or expect too much computation out of of the model in any
[01:50:54] computation out of of the model in any single individual token because there's
[01:50:56] single individual token because there's only so much computation that happens
[01:50:58] only so much computation that happens per token okay roughly fixed amount of
[01:51:01] per token okay roughly fixed amount of computation here
[01:51:03] computation here so that's why this answer here is
[01:51:06] so that's why this answer here is significantly worse and the reason for
[01:51:08] significantly worse and the reason for that is Imagine going from left to right
[01:51:10] that is Imagine going from left to right here um and I copy pasted it right here
[01:51:14] here um and I copy pasted it right here the answer is three Etc imagine the
[01:51:16] the answer is three Etc imagine the model having to go from left to right
[01:51:18] model having to go from left to right emitting these tokens one at a time it
[01:51:20] emitting these tokens one at a time it has to say or we're expecting to say the
[01:51:23] has to say or we're expecting to say the answer is space dollar sign and then
[01:51:28] answer is space dollar sign and then right here we're expecting it to
[01:51:29] right here we're expecting it to basically cram all of the computation of
[01:51:31] basically cram all of the computation of this problem into this single token it
[01:51:33] this problem into this single token it has to emit the correct answer three and
[01:51:36] has to emit the correct answer three and then once we've emitted the answer three
[01:51:38] then once we've emitted the answer three we're expecting it to say all these
[01:51:40] we're expecting it to say all these tokens but at this point we've already
[01:51:41] tokens but at this point we've already prod produced the answer and it's
[01:51:43] prod produced the answer and it's already in the context window for all
[01:51:45] already in the context window for all these tokens that follow so anything
[01:51:47] these tokens that follow so anything here is just um kind of post Hawk
[01:51:49] here is just um kind of post Hawk justification of why this is the answer
[01:51:52] justification of why this is the answer um because the answer is already created
[01:51:54] um because the answer is already created it's already in the token window so it's
[01:51:56] it's already in the token window so it's it's not actually being calculated here
[01:51:59] it's not actually being calculated here um and so if you are answering the
[01:52:01] um and so if you are answering the question directly and immediately you
[01:52:03] question directly and immediately you are training the model to to try to
[01:52:06] are training the model to to try to basically guess the answer in a single
[01:52:08] basically guess the answer in a single token and that is just not going to work
[01:52:10] token and that is just not going to work because of the finite amount of
[01:52:11] because of the finite amount of computation that happens per token
[01:52:14] computation that happens per token that's why this answer on the right is
[01:52:16] that's why this answer on the right is significantly better because we are
[01:52:17] significantly better because we are Distributing this computation across the
[01:52:19] Distributing this computation across the answer we're actually getting the model
[01:52:21] answer we're actually getting the model to sort of slowly come to the answer
[01:52:23] to sort of slowly come to the answer from the left to right we're getting
[01:52:25] from the left to right we're getting intermediate results we're saying okay
[01:52:27] intermediate results we're saying okay the total cost of oranges is four so 30
[01:52:29] the total cost of oranges is four so 30 - 4 is 9 and so we're creating
[01:52:32] - 4 is 9 and so we're creating intermediate calculations and each one
[01:52:34] intermediate calculations and each one of these calculations is by itself not
[01:52:36] of these calculations is by itself not that expensive and so we're actually
[01:52:38] that expensive and so we're actually basically kind of guessing a little bit
[01:52:40] basically kind of guessing a little bit the difficulty that the model is capable
[01:52:42] the difficulty that the model is capable of in any single one of these individual
[01:52:45] of in any single one of these individual tokens and there can never be too much
[01:52:47] tokens and there can never be too much work in any one of these tokens
[01:52:49] work in any one of these tokens computationally because then the model
[01:52:51] computationally because then the model won't be able to do that later at test
[01:52:53] won't be able to do that later at test time and so we're teaching the model
[01:52:55] time and so we're teaching the model here to spread out its reasoning and to
[01:52:58] here to spread out its reasoning and to spread out its computation over the
[01:53:00] spread out its computation over the tokens and in this way it only has very
[01:53:03] tokens and in this way it only has very simple problems in each token and they
[01:53:05] simple problems in each token and they can add up and then by the time it's
[01:53:08] can add up and then by the time it's near the end it has all the previous
[01:53:10] near the end it has all the previous results in its working memory and it's
[01:53:12] results in its working memory and it's much easier for it to determine that the
[01:53:13] much easier for it to determine that the answer is and here it is three so this
[01:53:16] answer is and here it is three so this is a significantly better label for our
[01:53:18] is a significantly better label for our computation this would be really bad and
[01:53:21] computation this would be really bad and is teaching the model to try to do all
[01:53:23] is teaching the model to try to do all the computation in a single token and
[01:53:25] the computation in a single token and it's really
[01:53:26] it's really bad so uh that's kind of like an
[01:53:29] bad so uh that's kind of like an interesting thing to keep in mind is in
[01:53:30] interesting thing to keep in mind is in your
[01:53:32] your prompts uh usually don't have to think
[01:53:34] prompts uh usually don't have to think about it explicitly because uh the
[01:53:36] about it explicitly because uh the people at open AI have labelers and so
[01:53:39] people at open AI have labelers and so on that actually worry about this and
[01:53:40] on that actually worry about this and they make sure that the answers are
[01:53:42] they make sure that the answers are spread out and so actually open AI will
[01:53:44] spread out and so actually open AI will kind of like do the right thing so when
[01:53:46] kind of like do the right thing so when I ask this question for chat GPT it's
[01:53:48] I ask this question for chat GPT it's actually going to go very slowly it's
[01:53:50] actually going to go very slowly it's going to be like okay let's define our
[01:53:51] going to be like okay let's define our variables set up the equation
[01:53:53] variables set up the equation and it's kind of creating all these
[01:53:54] and it's kind of creating all these intermediate results these are not for
[01:53:56] intermediate results these are not for you these are for the model if the model
[01:53:59] you these are for the model if the model is not creating these intermediate
[01:54:00] is not creating these intermediate results for itself it's not going to be
[01:54:02] results for itself it's not going to be able to reach three I also wanted to
[01:54:04] able to reach three I also wanted to show you that it's possible to be a bit
[01:54:06] show you that it's possible to be a bit mean to the model uh we can just ask for
[01:54:08] mean to the model uh we can just ask for things so as an example I said I gave it
[01:54:11] things so as an example I said I gave it the exact same uh prompt and I said
[01:54:13] the exact same uh prompt and I said answer the question in a single token
[01:54:15] answer the question in a single token just immediately give me the answer
[01:54:16] just immediately give me the answer nothing else and it turns out that for
[01:54:19] nothing else and it turns out that for this simple um prompt here it actually
[01:54:22] this simple um prompt here it actually was able to do it in single go so it
[01:54:24] was able to do it in single go so it just created a single I think this is
[01:54:25] just created a single I think this is two tokens right uh because the dollar
[01:54:28] two tokens right uh because the dollar sign is its own token so basically this
[01:54:30] sign is its own token so basically this model didn't give me a single token it
[01:54:32] model didn't give me a single token it gave me two tokens but it still produced
[01:54:34] gave me two tokens but it still produced the correct answer and it did that in a
[01:54:36] the correct answer and it did that in a single forward pass of the
[01:54:38] single forward pass of the network now that's because the numbers
[01:54:40] network now that's because the numbers here I think are very simple and so I
[01:54:42] here I think are very simple and so I made it a bit more difficult to be a bit
[01:54:43] made it a bit more difficult to be a bit mean to the model so I said Emily buys
[01:54:45] mean to the model so I said Emily buys 23 apples and 177 oranges and then I
[01:54:48] 23 apples and 177 oranges and then I just made the numbers a bit bigger and
[01:54:50] just made the numbers a bit bigger and I'm just making it harder for the model
[01:54:52] I'm just making it harder for the model I'm asking it to more computation in a
[01:54:53] I'm asking it to more computation in a single token and so I said the same
[01:54:56] single token and so I said the same thing and here it gave me five and five
[01:54:58] thing and here it gave me five and five is actually not correct so the model
[01:55:00] is actually not correct so the model failed to do all of this calculation in
[01:55:02] failed to do all of this calculation in a single forward pass of the network it
[01:55:04] a single forward pass of the network it failed to go from the input tokens and
[01:55:08] failed to go from the input tokens and then in a single forward pass of the
[01:55:09] then in a single forward pass of the network single go through the network it
[01:55:11] network single go through the network it couldn't produce the result and then I
[01:55:14] couldn't produce the result and then I said okay now don't worry about the the
[01:55:16] said okay now don't worry about the the token limit and just solve the problem
[01:55:18] token limit and just solve the problem as usual and then it goes all the
[01:55:20] as usual and then it goes all the intermediate results it simplifies and
[01:55:23] intermediate results it simplifies and every one of these intermediate results
[01:55:24] every one of these intermediate results here and intermediate calculations is
[01:55:26] here and intermediate calculations is much easier for the model and um it sort
[01:55:30] much easier for the model and um it sort of it's not too much work per token all
[01:55:32] of it's not too much work per token all of the tokens here are correct and it
[01:55:34] of the tokens here are correct and it arises the solution which is seven and I
[01:55:36] arises the solution which is seven and I just couldn't squeeze all of this work
[01:55:38] just couldn't squeeze all of this work it couldn't squeeze that into a single
[01:55:40] it couldn't squeeze that into a single forward passive Network so I think
[01:55:42] forward passive Network so I think that's kind of just a cute example and
[01:55:43] that's kind of just a cute example and something to kind of like think about
[01:55:45] something to kind of like think about and I think it's kind of again just
[01:55:47] and I think it's kind of again just elucidative in terms of how these uh
[01:55:49] elucidative in terms of how these uh models work the last thing that I would
[01:55:50] models work the last thing that I would say on this topic is that if I was in
[01:55:52] say on this topic is that if I was in practi is trying to actually solve this
[01:55:53] practi is trying to actually solve this in my day-to-day life I might actually
[01:55:55] in my day-to-day life I might actually not uh trust that the model that all the
[01:55:58] not uh trust that the model that all the intermediate calculations correctly here
[01:56:00] intermediate calculations correctly here so actually probably what I do is
[01:56:01] so actually probably what I do is something like this I would come here
[01:56:02] something like this I would come here and I would say use code and uh that's
[01:56:06] and I would say use code and uh that's because code is one of the possible
[01:56:09] because code is one of the possible tools that chachy PD can use and instead
[01:56:12] tools that chachy PD can use and instead of it having to do mental arithmetic
[01:56:14] of it having to do mental arithmetic like this mental arithmetic here I don't
[01:56:16] like this mental arithmetic here I don't fully trust it and especially if the
[01:56:17] fully trust it and especially if the numbers get really big there's no
[01:56:19] numbers get really big there's no guarantee that the model will do this
[01:56:20] guarantee that the model will do this correctly any one of these intermediates
[01:56:22] correctly any one of these intermediates steps might in principle fail we're
[01:56:25] steps might in principle fail we're using neural networks to do mental
[01:56:26] using neural networks to do mental arithmetic uh kind of like you doing
[01:56:28] arithmetic uh kind of like you doing mental arithmetic in your brain it might
[01:56:30] mental arithmetic in your brain it might just like uh screw up some of the
[01:56:31] just like uh screw up some of the intermediate results it's actually kind
[01:56:33] intermediate results it's actually kind of amazing that it can even do this kind
[01:56:34] of amazing that it can even do this kind of mental arithmetic I don't think I
[01:56:35] of mental arithmetic I don't think I could do this in my head but basically
[01:56:37] could do this in my head but basically the model is kind of like doing it in
[01:56:38] the model is kind of like doing it in its head and I don't trust that so I
[01:56:40] its head and I don't trust that so I wanted to use tools so you can say stuff
[01:56:42] wanted to use tools so you can say stuff like use
[01:56:43] like use code and uh I'm not sure what happened
[01:56:47] code and uh I'm not sure what happened there use
[01:56:50] there use code and so um like I mentioned there's
[01:56:53] code and so um like I mentioned there's a special tool and the uh the model can
[01:56:56] a special tool and the uh the model can write code and I can inspect that this
[01:56:59] write code and I can inspect that this code is correct and then uh it's not
[01:57:02] code is correct and then uh it's not relying on its mental arithmetic it is
[01:57:04] relying on its mental arithmetic it is using the python interpreter which is a
[01:57:05] using the python interpreter which is a very simple programming language to
[01:57:07] very simple programming language to basically uh write out the code that
[01:57:09] basically uh write out the code that calculates the result and I would
[01:57:11] calculates the result and I would personally trust this a lot more because
[01:57:12] personally trust this a lot more because this came out of a Python program which
[01:57:14] this came out of a Python program which I think has a lot more correctness
[01:57:15] I think has a lot more correctness guarantees than the mental arithmetic of
[01:57:18] guarantees than the mental arithmetic of a language model uh so just um another
[01:57:21] a language model uh so just um another kind of uh potential hint that if you
[01:57:23] kind of uh potential hint that if you have these kinds of problems uh you may
[01:57:25] have these kinds of problems uh you may want to basically just uh ask the model
[01:57:27] want to basically just uh ask the model to use the code interpreter and just
[01:57:29] to use the code interpreter and just like we saw with the web search the
[01:57:31] like we saw with the web search the model has special uh kind of tokens for
[01:57:34] model has special uh kind of tokens for calling uh like it will not actually
[01:57:37] calling uh like it will not actually generate these tokens from the language
[01:57:38] generate these tokens from the language model it will write the program and then
[01:57:41] model it will write the program and then it actually sends that program to a
[01:57:43] it actually sends that program to a different sort of part of the computer
[01:57:44] different sort of part of the computer that actually just runs that program and
[01:57:46] that actually just runs that program and brings back the result and then the
[01:57:48] brings back the result and then the model gets access to that result and can
[01:57:50] model gets access to that result and can tell you that okay the cost of each
[01:57:51] tell you that okay the cost of each apple is seven
[01:57:53] apple is seven um so that's another kind of tool and I
[01:57:55] um so that's another kind of tool and I would use this in practice for yourself
[01:57:58] would use this in practice for yourself and it's um yeah it's just uh less error
[01:58:02] and it's um yeah it's just uh less error prone I would say so that's why I called
[01:58:04] prone I would say so that's why I called this section models need tokens to think
[01:58:07] this section models need tokens to think distribute your competition across many
[01:58:09] distribute your competition across many tokens ask models to create intermediate
[01:58:11] tokens ask models to create intermediate results or whenever you can lean on
[01:58:14] results or whenever you can lean on tools and Tool use instead of allowing
[01:58:16] tools and Tool use instead of allowing the models to do all of the stuff in
[01:58:17] the models to do all of the stuff in their memory so if they try to do it all
[01:58:19] their memory so if they try to do it all in their memory I don't fully trust it
[01:58:21] in their memory I don't fully trust it and prefer to use tools whenever
[01:58:23] and prefer to use tools whenever possible I want to show you one more
[01:58:25] possible I want to show you one more example of where this actually comes up
[01:58:27] example of where this actually comes up and that's in counting so models
[01:58:29] and that's in counting so models actually are not very good at counting
[01:58:30] actually are not very good at counting for the exact same reason you're asking
[01:58:32] for the exact same reason you're asking for way too much in a single individual
[01:58:34] for way too much in a single individual token so let me show you a simple
[01:58:37] token so let me show you a simple example of that um how many dots are
[01:58:39] example of that um how many dots are below and then I just put in a bunch of
[01:58:41] below and then I just put in a bunch of dots and Chach says there are and then
[01:58:45] dots and Chach says there are and then it just tries to solve the problem in a
[01:58:46] it just tries to solve the problem in a single token so in a single token it has
[01:58:49] single token so in a single token it has to count the number of dots in its
[01:58:51] to count the number of dots in its context window
[01:58:53] context window um and it has to do that in the single
[01:58:55] um and it has to do that in the single forward pass of a network and a single
[01:58:57] forward pass of a network and a single forward pass of a network as we talked
[01:58:59] forward pass of a network as we talked about there's not that much computation
[01:59:00] about there's not that much computation that can happen there just think of that
[01:59:02] that can happen there just think of that as being like very little competation
[01:59:04] as being like very little competation that happens there so if I just look at
[01:59:06] that happens there so if I just look at what the model sees let's go to the LM
[01:59:09] what the model sees let's go to the LM go to tokenizer it sees uh
[01:59:14] go to tokenizer it sees uh this how many dots are below and then it
[01:59:16] this how many dots are below and then it turns out that these dots here this
[01:59:18] turns out that these dots here this group of I think 20 dots is a single
[01:59:20] group of I think 20 dots is a single token and then this group of whatever it
[01:59:23] token and then this group of whatever it is is another token and then for some
[01:59:25] is is another token and then for some reason they break up as this so I don't
[01:59:28] reason they break up as this so I don't actually this has to do with the details
[01:59:29] actually this has to do with the details of the tokenizer but it turns out that
[01:59:31] of the tokenizer but it turns out that these um the model basically sees the
[01:59:34] these um the model basically sees the token ID this this this and so on and
[01:59:38] token ID this this this and so on and then from these token IDs it's expected
[01:59:41] then from these token IDs it's expected to count the number and spoiler alert is
[01:59:44] to count the number and spoiler alert is not 161 it's actually I believe
[01:59:46] not 161 it's actually I believe 177 so here's what we can do instead uh
[01:59:48] 177 so here's what we can do instead uh we can say use code and you might expect
[01:59:52] we can say use code and you might expect that like why should this work and it's
[01:59:54] that like why should this work and it's actually kind of subtle and kind of
[01:59:55] actually kind of subtle and kind of interesting so when I say use code I
[01:59:57] interesting so when I say use code I actually expect this to work let's see
[01:59:59] actually expect this to work let's see okay 177 is correct so what happens here
[02:00:03] okay 177 is correct so what happens here is I've actually it doesn't look like it
[02:00:05] is I've actually it doesn't look like it but I've broken down the problem into a
[02:00:08] but I've broken down the problem into a problems that are easier for the model I
[02:00:10] problems that are easier for the model I know that the model can't count it can't
[02:00:12] know that the model can't count it can't do mental counting but I know that the
[02:00:14] do mental counting but I know that the model is actually pretty good at doing
[02:00:16] model is actually pretty good at doing copy pasting so what I'm doing here is
[02:00:18] copy pasting so what I'm doing here is when I say use code it creates a string
[02:00:20] when I say use code it creates a string in Python for this and the task of
[02:00:24] in Python for this and the task of basically copy pasting my input here to
[02:00:27] basically copy pasting my input here to here is very simple because for the
[02:00:30] here is very simple because for the model um it sees this string of uh it
[02:00:34] model um it sees this string of uh it sees it as just these four tokens or
[02:00:36] sees it as just these four tokens or whatever it is so it's very simple for
[02:00:37] whatever it is so it's very simple for the model to copy paste those token IDs
[02:00:41] the model to copy paste those token IDs and um kind of unpack them into Dots
[02:00:45] and um kind of unpack them into Dots here and so it creates this string and
[02:00:48] here and so it creates this string and then it calls python routine. count and
[02:00:50] then it calls python routine. count and then it comes up with the correct answer
[02:00:52] then it comes up with the correct answer so the python interpreter is doing the
[02:00:54] so the python interpreter is doing the counting it's not the models mental
[02:00:55] counting it's not the models mental arithmetic doing the counting so it's
[02:00:57] arithmetic doing the counting so it's again a simple example of um models need
[02:01:01] again a simple example of um models need tokens to think don't rely on their
[02:01:03] tokens to think don't rely on their mental arithmetic and um that's why also
[02:01:06] mental arithmetic and um that's why also the models are not very good at counting
[02:01:07] the models are not very good at counting if you need them to do counting tasks
[02:01:09] if you need them to do counting tasks always ask them to lean on the tool now
[02:01:12] always ask them to lean on the tool now the models also have many other little
[02:01:14] the models also have many other little cognitive deficits here and there and
[02:01:15] cognitive deficits here and there and these are kind of like sharp edges of
[02:01:17] these are kind of like sharp edges of the technology to be kind of aware of
[02:01:18] the technology to be kind of aware of over time so as an example the models
[02:01:21] over time so as an example the models are not very good with all kinds of
[02:01:22] are not very good with all kinds of spelling related tasks they're not very
[02:01:24] spelling related tasks they're not very good at it and I told you that we would
[02:01:27] good at it and I told you that we would loop back around to tokenization and the
[02:01:29] loop back around to tokenization and the reason to do for this is that the models
[02:01:31] reason to do for this is that the models they don't see the characters they see
[02:01:33] they don't see the characters they see tokens and they their entire world is
[02:01:36] tokens and they their entire world is about tokens which are these little text
[02:01:37] about tokens which are these little text chunks and so they don't see characters
[02:01:39] chunks and so they don't see characters like our eyes do and so very simple
[02:01:42] like our eyes do and so very simple character level tasks often fail so for
[02:01:45] character level tasks often fail so for example uh I'm giving it a string
[02:01:48] example uh I'm giving it a string ubiquitous and I'm asking it to print
[02:01:50] ubiquitous and I'm asking it to print only every third character starting with
[02:01:52] only every third character starting with the first one so we start with U and
[02:01:54] the first one so we start with U and then we should go every third so every
[02:01:57] then we should go every third so every so 1 2 3 Q should be next and then Etc
[02:02:01] so 1 2 3 Q should be next and then Etc so this I see is not correct and again
[02:02:04] so this I see is not correct and again my hypothesis is that this is again
[02:02:06] my hypothesis is that this is again Dental arithmetic here is failing number
[02:02:08] Dental arithmetic here is failing number one a little bit but number two I think
[02:02:10] one a little bit but number two I think the the more important issue here is
[02:02:12] the the more important issue here is that if you go to Tik
[02:02:14] that if you go to Tik tokenizer and you look at ubiquitous we
[02:02:16] tokenizer and you look at ubiquitous we see that it is three tokens right so you
[02:02:19] see that it is three tokens right so you and I see ubiquitous and we can easily
[02:02:21] and I see ubiquitous and we can easily access the individual letters because we
[02:02:24] access the individual letters because we kind of see them and when we have it in
[02:02:26] kind of see them and when we have it in the working memory of our visual sort of
[02:02:28] the working memory of our visual sort of field we can really easily index into
[02:02:30] field we can really easily index into every third letter and I can do that
[02:02:31] every third letter and I can do that task but the models don't have access to
[02:02:33] task but the models don't have access to the individual letters they see this as
[02:02:35] the individual letters they see this as these three tokens and uh remember these
[02:02:38] these three tokens and uh remember these models are trained from scratch on the
[02:02:40] models are trained from scratch on the internet and all these token uh
[02:02:43] internet and all these token uh basically the model has to discover how
[02:02:44] basically the model has to discover how many of all these different letters are
[02:02:46] many of all these different letters are packed into all these different tokens
[02:02:48] packed into all these different tokens and the reason we even use tokens is
[02:02:49] and the reason we even use tokens is mostly for efficiency uh but I think a
[02:02:51] mostly for efficiency uh but I think a lot of people areed interested to delete
[02:02:53] lot of people areed interested to delete tokens entirely like we should really
[02:02:55] tokens entirely like we should really have character level or bite level
[02:02:56] have character level or bite level models it's just that that would create
[02:02:58] models it's just that that would create very long sequences and people don't
[02:03:00] very long sequences and people don't know how to deal with that right now so
[02:03:02] know how to deal with that right now so while we have the token World any kind
[02:03:04] while we have the token World any kind of spelling tasks are not actually
[02:03:05] of spelling tasks are not actually expected to work super well so because I
[02:03:08] expected to work super well so because I know that spelling is not a strong suit
[02:03:09] know that spelling is not a strong suit because of tokenization I can again Ask
[02:03:12] because of tokenization I can again Ask it to lean On Tools so I can just say
[02:03:14] it to lean On Tools so I can just say use code and I would again expect this
[02:03:16] use code and I would again expect this to work because the task of copy pasting
[02:03:18] to work because the task of copy pasting ubiquitous into the python interpreter
[02:03:20] ubiquitous into the python interpreter is much easier and then we're leaning on
[02:03:22] is much easier and then we're leaning on python interpreter to manipulate the
[02:03:25] python interpreter to manipulate the characters of this string so when I say
[02:03:28] characters of this string so when I say use
[02:03:29] use code
[02:03:31] code ubiquitous yes it indexes into every
[02:03:33] ubiquitous yes it indexes into every third character and the actual truth is
[02:03:35] third character and the actual truth is u2s
[02:03:37] u2s uqs uh which looks correct to me so um
[02:03:41] uqs uh which looks correct to me so um again an example of spelling related
[02:03:43] again an example of spelling related tasks not working very well a very
[02:03:45] tasks not working very well a very famous example of that recently is how
[02:03:47] famous example of that recently is how many R are there in strawberry and this
[02:03:49] many R are there in strawberry and this went viral many times and basically the
[02:03:52] went viral many times and basically the models now get it correct they say there
[02:03:53] models now get it correct they say there are three Rs in Strawberry but for a
[02:03:55] are three Rs in Strawberry but for a very long time all the state-of-the-art
[02:03:57] very long time all the state-of-the-art models would insist that there are only
[02:03:58] models would insist that there are only two RS in strawberry and this caused a
[02:04:01] two RS in strawberry and this caused a lot of you know Ruckus because is that a
[02:04:04] lot of you know Ruckus because is that a word I think so because um it just kind
[02:04:07] word I think so because um it just kind of like why are the models so brilliant
[02:04:09] of like why are the models so brilliant and they can solve math Olympiad
[02:04:10] and they can solve math Olympiad questions but they can't like count RS
[02:04:13] questions but they can't like count RS in strawberry and the answer for that
[02:04:15] in strawberry and the answer for that again is I've got built up to it kind of
[02:04:17] again is I've got built up to it kind of slowly but number one the models don't
[02:04:19] slowly but number one the models don't see characters they see tokens and
[02:04:21] see characters they see tokens and number two they are not very good at
[02:04:23] number two they are not very good at counting and so here we are combining
[02:04:25] counting and so here we are combining the difficulty of seeing the characters
[02:04:27] the difficulty of seeing the characters with the difficulty of counting and
[02:04:29] with the difficulty of counting and that's why the models struggled with
[02:04:31] that's why the models struggled with this even though I think by now honestly
[02:04:33] this even though I think by now honestly I think open I may have hardcoded the
[02:04:35] I think open I may have hardcoded the answer here or I'm not sure what they
[02:04:36] answer here or I'm not sure what they did but um uh but this specific query
[02:04:40] did but um uh but this specific query now works
[02:04:42] now works so models are not very good at spelling
[02:04:44] so models are not very good at spelling and there there's a bunch of other
[02:04:46] and there there's a bunch of other little sharp edges and I don't want to
[02:04:47] little sharp edges and I don't want to go into all of them I just want to show
[02:04:48] go into all of them I just want to show you a few examples of things to be aware
[02:04:50] you a few examples of things to be aware of and uh when you're using these models
[02:04:52] of and uh when you're using these models in practice I don't actually want to
[02:04:54] in practice I don't actually want to have a comprehensive analysis here of
[02:04:56] have a comprehensive analysis here of all the ways that the models are kind of
[02:04:58] all the ways that the models are kind of like falling short I just want to make
[02:05:00] like falling short I just want to make the point that there are some Jagged
[02:05:01] the point that there are some Jagged edges here and there and we've discussed
[02:05:04] edges here and there and we've discussed a few of them and a few of them make
[02:05:05] a few of them and a few of them make sense but some of them also will just
[02:05:06] sense but some of them also will just not make as much sense and they're kind
[02:05:08] not make as much sense and they're kind of like you're left scratching your head
[02:05:10] of like you're left scratching your head even if you understand in- depth how
[02:05:12] even if you understand in- depth how these models work and and good example
[02:05:14] these models work and and good example of that recently is the following uh the
[02:05:16] of that recently is the following uh the models are not very good at very simple
[02:05:18] models are not very good at very simple questions like this and uh this is
[02:05:20] questions like this and uh this is shocking to a lot of people because
[02:05:22] shocking to a lot of people because these math uh these problems can solve
[02:05:24] these math uh these problems can solve complex math problems they can answer
[02:05:26] complex math problems they can answer PhD grade physics chemistry biology
[02:05:29] PhD grade physics chemistry biology questions much better than I can but
[02:05:31] questions much better than I can but sometimes they fall short in like super
[02:05:32] sometimes they fall short in like super simple problems like this so here we go
[02:05:35] simple problems like this so here we go 9.11 is bigger than 9.9 and it justifies
[02:05:38] 9.11 is bigger than 9.9 and it justifies it in some way but obviously and then at
[02:05:41] it in some way but obviously and then at the end okay it actually it flips its
[02:05:44] the end okay it actually it flips its decision later so um I don't believe
[02:05:47] decision later so um I don't believe that this is very reproducible sometimes
[02:05:49] that this is very reproducible sometimes it flips around its answer sometimes
[02:05:51] it flips around its answer sometimes gets it right sometimes get it get it
[02:05:52] gets it right sometimes get it get it wrong uh let's try
[02:06:00] again okay even though it might look larger okay so here it doesn't even
[02:06:02] larger okay so here it doesn't even correct itself in the end if you ask
[02:06:03] correct itself in the end if you ask many times sometimes it gets it right
[02:06:05] many times sometimes it gets it right too but how is it that the model can do
[02:06:07] too but how is it that the model can do so great at Olympiad grade problems but
[02:06:10] so great at Olympiad grade problems but then fail on very simple problems like
[02:06:12] then fail on very simple problems like this and uh I think this one is as I
[02:06:16] this and uh I think this one is as I mentioned a little bit of a head
[02:06:17] mentioned a little bit of a head scratcher it turns out that a bunch of
[02:06:18] scratcher it turns out that a bunch of people studied this in depth and I
[02:06:20] people studied this in depth and I haven't actually read the paper uh but
[02:06:22] haven't actually read the paper uh but what I was told by this team was that
[02:06:25] what I was told by this team was that when you scrutinize the activations
[02:06:28] when you scrutinize the activations inside the neural network when you look
[02:06:29] inside the neural network when you look at some of the features and what what
[02:06:31] at some of the features and what what features turn on or off and what neurons
[02:06:33] features turn on or off and what neurons turn on or off uh a bunch of neurons
[02:06:36] turn on or off uh a bunch of neurons inside the neural network light up that
[02:06:38] inside the neural network light up that are usually associated with Bible verses
[02:06:41] are usually associated with Bible verses U and so I think the model is kind of
[02:06:43] U and so I think the model is kind of like reminded that these almost look
[02:06:45] like reminded that these almost look like Bible verse markers and in a bip
[02:06:48] like Bible verse markers and in a bip verse setting 9.11 would come after 99.9
[02:06:52] verse setting 9.11 would come after 99.9 and so basically the model somehow finds
[02:06:54] and so basically the model somehow finds it like cognitively very distracting
[02:06:56] it like cognitively very distracting that in Bible verses 9.11 would be
[02:06:58] that in Bible verses 9.11 would be greater um even though here it's
[02:07:01] greater um even though here it's actually trying to justify it and come
[02:07:02] actually trying to justify it and come up to the answer with a math it still
[02:07:05] up to the answer with a math it still ends up with the wrong answer here so it
[02:07:07] ends up with the wrong answer here so it basically just doesn't fully make sense
[02:07:09] basically just doesn't fully make sense and it's not fully understood and um
[02:07:13] and it's not fully understood and um there's a few Jagged issues like that so
[02:07:15] there's a few Jagged issues like that so that's why treat this as a as what it is
[02:07:17] that's why treat this as a as what it is which is a St stochastic system that is
[02:07:19] which is a St stochastic system that is really magical but that you can't also
[02:07:21] really magical but that you can't also fully trust and you want to use it as a
[02:07:23] fully trust and you want to use it as a tool not as something that you kind of
[02:07:25] tool not as something that you kind of like letter rip on a problem and
[02:07:27] like letter rip on a problem and copypaste the results okay so we have
[02:07:29] copypaste the results okay so we have now covered two major stages of training
[02:07:32] now covered two major stages of training of large language models we saw that in
[02:07:35] of large language models we saw that in the first stage this is called the
[02:07:36] the first stage this is called the pre-training stage we are basically
[02:07:38] pre-training stage we are basically training on internet documents and when
[02:07:41] training on internet documents and when you train a language model on internet
[02:07:42] you train a language model on internet documents you get what's called a base
[02:07:44] documents you get what's called a base model and it's basically an internet
[02:07:46] model and it's basically an internet document simulator right now we saw that
[02:07:49] document simulator right now we saw that this is an interesting artifact and uh
[02:07:51] this is an interesting artifact and uh this takes many months to train on
[02:07:53] this takes many months to train on thousands of computers and it's kind of
[02:07:55] thousands of computers and it's kind of a lossy compression of the internet and
[02:07:57] a lossy compression of the internet and it's extremely interesting but it's not
[02:07:59] it's extremely interesting but it's not directly useful because we don't want to
[02:08:01] directly useful because we don't want to sample internet documents we want to ask
[02:08:03] sample internet documents we want to ask questions of an AI and have it respond
[02:08:05] questions of an AI and have it respond to our questions so for that we need an
[02:08:07] to our questions so for that we need an assistant and we saw that we can
[02:08:09] assistant and we saw that we can actually construct an assistant in the
[02:08:11] actually construct an assistant in the process of a post
[02:08:14] process of a post training and specifically in the process
[02:08:17] training and specifically in the process of supervised fine-tuning as we call
[02:08:19] of supervised fine-tuning as we call it so in this stage we saw that it's
[02:08:23] it so in this stage we saw that it's algorithmically identical to
[02:08:24] algorithmically identical to pre-training nothing is going to change
[02:08:26] pre-training nothing is going to change the only thing that changes is the data
[02:08:28] the only thing that changes is the data set so instead of Internet documents we
[02:08:30] set so instead of Internet documents we now want to create and curate a very
[02:08:33] now want to create and curate a very nice data set of conversations so we
[02:08:35] nice data set of conversations so we want Millions conversations on all kinds
[02:08:38] want Millions conversations on all kinds of diverse topics between a human and an
[02:08:42] of diverse topics between a human and an assistant and fundamentally these
[02:08:44] assistant and fundamentally these conversations are created by humans so
[02:08:47] conversations are created by humans so humans write the prompts and humans
[02:08:50] humans write the prompts and humans write the ideal response responses and
[02:08:52] write the ideal response responses and they do that based on labeling
[02:08:55] they do that based on labeling documentations now in the modern stack
[02:08:57] documentations now in the modern stack it's not actually done fully and
[02:08:59] it's not actually done fully and manually by humans right they actually
[02:09:01] manually by humans right they actually now have a lot of help from these tools
[02:09:03] now have a lot of help from these tools so we can use language models um to help
[02:09:06] so we can use language models um to help us create these data sets and that's
[02:09:07] us create these data sets and that's done extensively but fundamentally it's
[02:09:09] done extensively but fundamentally it's all still coming from Human curation at
[02:09:11] all still coming from Human curation at the end so we create these conversations
[02:09:14] the end so we create these conversations that now becomes our data set we fine
[02:09:16] that now becomes our data set we fine tune on it or continue training on it
[02:09:18] tune on it or continue training on it and we get an assistant and then we kind
[02:09:20] and we get an assistant and then we kind of shifted gears and started talking
[02:09:22] of shifted gears and started talking about some of the kind of cognitive
[02:09:23] about some of the kind of cognitive implications of what this assistant is
[02:09:25] implications of what this assistant is like and we saw that for example the
[02:09:27] like and we saw that for example the assistant will hallucinate if you don't
[02:09:30] assistant will hallucinate if you don't take some sort of mitigations towards it
[02:09:32] take some sort of mitigations towards it so we saw that hallucinations would be
[02:09:34] so we saw that hallucinations would be common and then we looked at some of the
[02:09:36] common and then we looked at some of the mitigations of those hallucinations and
[02:09:38] mitigations of those hallucinations and then we saw that the models are quite
[02:09:39] then we saw that the models are quite impressive and can do a lot of stuff in
[02:09:41] impressive and can do a lot of stuff in their head but we saw that they can also
[02:09:43] their head but we saw that they can also Lean On Tools to become better so for
[02:09:46] Lean On Tools to become better so for example we can lo lean on a web search
[02:09:48] example we can lo lean on a web search in order to hallucinate less and to
[02:09:51] in order to hallucinate less and to maybe bring up some more um recent
[02:09:53] maybe bring up some more um recent information or something like that or we
[02:09:55] information or something like that or we can lean on tools like code interpreter
[02:09:57] can lean on tools like code interpreter so the code can so the llm can write
[02:09:59] so the code can so the llm can write some code and actually run it and see
[02:10:01] some code and actually run it and see the
[02:10:02] the results so these are some of the topics
[02:10:04] results so these are some of the topics we looked at so far um now what I'd like
[02:10:06] we looked at so far um now what I'd like to do is I'd like to cover the last and
[02:10:09] to do is I'd like to cover the last and major stage of this Pipeline and that is
[02:10:13] major stage of this Pipeline and that is reinforcement learning so reinforcement
[02:10:16] reinforcement learning so reinforcement learning is still kind of thought to be
[02:10:17] learning is still kind of thought to be under the umbrella of posttraining uh
[02:10:20] under the umbrella of posttraining uh but it is the last third major stage and
[02:10:22] but it is the last third major stage and it's a different way of training
[02:10:24] it's a different way of training language models and usually follows as
[02:10:27] language models and usually follows as this third step so inside companies like
[02:10:29] this third step so inside companies like open AI you will start here and these
[02:10:31] open AI you will start here and these are all separate teams so there's a team
[02:10:33] are all separate teams so there's a team doing data for pre-training and a team
[02:10:36] doing data for pre-training and a team doing training for pre-training and then
[02:10:38] doing training for pre-training and then there's a team doing all the
[02:10:40] there's a team doing all the conversation generation in a in a
[02:10:42] conversation generation in a in a different team that is kind of doing the
[02:10:44] different team that is kind of doing the supervis fine tuning and there will be a
[02:10:46] supervis fine tuning and there will be a team for the reinforcement learning as
[02:10:47] team for the reinforcement learning as well so it's kind of like a handoff of
[02:10:49] well so it's kind of like a handoff of these models you get your base model the
[02:10:51] these models you get your base model the then you find you need to be an
[02:10:52] then you find you need to be an assistant and then you go into
[02:10:54] assistant and then you go into reinforcement learning which we'll talk
[02:10:55] reinforcement learning which we'll talk about uh
[02:10:57] about uh now so that's kind of like the major
[02:10:59] now so that's kind of like the major flow and so let's now focus on
[02:11:01] flow and so let's now focus on reinforcement learning the last major
[02:11:03] reinforcement learning the last major stage of training and let me first
[02:11:06] stage of training and let me first actually motivate it and why we would
[02:11:07] actually motivate it and why we would want to do reinforcement learning and
[02:11:09] want to do reinforcement learning and what it looks like on a high level so I
[02:11:11] what it looks like on a high level so I would now like to try to motivate the
[02:11:13] would now like to try to motivate the reinforcement learning stage and what it
[02:11:14] reinforcement learning stage and what it corresponds to with something that
[02:11:15] corresponds to with something that you're probably familiar with and that
[02:11:17] you're probably familiar with and that is basically going to school so just
[02:11:19] is basically going to school so just like you went to school to become um
[02:11:21] like you went to school to become um really good at something we want to take
[02:11:23] really good at something we want to take large language models through school and
[02:11:26] large language models through school and really what we're doing is um we're um
[02:11:30] really what we're doing is um we're um we have a few paradigms of ways of uh
[02:11:32] we have a few paradigms of ways of uh giving them knowledge or transferring
[02:11:34] giving them knowledge or transferring skills so in particular when we're
[02:11:36] skills so in particular when we're working with textbooks in school you'll
[02:11:38] working with textbooks in school you'll see that there are three major kind of
[02:11:41] see that there are three major kind of uh pieces of information in these
[02:11:43] uh pieces of information in these textbooks three classes of information
[02:11:46] textbooks three classes of information the first thing you'll see is you'll see
[02:11:47] the first thing you'll see is you'll see a lot of exposition um and by the way
[02:11:49] a lot of exposition um and by the way this is a totally random book I pulled
[02:11:51] this is a totally random book I pulled from the internet I I think it's some
[02:11:52] from the internet I I think it's some kind of an organic chemistry or
[02:11:53] kind of an organic chemistry or something I'm not sure uh but the
[02:11:55] something I'm not sure uh but the important thing is that you'll see that
[02:11:57] important thing is that you'll see that most of the text most of it is kind of
[02:11:59] most of the text most of it is kind of just like the meat of it is exposition
[02:12:01] just like the meat of it is exposition it's kind of like background knowledge
[02:12:03] it's kind of like background knowledge Etc as you are reading through the words
[02:12:06] Etc as you are reading through the words of this Exposition you can think of that
[02:12:08] of this Exposition you can think of that roughly as training on that data so um
[02:12:12] roughly as training on that data so um and that's why when you're reading
[02:12:14] and that's why when you're reading through this stuff this background
[02:12:15] through this stuff this background knowledge and this all this context
[02:12:16] knowledge and this all this context information it's kind of equivalent to
[02:12:19] information it's kind of equivalent to pre-training so it's it's where we build
[02:12:21] pre-training so it's it's where we build sort of like a knowledge base of this
[02:12:24] sort of like a knowledge base of this data and get a sense of the topic the
[02:12:27] data and get a sense of the topic the next major kind of information that you
[02:12:29] next major kind of information that you will see is these uh problems and with
[02:12:33] will see is these uh problems and with their worked Solutions so basically a
[02:12:36] their worked Solutions so basically a human expert in this case uh the author
[02:12:37] human expert in this case uh the author of this book has given us not just a
[02:12:39] of this book has given us not just a problem but has also worked through the
[02:12:41] problem but has also worked through the solution and the solution is basically
[02:12:44] solution and the solution is basically like equivalent to having like this
[02:12:46] like equivalent to having like this ideal response for an assistant so it's
[02:12:48] ideal response for an assistant so it's basically the expert is showing us how
[02:12:50] basically the expert is showing us how to solve the problem in it's uh kind of
[02:12:52] to solve the problem in it's uh kind of like um in its full form so as we are
[02:12:55] like um in its full form so as we are reading the solution we are basically
[02:12:58] reading the solution we are basically training on the expert data and then
[02:13:01] training on the expert data and then later we can try to imitate the expert
[02:13:04] later we can try to imitate the expert um and basically um that's that roughly
[02:13:07] um and basically um that's that roughly correspond to having the sft model
[02:13:09] correspond to having the sft model that's what it would be doing so
[02:13:11] that's what it would be doing so basically we've already done
[02:13:12] basically we've already done pre-training and we've already covered
[02:13:15] pre-training and we've already covered this um imitation of experts and how
[02:13:17] this um imitation of experts and how they solve these problems and the third
[02:13:20] they solve these problems and the third stage of reinforcement learning is
[02:13:22] stage of reinforcement learning is basically the practice problems so
[02:13:24] basically the practice problems so sometimes you'll see this is just a
[02:13:26] sometimes you'll see this is just a single practice problem here but of
[02:13:27] single practice problem here but of course there will be usually many
[02:13:29] course there will be usually many practice problems at the end of each
[02:13:30] practice problems at the end of each chapter in any textbook and practice
[02:13:33] chapter in any textbook and practice problems of course we know are critical
[02:13:34] problems of course we know are critical for learning because what are they
[02:13:36] for learning because what are they getting you to do they're getting you to
[02:13:38] getting you to do they're getting you to practice uh to practice yourself and
[02:13:40] practice uh to practice yourself and discover ways of solving these problems
[02:13:42] discover ways of solving these problems yourself and so what you get in a
[02:13:44] yourself and so what you get in a practice problem is you get a problem
[02:13:46] practice problem is you get a problem description but you're not given the
[02:13:49] description but you're not given the solution but you are given the final
[02:13:51] solution but you are given the final answer answer usually in the answer key
[02:13:53] answer answer usually in the answer key of the textbook and so you know the
[02:13:55] of the textbook and so you know the final answer that you're trying to get
[02:13:57] final answer that you're trying to get to and you have the problem statement
[02:13:59] to and you have the problem statement but you don't have the solution you are
[02:14:00] but you don't have the solution you are trying to practice the solution you're
[02:14:03] trying to practice the solution you're trying out many different things and
[02:14:04] trying out many different things and you're seeing what gets you to the final
[02:14:07] you're seeing what gets you to the final solution the best and so you're
[02:14:10] solution the best and so you're discovering how to solve these problems
[02:14:12] discovering how to solve these problems so and in the process of that you're
[02:14:13] so and in the process of that you're relying on number one the background
[02:14:15] relying on number one the background information which comes from
[02:14:16] information which comes from pre-training and number two maybe a
[02:14:18] pre-training and number two maybe a little bit of imitation of human experts
[02:14:20] little bit of imitation of human experts and you can probably try similar kinds
[02:14:22] and you can probably try similar kinds of solutions and so on so we've done
[02:14:25] of solutions and so on so we've done this and this and now in this section
[02:14:27] this and this and now in this section we're going to try to practice and so
[02:14:30] we're going to try to practice and so we're going to be given prompts we're
[02:14:32] we're going to be given prompts we're going to be given Solutions U sorry the
[02:14:35] going to be given Solutions U sorry the final answers but we're not going to be
[02:14:36] final answers but we're not going to be given expert Solutions we have to
[02:14:39] given expert Solutions we have to practice and try stuff out and that's
[02:14:41] practice and try stuff out and that's what reinforcement learning is about
[02:14:43] what reinforcement learning is about okay so let's go back to the problem
[02:14:44] okay so let's go back to the problem that we worked with previously just so
[02:14:46] that we worked with previously just so we have a concrete example to talk
[02:14:48] we have a concrete example to talk through as we explore sort of the topic
[02:14:50] through as we explore sort of the topic here so um I'm here in the Teck
[02:14:53] here so um I'm here in the Teck tokenizer because I'd also like to well
[02:14:55] tokenizer because I'd also like to well I get a text box which is useful but
[02:14:57] I get a text box which is useful but number two I want to remind you again
[02:14:59] number two I want to remind you again that we're always working with
[02:15:00] that we're always working with onedimensional token sequences and so um
[02:15:03] onedimensional token sequences and so um I actually like prefer this view because
[02:15:04] I actually like prefer this view because this is like the native view of the llm
[02:15:06] this is like the native view of the llm if that makes sense like this is what it
[02:15:08] if that makes sense like this is what it actually sees it sees token IDs right
[02:15:11] actually sees it sees token IDs right okay so Emily buys three apples and two
[02:15:14] okay so Emily buys three apples and two oranges each orange is $2 the total cost
[02:15:17] oranges each orange is $2 the total cost of all the fruit is $13 what is the cost
[02:15:20] of all the fruit is $13 what is the cost of each apple
[02:15:22] of each apple and what I'd like to what I like you to
[02:15:23] and what I'd like to what I like you to appreciate here is these are like four
[02:15:26] appreciate here is these are like four possible candidate Solutions as an
[02:15:29] possible candidate Solutions as an example and they all reach the answer
[02:15:32] example and they all reach the answer three now what I'd like you to
[02:15:33] three now what I'd like you to appreciate at this point is that if I am
[02:15:35] appreciate at this point is that if I am the human data labeler that is creating
[02:15:38] the human data labeler that is creating a conversation to be entered into the
[02:15:39] a conversation to be entered into the training set I don't actually really
[02:15:42] training set I don't actually really know which of these
[02:15:44] know which of these conversations to um to add to the data
[02:15:48] conversations to um to add to the data set some of these conversations kind of
[02:15:50] set some of these conversations kind of set up a system equations some of them
[02:15:53] set up a system equations some of them sort of like just talk through it in
[02:15:54] sort of like just talk through it in English and some of them just kind of
[02:15:56] English and some of them just kind of like skip right through to the
[02:15:58] like skip right through to the solution um if you look at chbt for
[02:16:01] solution um if you look at chbt for example and you give it this question it
[02:16:04] example and you give it this question it defines a system of variables and it
[02:16:05] defines a system of variables and it kind of like does this little thing what
[02:16:07] kind of like does this little thing what we have to appreciate and uh
[02:16:09] we have to appreciate and uh differentiate between though is um the
[02:16:12] differentiate between though is um the first purpose of a solution is to reach
[02:16:14] first purpose of a solution is to reach the right answer of course we want to
[02:16:16] the right answer of course we want to get the final answer three that is the
[02:16:18] get the final answer three that is the that is the important purpose here but
[02:16:20] that is the important purpose here but there's kind of like a secondary purpose
[02:16:21] there's kind of like a secondary purpose as well where here we are also just kind
[02:16:24] as well where here we are also just kind of trying to make it like nice uh for
[02:16:26] of trying to make it like nice uh for the human because we're kind of assuming
[02:16:28] the human because we're kind of assuming that the person wants to see the
[02:16:29] that the person wants to see the solution they want to see the
[02:16:30] solution they want to see the intermediate steps we want to present it
[02:16:32] intermediate steps we want to present it nicely Etc so there are two separate
[02:16:34] nicely Etc so there are two separate things going on here number one is the
[02:16:36] things going on here number one is the presentation for the human but number
[02:16:38] presentation for the human but number two we're trying to actually get the
[02:16:39] two we're trying to actually get the right answer um so let's for the moment
[02:16:42] right answer um so let's for the moment focus on just reaching the final answer
[02:16:45] focus on just reaching the final answer if we're only care if we only care about
[02:16:47] if we're only care if we only care about the final answer then which of these is
[02:16:50] the final answer then which of these is the optimal or the best prompt um sorry
[02:16:54] the optimal or the best prompt um sorry the best solution for the llm to reach
[02:16:56] the best solution for the llm to reach the right
[02:16:58] the right answer um and what I'm trying to get at
[02:17:00] answer um and what I'm trying to get at is we don't know me as a human labeler I
[02:17:03] is we don't know me as a human labeler I would not know which one of these is
[02:17:04] would not know which one of these is best so as an example we saw earlier on
[02:17:07] best so as an example we saw earlier on when we looked at
[02:17:09] when we looked at um the token sequences here and the
[02:17:12] um the token sequences here and the mental arithmetic and reasoning we saw
[02:17:14] mental arithmetic and reasoning we saw that for each token we can only spend
[02:17:16] that for each token we can only spend basically a finite number of finite
[02:17:18] basically a finite number of finite amount of compute here that is not very
[02:17:20] amount of compute here that is not very large or you should think about it that
[02:17:21] large or you should think about it that way way and so we can't actually make
[02:17:24] way way and so we can't actually make too big of a leap in any one token is is
[02:17:27] too big of a leap in any one token is is maybe the way to think about it so as an
[02:17:29] maybe the way to think about it so as an example in this one what's really nice
[02:17:31] example in this one what's really nice about it is that it's very few tokens so
[02:17:33] about it is that it's very few tokens so it's going to take us very short amount
[02:17:34] it's going to take us very short amount of time to get to the answer but right
[02:17:37] of time to get to the answer but right here when we're doing 30 - 4 IDE 3
[02:17:40] here when we're doing 30 - 4 IDE 3 equals right in this token here we're
[02:17:43] equals right in this token here we're actually asking for a lot of computation
[02:17:44] actually asking for a lot of computation to happen on that single individual
[02:17:46] to happen on that single individual token and so maybe this is a bad example
[02:17:48] token and so maybe this is a bad example to give to the llm because it's kind of
[02:17:49] to give to the llm because it's kind of incentivizing it to skip through the
[02:17:51] incentivizing it to skip through the calculations very quickly and it's going
[02:17:52] calculations very quickly and it's going to actually make up mistakes make
[02:17:54] to actually make up mistakes make mistakes in this mental arithmetic uh so
[02:17:57] mistakes in this mental arithmetic uh so maybe it would work better to like
[02:17:58] maybe it would work better to like spread out the spread it out more maybe
[02:18:01] spread out the spread it out more maybe it would be better to set it up as an
[02:18:02] it would be better to set it up as an equation maybe it would be better to
[02:18:04] equation maybe it would be better to talk through it we fundamentally don't
[02:18:06] talk through it we fundamentally don't know and we don't know because what is
[02:18:10] know and we don't know because what is easy for you or I as or as human
[02:18:12] easy for you or I as or as human labelers what's easy for us or hard for
[02:18:14] labelers what's easy for us or hard for us is different than what's easy or hard
[02:18:17] us is different than what's easy or hard for the llm it cognition is different um
[02:18:20] for the llm it cognition is different um and the token sequences are kind of like
[02:18:23] and the token sequences are kind of like different hard for it and so some of the
[02:18:28] different hard for it and so some of the token sequences here that are trivial
[02:18:30] token sequences here that are trivial for me might be um very too much of a
[02:18:34] for me might be um very too much of a leap for the llm so right here this
[02:18:37] leap for the llm so right here this token would be way too hard but
[02:18:38] token would be way too hard but conversely many of the tokens that I'm
[02:18:41] conversely many of the tokens that I'm creating here might be just trivial to
[02:18:43] creating here might be just trivial to the llm and we're just wasting tokens
[02:18:45] the llm and we're just wasting tokens like why waste all these tokens when
[02:18:47] like why waste all these tokens when this is all trivial so if the only thing
[02:18:49] this is all trivial so if the only thing we care care about is the final answer
[02:18:52] we care care about is the final answer and we're separating out the issue of
[02:18:53] and we're separating out the issue of the presentation to the human um then we
[02:18:56] the presentation to the human um then we don't actually really know how to
[02:18:57] don't actually really know how to annotate this example we don't know what
[02:18:59] annotate this example we don't know what solution to get to the llm because we
[02:19:01] solution to get to the llm because we are not the
[02:19:02] are not the llm and it's clear here in the case of
[02:19:05] llm and it's clear here in the case of like the math example but this is
[02:19:07] like the math example but this is actually like a very pervasive issue
[02:19:09] actually like a very pervasive issue like for our knowledge is not lm's
[02:19:12] like for our knowledge is not lm's knowledge like the llm actually has a
[02:19:14] knowledge like the llm actually has a ton of knowledge of PhD in math and
[02:19:15] ton of knowledge of PhD in math and physics chemistry and whatnot so in many
[02:19:17] physics chemistry and whatnot so in many ways it actually knows more than I do
[02:19:19] ways it actually knows more than I do and I'm I'm potentially not utilizing
[02:19:22] and I'm I'm potentially not utilizing that knowledge in its problem solving
[02:19:24] that knowledge in its problem solving but conversely I might be injecting a
[02:19:26] but conversely I might be injecting a bunch of knowledge in my solutions that
[02:19:28] bunch of knowledge in my solutions that the LM doesn't know in its parameters
[02:19:31] the LM doesn't know in its parameters and then those are like sudden leaps
[02:19:33] and then those are like sudden leaps that are very confusing to the model and
[02:19:36] that are very confusing to the model and so our cognitions are different and I
[02:19:39] so our cognitions are different and I don't really know what to put here if
[02:19:41] don't really know what to put here if all we care about is the reaching the
[02:19:43] all we care about is the reaching the final solution and doing it economically
[02:19:46] final solution and doing it economically ideally and so long story short we are
[02:19:50] ideally and so long story short we are not in a good position to create these
[02:19:53] not in a good position to create these uh token sequences for the LM and
[02:19:55] uh token sequences for the LM and they're useful by imitation to
[02:19:57] they're useful by imitation to initialize the system but we really want
[02:20:00] initialize the system but we really want the llm to discover the token sequences
[02:20:02] the llm to discover the token sequences that work for it we need to find it
[02:20:05] that work for it we need to find it needs to find for itself what token
[02:20:07] needs to find for itself what token sequence reliably gets to the answer
[02:20:10] sequence reliably gets to the answer given the prompt and it needs to
[02:20:12] given the prompt and it needs to discover that in the process of
[02:20:13] discover that in the process of reinforcement learning and of trial and
[02:20:14] reinforcement learning and of trial and error so let's see how this example
[02:20:18] error so let's see how this example would work like in reinforcement
[02:20:20] would work like in reinforcement learning
[02:20:21] learning okay so we're now back in the huging
[02:20:23] okay so we're now back in the huging face inference playground and uh that
[02:20:26] face inference playground and uh that just allows me to very easily call uh
[02:20:28] just allows me to very easily call uh different kinds of models so as an
[02:20:30] different kinds of models so as an example here on the top right I chose
[02:20:31] example here on the top right I chose the Gemma 2 2 billion parameter model so
[02:20:34] the Gemma 2 2 billion parameter model so two billion is very very small so this
[02:20:36] two billion is very very small so this is a tiny model but it's okay so we're
[02:20:39] is a tiny model but it's okay so we're going to give it um the way that
[02:20:41] going to give it um the way that reinforcement learning will basically
[02:20:42] reinforcement learning will basically work is actually quite quite simple um
[02:20:45] work is actually quite quite simple um we need to try many different kinds of
[02:20:47] we need to try many different kinds of solutions and we want to see which
[02:20:49] solutions and we want to see which Solutions work well or not
[02:20:51] Solutions work well or not so we're basically going to take the
[02:20:53] so we're basically going to take the prompt we're going to run the
[02:20:55] prompt we're going to run the model and the model generates a solution
[02:20:59] model and the model generates a solution and then we're going to inspect the
[02:21:00] and then we're going to inspect the solution and we know that the correct
[02:21:02] solution and we know that the correct answer for this one is $3 and so indeed
[02:21:05] answer for this one is $3 and so indeed the model gets it correct it says it's
[02:21:07] the model gets it correct it says it's $3 so this is correct so that's just one
[02:21:10] $3 so this is correct so that's just one attempt at DIS solution so now we're
[02:21:12] attempt at DIS solution so now we're going to delete this and we're going to
[02:21:13] going to delete this and we're going to rerun it again let's try a second
[02:21:15] rerun it again let's try a second attempt so the model solves it in a bit
[02:21:18] attempt so the model solves it in a bit slightly different way right every
[02:21:20] slightly different way right every single attempt will be a different
[02:21:22] single attempt will be a different generation because these models are
[02:21:23] generation because these models are stochastic systems remember that at
[02:21:25] stochastic systems remember that at every single token here we have a
[02:21:26] every single token here we have a probability distribution and we're
[02:21:28] probability distribution and we're sampling from that distribution so we
[02:21:30] sampling from that distribution so we end up kind kind of going down slightly
[02:21:32] end up kind kind of going down slightly different paths and so this is a second
[02:21:34] different paths and so this is a second solution that also ends in the correct
[02:21:36] solution that also ends in the correct answer now we're going to delete that
[02:21:39] answer now we're going to delete that let's go a third
[02:21:40] let's go a third time okay so again slightly different
[02:21:42] time okay so again slightly different solution but also gets it
[02:21:44] solution but also gets it correct now we can actually repeat this
[02:21:47] correct now we can actually repeat this uh many times and so in practice you
[02:21:49] uh many times and so in practice you might actually sample thousand of
[02:21:51] might actually sample thousand of independent Solutions or even like
[02:21:53] independent Solutions or even like million solutions for just a single
[02:21:55] million solutions for just a single prompt um and some of them will be
[02:21:58] prompt um and some of them will be correct and some of them will not be
[02:21:59] correct and some of them will not be very correct and basically what we want
[02:22:01] very correct and basically what we want to do is we want to encourage the
[02:22:02] to do is we want to encourage the solutions that lead to correct answers
[02:22:05] solutions that lead to correct answers so let's take a look at what that looks
[02:22:07] so let's take a look at what that looks like so if we come back over here here's
[02:22:09] like so if we come back over here here's kind of like a cartoon diagram of what
[02:22:11] kind of like a cartoon diagram of what this is looking like we have a prompt
[02:22:14] this is looking like we have a prompt and then we tried many different
[02:22:15] and then we tried many different solutions in
[02:22:17] solutions in parallel and some of the solutions um
[02:22:20] parallel and some of the solutions um might go well so they get the right
[02:22:22] might go well so they get the right answer which is in green and some of the
[02:22:24] answer which is in green and some of the solutions might go poorly and may not
[02:22:26] solutions might go poorly and may not reach the right answer which is red now
[02:22:29] reach the right answer which is red now this problem here unfortunately is not
[02:22:30] this problem here unfortunately is not the best example because it's a trivial
[02:22:32] the best example because it's a trivial prompt and as we saw uh even like a two
[02:22:35] prompt and as we saw uh even like a two billion parameter model always gets it
[02:22:36] billion parameter model always gets it right so it's not the best example in
[02:22:38] right so it's not the best example in that sense but let's just exercise some
[02:22:40] that sense but let's just exercise some imagination here and let's just suppose
[02:22:44] imagination here and let's just suppose that the um green ones are good and the
[02:22:47] that the um green ones are good and the red ones are
[02:22:49] red ones are bad okay so we generated 15 Solutions
[02:22:52] bad okay so we generated 15 Solutions only four of them got the right answer
[02:22:55] only four of them got the right answer and so now what we want to do is
[02:22:57] and so now what we want to do is basically we want to encourage the kinds
[02:22:58] basically we want to encourage the kinds of solutions that lead to right answers
[02:23:01] of solutions that lead to right answers so whatever token sequences happened in
[02:23:03] so whatever token sequences happened in these red Solutions obviously something
[02:23:05] these red Solutions obviously something went wrong along the way somewhere and
[02:23:08] went wrong along the way somewhere and uh this was not a good path to take
[02:23:10] uh this was not a good path to take through the solution and whatever token
[02:23:12] through the solution and whatever token sequences there were in these Green
[02:23:13] sequences there were in these Green Solutions well things went uh pretty
[02:23:15] Solutions well things went uh pretty well in this situation and so we want to
[02:23:18] well in this situation and so we want to do more things like it in prompts like
[02:23:21] do more things like it in prompts like this and the way we encourage this kind
[02:23:24] this and the way we encourage this kind of a behavior in the future is we
[02:23:25] of a behavior in the future is we basically train on these sequences um
[02:23:28] basically train on these sequences um but these training sequencies now are
[02:23:30] but these training sequencies now are not coming from expert human annotators
[02:23:32] not coming from expert human annotators there's no human who decided that this
[02:23:34] there's no human who decided that this is the correct solution this solution
[02:23:36] is the correct solution this solution came from the model itself so the model
[02:23:38] came from the model itself so the model is practicing here it's tried out a few
[02:23:40] is practicing here it's tried out a few Solutions four of them seem to have
[02:23:42] Solutions four of them seem to have worked and now the model will kind of
[02:23:44] worked and now the model will kind of like train on them and this corresponds
[02:23:46] like train on them and this corresponds to a student basically looking at their
[02:23:47] to a student basically looking at their Solutions and being like okay well this
[02:23:49] Solutions and being like okay well this one worked really well so this is this
[02:23:50] one worked really well so this is this is how I should be solving these kinds
[02:23:52] is how I should be solving these kinds of problems and uh here in this example
[02:23:56] of problems and uh here in this example there are many different ways to
[02:23:57] there are many different ways to actually like really tweak the
[02:23:59] actually like really tweak the methodology a little bit here but just
[02:24:01] methodology a little bit here but just to give the core idea across maybe it's
[02:24:02] to give the core idea across maybe it's simplest to just think about take the
[02:24:05] simplest to just think about take the taking the single best solution out of
[02:24:06] taking the single best solution out of these four uh like say this one that's
[02:24:09] these four uh like say this one that's why it was yellow uh so this is the the
[02:24:12] why it was yellow uh so this is the the solution that not only led to the right
[02:24:14] solution that not only led to the right answer but may maybe had some other nice
[02:24:16] answer but may maybe had some other nice properties maybe it was the shortest one
[02:24:18] properties maybe it was the shortest one or it looked nicest in some ways or uh
[02:24:21] or it looked nicest in some ways or uh there's other criteria you could think
[02:24:22] there's other criteria you could think of as an example but we're going to
[02:24:24] of as an example but we're going to decide that this the top solution we're
[02:24:25] decide that this the top solution we're going to train on it and then uh the
[02:24:28] going to train on it and then uh the model will be slightly more likely once
[02:24:31] model will be slightly more likely once you do the parameter update to take this
[02:24:34] you do the parameter update to take this path in this kind of a setting in the
[02:24:36] path in this kind of a setting in the future but you have to remember that
[02:24:38] future but you have to remember that we're going to run many different
[02:24:40] we're going to run many different diverse prompts across lots of math
[02:24:42] diverse prompts across lots of math problems and physics problems and
[02:24:43] problems and physics problems and whatever wherever there might be so tens
[02:24:46] whatever wherever there might be so tens of thousands of prompts maybe have in
[02:24:48] of thousands of prompts maybe have in mind there's thousands of solutions
[02:24:51] mind there's thousands of solutions prompt and so this is all happening kind
[02:24:53] prompt and so this is all happening kind of like at the same time and as we're
[02:24:55] of like at the same time and as we're iterating this process the model is
[02:24:58] iterating this process the model is discovering for itself what kinds of
[02:25:00] discovering for itself what kinds of token sequences lead it to correct
[02:25:03] token sequences lead it to correct answers it's not coming from a human
[02:25:05] answers it's not coming from a human annotator the the model is kind of like
[02:25:08] annotator the the model is kind of like playing in this playground and it knows
[02:25:10] playing in this playground and it knows what it's trying to get to and it's
[02:25:13] what it's trying to get to and it's discovering sequences that work for it
[02:25:15] discovering sequences that work for it uh these are sequences that don't make
[02:25:17] uh these are sequences that don't make any mental leaps uh they they seem to
[02:25:20] any mental leaps uh they they seem to work reliably and statistically and uh
[02:25:23] work reliably and statistically and uh fully utilize the knowledge of the model
[02:25:25] fully utilize the knowledge of the model as it has it and so uh this is the
[02:25:28] as it has it and so uh this is the process of reinforcement
[02:25:30] process of reinforcement learning it's basically a guess and
[02:25:32] learning it's basically a guess and check we're going to guess many
[02:25:33] check we're going to guess many different types of solutions we're going
[02:25:34] different types of solutions we're going to check them and we're going to do more
[02:25:36] to check them and we're going to do more of what worked in the future and that is
[02:25:39] of what worked in the future and that is uh reinforcement learning so in the
[02:25:41] uh reinforcement learning so in the context of what came before we see now
[02:25:43] context of what came before we see now that the sft model the supervised fine
[02:25:45] that the sft model the supervised fine tuning model it's still helpful because
[02:25:47] tuning model it's still helpful because it still kind of like initializes the
[02:25:49] it still kind of like initializes the model a little bit into to the vicinity
[02:25:51] model a little bit into to the vicinity of the correct Solutions so it's kind of
[02:25:53] of the correct Solutions so it's kind of like a initialization of um of the model
[02:25:57] like a initialization of um of the model in the sense that it kind of gets the
[02:25:58] in the sense that it kind of gets the model to you know take Solutions like
[02:26:01] model to you know take Solutions like write out Solutions and maybe it has an
[02:26:03] write out Solutions and maybe it has an understanding of setting up a system of
[02:26:04] understanding of setting up a system of equations or maybe it kind of like talks
[02:26:06] equations or maybe it kind of like talks through a solution so it gets you into
[02:26:08] through a solution so it gets you into the vicinity of correct Solutions but
[02:26:10] the vicinity of correct Solutions but reinforcement learning is where
[02:26:12] reinforcement learning is where everything gets dialed in we really
[02:26:14] everything gets dialed in we really discover the solutions that work for the
[02:26:15] discover the solutions that work for the model get the right answers we encourage
[02:26:18] model get the right answers we encourage them and then the model just kind of
[02:26:19] them and then the model just kind of like gets better over time time okay so
[02:26:22] like gets better over time time okay so that is the high Lev process for how we
[02:26:23] that is the high Lev process for how we train large language models in short we
[02:26:26] train large language models in short we train them kind of very similar to how
[02:26:28] train them kind of very similar to how we train children and basically the only
[02:26:31] we train children and basically the only difference is that children go through
[02:26:32] difference is that children go through chapters of books and they do all these
[02:26:35] chapters of books and they do all these different types of training exercises um
[02:26:38] different types of training exercises um kind of within the chapter of each book
[02:26:40] kind of within the chapter of each book but instead when we train AIS it's
[02:26:41] but instead when we train AIS it's almost like we kind of do it stage by
[02:26:43] almost like we kind of do it stage by stage depending on the type of that
[02:26:45] stage depending on the type of that stage so first what we do is we do
[02:26:48] stage so first what we do is we do pre-training which as we saw is
[02:26:49] pre-training which as we saw is equivalent to uh basically reading all
[02:26:51] equivalent to uh basically reading all the expository material so we look at
[02:26:54] the expository material so we look at all the textbooks at the same time and
[02:26:55] all the textbooks at the same time and we read all the exposition and we try to
[02:26:58] we read all the exposition and we try to build a knowledge base the second thing
[02:27:00] build a knowledge base the second thing then is we go into the sft stage which
[02:27:03] then is we go into the sft stage which is really looking at all the fixed uh
[02:27:05] is really looking at all the fixed uh sort of like solutions from Human
[02:27:07] sort of like solutions from Human Experts of all the different kinds of
[02:27:10] Experts of all the different kinds of worked Solutions across all the
[02:27:12] worked Solutions across all the textbooks and we just kind of get an sft
[02:27:15] textbooks and we just kind of get an sft model which is able to imitate the
[02:27:16] model which is able to imitate the experts but does so kind of blindly it
[02:27:18] experts but does so kind of blindly it just kind of like does its best guess
[02:27:21] just kind of like does its best guess uh kind of just like trying to mimic
[02:27:23] uh kind of just like trying to mimic statistically the expert behavior and so
[02:27:25] statistically the expert behavior and so that's what you get when you look at all
[02:27:26] that's what you get when you look at all the work Solutions and then finally in
[02:27:29] the work Solutions and then finally in the last stage we do all the practice
[02:27:31] the last stage we do all the practice problems in the RL stage across all the
[02:27:33] problems in the RL stage across all the textbooks we only do the practice
[02:27:35] textbooks we only do the practice problems and that's how we get the RL
[02:27:38] problems and that's how we get the RL model so on a high level the way we
[02:27:40] model so on a high level the way we train llms is very much equivalent uh to
[02:27:43] train llms is very much equivalent uh to the process that we train uh that we use
[02:27:45] the process that we train uh that we use for training of children the next point
[02:27:48] for training of children the next point I would like to make is that actually
[02:27:49] I would like to make is that actually these first two stat ages pre-training
[02:27:51] these first two stat ages pre-training and surprise fine-tuning they've been
[02:27:53] and surprise fine-tuning they've been around for years and they are very
[02:27:54] around for years and they are very standard and everyone does them all the
[02:27:55] standard and everyone does them all the different llm providers it is this last
[02:27:58] different llm providers it is this last stage the RL training that is a lot more
[02:28:01] stage the RL training that is a lot more early in its process of development and
[02:28:03] early in its process of development and is not standard yet in the field and so
[02:28:07] is not standard yet in the field and so um this stage is a lot more kind of
[02:28:09] um this stage is a lot more kind of early and nent and the reason for that
[02:28:12] early and nent and the reason for that is because I actually skipped over a ton
[02:28:13] is because I actually skipped over a ton of little details here in this process
[02:28:15] of little details here in this process the high level idea is very simple it's
[02:28:17] the high level idea is very simple it's trial and there learning but there's a
[02:28:19] trial and there learning but there's a ton of details and little math
[02:28:20] ton of details and little math mathematical kind of like nuances to
[02:28:22] mathematical kind of like nuances to exactly how you pick the solutions that
[02:28:23] exactly how you pick the solutions that are the best and how much you train on
[02:28:25] are the best and how much you train on them and what is the prompt distribution
[02:28:27] them and what is the prompt distribution and how to set up the training run such
[02:28:29] and how to set up the training run such that this actually works so there's a
[02:28:31] that this actually works so there's a lot of little details and knobs to the
[02:28:33] lot of little details and knobs to the core idea that is very very simple and
[02:28:35] core idea that is very very simple and so getting the details right here uh is
[02:28:38] so getting the details right here uh is not trivial and so a lot of companies
[02:28:40] not trivial and so a lot of companies like for example open and other LM
[02:28:42] like for example open and other LM providers have experimented internally
[02:28:44] providers have experimented internally with reinforcement learning fine tuning
[02:28:46] with reinforcement learning fine tuning for llms for a while but they've not
[02:28:49] for llms for a while but they've not talked about it publicly
[02:28:51] talked about it publicly um it's all kind of done inside the
[02:28:52] um it's all kind of done inside the company and so that's why the paper from
[02:28:55] company and so that's why the paper from Deep seek that came out very very
[02:28:57] Deep seek that came out very very recently was such a big deal because
[02:28:59] recently was such a big deal because this is a paper from this company called
[02:29:01] this is a paper from this company called DC Kai in China and this paper really
[02:29:05] DC Kai in China and this paper really talked very publicly about reinforcement
[02:29:07] talked very publicly about reinforcement learning fine training for large
[02:29:08] learning fine training for large language models and how incredibly
[02:29:10] language models and how incredibly important it is for large language
[02:29:12] important it is for large language models and how it brings out a lot of
[02:29:15] models and how it brings out a lot of reasoning capabilities in the models
[02:29:16] reasoning capabilities in the models we'll go into this in a second so this
[02:29:18] we'll go into this in a second so this paper reinvigorated the public interest
[02:29:21] paper reinvigorated the public interest of using RL for llms and gave a lot of
[02:29:25] of using RL for llms and gave a lot of the um sort of n-r details that are
[02:29:28] the um sort of n-r details that are needed to reproduce their results and
[02:29:30] needed to reproduce their results and actually get the stage to work for large
[02:29:31] actually get the stage to work for large langage models so let me take you
[02:29:33] langage models so let me take you briefly through this uh deep seek R1
[02:29:35] briefly through this uh deep seek R1 paper and what happens when you actually
[02:29:37] paper and what happens when you actually correctly apply RL to language models
[02:29:39] correctly apply RL to language models and what that looks like and what that
[02:29:40] and what that looks like and what that gives you so the first thing I'll scroll
[02:29:41] gives you so the first thing I'll scroll to is this uh kind of figure two here
[02:29:44] to is this uh kind of figure two here where we are looking at the Improvement
[02:29:46] where we are looking at the Improvement in how the models are solving
[02:29:47] in how the models are solving mathematical problems so this is the
[02:29:49] mathematical problems so this is the accuracy of solving mathematical
[02:29:51] accuracy of solving mathematical problems on the a accuracy and then we
[02:29:54] problems on the a accuracy and then we can go to the web page and we can see
[02:29:55] can go to the web page and we can see the kinds of problems that are actually
[02:29:57] the kinds of problems that are actually in these um these the kinds of math
[02:29:59] in these um these the kinds of math problems that are being measured here so
[02:30:01] problems that are being measured here so these are simple math problems you can
[02:30:02] these are simple math problems you can um pause the video if you like but these
[02:30:05] um pause the video if you like but these are the kinds of problems that basically
[02:30:06] are the kinds of problems that basically the models are being asked to solve and
[02:30:08] the models are being asked to solve and you can see that in the beginning
[02:30:09] you can see that in the beginning they're not doing very well but then as
[02:30:11] they're not doing very well but then as you update the model with this many
[02:30:13] you update the model with this many thousands of steps their accuracy kind
[02:30:15] thousands of steps their accuracy kind of continues to climb so the models are
[02:30:17] of continues to climb so the models are improving and they're solving these
[02:30:19] improving and they're solving these problems with a higher accuracy
[02:30:21] problems with a higher accuracy as you do this trial and error on a
[02:30:23] as you do this trial and error on a large data set of these kinds of
[02:30:24] large data set of these kinds of problems and the models are discovering
[02:30:26] problems and the models are discovering how to solve math problems but even more
[02:30:29] how to solve math problems but even more incredible than the quantitative kind of
[02:30:32] incredible than the quantitative kind of results of solving these problems with a
[02:30:34] results of solving these problems with a higher accuracy is the qualitative means
[02:30:36] higher accuracy is the qualitative means by which the model achieves these
[02:30:37] by which the model achieves these results so when we scroll down uh one of
[02:30:40] results so when we scroll down uh one of the figures here that is kind of
[02:30:41] the figures here that is kind of interesting is that later on in the
[02:30:44] interesting is that later on in the optimization the model seems to be uh
[02:30:47] optimization the model seems to be uh using average length per response uh
[02:30:50] using average length per response uh goes up up so the model seems to be
[02:30:51] goes up up so the model seems to be using more tokens to get its higher
[02:30:55] using more tokens to get its higher accuracy results so it's learning to
[02:30:56] accuracy results so it's learning to create very very long Solutions why are
[02:30:59] create very very long Solutions why are these Solutions very long we can look at
[02:31:01] these Solutions very long we can look at them qualitatively here so basically
[02:31:03] them qualitatively here so basically what they discover is that the model
[02:31:05] what they discover is that the model solution get very very long partially
[02:31:07] solution get very very long partially because so here's a question and here's
[02:31:09] because so here's a question and here's kind of the answer from the model what
[02:31:11] kind of the answer from the model what the model learns to do um and this is an
[02:31:14] the model learns to do um and this is an immerging property of new optimization
[02:31:16] immerging property of new optimization it just discovers that this is good for
[02:31:18] it just discovers that this is good for problem solving is it starts to do stuff
[02:31:20] problem solving is it starts to do stuff like this wait wait wait that's Nota
[02:31:22] like this wait wait wait that's Nota moment I can flag here let's reevaluate
[02:31:24] moment I can flag here let's reevaluate this step by step to identify the
[02:31:25] this step by step to identify the correct sum can be so what is the model
[02:31:27] correct sum can be so what is the model doing here right the model is basically
[02:31:30] doing here right the model is basically re-evaluating steps it has learned that
[02:31:32] re-evaluating steps it has learned that it works better for accuracy to try out
[02:31:35] it works better for accuracy to try out lots of ideas try something from
[02:31:37] lots of ideas try something from different perspectives retrace reframe
[02:31:40] different perspectives retrace reframe backtrack is doing a lot of the things
[02:31:42] backtrack is doing a lot of the things that you and I are doing in the process
[02:31:43] that you and I are doing in the process of problem solving for mathematical
[02:31:45] of problem solving for mathematical questions but it's rediscovering what
[02:31:47] questions but it's rediscovering what happens in your head not what you put
[02:31:49] happens in your head not what you put down on the solution and there is no
[02:31:51] down on the solution and there is no human who can hardcode this stuff in the
[02:31:53] human who can hardcode this stuff in the ideal assistant response this is only
[02:31:55] ideal assistant response this is only something that can be discovered in the
[02:31:56] something that can be discovered in the process of reinforcement learning
[02:31:58] process of reinforcement learning because you wouldn't know what to put
[02:32:00] because you wouldn't know what to put here this just turns out to work for the
[02:32:02] here this just turns out to work for the model and it improves its accuracy in
[02:32:04] model and it improves its accuracy in problem solving so the model learns what
[02:32:07] problem solving so the model learns what we call these chains of thought in your
[02:32:09] we call these chains of thought in your head and it's an emergent property of
[02:32:11] head and it's an emergent property of the optim of the optimization and that's
[02:32:14] the optim of the optimization and that's what's bloating up the response length
[02:32:16] what's bloating up the response length but that's also what's increasing the
[02:32:18] but that's also what's increasing the accuracy of the problem problem solving
[02:32:21] accuracy of the problem problem solving so what's incredible here is basically
[02:32:22] so what's incredible here is basically the model is discovering ways to think
[02:32:25] the model is discovering ways to think it's learning what I like to call
[02:32:26] it's learning what I like to call cognitive strategies of how you
[02:32:28] cognitive strategies of how you manipulate a problem and how you
[02:32:30] manipulate a problem and how you approach it from different perspectives
[02:32:32] approach it from different perspectives how you pull in some analogies or do
[02:32:34] how you pull in some analogies or do different kinds of things like that and
[02:32:36] different kinds of things like that and how you kind of uh try out many
[02:32:37] how you kind of uh try out many different things over time uh check a
[02:32:39] different things over time uh check a result from different perspectives and
[02:32:41] result from different perspectives and how you kind of uh solve problems but
[02:32:43] how you kind of uh solve problems but here it's kind of discovered by the RL
[02:32:45] here it's kind of discovered by the RL so extremely incredible to see this
[02:32:47] so extremely incredible to see this emerge in the optimization without
[02:32:49] emerge in the optimization without having to hardcode it anywhere the only
[02:32:51] having to hardcode it anywhere the only thing we've given it are the correct
[02:32:52] thing we've given it are the correct answers and this comes out from trying
[02:32:55] answers and this comes out from trying to just solve them correctly which is
[02:32:57] to just solve them correctly which is incredible
[02:32:58] incredible um now let's go back to actually the
[02:33:01] um now let's go back to actually the problem that we've been working with and
[02:33:02] problem that we've been working with and let's take a look at what it would look
[02:33:04] let's take a look at what it would look like uh for uh for this kind of a model
[02:33:08] like uh for uh for this kind of a model what we call reasoning or thinking model
[02:33:10] what we call reasoning or thinking model to solve that problem okay so recall
[02:33:12] to solve that problem okay so recall that this is the problem we've been
[02:33:13] that this is the problem we've been working with and when I pasted it into
[02:33:15] working with and when I pasted it into chat GPT 40 I'm getting this kind of a
[02:33:18] chat GPT 40 I'm getting this kind of a response let's take a look at what
[02:33:19] response let's take a look at what happens when you give this same query to
[02:33:22] happens when you give this same query to what's called a reasoning or a thinking
[02:33:24] what's called a reasoning or a thinking model this is a model that was trained
[02:33:25] model this is a model that was trained with reinforcement learning so this
[02:33:28] with reinforcement learning so this model described in this paper DC car1 is
[02:33:31] model described in this paper DC car1 is available on chat. dec.com uh so this is
[02:33:34] available on chat. dec.com uh so this is kind of like the company uh that
[02:33:36] kind of like the company uh that developed is hosting it you have to make
[02:33:38] developed is hosting it you have to make sure that the Deep think button is
[02:33:39] sure that the Deep think button is turned on to get the R1 model as it's
[02:33:42] turned on to get the R1 model as it's called we can paste it here and run
[02:33:44] called we can paste it here and run it and so let's take a look at what
[02:33:47] it and so let's take a look at what happens now and what is the output of
[02:33:48] happens now and what is the output of the model okay so here's it says so this
[02:33:51] the model okay so here's it says so this is previously what we get using
[02:33:53] is previously what we get using basically what's an sft approach a
[02:33:55] basically what's an sft approach a supervised funing approach this is like
[02:33:57] supervised funing approach this is like mimicking an expert solution this is
[02:33:59] mimicking an expert solution this is what we get from the RL model okay let
[02:34:02] what we get from the RL model okay let me try to figure this out so Emily buys
[02:34:03] me try to figure this out so Emily buys three apples and two oranges each orange
[02:34:05] three apples and two oranges each orange cost $2 total is 13 I need to find out
[02:34:08] cost $2 total is 13 I need to find out blah blah blah so here you you um as
[02:34:11] blah blah blah so here you you um as you're reading this you can't escape
[02:34:14] you're reading this you can't escape thinking that this model is
[02:34:16] thinking that this model is thinking um is definitely pursuing the
[02:34:19] thinking um is definitely pursuing the solution solution it deres that it must
[02:34:22] solution solution it deres that it must cost $3 and then it says wait a second
[02:34:24] cost $3 and then it says wait a second let me check my math again to be sure
[02:34:25] let me check my math again to be sure and then it tries it from a slightly
[02:34:26] and then it tries it from a slightly different perspective and then it says
[02:34:29] different perspective and then it says yep all that checks out I think that's
[02:34:31] yep all that checks out I think that's the answer I don't see any mistakes let
[02:34:33] the answer I don't see any mistakes let me see if there's another way to
[02:34:34] me see if there's another way to approach the problem maybe setting up an
[02:34:36] approach the problem maybe setting up an equation let's let the cost of one apple
[02:34:39] equation let's let the cost of one apple be $8 then blah blah blah yep same
[02:34:42] be $8 then blah blah blah yep same answer so definitely each apple is $3
[02:34:45] answer so definitely each apple is $3 all right confident that that's correct
[02:34:47] all right confident that that's correct and then what it does once it sort of um
[02:34:50] and then what it does once it sort of um did the thinking process is it writes up
[02:34:52] did the thinking process is it writes up the nice solution for the human and so
[02:34:54] the nice solution for the human and so this is now considering so this is more
[02:34:56] this is now considering so this is more about the correctness aspect and this is
[02:34:59] about the correctness aspect and this is more about the presentation aspect where
[02:35:01] more about the presentation aspect where it kind of like writes it out nicely and
[02:35:03] it kind of like writes it out nicely and uh boxes in the correct answer at the
[02:35:05] uh boxes in the correct answer at the bottom and so what's incredible about
[02:35:07] bottom and so what's incredible about this is we get this like thinking
[02:35:08] this is we get this like thinking process of the model and this is what's
[02:35:11] process of the model and this is what's coming from the reinforcement learning
[02:35:12] coming from the reinforcement learning process this is what's bloating up the
[02:35:15] process this is what's bloating up the length of the token sequences they're
[02:35:17] length of the token sequences they're doing thinking and they're trying
[02:35:18] doing thinking and they're trying different ways this is what's giving you
[02:35:21] different ways this is what's giving you higher accuracy in problem
[02:35:22] higher accuracy in problem solving and this is where we are seeing
[02:35:25] solving and this is where we are seeing these aha moments and these different
[02:35:27] these aha moments and these different strategies and these um ideas for how
[02:35:30] strategies and these um ideas for how you can make sure that you're getting
[02:35:31] you can make sure that you're getting the correct
[02:35:32] the correct answer the last point I wanted to make
[02:35:34] answer the last point I wanted to make is some people are a little bit nervous
[02:35:36] is some people are a little bit nervous about putting you know very sensitive
[02:35:39] about putting you know very sensitive data into chat.com because this is a
[02:35:41] data into chat.com because this is a Chinese company so people don't um
[02:35:43] Chinese company so people don't um people are a little bit careful and Cy
[02:35:45] people are a little bit careful and Cy with that a little bit um deep seek R1
[02:35:48] with that a little bit um deep seek R1 is a model that was released by this
[02:35:50] is a model that was released by this company so this is an open source model
[02:35:52] company so this is an open source model or open weights model it is available
[02:35:55] or open weights model it is available for anyone to download and use you will
[02:35:57] for anyone to download and use you will not be able to like run it in its full
[02:36:00] not be able to like run it in its full um sort of the full model in full
[02:36:03] um sort of the full model in full Precision you won't run that on a
[02:36:04] Precision you won't run that on a MacBook but uh or like a local device
[02:36:07] MacBook but uh or like a local device because this is a fairly large model but
[02:36:09] because this is a fairly large model but many companies are hosting the full
[02:36:11] many companies are hosting the full largest model one of those companies
[02:36:13] largest model one of those companies that I like to use is called
[02:36:15] that I like to use is called together. so when you go to together.
[02:36:17] together. so when you go to together. you sign up and you go to playgrounds
[02:36:19] you sign up and you go to playgrounds you can can select here in the chat deep
[02:36:22] you can can select here in the chat deep seek R1 and there's many different kinds
[02:36:24] seek R1 and there's many different kinds of other models that you can select here
[02:36:25] of other models that you can select here these are all state-of-the-art models so
[02:36:27] these are all state-of-the-art models so this is kind of similar to the hugging
[02:36:28] this is kind of similar to the hugging face inference playground that we've
[02:36:30] face inference playground that we've been playing with so far but together. a
[02:36:32] been playing with so far but together. a will usually host all the
[02:36:33] will usually host all the state-of-the-art models so select DT
[02:36:36] state-of-the-art models so select DT car1 um you can try to ignore a lot of
[02:36:38] car1 um you can try to ignore a lot of these I think the default settings will
[02:36:40] these I think the default settings will often be okay and we can put in this and
[02:36:44] often be okay and we can put in this and because the model was released by Deep
[02:36:45] because the model was released by Deep seek what you're getting here should be
[02:36:48] seek what you're getting here should be basically equivalent to what you're
[02:36:49] basically equivalent to what you're getting here now because of the
[02:36:51] getting here now because of the randomness in the sampling we're going
[02:36:52] randomness in the sampling we're going to get something slightly different uh
[02:36:54] to get something slightly different uh but in principle this should be uh
[02:36:56] but in principle this should be uh identical in terms of the power of the
[02:36:57] identical in terms of the power of the model and you should be able to see the
[02:36:59] model and you should be able to see the same things quantitatively and
[02:37:00] same things quantitatively and qualitatively uh but uh this model is
[02:37:02] qualitatively uh but uh this model is coming from kind of a an American
[02:37:05] coming from kind of a an American company so that's deep seek and that's
[02:37:07] company so that's deep seek and that's the what's called a reasoning
[02:37:09] the what's called a reasoning model now when I go back to chat uh let
[02:37:12] model now when I go back to chat uh let me go to chat here okay so the models
[02:37:15] me go to chat here okay so the models that you're going to see in the drop
[02:37:16] that you're going to see in the drop down here some of them like 01 03 mini
[02:37:19] down here some of them like 01 03 mini O3 mini High Etc they are talking about
[02:37:21] O3 mini High Etc they are talking about uses Advanced reasoning now what this is
[02:37:24] uses Advanced reasoning now what this is referring to uses Advanced reasoning is
[02:37:26] referring to uses Advanced reasoning is it's referring to the fact that it was
[02:37:28] it's referring to the fact that it was trained by reinforcement learning with
[02:37:30] trained by reinforcement learning with techniques very similar to those of deep
[02:37:32] techniques very similar to those of deep C car1 per public statements of opening
[02:37:34] C car1 per public statements of opening ey employees uh so these are thinking
[02:37:38] ey employees uh so these are thinking models trained with RL and these models
[02:37:40] models trained with RL and these models like GPT 4 or GPT 4 40 mini that you're
[02:37:43] like GPT 4 or GPT 4 40 mini that you're getting in the free tier you should
[02:37:44] getting in the free tier you should think of them as mostly sft models
[02:37:46] think of them as mostly sft models supervised fine tuning models they don't
[02:37:48] supervised fine tuning models they don't actually do this like thinking as as you
[02:37:50] actually do this like thinking as as you see in the RL models and even though
[02:37:52] see in the RL models and even though there's a little bit of reinforcement
[02:37:53] there's a little bit of reinforcement learning involved with these models and
[02:37:55] learning involved with these models and I'll go that into that in a second these
[02:37:57] I'll go that into that in a second these are mostly sft models I think you should
[02:37:59] are mostly sft models I think you should think about it that way so in the same
[02:38:01] think about it that way so in the same way as what we saw here we can pick one
[02:38:03] way as what we saw here we can pick one of the thinking models like say 03 mini
[02:38:05] of the thinking models like say 03 mini high and these models by the way might
[02:38:07] high and these models by the way might not be available to you unless you pay a
[02:38:10] not be available to you unless you pay a Chachi PT subscription of either $20 per
[02:38:12] Chachi PT subscription of either $20 per month or $200 per month for some of the
[02:38:14] month or $200 per month for some of the top models so we can pick a thinking
[02:38:17] top models so we can pick a thinking model and run now what's going to happen
[02:38:20] model and run now what's going to happen here is it's going to say reasoning and
[02:38:22] here is it's going to say reasoning and it's going to start to do stuff like
[02:38:23] it's going to start to do stuff like this and um what we're seeing here is
[02:38:27] this and um what we're seeing here is not exactly the stuff we're seeing here
[02:38:29] not exactly the stuff we're seeing here so even though under the hood the model
[02:38:32] so even though under the hood the model produces these kinds of uh kind of
[02:38:34] produces these kinds of uh kind of chains of thought opening ey chooses to
[02:38:36] chains of thought opening ey chooses to not show the exact chains of thought in
[02:38:39] not show the exact chains of thought in the web interface it shows little
[02:38:41] the web interface it shows little summaries of that of those chains of
[02:38:43] summaries of that of those chains of thought and open kind of does this I
[02:38:45] thought and open kind of does this I think partly because uh they are worried
[02:38:47] think partly because uh they are worried about what's called the distillation
[02:38:48] about what's called the distillation risk that is that someone could come in
[02:38:50] risk that is that someone could come in and actually try to imitate those
[02:38:52] and actually try to imitate those reasoning traces and recover a lot of
[02:38:54] reasoning traces and recover a lot of the reasoning performance by just
[02:38:55] the reasoning performance by just imitating the reasoning uh chains of
[02:38:58] imitating the reasoning uh chains of thought and so they kind of hide them
[02:38:59] thought and so they kind of hide them and they only show little summaries of
[02:39:01] and they only show little summaries of them so you're not getting exactly what
[02:39:02] them so you're not getting exactly what you would get in deep seek as with
[02:39:04] you would get in deep seek as with respect to the reasoning itself and then
[02:39:07] respect to the reasoning itself and then they write up the
[02:39:09] they write up the solution so these are kind of like
[02:39:11] solution so these are kind of like equivalent even though we're not seeing
[02:39:12] equivalent even though we're not seeing the full under the hood details now in
[02:39:14] the full under the hood details now in terms of the performance uh these models
[02:39:17] terms of the performance uh these models and deep seek models are currently rly
[02:39:20] and deep seek models are currently rly on par I would say it's kind of hard to
[02:39:21] on par I would say it's kind of hard to tell because of the evaluations but if
[02:39:23] tell because of the evaluations but if you're paying $200 per month to open AI
[02:39:25] you're paying $200 per month to open AI some of these models I believe are
[02:39:26] some of these models I believe are currently they basically still look
[02:39:28] currently they basically still look better uh but deep seek R1 for now is
[02:39:31] better uh but deep seek R1 for now is still a very solid choice for a thinking
[02:39:33] still a very solid choice for a thinking model that would be available to you um
[02:39:36] model that would be available to you um sort of um either on this website or any
[02:39:39] sort of um either on this website or any other website because the model is open
[02:39:41] other website because the model is open weights you can just download it so
[02:39:44] weights you can just download it so that's thinking models so what is the
[02:39:46] that's thinking models so what is the summary so far well we've talked about
[02:39:48] summary so far well we've talked about reinforcement learning and the fact that
[02:39:51] reinforcement learning and the fact that thinking emerges in the process of the
[02:39:52] thinking emerges in the process of the optimization on when we basically run RL
[02:39:55] optimization on when we basically run RL on many math uh and kind of code
[02:39:57] on many math uh and kind of code problems that have verifiable Solutions
[02:40:00] problems that have verifiable Solutions so there's like an answer three
[02:40:02] so there's like an answer three Etc now these thinking models you can
[02:40:05] Etc now these thinking models you can access in for example deep seek or any
[02:40:07] access in for example deep seek or any inference provider like together. a and
[02:40:10] inference provider like together. a and choosing deep seek over there these
[02:40:13] choosing deep seek over there these thinking models are also available uh in
[02:40:15] thinking models are also available uh in chpt under any of the 01 or O3
[02:40:18] chpt under any of the 01 or O3 models but these GPT 4 R models Etc
[02:40:21] models but these GPT 4 R models Etc they're not thinking models you should
[02:40:22] they're not thinking models you should think of them as mostly sft models now
[02:40:25] think of them as mostly sft models now if you are um if you have a prompt that
[02:40:28] if you are um if you have a prompt that requires Advanced reasoning and so on
[02:40:30] requires Advanced reasoning and so on you should probably use some of the
[02:40:31] you should probably use some of the thinking models or at least try them out
[02:40:33] thinking models or at least try them out but empirically for a lot of my use when
[02:40:35] but empirically for a lot of my use when you're asking a simpler question there's
[02:40:37] you're asking a simpler question there's like a knowledge based question or
[02:40:38] like a knowledge based question or something like that this might be
[02:40:39] something like that this might be Overkill like there's no need to think
[02:40:40] Overkill like there's no need to think 30 seconds about some factual question
[02:40:43] 30 seconds about some factual question so for that I will uh sometimes default
[02:40:45] so for that I will uh sometimes default to just GPT 40 so empirically about 80
[02:40:47] to just GPT 40 so empirically about 80 90% of my use is just gp4
[02:40:50] 90% of my use is just gp4 and when I come across a very difficult
[02:40:51] and when I come across a very difficult problem like in math and code Etc I will
[02:40:53] problem like in math and code Etc I will reach for the thinking models but then I
[02:40:56] reach for the thinking models but then I have to wait a bit longer because
[02:40:57] have to wait a bit longer because they're thinking um so you can access
[02:41:00] they're thinking um so you can access these on chat on deep seek also I wanted
[02:41:03] these on chat on deep seek also I wanted to point out that um AI studio.
[02:41:06] to point out that um AI studio. go.com even though it looks really busy
[02:41:08] go.com even though it looks really busy really ugly because Google's just unable
[02:41:11] really ugly because Google's just unable to do this kind of stuff well it's like
[02:41:13] to do this kind of stuff well it's like what is happening but if you choose
[02:41:15] what is happening but if you choose model and you choose here Gemini 2.0
[02:41:18] model and you choose here Gemini 2.0 flash thinking experimental 01 21 if you
[02:41:20] flash thinking experimental 01 21 if you choose that one that's also a a kind of
[02:41:22] choose that one that's also a a kind of early experiment experimental of a
[02:41:25] early experiment experimental of a thinking model by Google so we can go
[02:41:27] thinking model by Google so we can go here and we can give it the same problem
[02:41:29] here and we can give it the same problem and click run and this is also a
[02:41:31] and click run and this is also a thinking problem a thinking model that
[02:41:34] thinking problem a thinking model that will also do something
[02:41:35] will also do something similar and comes out with the right
[02:41:37] similar and comes out with the right answer here so basically Gemini also
[02:41:40] answer here so basically Gemini also offers a thinking model anthropic
[02:41:42] offers a thinking model anthropic currently does not offer a thinking
[02:41:44] currently does not offer a thinking model but basically this is kind of like
[02:41:45] model but basically this is kind of like the frontier development of these llms I
[02:41:48] the frontier development of these llms I think RL is kind of like this new
[02:41:49] think RL is kind of like this new exciting stage but getting the details
[02:41:52] exciting stage but getting the details right is difficult and that's why all
[02:41:54] right is difficult and that's why all these models and thinking models are
[02:41:55] these models and thinking models are currently experimental as of 2025 very
[02:41:58] currently experimental as of 2025 very early 2025 um but this is kind of like
[02:42:01] early 2025 um but this is kind of like the frontier development of pushing the
[02:42:03] the frontier development of pushing the performance on these very difficult
[02:42:04] performance on these very difficult problems using reasoning that is
[02:42:06] problems using reasoning that is emerging in these optimizations one more
[02:42:08] emerging in these optimizations one more connection that I wanted to bring up is
[02:42:10] connection that I wanted to bring up is that the discovery that reinforcement
[02:42:12] that the discovery that reinforcement learning is extremely powerful way of
[02:42:14] learning is extremely powerful way of learning is not new to the field of AI
[02:42:18] learning is not new to the field of AI and one place what we've already seen
[02:42:19] and one place what we've already seen this demonstrated is in the game of Go
[02:42:22] this demonstrated is in the game of Go and famously Deep Mind developed the
[02:42:24] and famously Deep Mind developed the system alphago and you can watch a movie
[02:42:26] system alphago and you can watch a movie about it um where the system is learning
[02:42:30] about it um where the system is learning to play the game of go against top human
[02:42:32] to play the game of go against top human players and um when we go to the paper
[02:42:36] players and um when we go to the paper underlying alphago so in this paper when
[02:42:40] underlying alphago so in this paper when we scroll
[02:42:41] we scroll down we actually find a really
[02:42:43] down we actually find a really interesting
[02:42:44] interesting plot um that I think uh is kind of
[02:42:47] plot um that I think uh is kind of familiar uh to us and we're kind of like
[02:42:49] familiar uh to us and we're kind of like we discovering in the more open domain
[02:42:52] we discovering in the more open domain of arbitrary problem solving instead of
[02:42:54] of arbitrary problem solving instead of on the closed specific domain of the
[02:42:55] on the closed specific domain of the game of Go but basically what they saw
[02:42:58] game of Go but basically what they saw and we're going to see this in llms as
[02:42:59] and we're going to see this in llms as well as this becomes more mature is this
[02:43:03] well as this becomes more mature is this is the ELO rating of playing game of Go
[02:43:05] is the ELO rating of playing game of Go and this is leas dull an extremely
[02:43:07] and this is leas dull an extremely strong human player and here what they
[02:43:10] strong human player and here what they are comparing is the strength of a model
[02:43:12] are comparing is the strength of a model learned trained by supervised learning
[02:43:14] learned trained by supervised learning and a model trained by reinforcement
[02:43:16] and a model trained by reinforcement learning so the supervised learning
[02:43:18] learning so the supervised learning model is imitating human expert players
[02:43:21] model is imitating human expert players so if you just get a huge amount of
[02:43:22] so if you just get a huge amount of games played by expert players in the
[02:43:24] games played by expert players in the game of Go and you try to imitate them
[02:43:26] game of Go and you try to imitate them you are going to get better but then you
[02:43:29] you are going to get better but then you top out and you never quite get better
[02:43:32] top out and you never quite get better than some of the top top top players of
[02:43:34] than some of the top top top players of in the game of Go like LEL so you're
[02:43:36] in the game of Go like LEL so you're never going to reach there because
[02:43:38] never going to reach there because you're just imitating human players you
[02:43:39] you're just imitating human players you can't fundamentally go beyond a human
[02:43:41] can't fundamentally go beyond a human player if you're just imitating human
[02:43:43] player if you're just imitating human players but in a process of
[02:43:44] players but in a process of reinforcement learning is significantly
[02:43:46] reinforcement learning is significantly more powerful in reinforcement learning
[02:43:48] more powerful in reinforcement learning for a game of Go it means that the
[02:43:50] for a game of Go it means that the system is playing moves that empirically
[02:43:53] system is playing moves that empirically and statistically lead to win to winning
[02:43:56] and statistically lead to win to winning the game and so alphago is a system
[02:44:00] the game and so alphago is a system where it kind of plays against it itself
[02:44:02] where it kind of plays against it itself and it's using reinforcement learning to
[02:44:04] and it's using reinforcement learning to create
[02:44:05] create rollouts so it's the exact same diagram
[02:44:07] rollouts so it's the exact same diagram here but there's no prompt it's just uh
[02:44:10] here but there's no prompt it's just uh because there's no prompt it's just a
[02:44:11] because there's no prompt it's just a fixed game of Go but it's trying out
[02:44:14] fixed game of Go but it's trying out lots of solutions it's trying out lots
[02:44:15] lots of solutions it's trying out lots of plays and then the games that lead to
[02:44:18] of plays and then the games that lead to a win instead of a specific answer are
[02:44:21] a win instead of a specific answer are reinforced they're they're made stronger
[02:44:24] reinforced they're they're made stronger and so um the system is learning
[02:44:27] and so um the system is learning basically the sequences of actions that
[02:44:28] basically the sequences of actions that empirically and statistically lead to
[02:44:30] empirically and statistically lead to winning the game and reinforcement
[02:44:33] winning the game and reinforcement learning is not going to be constrained
[02:44:34] learning is not going to be constrained by human performance and reinforcement
[02:44:36] by human performance and reinforcement learning can do significantly better and
[02:44:38] learning can do significantly better and overcome even the top players like Lisa
[02:44:41] overcome even the top players like Lisa Dole and so uh probably they could have
[02:44:45] Dole and so uh probably they could have run this longer and they just chose to
[02:44:46] run this longer and they just chose to crop it at some point because this costs
[02:44:48] crop it at some point because this costs money but this is very powerful
[02:44:49] money but this is very powerful demonstration of reinforcement learning
[02:44:51] demonstration of reinforcement learning and we're only starting to kind of see
[02:44:53] and we're only starting to kind of see hints of this diagram in larger language
[02:44:56] hints of this diagram in larger language models for reasoning problems so we're
[02:44:59] models for reasoning problems so we're not going to get too far by just
[02:45:00] not going to get too far by just imitating experts we need to go beyond
[02:45:02] imitating experts we need to go beyond that set up these like little game
[02:45:04] that set up these like little game environments and get let let the system
[02:45:07] environments and get let let the system discover reasoning traces or like ways
[02:45:10] discover reasoning traces or like ways of solving problems uh that are unique
[02:45:14] of solving problems uh that are unique and that uh just basically work
[02:45:16] and that uh just basically work well now on this aspect of uniqueness
[02:45:19] well now on this aspect of uniqueness notice that when you're doing
[02:45:20] notice that when you're doing reinforcement learning nothing prevents
[02:45:22] reinforcement learning nothing prevents you from veering off the distribution of
[02:45:25] you from veering off the distribution of how humans are playing the game and so
[02:45:27] how humans are playing the game and so when we go back to uh this alphao search
[02:45:29] when we go back to uh this alphao search here one of the suggested modifications
[02:45:32] here one of the suggested modifications is called move 37 and move 37 in alphao
[02:45:35] is called move 37 and move 37 in alphao is referring to a specific point in time
[02:45:37] is referring to a specific point in time where alphago basically played a move
[02:45:41] where alphago basically played a move that uh no human expert would play uh so
[02:45:44] that uh no human expert would play uh so the probability of this move uh to be
[02:45:46] the probability of this move uh to be played by a human player was evaluated
[02:45:48] played by a human player was evaluated to be about 1 in 10th ,000 so it's a
[02:45:50] to be about 1 in 10th ,000 so it's a very rare move but in retrospect it was
[02:45:52] very rare move but in retrospect it was a brilliant move so alphago in the
[02:45:54] a brilliant move so alphago in the process of reinforcement learning
[02:45:56] process of reinforcement learning discovered kind of like a strategy of
[02:45:58] discovered kind of like a strategy of playing that was unknown to humans and
[02:46:00] playing that was unknown to humans and but is in retrospect uh brilliant I
[02:46:02] but is in retrospect uh brilliant I recommend this YouTube video um leis do
[02:46:05] recommend this YouTube video um leis do versus alphao move 37 reactions and
[02:46:07] versus alphao move 37 reactions and Analysis and this is kind of what it
[02:46:09] Analysis and this is kind of what it looked like when alphao played this
[02:46:11] looked like when alphao played this move
[02:46:14] move value that's a very that's a very
[02:46:17] value that's a very that's a very surprising move I thought I thought it
[02:46:20] surprising move I thought I thought it was I thought it was a
[02:46:22] was I thought it was a mistake when I see this move anyway so
[02:46:25] mistake when I see this move anyway so basically people are kind of freaking
[02:46:26] basically people are kind of freaking out because it's a it's a move that a
[02:46:29] out because it's a it's a move that a human would not play that alphago played
[02:46:31] human would not play that alphago played because in its training uh this move
[02:46:34] because in its training uh this move seemed to be a good idea it just happens
[02:46:36] seemed to be a good idea it just happens not to be a kind of thing that a humans
[02:46:37] not to be a kind of thing that a humans would would do and so that is again the
[02:46:39] would would do and so that is again the power of reinforcement learning and in
[02:46:41] power of reinforcement learning and in principle we can actually see the
[02:46:43] principle we can actually see the equivalence of that if we continue
[02:46:44] equivalence of that if we continue scaling this Paradigm in language models
[02:46:47] scaling this Paradigm in language models and what that looks like is kind of
[02:46:48] and what that looks like is kind of unknown so so um what does it mean to
[02:46:51] unknown so so um what does it mean to solve problems in such a way that uh
[02:46:55] solve problems in such a way that uh even humans would not be able to get how
[02:46:57] even humans would not be able to get how can you be better at reasoning or
[02:46:58] can you be better at reasoning or thinking than humans how can you go
[02:47:00] thinking than humans how can you go beyond just uh a thinking human like
[02:47:04] beyond just uh a thinking human like maybe it means discovering analogies
[02:47:06] maybe it means discovering analogies that humans would not be able to uh
[02:47:08] that humans would not be able to uh create or maybe it's like a new thinking
[02:47:10] create or maybe it's like a new thinking strategy it's kind of hard to think
[02:47:11] strategy it's kind of hard to think through uh maybe it's a holy new
[02:47:14] through uh maybe it's a holy new language that actually is not even
[02:47:16] language that actually is not even English maybe it discovers its own
[02:47:18] English maybe it discovers its own language that is a lot better at
[02:47:20] language that is a lot better at thinking um because the model is
[02:47:23] thinking um because the model is unconstrained to even like stick with
[02:47:24] unconstrained to even like stick with English uh so maybe it takes a different
[02:47:27] English uh so maybe it takes a different language to think in or it discovers its
[02:47:29] language to think in or it discovers its own language so in principle the
[02:47:31] own language so in principle the behavior of the system is a lot less
[02:47:33] behavior of the system is a lot less defined it is open to do whatever works
[02:47:37] defined it is open to do whatever works and it is open to also slowly Drift from
[02:47:40] and it is open to also slowly Drift from the distribution of its training data
[02:47:41] the distribution of its training data which is English but all of that can
[02:47:44] which is English but all of that can only be done if we have a very large
[02:47:46] only be done if we have a very large diverse set of problems in which the
[02:47:48] diverse set of problems in which the these strategy can be refined and
[02:47:50] these strategy can be refined and perfected and so that is a lot of the
[02:47:52] perfected and so that is a lot of the frontier LM research that's going on
[02:47:54] frontier LM research that's going on right now is trying to kind of create
[02:47:56] right now is trying to kind of create those kinds of prompt distributions that
[02:47:57] those kinds of prompt distributions that are large and diverse these are all kind
[02:47:59] are large and diverse these are all kind of like game environments in which the
[02:48:01] of like game environments in which the llms can practice their thinking and uh
[02:48:04] llms can practice their thinking and uh it's kind of like writing you know these
[02:48:06] it's kind of like writing you know these practice problems we have to create
[02:48:08] practice problems we have to create practice problems for all of domains of
[02:48:10] practice problems for all of domains of knowledge and if we have practice
[02:48:12] knowledge and if we have practice problems and tons of them the models
[02:48:14] problems and tons of them the models will be able to reinforcement learning
[02:48:16] will be able to reinforcement learning reinforcement learn on them and kind of
[02:48:18] reinforcement learn on them and kind of uh create these kinds of uh diagrams but
[02:48:22] uh create these kinds of uh diagrams but in the domain of open thinking instead
[02:48:24] in the domain of open thinking instead of a closed domain like game of Go
[02:48:27] of a closed domain like game of Go there's one more section within
[02:48:28] there's one more section within reinforcement learning that I wanted to
[02:48:29] reinforcement learning that I wanted to cover and that is that of learning in
[02:48:32] cover and that is that of learning in unverifiable domains so so far all of
[02:48:35] unverifiable domains so so far all of the problems that we've looked at are in
[02:48:37] the problems that we've looked at are in what's called verifiable domains that is
[02:48:39] what's called verifiable domains that is any candidate solution we can score very
[02:48:41] any candidate solution we can score very easily against a concrete answer so for
[02:48:44] easily against a concrete answer so for example answer is three and we can very
[02:48:46] example answer is three and we can very easily score these Solutions against the
[02:48:48] easily score these Solutions against the answer of three
[02:48:50] answer of three either we require the models to like box
[02:48:52] either we require the models to like box in their answers and then we just check
[02:48:54] in their answers and then we just check for equality of whatever is in the box
[02:48:56] for equality of whatever is in the box with the answer or you can also use uh
[02:48:58] with the answer or you can also use uh kind of what's called an llm judge so
[02:49:01] kind of what's called an llm judge so the llm judge looks at a solution and it
[02:49:03] the llm judge looks at a solution and it gets the answer and just basically
[02:49:05] gets the answer and just basically scores the solution for whether it's
[02:49:07] scores the solution for whether it's consistent with the answer or not and
[02:49:08] consistent with the answer or not and llms uh empirically are good enough at
[02:49:11] llms uh empirically are good enough at the current capability that they can do
[02:49:12] the current capability that they can do this fairly reliably so we can apply
[02:49:14] this fairly reliably so we can apply those kinds of techniques as well in any
[02:49:16] those kinds of techniques as well in any case we have a concrete answer and we're
[02:49:18] case we have a concrete answer and we're just checking Solutions again against it
[02:49:19] just checking Solutions again against it and we can do this automatically with no
[02:49:21] and we can do this automatically with no kind of humans in the loop the problem
[02:49:24] kind of humans in the loop the problem is that we can't apply the strategy in
[02:49:25] is that we can't apply the strategy in what's called unverifiable domains so
[02:49:28] what's called unverifiable domains so usually these are for example creative
[02:49:29] usually these are for example creative writing tasks like write a joke about
[02:49:31] writing tasks like write a joke about Pelicans or write a poem or summarize a
[02:49:33] Pelicans or write a poem or summarize a paragraph or something like that in
[02:49:35] paragraph or something like that in these kinds of domains it becomes harder
[02:49:37] these kinds of domains it becomes harder to score our different solutions to this
[02:49:40] to score our different solutions to this problem so for example writing a joke
[02:49:42] problem so for example writing a joke about Pelicans we can generate lots of
[02:49:43] about Pelicans we can generate lots of different uh jokes of course that's fine
[02:49:46] different uh jokes of course that's fine for example we can go to chbt and we can
[02:49:48] for example we can go to chbt and we can get it to uh generate a joke about
[02:49:51] get it to uh generate a joke about Pelicans uh so much stuff in their beaks
[02:49:54] Pelicans uh so much stuff in their beaks because they don't bellan in
[02:49:57] because they don't bellan in backpacks what
[02:49:59] backpacks what okay we can uh we can try something else
[02:50:03] okay we can uh we can try something else why don't Pelicans ever pay for their
[02:50:04] why don't Pelicans ever pay for their drinks because they always B it to
[02:50:06] drinks because they always B it to someone else haha okay so these models
[02:50:10] someone else haha okay so these models are not obviously not very good at humor
[02:50:12] are not obviously not very good at humor actually I think it's pretty fascinating
[02:50:13] actually I think it's pretty fascinating because I think humor is secretly very
[02:50:15] because I think humor is secretly very difficult and the model have the
[02:50:17] difficult and the model have the capability I think anyway in any case
[02:50:20] capability I think anyway in any case you could imagine creating lots of jokes
[02:50:23] you could imagine creating lots of jokes the problem that we are facing is how do
[02:50:25] the problem that we are facing is how do we score them now in principle we could
[02:50:28] we score them now in principle we could of course get a human to look at all
[02:50:29] of course get a human to look at all these jokes just like I did right now
[02:50:32] these jokes just like I did right now the problem with that is if you are
[02:50:33] the problem with that is if you are doing reinforcement learning you're
[02:50:35] doing reinforcement learning you're going to be doing many thousands of
[02:50:36] going to be doing many thousands of updates and for each update you want to
[02:50:38] updates and for each update you want to be looking at say thousands of prompts
[02:50:41] be looking at say thousands of prompts and for each prompt you want to be
[02:50:42] and for each prompt you want to be potentially looking at looking at
[02:50:43] potentially looking at looking at hundred or thousands of different kinds
[02:50:45] hundred or thousands of different kinds of generations and so there's just like
[02:50:48] of generations and so there's just like way too many of these to look at and so
[02:50:51] way too many of these to look at and so um in principle you could have a human
[02:50:52] um in principle you could have a human inspect all of them and score them and
[02:50:54] inspect all of them and score them and decide that okay maybe this one is funny
[02:50:56] decide that okay maybe this one is funny and uh maybe this one is funny and this
[02:50:58] and uh maybe this one is funny and this one is funny and we could train on them
[02:51:01] one is funny and we could train on them to get the model to become slightly
[02:51:03] to get the model to become slightly better at jokes um in the context of
[02:51:05] better at jokes um in the context of pelicans at least um the problem is that
[02:51:09] pelicans at least um the problem is that it's just like way too much human time
[02:51:11] it's just like way too much human time this is an unscalable strategy we need
[02:51:12] this is an unscalable strategy we need some kind of an automatic strategy for
[02:51:14] some kind of an automatic strategy for doing this and one sort of solution to
[02:51:17] doing this and one sort of solution to this was proposed in this paper
[02:51:19] this was proposed in this paper uh that introduced what's called
[02:51:20] uh that introduced what's called reinforcement learning from Human
[02:51:22] reinforcement learning from Human feedback and so this was a paper from
[02:51:24] feedback and so this was a paper from open at the time and many of these
[02:51:25] open at the time and many of these people are now um co-founders in
[02:51:28] people are now um co-founders in anthropic um and this kind of proposed a
[02:51:31] anthropic um and this kind of proposed a approach for uh basically doing
[02:51:33] approach for uh basically doing reinforcement learning in unverifiable
[02:51:35] reinforcement learning in unverifiable domains so let's take a look at how that
[02:51:37] domains so let's take a look at how that works so this is the cartoon diagram of
[02:51:40] works so this is the cartoon diagram of the core ideas involved so as I
[02:51:42] the core ideas involved so as I mentioned the native approach is if we
[02:51:44] mentioned the native approach is if we just set Infinity human time we could
[02:51:46] just set Infinity human time we could just run RL in these domains just fine
[02:51:49] just run RL in these domains just fine so for example we can run RL as usual if
[02:51:52] so for example we can run RL as usual if I have Infinity humans I would I just
[02:51:54] I have Infinity humans I would I just want to do and these are just cartoon
[02:51:55] want to do and these are just cartoon numbers I want to do 1,000 updates where
[02:51:58] numbers I want to do 1,000 updates where each update will be on 1,000 prompts and
[02:52:01] each update will be on 1,000 prompts and in for each prompt we're going to have
[02:52:02] in for each prompt we're going to have 1,000 roll outs that we're scoring so we
[02:52:06] 1,000 roll outs that we're scoring so we can run RL with this kind of a setup the
[02:52:09] can run RL with this kind of a setup the problem is in the process of doing this
[02:52:11] problem is in the process of doing this I will need to run one I will need to
[02:52:13] I will need to run one I will need to ask a human to evaluate a joke a total
[02:52:15] ask a human to evaluate a joke a total of 1 billion times and so that's a lot
[02:52:18] of 1 billion times and so that's a lot of people looking at really terrible
[02:52:20] of people looking at really terrible jokes so we don't want to do that so
[02:52:22] jokes so we don't want to do that so instead we want to take the arlef
[02:52:24] instead we want to take the arlef approach so um in our Rel of approach we
[02:52:28] approach so um in our Rel of approach we are kind of like the the core trick is
[02:52:30] are kind of like the the core trick is that of indirection so we're going to
[02:52:33] that of indirection so we're going to involve humans just a little bit and the
[02:52:35] involve humans just a little bit and the way we cheat is that we basically train
[02:52:37] way we cheat is that we basically train a whole separate neural network that we
[02:52:39] a whole separate neural network that we call a reward model and this neural
[02:52:42] call a reward model and this neural network will kind of like imitate human
[02:52:44] network will kind of like imitate human scores so we're going to ask humans to
[02:52:47] scores so we're going to ask humans to score um roll
[02:52:49] score um roll we're going to then imitate human scores
[02:52:52] we're going to then imitate human scores using a neural network and this neural
[02:52:54] using a neural network and this neural network will become a kind of simulator
[02:52:56] network will become a kind of simulator of human
[02:52:57] of human preferences and now that we have a
[02:52:58] preferences and now that we have a neural network simulator we can do RL
[02:53:01] neural network simulator we can do RL against it so instead of asking a real
[02:53:03] against it so instead of asking a real human we're asking a simulated human for
[02:53:06] human we're asking a simulated human for their score of a joke as an example and
[02:53:10] their score of a joke as an example and so once we have a simulator we're often
[02:53:12] so once we have a simulator we're often racist because we can query it as many
[02:53:14] racist because we can query it as many times as we want to and it's all whole
[02:53:16] times as we want to and it's all whole automatic process and we can now do
[02:53:18] automatic process and we can now do reinforcement learning with respect to
[02:53:19] reinforcement learning with respect to the simulator and the simulator as you
[02:53:21] the simulator and the simulator as you might expect is not going to be a
[02:53:22] might expect is not going to be a perfect human but if it's at least
[02:53:24] perfect human but if it's at least statistically similar to human judgment
[02:53:27] statistically similar to human judgment then you might expect that this will do
[02:53:28] then you might expect that this will do something and in practice indeed uh it
[02:53:30] something and in practice indeed uh it does so once we have a simulator we can
[02:53:33] does so once we have a simulator we can do RL and everything works great so let
[02:53:35] do RL and everything works great so let me show you a cartoon diagram a little
[02:53:37] me show you a cartoon diagram a little bit of what this process looks like
[02:53:39] bit of what this process looks like although the details are not 100 like
[02:53:41] although the details are not 100 like super important it's just a core idea of
[02:53:43] super important it's just a core idea of how this works so here I have a cartoon
[02:53:44] how this works so here I have a cartoon diagram of a hypothetical example of
[02:53:46] diagram of a hypothetical example of what training the reward model would
[02:53:48] what training the reward model would look like so we have a prompt like write
[02:53:50] look like so we have a prompt like write a joke about picans and then here we
[02:53:52] a joke about picans and then here we have five separate roll outs so these
[02:53:54] have five separate roll outs so these are all five different jokes just like
[02:53:57] are all five different jokes just like this one now the first thing we're going
[02:53:59] this one now the first thing we're going to do is we are going to ask a human to
[02:54:03] to do is we are going to ask a human to uh order these jokes from the best to
[02:54:05] uh order these jokes from the best to worst so this is uh so here this human
[02:54:08] worst so this is uh so here this human thought that this joke is the best the
[02:54:11] thought that this joke is the best the funniest so number one joke this is
[02:54:14] funniest so number one joke this is number two joke number three joke four
[02:54:17] number two joke number three joke four and five so this is the worst joke
[02:54:19] and five so this is the worst joke we're asking humans to order instead of
[02:54:21] we're asking humans to order instead of give scores directly because it's a bit
[02:54:23] give scores directly because it's a bit of an easier task it's easier for a
[02:54:24] of an easier task it's easier for a human to give an ordering than to give
[02:54:26] human to give an ordering than to give precise scores now that is now the
[02:54:29] precise scores now that is now the supervision for the model so the human
[02:54:31] supervision for the model so the human has ordered them and that is kind of
[02:54:33] has ordered them and that is kind of like their contribution to the training
[02:54:34] like their contribution to the training process but now separately what we're
[02:54:36] process but now separately what we're going to do is we're going to ask a
[02:54:38] going to do is we're going to ask a reward model uh about its scoring of
[02:54:41] reward model uh about its scoring of these jokes now the reward model is a
[02:54:43] these jokes now the reward model is a whole separate neural network completely
[02:54:45] whole separate neural network completely separate neural net um and it's also
[02:54:48] separate neural net um and it's also probably a transform
[02:54:49] probably a transform uh but it's not a language model in the
[02:54:51] uh but it's not a language model in the sense that it generates diverse language
[02:54:53] sense that it generates diverse language Etc it's just a scoring model so the
[02:54:57] Etc it's just a scoring model so the reward model will take as an input The
[02:54:59] reward model will take as an input The Prompt number one and number two a
[02:55:02] Prompt number one and number two a candidate joke so um those are the two
[02:55:05] candidate joke so um those are the two inputs that go into the reward model so
[02:55:07] inputs that go into the reward model so here for example the reward model would
[02:55:09] here for example the reward model would be taken this prompt and this joke now
[02:55:12] be taken this prompt and this joke now the output of a reward model is a single
[02:55:14] the output of a reward model is a single number and this number is thought of as
[02:55:16] number and this number is thought of as a score and it can range for example
[02:55:18] a score and it can range for example from Z to one so zero would be the worst
[02:55:21] from Z to one so zero would be the worst score and one would be the best score so
[02:55:24] score and one would be the best score so here are some examples of what a
[02:55:25] here are some examples of what a hypothetical reward model at some stage
[02:55:27] hypothetical reward model at some stage in the training process would give uh s
[02:55:29] in the training process would give uh s scoring to these jokes so 0.1 is a very
[02:55:33] scoring to these jokes so 0.1 is a very low score 08 is a really high score and
[02:55:36] low score 08 is a really high score and so on and so now um we compare the
[02:55:41] so on and so now um we compare the scores given by the reward model with uh
[02:55:43] scores given by the reward model with uh the ordering given by the human and
[02:55:45] the ordering given by the human and there's a precise mathematical way to
[02:55:47] there's a precise mathematical way to actually calculate this uh basically set
[02:55:49] actually calculate this uh basically set up a loss function and calculate a kind
[02:55:52] up a loss function and calculate a kind of like a correspondence here and uh
[02:55:54] of like a correspondence here and uh update a model based on it but I just
[02:55:56] update a model based on it but I just want to give you the intuition which is
[02:55:58] want to give you the intuition which is that as an example here for this second
[02:56:01] that as an example here for this second joke the the human thought that it was
[02:56:02] joke the the human thought that it was the funniest and the model kind of
[02:56:04] the funniest and the model kind of agreed right 08 is a relatively high
[02:56:06] agreed right 08 is a relatively high score but this score should have been
[02:56:07] score but this score should have been even higher right so after an update we
[02:56:11] even higher right so after an update we would expect that maybe this score
[02:56:12] would expect that maybe this score should have been will actually grow
[02:56:14] should have been will actually grow after an update of the network to be
[02:56:15] after an update of the network to be like say 081 or
[02:56:17] like say 081 or something um for this one here they
[02:56:19] something um for this one here they actually are in a massive disagreement
[02:56:21] actually are in a massive disagreement because the human thought that this was
[02:56:22] because the human thought that this was number two but here the the score is
[02:56:25] number two but here the the score is only 0.1 and so this score needs to be
[02:56:28] only 0.1 and so this score needs to be much higher so after an update on top of
[02:56:31] much higher so after an update on top of this um kind of a supervision this might
[02:56:34] this um kind of a supervision this might grow a lot more like maybe it's 0.15 or
[02:56:35] grow a lot more like maybe it's 0.15 or something like
[02:56:36] something like that um and then here the human thought
[02:56:40] that um and then here the human thought that this one was the worst joke but
[02:56:42] that this one was the worst joke but here the model actually gave it a fairly
[02:56:44] here the model actually gave it a fairly High number so you might expect that
[02:56:45] High number so you might expect that after the update uh this would come down
[02:56:48] after the update uh this would come down to maybe 3 3.5 or something like that so
[02:56:50] to maybe 3 3.5 or something like that so basically we're doing what we did before
[02:56:52] basically we're doing what we did before we're slightly nudging the predictions
[02:56:55] we're slightly nudging the predictions from the models using a neural network
[02:56:57] from the models using a neural network training
[02:56:58] training process and we're trying to make the
[02:57:01] process and we're trying to make the reward model scores be consistent with
[02:57:03] reward model scores be consistent with human
[02:57:04] human ordering and so um as we update the
[02:57:07] ordering and so um as we update the reward model on human data it becomes
[02:57:10] reward model on human data it becomes better and better simulator of the
[02:57:12] better and better simulator of the scores and orders uh that humans provide
[02:57:15] scores and orders uh that humans provide and then becomes kind of like the the
[02:57:17] and then becomes kind of like the the neural the simulator of human
[02:57:19] neural the simulator of human preferences which we can then do RL
[02:57:21] preferences which we can then do RL against but critically we're not asking
[02:57:23] against but critically we're not asking humans one billion times to look at a
[02:57:25] humans one billion times to look at a joke we're maybe looking at th000
[02:57:27] joke we're maybe looking at th000 prompts and five roll outs each so maybe
[02:57:29] prompts and five roll outs each so maybe 5,000 jokes that humans have to look at
[02:57:31] 5,000 jokes that humans have to look at in total and they just give the ordering
[02:57:33] in total and they just give the ordering and then we're training the model to be
[02:57:34] and then we're training the model to be consistent with that ordering and I'm
[02:57:36] consistent with that ordering and I'm skipping over the mathematical details
[02:57:39] skipping over the mathematical details but I just want you to understand a high
[02:57:40] but I just want you to understand a high level idea that uh this reward model is
[02:57:43] level idea that uh this reward model is do is basically giving us this scour and
[02:57:45] do is basically giving us this scour and we have a way of training it to be
[02:57:47] we have a way of training it to be consistent with human orderings
[02:57:49] consistent with human orderings and that's how rhf works okay so that is
[02:57:51] and that's how rhf works okay so that is the rough idea we basically train
[02:57:53] the rough idea we basically train simulators of humans and RL with respect
[02:57:56] simulators of humans and RL with respect to those
[02:57:57] to those simulators now I want to talk about
[02:57:59] simulators now I want to talk about first the upside of reinforcement
[02:58:01] first the upside of reinforcement learning from Human
[02:58:04] learning from Human feedback the first thing is that this
[02:58:06] feedback the first thing is that this allows us to run reinforcement learning
[02:58:08] allows us to run reinforcement learning which we know is incredibly powerful
[02:58:09] which we know is incredibly powerful kind of set of techniques and it allows
[02:58:11] kind of set of techniques and it allows us to do it in arbitrary domains and
[02:58:13] us to do it in arbitrary domains and including the ones that are unverifiable
[02:58:16] including the ones that are unverifiable so things like summarization and poem
[02:58:18] so things like summarization and poem writing joke writing or any other
[02:58:19] writing joke writing or any other creative writing really uh in domains
[02:58:21] creative writing really uh in domains outside of math and code
[02:58:23] outside of math and code Etc now empirically what we see when we
[02:58:26] Etc now empirically what we see when we actually apply rhf is that this is a way
[02:58:28] actually apply rhf is that this is a way to improve the performance of the model
[02:58:31] to improve the performance of the model and uh I have a top answer for why that
[02:58:34] and uh I have a top answer for why that might be but I don't actually know that
[02:58:36] might be but I don't actually know that it is like super well established on
[02:58:38] it is like super well established on like why this is you can empirically
[02:58:40] like why this is you can empirically observe that when you do rhf correctly
[02:58:42] observe that when you do rhf correctly the models you get are just like a
[02:58:43] the models you get are just like a little bit better um but as to why is I
[02:58:46] little bit better um but as to why is I think like not as clear so here's my
[02:58:47] think like not as clear so here's my best guess my best guess is that this is
[02:58:50] best guess my best guess is that this is possibly mostly due to the discriminator
[02:58:52] possibly mostly due to the discriminator generator
[02:58:53] generator Gap what that means is that in many
[02:58:56] Gap what that means is that in many cases it is significantly easier to
[02:58:58] cases it is significantly easier to discriminate than to generate for humans
[02:59:01] discriminate than to generate for humans so in particular an example of this is
[02:59:05] so in particular an example of this is um in when we do supervised fine-tuning
[02:59:08] um in when we do supervised fine-tuning right
[02:59:09] right sft we're asking humans to generate the
[02:59:12] sft we're asking humans to generate the ideal assistant response and in many
[02:59:15] ideal assistant response and in many cases here um as I've shown it uh the
[02:59:19] cases here um as I've shown it uh the ideal response is very simple to write
[02:59:20] ideal response is very simple to write but in many cases might not be so for
[02:59:22] but in many cases might not be so for example in summarization or poem writing
[02:59:24] example in summarization or poem writing or joke writing like how are you as a
[02:59:26] or joke writing like how are you as a human assist as a human labeler um
[02:59:29] human assist as a human labeler um supposed to give the ideal response in
[02:59:31] supposed to give the ideal response in these cases it requires creative human
[02:59:33] these cases it requires creative human writing to do that and so rhf kind of
[02:59:36] writing to do that and so rhf kind of sidesteps this because we get um we get
[02:59:39] sidesteps this because we get um we get to ask people a significantly easier
[02:59:41] to ask people a significantly easier question as a data labelers they're not
[02:59:43] question as a data labelers they're not asked to write poems directly they're
[02:59:45] asked to write poems directly they're just given five poems from the model and
[02:59:47] just given five poems from the model and they're just asked to order them and so
[02:59:49] they're just asked to order them and so that's just a much easier task for a
[02:59:51] that's just a much easier task for a human labeler to do and so what I think
[02:59:54] human labeler to do and so what I think this allows you to do basically is it um
[02:59:57] this allows you to do basically is it um it kind of like allows a lot more higher
[03:00:00] it kind of like allows a lot more higher accuracy data because we're not asking
[03:00:02] accuracy data because we're not asking people to do the generation task which
[03:00:04] people to do the generation task which can be extremely difficult like we're
[03:00:06] can be extremely difficult like we're not asking them to do creative writing
[03:00:08] not asking them to do creative writing we're just trying to get them to
[03:00:09] we're just trying to get them to distinguish between creative writings
[03:00:11] distinguish between creative writings and uh find the ones that are best and
[03:00:14] and uh find the ones that are best and that is the signal that humans are
[03:00:16] that is the signal that humans are providing just the ordering and that is
[03:00:18] providing just the ordering and that is their input into the system and then the
[03:00:20] their input into the system and then the system in rhf just discovers the kinds
[03:00:23] system in rhf just discovers the kinds of responses that would be graded well
[03:00:26] of responses that would be graded well by humans and so that step of
[03:00:29] by humans and so that step of indirection allows the models to become
[03:00:31] indirection allows the models to become a bit better so that is the upside of
[03:00:34] a bit better so that is the upside of our LF it allows us to run RL it
[03:00:36] our LF it allows us to run RL it empirically results in better models and
[03:00:38] empirically results in better models and it allows uh people to contribute their
[03:00:40] it allows uh people to contribute their supervision uh even without having to do
[03:00:42] supervision uh even without having to do extremely difficult tasks um in the case
[03:00:45] extremely difficult tasks um in the case of writing ideal responses unfortunately
[03:00:48] of writing ideal responses unfortunately our HF also comes with significant
[03:00:50] our HF also comes with significant downsides and so um the main one is that
[03:00:54] downsides and so um the main one is that basically we are doing reinforcement
[03:00:56] basically we are doing reinforcement learning not with respect to humans and
[03:00:58] learning not with respect to humans and actual human judgment but with respect
[03:00:59] actual human judgment but with respect to a lossy simulation of humans right
[03:01:02] to a lossy simulation of humans right and this lossy simulation could be
[03:01:03] and this lossy simulation could be misleading because it's just a it's just
[03:01:05] misleading because it's just a it's just a simulation right it's just a language
[03:01:07] a simulation right it's just a language model that's kind of outputting scores
[03:01:09] model that's kind of outputting scores and it might not perfectly reflect the
[03:01:11] and it might not perfectly reflect the opinion of an actual human with an
[03:01:13] opinion of an actual human with an actual brain in all the possible
[03:01:15] actual brain in all the possible different cases so that's number one
[03:01:17] different cases so that's number one which is actually something even more
[03:01:19] which is actually something even more subtle and devious going on that uh
[03:01:21] subtle and devious going on that uh really
[03:01:22] really dramatically holds back our LF as a
[03:01:25] dramatically holds back our LF as a technique that we can really scale to
[03:01:28] technique that we can really scale to significantly um kind of Smart Systems
[03:01:31] significantly um kind of Smart Systems and that is that reinforcement learning
[03:01:33] and that is that reinforcement learning is extremely good at discovering a way
[03:01:35] is extremely good at discovering a way to game the model to game the simulation
[03:01:38] to game the model to game the simulation so this reward model that we're
[03:01:40] so this reward model that we're constructing here that gives the course
[03:01:43] constructing here that gives the course these models are Transformers these
[03:01:46] these models are Transformers these Transformers are massive neurals they
[03:01:48] Transformers are massive neurals they have billions of parameters and they
[03:01:50] have billions of parameters and they imitate humans but they do so in a kind
[03:01:52] imitate humans but they do so in a kind of like a simulation way now the problem
[03:01:54] of like a simulation way now the problem is that these are massive complicated
[03:01:56] is that these are massive complicated systems right there's a billion
[03:01:57] systems right there's a billion parameters here that are outputting a
[03:01:59] parameters here that are outputting a single
[03:02:00] single score it turns out that there are ways
[03:02:03] score it turns out that there are ways to gain these models you can find kinds
[03:02:06] to gain these models you can find kinds of inputs that were not part of their
[03:02:08] of inputs that were not part of their training set and these inputs
[03:02:11] training set and these inputs inexplicably get very high scores but in
[03:02:14] inexplicably get very high scores but in a fake way so very often what you find
[03:02:17] a fake way so very often what you find if you run our lch for very long so for
[03:02:19] if you run our lch for very long so for example if we do 1,000 updates which is
[03:02:21] example if we do 1,000 updates which is like say a lot of updates you might
[03:02:24] like say a lot of updates you might expect that your jokes are getting
[03:02:25] expect that your jokes are getting better and that you're getting like real
[03:02:27] better and that you're getting like real bangers about Pelicans but that's not
[03:02:29] bangers about Pelicans but that's not EXA exactly what happens what happens is
[03:02:32] EXA exactly what happens what happens is that uh in the first few hundred steps
[03:02:34] that uh in the first few hundred steps the jokes about Pelicans are probably
[03:02:35] the jokes about Pelicans are probably improving a little bit and then they
[03:02:37] improving a little bit and then they actually dramatically fall off the cliff
[03:02:39] actually dramatically fall off the cliff and you start to get extremely
[03:02:40] and you start to get extremely nonsensical results like for example you
[03:02:43] nonsensical results like for example you start to get um the top joke about
[03:02:45] start to get um the top joke about Pelicans starts to be the
[03:02:48] Pelicans starts to be the and this makes no sense right like when
[03:02:49] and this makes no sense right like when you look at it why should this be a top
[03:02:51] you look at it why should this be a top joke but when you take the the and you
[03:02:54] joke but when you take the the and you plug it into your reward model you'd
[03:02:56] plug it into your reward model you'd expect score of zero but actually the
[03:02:57] expect score of zero but actually the reward model loves this as a joke it
[03:03:00] reward model loves this as a joke it will tell you that the the the theth is
[03:03:03] will tell you that the the the theth is a score of 1. Z this is a top joke and
[03:03:06] a score of 1. Z this is a top joke and this makes no sense right but it's
[03:03:08] this makes no sense right but it's because these models are just
[03:03:09] because these models are just simulations of humans and they're
[03:03:11] simulations of humans and they're massive neural lots and you can find
[03:03:12] massive neural lots and you can find inputs at the bottom that kind of like
[03:03:15] inputs at the bottom that kind of like get into the part of the input space
[03:03:17] get into the part of the input space that kind of gives you nonsensical
[03:03:18] that kind of gives you nonsensical results these examples are what's called
[03:03:20] results these examples are what's called adversarial examples and I'm not going
[03:03:22] adversarial examples and I'm not going to go into the topic too much but these
[03:03:24] to go into the topic too much but these are adversarial inputs to the model they
[03:03:26] are adversarial inputs to the model they are specific little inputs that kind of
[03:03:29] are specific little inputs that kind of go between the nooks and crannies of the
[03:03:31] go between the nooks and crannies of the model and give nonsensical results at
[03:03:33] model and give nonsensical results at the top now here's what you might
[03:03:35] the top now here's what you might imagine doing you say okay the the the
[03:03:37] imagine doing you say okay the the the is obviously not score of one um it's
[03:03:39] is obviously not score of one um it's obviously a low score so let's take the
[03:03:41] obviously a low score so let's take the the the the the let's add it to the data
[03:03:43] the the the the let's add it to the data set and give it an ordering that is
[03:03:46] set and give it an ordering that is extremely bad like a score of five and
[03:03:48] extremely bad like a score of five and indeed your model will learn that the D
[03:03:50] indeed your model will learn that the D should have a very low score and it will
[03:03:52] should have a very low score and it will give it score of zero the problem is
[03:03:54] give it score of zero the problem is that there will always be basically
[03:03:55] that there will always be basically infinite number of nonsensical
[03:03:58] infinite number of nonsensical adversarial examples hiding in the model
[03:04:01] adversarial examples hiding in the model if you iterate this process many times
[03:04:02] if you iterate this process many times and you keep adding nonsensical stuff to
[03:04:04] and you keep adding nonsensical stuff to your reward model and giving it very low
[03:04:06] your reward model and giving it very low scores you can you'll never win the game
[03:04:09] scores you can you'll never win the game uh you can do this many many rounds and
[03:04:11] uh you can do this many many rounds and reinforcement learning if you run it
[03:04:13] reinforcement learning if you run it long enough will always find a way to
[03:04:14] long enough will always find a way to gain the model it will discover
[03:04:16] gain the model it will discover adversarial examples it will get get
[03:04:18] adversarial examples it will get get really high scores uh with nonsensical
[03:04:21] really high scores uh with nonsensical results and fundamentally this is
[03:04:23] results and fundamentally this is because our scoring function is a giant
[03:04:26] because our scoring function is a giant neural nut and RL is extremely good at
[03:04:29] neural nut and RL is extremely good at finding just the ways to trick it uh so
[03:04:34] finding just the ways to trick it uh so long story short you always run rhf put
[03:04:37] long story short you always run rhf put for maybe a few hundred updates the
[03:04:39] for maybe a few hundred updates the model is getting better and then you
[03:04:40] model is getting better and then you have to crop it and you are done you
[03:04:42] have to crop it and you are done you can't run too much against this reward
[03:04:46] can't run too much against this reward model because the optimization will
[03:04:48] model because the optimization will start to game it and you basically crop
[03:04:50] start to game it and you basically crop it and you call it and you ship it um
[03:04:54] it and you call it and you ship it um and uh you can improve the reward model
[03:04:56] and uh you can improve the reward model but you kind of like come across these
[03:04:58] but you kind of like come across these situations eventually at some point so
[03:05:01] situations eventually at some point so rhf basically what I usually say is that
[03:05:04] rhf basically what I usually say is that RF is not RL and what I mean by that is
[03:05:07] RF is not RL and what I mean by that is I mean RF is RL obviously but it's not
[03:05:09] I mean RF is RL obviously but it's not RL in the magical sense this is not RL
[03:05:12] RL in the magical sense this is not RL that you can run
[03:05:14] that you can run indefinitely these kinds of problems
[03:05:16] indefinitely these kinds of problems like where you are getting con correct
[03:05:18] like where you are getting con correct answer you cannot gain this as easily
[03:05:21] answer you cannot gain this as easily you either got the correct answer or you
[03:05:22] you either got the correct answer or you didn't and the scoring function is much
[03:05:24] didn't and the scoring function is much much simpler you're just looking at the
[03:05:25] much simpler you're just looking at the boxed area and seeing if the result is
[03:05:27] boxed area and seeing if the result is correct so it's very difficult to gain
[03:05:30] correct so it's very difficult to gain these functions but uh gaming a reward
[03:05:32] these functions but uh gaming a reward model is possible now in these
[03:05:34] model is possible now in these verifiable domains you can run RL
[03:05:36] verifiable domains you can run RL indefinitely you could run for tens of
[03:05:39] indefinitely you could run for tens of thousands hundreds of thousands of steps
[03:05:40] thousands hundreds of thousands of steps and discover all kinds of really crazy
[03:05:42] and discover all kinds of really crazy strategies that we might not even ever
[03:05:44] strategies that we might not even ever think about of Performing really well
[03:05:46] think about of Performing really well for all these problems in the game of Go
[03:05:49] for all these problems in the game of Go there's no way to to beat to basically
[03:05:51] there's no way to to beat to basically game uh the winning of a game or the
[03:05:53] game uh the winning of a game or the losing of a game we have a perfect
[03:05:55] losing of a game we have a perfect simulator we know all the different uh
[03:05:58] simulator we know all the different uh where all the stones are placed and we
[03:05:59] where all the stones are placed and we can calculate uh whether someone has won
[03:06:01] can calculate uh whether someone has won or not there's no way to gain that and
[03:06:04] or not there's no way to gain that and so you can do RL indefinitely and you
[03:06:06] so you can do RL indefinitely and you can eventually be beat even leol but
[03:06:09] can eventually be beat even leol but with models like this which are gameable
[03:06:12] with models like this which are gameable you cannot repeat this process
[03:06:14] you cannot repeat this process indefinitely so I kind of see rhf as not
[03:06:17] indefinitely so I kind of see rhf as not real RL because the reward function is
[03:06:19] real RL because the reward function is gameable so it's kind of more like in
[03:06:21] gameable so it's kind of more like in the realm of like little fine-tuning
[03:06:23] the realm of like little fine-tuning it's a little it's a little Improvement
[03:06:26] it's a little it's a little Improvement but it's not something that is
[03:06:27] but it's not something that is fundamentally set up correctly where you
[03:06:30] fundamentally set up correctly where you can insert more compute run for longer
[03:06:32] can insert more compute run for longer and get much better and magical results
[03:06:35] and get much better and magical results so it's it's uh it's not RL in that
[03:06:37] so it's it's uh it's not RL in that sense it's not RL in the sense that it
[03:06:39] sense it's not RL in the sense that it lacks magic um it can find you in your
[03:06:41] lacks magic um it can find you in your model and get a better performance and
[03:06:44] model and get a better performance and indeed if we go back to chat GPT the GPT
[03:06:47] indeed if we go back to chat GPT the GPT 40 model has gone through rhf because it
[03:06:50] 40 model has gone through rhf because it works well but it's just not RL in the
[03:06:53] works well but it's just not RL in the same sense rlf is like a little fine
[03:06:55] same sense rlf is like a little fine tune that slightly improves your model
[03:06:56] tune that slightly improves your model is maybe like the way I would think
[03:06:58] is maybe like the way I would think about it okay so that's most of the
[03:07:00] about it okay so that's most of the technical content that I wanted to cover
[03:07:02] technical content that I wanted to cover I took you through the three major
[03:07:03] I took you through the three major stages and paradigms of training these
[03:07:05] stages and paradigms of training these models pre-training supervised fine
[03:07:07] models pre-training supervised fine tuning and reinforcement learning and I
[03:07:09] tuning and reinforcement learning and I showed you that they Loosely correspond
[03:07:11] showed you that they Loosely correspond to the process we already use for
[03:07:13] to the process we already use for teaching children and so in particular
[03:07:15] teaching children and so in particular we talked about pre-training being sort
[03:07:17] we talked about pre-training being sort of like the basic knowledge acquisition
[03:07:19] of like the basic knowledge acquisition of reading Exposition supervised fine
[03:07:21] of reading Exposition supervised fine tuning being the process of looking at
[03:07:23] tuning being the process of looking at lots and lots of worked examples and
[03:07:25] lots and lots of worked examples and imitating experts and practice problems
[03:07:29] imitating experts and practice problems the only difference is that we now have
[03:07:30] the only difference is that we now have to effectively write textbooks for llms
[03:07:33] to effectively write textbooks for llms and AIS across all the disciplines of
[03:07:35] and AIS across all the disciplines of human knowledge and also in all the
[03:07:37] human knowledge and also in all the cases where we actually would like them
[03:07:39] cases where we actually would like them to work like code and math and you know
[03:07:43] to work like code and math and you know basically all the other disciplines so
[03:07:44] basically all the other disciplines so we're in the process of writing
[03:07:45] we're in the process of writing textbooks for them refining all the
[03:07:48] textbooks for them refining all the algorithms that I've presented on the
[03:07:49] algorithms that I've presented on the high level and then of course doing a
[03:07:51] high level and then of course doing a really really good job at the execution
[03:07:53] really really good job at the execution of training these models at scale and
[03:07:55] of training these models at scale and efficiently so in particular I didn't go
[03:07:57] efficiently so in particular I didn't go into too many details but these are
[03:07:59] into too many details but these are extremely large and complicated
[03:08:01] extremely large and complicated distributed uh sort of
[03:08:04] distributed uh sort of um jobs that have to run over tens of
[03:08:07] um jobs that have to run over tens of thousands or even hundreds of thousands
[03:08:08] thousands or even hundreds of thousands of gpus and the engineering that goes
[03:08:11] of gpus and the engineering that goes into this is really at the stateof the
[03:08:13] into this is really at the stateof the art of what's possible with computers at
[03:08:14] art of what's possible with computers at that scale so I didn't cover that aspect
[03:08:18] that scale so I didn't cover that aspect too much
[03:08:19] too much but um this is very kind of serious and
[03:08:23] but um this is very kind of serious and they were underlying all these very
[03:08:24] they were underlying all these very simple algorithms
[03:08:26] simple algorithms ultimately now I also talked about sort
[03:08:29] ultimately now I also talked about sort of like the theory of mind a little bit
[03:08:30] of like the theory of mind a little bit of these models and the thing I want you
[03:08:32] of these models and the thing I want you to take away is that these models are
[03:08:34] to take away is that these models are really good but they're extremely useful
[03:08:35] really good but they're extremely useful as tools for your work you shouldn't uh
[03:08:38] as tools for your work you shouldn't uh sort of trust them fully and I showed
[03:08:40] sort of trust them fully and I showed you some examples of that even though we
[03:08:41] you some examples of that even though we have mitigations for hallucinations the
[03:08:43] have mitigations for hallucinations the models are not perfect and they will
[03:08:45] models are not perfect and they will hallucinate still it's gotten better
[03:08:47] hallucinate still it's gotten better over time and it will continue to get
[03:08:48] over time and it will continue to get better but they can
[03:08:50] better but they can hallucinate in other words in in
[03:08:52] hallucinate in other words in in addition to that I covered kind of like
[03:08:54] addition to that I covered kind of like what I call the Swiss cheese uh sort of
[03:08:56] what I call the Swiss cheese uh sort of model of llm capabilities that you
[03:08:58] model of llm capabilities that you should have in your mind the models are
[03:08:59] should have in your mind the models are incredibly good across so many different
[03:09:01] incredibly good across so many different disciplines but then fail randomly
[03:09:03] disciplines but then fail randomly almost in some unique cases so for
[03:09:05] almost in some unique cases so for example what is bigger 9.11 or 9.9 like
[03:09:08] example what is bigger 9.11 or 9.9 like the model doesn't know but
[03:09:09] the model doesn't know but simultaneously it can turn around and
[03:09:11] simultaneously it can turn around and solve Olympiad questions and so this is
[03:09:14] solve Olympiad questions and so this is a hole in the Swiss cheese and there are
[03:09:16] a hole in the Swiss cheese and there are many of them and you don't want to trip
[03:09:18] many of them and you don't want to trip over them so don't um treat these models
[03:09:22] over them so don't um treat these models as infallible models check their work
[03:09:24] as infallible models check their work use them as tools use them for
[03:09:26] use them as tools use them for inspiration use them for the first draft
[03:09:28] inspiration use them for the first draft but uh work with them as tools and be
[03:09:30] but uh work with them as tools and be ultimately respons responsible for the
[03:09:33] ultimately respons responsible for the you know product of your
[03:09:35] you know product of your work and that's roughly what I wanted to
[03:09:39] work and that's roughly what I wanted to talk about this is how they're trained
[03:09:41] talk about this is how they're trained and this is what they are let's now turn
[03:09:43] and this is what they are let's now turn to what are some of the future
[03:09:44] to what are some of the future capabilities of these models uh probably
[03:09:47] capabilities of these models uh probably what's coming down the pipe and also
[03:09:48] what's coming down the pipe and also where can you find these models I have a
[03:09:50] where can you find these models I have a few blow points on some of the things
[03:09:51] few blow points on some of the things that you can expect coming down the pipe
[03:09:53] that you can expect coming down the pipe the first thing you'll notice is that
[03:09:55] the first thing you'll notice is that the models will very rapidly become
[03:09:57] the models will very rapidly become multimodal everything I talked about
[03:09:59] multimodal everything I talked about above concerned text but very soon we'll
[03:10:01] above concerned text but very soon we'll have llms that can not just handle text
[03:10:04] have llms that can not just handle text but they can also operate natively and
[03:10:06] but they can also operate natively and very easily over audio so they can hear
[03:10:08] very easily over audio so they can hear and speak and also images so they can
[03:10:10] and speak and also images so they can see and paint and we're already seeing
[03:10:13] see and paint and we're already seeing the beginnings of all of this uh but
[03:10:15] the beginnings of all of this uh but this will be all done natively inside
[03:10:17] this will be all done natively inside inside the language model and this will
[03:10:19] inside the language model and this will enable kind of like natural
[03:10:21] enable kind of like natural conversations and roughly speaking the
[03:10:22] conversations and roughly speaking the reason that this is actually no
[03:10:24] reason that this is actually no different from everything we've covered
[03:10:25] different from everything we've covered above is that as a baseline you can
[03:10:28] above is that as a baseline you can tokenize audio and images and apply the
[03:10:31] tokenize audio and images and apply the exact same approaches of everything that
[03:10:33] exact same approaches of everything that we've talked about above so it's not a
[03:10:35] we've talked about above so it's not a fundamental change it's just uh it's
[03:10:36] fundamental change it's just uh it's just a to we have to add some tokens so
[03:10:39] just a to we have to add some tokens so as an example for tokenizing audio we
[03:10:41] as an example for tokenizing audio we can look at slices of the spectrogram of
[03:10:43] can look at slices of the spectrogram of the audio signal and we can tokenize
[03:10:46] the audio signal and we can tokenize that and just add more tokens that
[03:10:48] that and just add more tokens that suddenly represent audio and just add
[03:10:50] suddenly represent audio and just add them into the context windows and train
[03:10:51] them into the context windows and train on them just like above the same for
[03:10:53] on them just like above the same for images we can use patches and we can
[03:10:56] images we can use patches and we can separately tokenize patches and then
[03:10:59] separately tokenize patches and then what is an image an image is just a
[03:11:01] what is an image an image is just a sequence of tokens and this actually
[03:11:03] sequence of tokens and this actually kind of works and there's a lot of early
[03:11:05] kind of works and there's a lot of early work in this direction and so we can
[03:11:07] work in this direction and so we can just create streams of tokens that are
[03:11:09] just create streams of tokens that are representing audio images as well as
[03:11:10] representing audio images as well as text and interpers them and handle them
[03:11:12] text and interpers them and handle them all simultaneously in a single model so
[03:11:15] all simultaneously in a single model so that's one example of multimodality
[03:11:17] that's one example of multimodality uh second something that people are very
[03:11:19] uh second something that people are very interested in
[03:11:20] interested in is currently most of the work is that
[03:11:22] is currently most of the work is that we're handing individual tasks to the
[03:11:24] we're handing individual tasks to the models on kind of like a silver platter
[03:11:26] models on kind of like a silver platter like please solve this task for me and
[03:11:28] like please solve this task for me and the model sort of like does this little
[03:11:29] the model sort of like does this little task but it's up to us to still sort of
[03:11:32] task but it's up to us to still sort of like organize a coherent execution of
[03:11:35] like organize a coherent execution of tasks to perform jobs and the models are
[03:11:38] tasks to perform jobs and the models are not yet at the capability required to do
[03:11:41] not yet at the capability required to do this in a coherent error correcting way
[03:11:44] this in a coherent error correcting way over long periods of time so they're not
[03:11:47] over long periods of time so they're not able to fully string together tasks to
[03:11:49] able to fully string together tasks to perform these longer running jobs but
[03:11:51] perform these longer running jobs but they're getting there and this is
[03:11:52] they're getting there and this is improving uh over time but uh probably
[03:11:55] improving uh over time but uh probably what's going to happen here is we're
[03:11:56] what's going to happen here is we're going to start to see what's called
[03:11:57] going to start to see what's called agents which perform tasks over time and
[03:12:00] agents which perform tasks over time and you you supervise them and you watch
[03:12:03] you you supervise them and you watch their work and they come up to once in a
[03:12:05] their work and they come up to once in a while report progress and so on so we're
[03:12:07] while report progress and so on so we're going to see more long running agents uh
[03:12:10] going to see more long running agents uh tasks that don't just take you know a
[03:12:11] tasks that don't just take you know a few seconds of response but many tens of
[03:12:13] few seconds of response but many tens of seconds or even minutes or hours over
[03:12:15] seconds or even minutes or hours over time uh but these uh models are not
[03:12:18] time uh but these uh models are not infallible as we talked about above so
[03:12:20] infallible as we talked about above so all of this will require supervision so
[03:12:22] all of this will require supervision so for example in factories people talk
[03:12:23] for example in factories people talk about the human to robot ratio uh for
[03:12:26] about the human to robot ratio uh for automation I think we're going to see
[03:12:28] automation I think we're going to see something similar in the digital space
[03:12:30] something similar in the digital space where we are going to be talking about
[03:12:31] where we are going to be talking about human to agent ratios where humans
[03:12:33] human to agent ratios where humans becomes a lot more supervisors of agent
[03:12:36] becomes a lot more supervisors of agent tasks um in the digital
[03:12:38] tasks um in the digital domain uh next um I think everything is
[03:12:41] domain uh next um I think everything is going to become a lot more pervasive and
[03:12:43] going to become a lot more pervasive and invisible so it's kind of like
[03:12:45] invisible so it's kind of like integrated into the tools and everywhere
[03:12:49] integrated into the tools and everywhere um and in addition kind of like computer
[03:12:51] um and in addition kind of like computer using so right now these models aren't
[03:12:54] using so right now these models aren't able to take actions on your behalf but
[03:12:56] able to take actions on your behalf but I think this is a separate bullet point
[03:12:59] I think this is a separate bullet point um if you saw chpt launch the operator
[03:13:02] um if you saw chpt launch the operator then uh that's one early example of that
[03:13:04] then uh that's one early example of that where you can actually hand off control
[03:13:06] where you can actually hand off control to the model to perform you know
[03:13:08] to the model to perform you know keyboard and mouse actions on your
[03:13:10] keyboard and mouse actions on your behalf so that's also something that
[03:13:11] behalf so that's also something that that I think is very interesting the
[03:13:13] that I think is very interesting the last point I have here is just a general
[03:13:15] last point I have here is just a general comment that there's still a lot of
[03:13:16] comment that there's still a lot of research to potentially do in this
[03:13:17] research to potentially do in this domain main one example of that uh is
[03:13:20] domain main one example of that uh is something along the lines of test time
[03:13:21] something along the lines of test time training so remember that everything
[03:13:23] training so remember that everything we've done above and that we talked
[03:13:24] we've done above and that we talked about has two major stages there's first
[03:13:27] about has two major stages there's first the training stage where we tune the
[03:13:29] the training stage where we tune the parameters of the model to perform the
[03:13:30] parameters of the model to perform the tasks well once we get the parameters we
[03:13:33] tasks well once we get the parameters we fix them and then we deploy the model
[03:13:35] fix them and then we deploy the model for inference from there the model is
[03:13:38] for inference from there the model is fixed it doesn't change anymore it
[03:13:40] fixed it doesn't change anymore it doesn't learn from all the stuff that
[03:13:41] doesn't learn from all the stuff that it's doing a test time it's a fixed um
[03:13:43] it's doing a test time it's a fixed um number of parameters and the only thing
[03:13:45] number of parameters and the only thing that is changing is now the token inside
[03:13:48] that is changing is now the token inside the context windows and so the only type
[03:13:50] the context windows and so the only type of learning or test time learning that
[03:13:52] of learning or test time learning that the model has access to is the in
[03:13:54] the model has access to is the in context learning of its uh kind of like
[03:13:57] context learning of its uh kind of like uh dynamically adjustable context window
[03:13:59] uh dynamically adjustable context window depending on like what it's doing at
[03:14:00] depending on like what it's doing at test time so but I think this is still
[03:14:03] test time so but I think this is still different from humans who actually are
[03:14:05] different from humans who actually are able to like actually learn uh depending
[03:14:07] able to like actually learn uh depending on what they're doing especially when
[03:14:08] on what they're doing especially when you sleep for example like your brain is
[03:14:09] you sleep for example like your brain is updating your parameters or something
[03:14:11] updating your parameters or something like that right so there's no kind of
[03:14:13] like that right so there's no kind of equivalent of that currently in these
[03:14:14] equivalent of that currently in these models and tools so there's a lot of
[03:14:17] models and tools so there's a lot of like um more wonky ideas I think that
[03:14:18] like um more wonky ideas I think that are to be explored still and uh in
[03:14:21] are to be explored still and uh in particular I think this will be
[03:14:22] particular I think this will be necessary because the context window is
[03:14:24] necessary because the context window is a finite and precious resource and
[03:14:26] a finite and precious resource and especially once we start to tackle very
[03:14:28] especially once we start to tackle very long running multimodal tasks and we're
[03:14:30] long running multimodal tasks and we're putting in videos and these token
[03:14:32] putting in videos and these token windows will basically start to grow
[03:14:34] windows will basically start to grow extremely large like not thousands or
[03:14:36] extremely large like not thousands or even hundreds of thousands but
[03:14:38] even hundreds of thousands but significantly beyond that and the only
[03:14:40] significantly beyond that and the only trick uh the only kind of trick we have
[03:14:42] trick uh the only kind of trick we have Avail to us right now is to make the
[03:14:43] Avail to us right now is to make the context Windows longer but I think that
[03:14:46] context Windows longer but I think that that approach by itself will will not
[03:14:47] that approach by itself will will not will not scale to actual long running
[03:14:50] will not scale to actual long running tasks that are multimodal over time and
[03:14:52] tasks that are multimodal over time and so I think new ideas are needed in some
[03:14:54] so I think new ideas are needed in some of those disciplines um in some of those
[03:14:57] of those disciplines um in some of those kind of cases in the main where these
[03:14:58] kind of cases in the main where these tasks are going to require very long
[03:15:01] tasks are going to require very long contexts so those are some examples of
[03:15:03] contexts so those are some examples of some of the things you can um expect
[03:15:05] some of the things you can um expect coming down the pipe let's now turn to
[03:15:07] coming down the pipe let's now turn to where you can actually uh kind of keep
[03:15:09] where you can actually uh kind of keep track of this progress and um you know
[03:15:12] track of this progress and um you know be up to date with the latest and grest
[03:15:14] be up to date with the latest and grest of what's happening in the field so I
[03:15:15] of what's happening in the field so I would say the three resources that I
[03:15:17] would say the three resources that I have consistently used to stay up to
[03:15:19] have consistently used to stay up to date are number one El Marina uh so let
[03:15:22] date are number one El Marina uh so let me show you El
[03:15:24] me show you El Marina this is basically an llm leader
[03:15:26] Marina this is basically an llm leader board and it ranks all the top models
[03:15:30] board and it ranks all the top models and the ranking is based on human
[03:15:32] and the ranking is based on human comparisons so humans prompt these
[03:15:34] comparisons so humans prompt these models and they get to judge which one
[03:15:36] models and they get to judge which one gives a better answer they don't know
[03:15:38] gives a better answer they don't know which model is which they're just
[03:15:39] which model is which they're just looking at which model is the better
[03:15:40] looking at which model is the better answer and you can calculate a ranking
[03:15:43] answer and you can calculate a ranking and then you get some results and so
[03:15:45] and then you get some results and so what you can hear is what you can see
[03:15:46] what you can hear is what you can see here is the different organizations like
[03:15:48] here is the different organizations like Google Gemini for example that produce
[03:15:50] Google Gemini for example that produce these models when you click on any one
[03:15:51] these models when you click on any one of these it takes you to the place where
[03:15:54] of these it takes you to the place where that model is
[03:15:55] that model is hosted and then here we see Google is
[03:15:57] hosted and then here we see Google is currently on top with open AI right
[03:15:59] currently on top with open AI right behind here we see deep seek in position
[03:16:02] behind here we see deep seek in position number three now the reason this is a
[03:16:04] number three now the reason this is a big deal is the last column here you see
[03:16:06] big deal is the last column here you see license deep seek is an MIT license
[03:16:08] license deep seek is an MIT license model it's open weights anyone can use
[03:16:11] model it's open weights anyone can use these weights uh anyone can download
[03:16:13] these weights uh anyone can download them anyone can host their own version
[03:16:15] them anyone can host their own version of Deep seek and they can use it in what
[03:16:17] of Deep seek and they can use it in what whatever way they like and so it's not a
[03:16:19] whatever way they like and so it's not a proprietary model that you don't have
[03:16:20] proprietary model that you don't have access to it's it's basically an open
[03:16:22] access to it's it's basically an open weight release and so this is kind of
[03:16:25] weight release and so this is kind of unprecedented that a model this strong
[03:16:27] unprecedented that a model this strong was released with open weights so pretty
[03:16:30] was released with open weights so pretty cool from the team next up we have a few
[03:16:32] cool from the team next up we have a few more models from Google and open Ai and
[03:16:34] more models from Google and open Ai and then when you continue to scroll down
[03:16:35] then when you continue to scroll down you start to see some other Usual
[03:16:37] you start to see some other Usual Suspects so xai here anthropic with son
[03:16:41] Suspects so xai here anthropic with son it uh here at number
[03:16:43] it uh here at number 14 and
[03:16:46] 14 and um then
[03:16:48] um then meta with llama over here so llama
[03:16:51] meta with llama over here so llama similar to deep seek is an open weights
[03:16:53] similar to deep seek is an open weights model and so uh but it's down here as
[03:16:56] model and so uh but it's down here as opposed to up here now I will say that
[03:16:58] opposed to up here now I will say that this leaderboard was really good for a
[03:17:01] this leaderboard was really good for a long time I do think that in the last
[03:17:04] long time I do think that in the last few months it's become a little bit
[03:17:05] few months it's become a little bit gamed um and I don't trust it as much as
[03:17:08] gamed um and I don't trust it as much as I used to I think um just empirically I
[03:17:12] I used to I think um just empirically I feel like a lot of people for example
[03:17:13] feel like a lot of people for example are using a Sonet from anthropic and
[03:17:15] are using a Sonet from anthropic and that it's a really good model so but
[03:17:17] that it's a really good model so but that's all the way down here um in
[03:17:20] that's all the way down here um in number 14 and conversely I think not as
[03:17:22] number 14 and conversely I think not as many people are using Gemini but it's
[03:17:24] many people are using Gemini but it's racking really really high uh so I think
[03:17:27] racking really really high uh so I think use this as a first pass uh but uh sort
[03:17:30] use this as a first pass uh but uh sort of try out a few of the models for your
[03:17:33] of try out a few of the models for your tasks and see which one performs better
[03:17:36] tasks and see which one performs better the second thing that I would point to
[03:17:37] the second thing that I would point to is the uh AI news uh newsletter so AI
[03:17:41] is the uh AI news uh newsletter so AI news is not very creatively named but it
[03:17:43] news is not very creatively named but it is a very good newsletter produced by
[03:17:45] is a very good newsletter produced by swix and friends so thank you for
[03:17:46] swix and friends so thank you for maintaining it
[03:17:47] maintaining it and it's been very helpful to me because
[03:17:49] and it's been very helpful to me because it is extremely comprehensive so if you
[03:17:51] it is extremely comprehensive so if you go to archives uh you see that it's
[03:17:53] go to archives uh you see that it's produced almost every other day and um
[03:17:56] produced almost every other day and um it is very comprehensive and some of it
[03:17:58] it is very comprehensive and some of it is written by humans and curated by
[03:18:00] is written by humans and curated by humans but a lot of it is constructed
[03:18:01] humans but a lot of it is constructed automatically with llms so you'll see
[03:18:03] automatically with llms so you'll see that these are very comprehensive and
[03:18:05] that these are very comprehensive and you're probably not missing anything
[03:18:06] you're probably not missing anything major if you go through it of course
[03:18:09] major if you go through it of course you're probably not going to go through
[03:18:10] you're probably not going to go through it because it's so long but I do think
[03:18:12] it because it's so long but I do think that these summaries all the way up top
[03:18:14] that these summaries all the way up top are quite good and I think have some
[03:18:16] are quite good and I think have some human oversight uh so this has been very
[03:18:19] human oversight uh so this has been very helpful to me and the last thing I would
[03:18:20] helpful to me and the last thing I would point to is just X and Twitter uh a lot
[03:18:23] point to is just X and Twitter uh a lot of um AI happens on X and so I would
[03:18:26] of um AI happens on X and so I would just follow people who you like and
[03:18:27] just follow people who you like and trust and get all your latest and
[03:18:30] trust and get all your latest and greatest uh on X as well so those are
[03:18:32] greatest uh on X as well so those are the major places that have worked for me
[03:18:34] the major places that have worked for me over time and finally a few words on
[03:18:36] over time and finally a few words on where you can find the models and where
[03:18:38] where you can find the models and where can you use them so the first one I
[03:18:39] can you use them so the first one I would say is for any of the biggest
[03:18:41] would say is for any of the biggest proprietary models you just have to go
[03:18:43] proprietary models you just have to go to the website of that LM provider so
[03:18:45] to the website of that LM provider so for example for open a that's uh chat
[03:18:47] for example for open a that's uh chat I believe actually works now uh so
[03:18:49] I believe actually works now uh so that's for open
[03:18:51] that's for open AI now for or you know for um for Gemini
[03:18:55] AI now for or you know for um for Gemini I think it's gem. google.com or AI
[03:18:58] I think it's gem. google.com or AI Studio I think they have two for some
[03:19:00] Studio I think they have two for some reason that I don't fly understand no
[03:19:01] reason that I don't fly understand no one does um for the open weights models
[03:19:05] one does um for the open weights models like deep SE CL Etc you have to go to
[03:19:07] like deep SE CL Etc you have to go to some kind of an inference provider of
[03:19:08] some kind of an inference provider of LMS so my favorite one is together
[03:19:10] LMS so my favorite one is together together. a and I showed you that when
[03:19:12] together. a and I showed you that when you go to the playground of together. a
[03:19:14] you go to the playground of together. a then you can sort of pick lots of
[03:19:16] then you can sort of pick lots of different models and all of these are
[03:19:17] different models and all of these are open models of different types and you
[03:19:19] open models of different types and you can talk to them here as an
[03:19:22] can talk to them here as an example um now if you'd like to use a
[03:19:25] example um now if you'd like to use a base model like um you know a base model
[03:19:28] base model like um you know a base model then this is where I think it's not as
[03:19:30] then this is where I think it's not as common to find base models even on these
[03:19:31] common to find base models even on these inference providers they are all
[03:19:33] inference providers they are all targeting assistants and chat and so I
[03:19:36] targeting assistants and chat and so I think even here I can't I couldn't see
[03:19:38] think even here I can't I couldn't see base models here so for base models I
[03:19:40] base models here so for base models I usually go to hyperbolic because they
[03:19:42] usually go to hyperbolic because they serve my llama 3.1 base and I love that
[03:19:45] serve my llama 3.1 base and I love that model and you can just talk to it here
[03:19:47] model and you can just talk to it here so as far as I know this is this is a
[03:19:49] so as far as I know this is this is a good place for a base model and I wish
[03:19:51] good place for a base model and I wish more people hosted base models because
[03:19:53] more people hosted base models because they are useful and interesting to work
[03:19:55] they are useful and interesting to work with in some cases finally you can also
[03:19:58] with in some cases finally you can also take some of the models that are smaller
[03:20:00] take some of the models that are smaller and you can run them locally and so for
[03:20:02] and you can run them locally and so for example deep seek the biggest model
[03:20:04] example deep seek the biggest model you're not going to be able to run
[03:20:05] you're not going to be able to run locally on your MacBook but there are
[03:20:08] locally on your MacBook but there are smaller versions of the deep seek model
[03:20:09] smaller versions of the deep seek model that are what's called distilled and
[03:20:11] that are what's called distilled and then also you can run these models at
[03:20:12] then also you can run these models at smaller Precision so not at the native
[03:20:14] smaller Precision so not at the native Precision of for example fp8 on deep
[03:20:17] Precision of for example fp8 on deep seek or you know bf16 llama but much
[03:20:20] seek or you know bf16 llama but much much lower than that um and don't worry
[03:20:24] much lower than that um and don't worry if you don't fully understand those
[03:20:25] if you don't fully understand those details but you can run smaller versions
[03:20:27] details but you can run smaller versions that have been distilled and then at
[03:20:28] that have been distilled and then at even lower precision and then you can
[03:20:30] even lower precision and then you can fit them on your uh computer and so you
[03:20:33] fit them on your uh computer and so you can actually run pretty okay models on
[03:20:35] can actually run pretty okay models on your laptop and my favorite I think
[03:20:37] your laptop and my favorite I think place I go to usually is LM studio uh
[03:20:40] place I go to usually is LM studio uh which is basically an app you can get
[03:20:42] which is basically an app you can get and I think it kind of actually looks
[03:20:44] and I think it kind of actually looks really ugly and it's I don't like that
[03:20:45] really ugly and it's I don't like that it shows you all these models that are
[03:20:47] it shows you all these models that are basically not that useful like everyone
[03:20:48] basically not that useful like everyone just wants to run deep seek so I don't
[03:20:50] just wants to run deep seek so I don't know why they give you these 500
[03:20:51] know why they give you these 500 different types of models they're really
[03:20:53] different types of models they're really complicated to search for and you have
[03:20:54] complicated to search for and you have to choose different distillations and
[03:20:56] to choose different distillations and different uh precisions and it's all
[03:20:58] different uh precisions and it's all really confusing but once you actually
[03:21:00] really confusing but once you actually understand how it works and that's a
[03:21:01] understand how it works and that's a whole separate video then you can
[03:21:03] whole separate video then you can actually load up a model like here I
[03:21:04] actually load up a model like here I loaded up a llama 3 uh2 instruct 1
[03:21:08] loaded up a llama 3 uh2 instruct 1 billion and um you can just talk to it
[03:21:12] billion and um you can just talk to it so I ask for Pelican jokes and I can ask
[03:21:14] so I ask for Pelican jokes and I can ask for another one and it gives me another
[03:21:16] for another one and it gives me another one Etc all of this that happens here is
[03:21:19] one Etc all of this that happens here is locally on your computer so we're not
[03:21:21] locally on your computer so we're not actually going to anywhere anyone else
[03:21:22] actually going to anywhere anyone else this is running on the GPU on the
[03:21:24] this is running on the GPU on the MacBook Pro so that's very nice and you
[03:21:27] MacBook Pro so that's very nice and you can then eject the model when you're
[03:21:28] can then eject the model when you're done and that frees up the ram so LM
[03:21:31] done and that frees up the ram so LM studio is probably like my favorite one
[03:21:33] studio is probably like my favorite one even though I don't I think it's got a
[03:21:34] even though I don't I think it's got a lot of uiux issues and it's really
[03:21:36] lot of uiux issues and it's really geared towards uh professionals almost
[03:21:39] geared towards uh professionals almost uh but if you watch some videos on
[03:21:41] uh but if you watch some videos on YouTube I think you can figure out how
[03:21:42] YouTube I think you can figure out how to how to use this
[03:21:43] to how to use this interface uh so those are a few words on
[03:21:46] interface uh so those are a few words on where to find them so let me now loop
[03:21:48] where to find them so let me now loop back around to where we started the
[03:21:50] back around to where we started the question was when we go to chashi
[03:21:51] question was when we go to chashi pta.com and we enter some kind of a
[03:21:54] pta.com and we enter some kind of a query and we hit go what exactly is
[03:21:58] query and we hit go what exactly is happening here what are we seeing what
[03:22:00] happening here what are we seeing what are we talking to how does this work and
[03:22:03] are we talking to how does this work and I hope that this video gave you some
[03:22:05] I hope that this video gave you some appreciation for some of the under the
[03:22:06] appreciation for some of the under the hood details of how these models are
[03:22:08] hood details of how these models are trained and what this is that is coming
[03:22:10] trained and what this is that is coming back so in particular we now know that
[03:22:12] back so in particular we now know that your query is taken and is first chopped
[03:22:15] your query is taken and is first chopped up into tokens so we go to to tick
[03:22:18] up into tokens so we go to to tick tokenizer and here where is the place in
[03:22:21] tokenizer and here where is the place in the in the um sort of format that is for
[03:22:24] the in the um sort of format that is for the user query we basically put in our
[03:22:28] the user query we basically put in our query right there so our query goes into
[03:22:31] query right there so our query goes into what we discussed here is the
[03:22:33] what we discussed here is the conversation protocol format which is
[03:22:35] conversation protocol format which is this way that we maintain conversation
[03:22:37] this way that we maintain conversation objects so this gets inserted there and
[03:22:40] objects so this gets inserted there and then this whole thing ends up being just
[03:22:41] then this whole thing ends up being just a token sequence a onedimensional token
[03:22:43] a token sequence a onedimensional token sequence under the hood so Chachi PT saw
[03:22:46] sequence under the hood so Chachi PT saw this token sequence and then when we hit
[03:22:48] this token sequence and then when we hit go it basically continues appending
[03:22:51] go it basically continues appending tokens into this list it continues the
[03:22:53] tokens into this list it continues the sequence it acts like a token
[03:22:55] sequence it acts like a token autocomplete so in particular it gave us
[03:22:58] autocomplete so in particular it gave us this response so we can basically just
[03:23:00] this response so we can basically just put it here and we see the tokens that
[03:23:02] put it here and we see the tokens that it continued uh these are the tokens
[03:23:04] it continued uh these are the tokens that it continued with
[03:23:06] that it continued with roughly now the question
[03:23:08] roughly now the question becomes okay why are these the tokens
[03:23:11] becomes okay why are these the tokens that the model responded with what are
[03:23:13] that the model responded with what are these tokens where are they coming from
[03:23:15] these tokens where are they coming from uh what are we talking to and how do we
[03:23:17] uh what are we talking to and how do we program this system and so that's where
[03:23:19] program this system and so that's where we shifted gears and we talked about the
[03:23:21] we shifted gears and we talked about the under thehood pieces of it so the first
[03:23:24] under thehood pieces of it so the first stage of this process and there are
[03:23:26] stage of this process and there are three stages is the pre-training stage
[03:23:27] three stages is the pre-training stage which fundamentally has to do with just
[03:23:29] which fundamentally has to do with just knowledge acquisition from the internet
[03:23:31] knowledge acquisition from the internet into the parameters of this neural
[03:23:33] into the parameters of this neural network and so the neural net
[03:23:35] network and so the neural net internalizes a lot of Knowledge from the
[03:23:37] internalizes a lot of Knowledge from the internet but where the personality
[03:23:39] internet but where the personality really comes in is in the process of
[03:23:41] really comes in is in the process of supervised fine-tuning here and so what
[03:23:45] supervised fine-tuning here and so what what happens here is that basically the
[03:23:47] what happens here is that basically the a company like openai will curate a
[03:23:49] a company like openai will curate a large data set of conversations like say
[03:23:51] large data set of conversations like say 1 million conversation across very
[03:23:53] 1 million conversation across very diverse topics and there will be
[03:23:56] diverse topics and there will be conversations between a human and an
[03:23:58] conversations between a human and an assistant and even though there's a lot
[03:23:59] assistant and even though there's a lot of synthetic data generation used
[03:24:01] of synthetic data generation used throughout this entire process and a lot
[03:24:03] throughout this entire process and a lot of llm help and so on fundamentally this
[03:24:05] of llm help and so on fundamentally this is a human data curation task with lots
[03:24:08] is a human data curation task with lots of humans involved and in particular
[03:24:10] of humans involved and in particular these humans are data labelers hired by
[03:24:12] these humans are data labelers hired by open AI who are given labeling
[03:24:14] open AI who are given labeling instructions that they learn and they
[03:24:17] instructions that they learn and they task is to create ideal assistant
[03:24:18] task is to create ideal assistant responses for any arbitrary prompts so
[03:24:21] responses for any arbitrary prompts so they are teaching the neural network by
[03:24:24] they are teaching the neural network by example how to respond to
[03:24:27] example how to respond to prompts so what is the way to think
[03:24:30] prompts so what is the way to think about what came back here like what is
[03:24:33] about what came back here like what is this well I think the right way to think
[03:24:35] this well I think the right way to think about it is that this is the neural
[03:24:37] about it is that this is the neural network simulation of a data labeler at
[03:24:41] network simulation of a data labeler at openai so it's as if I gave this query
[03:24:44] openai so it's as if I gave this query to a data Li open and this data labeler
[03:24:47] to a data Li open and this data labeler first reads all of the labeling
[03:24:49] first reads all of the labeling instructions from open Ai and then
[03:24:51] instructions from open Ai and then spends 2 hours writing up the ideal
[03:24:53] spends 2 hours writing up the ideal assistant response to this query and uh
[03:24:57] assistant response to this query and uh giving it to me now we're not actually
[03:25:00] giving it to me now we're not actually doing that right because we didn't wait
[03:25:01] doing that right because we didn't wait two hours so what we're getting here is
[03:25:03] two hours so what we're getting here is a neural network simulation of that
[03:25:05] a neural network simulation of that process and we have to keep in mind that
[03:25:08] process and we have to keep in mind that these neural networks don't function
[03:25:10] these neural networks don't function like human brains do they are different
[03:25:12] like human brains do they are different what's easy or hard for them is
[03:25:14] what's easy or hard for them is different from what's easy or hard for
[03:25:16] different from what's easy or hard for humans and so we really are just getting
[03:25:18] humans and so we really are just getting a simulation so here I shown you this is
[03:25:21] a simulation so here I shown you this is a token stream and this is fundamentally
[03:25:23] a token stream and this is fundamentally the neural network with a bunch of
[03:25:25] the neural network with a bunch of activations and neurons in between this
[03:25:27] activations and neurons in between this is a fixed mathematical expression that
[03:25:29] is a fixed mathematical expression that mixes inputs from tokens with parameters
[03:25:33] mixes inputs from tokens with parameters of the model and they get mixed up and
[03:25:36] of the model and they get mixed up and get you the next token in a sequence but
[03:25:38] get you the next token in a sequence but this is a finite amount of compute that
[03:25:39] this is a finite amount of compute that happens for every single token and so
[03:25:42] happens for every single token and so this is some kind of a lossy simulation
[03:25:44] this is some kind of a lossy simulation of a human that is kind of like
[03:25:46] of a human that is kind of like restricted in this way and so whatever
[03:25:49] restricted in this way and so whatever the humans
[03:25:50] the humans write the language model is kind of
[03:25:52] write the language model is kind of imitating on this token level with only
[03:25:55] imitating on this token level with only this this specific computation for every
[03:25:58] this this specific computation for every single token and
[03:26:01] single token and sequence we also saw that as a result of
[03:26:04] sequence we also saw that as a result of this and the cognitive differences the
[03:26:06] this and the cognitive differences the models will suffer in a variety of ways
[03:26:08] models will suffer in a variety of ways and uh you have to be very careful with
[03:26:10] and uh you have to be very careful with their use so for example we saw that
[03:26:12] their use so for example we saw that they will suffer from hallucinations and
[03:26:14] they will suffer from hallucinations and they also we have the sense of a Swiss
[03:26:17] they also we have the sense of a Swiss model of the LM capabilities where
[03:26:19] model of the LM capabilities where basically there's like holes in the
[03:26:21] basically there's like holes in the cheese sometimes the models will just
[03:26:23] cheese sometimes the models will just arbitrarily like do something dumb uh so
[03:26:26] arbitrarily like do something dumb uh so even though they're doing lots of
[03:26:27] even though they're doing lots of magical stuff sometimes they just can't
[03:26:29] magical stuff sometimes they just can't so maybe you're not giving them enough
[03:26:31] so maybe you're not giving them enough tokens to think and maybe they're going
[03:26:32] tokens to think and maybe they're going to just make stuff up because they're
[03:26:34] to just make stuff up because they're mental arithmetic breaks uh maybe they
[03:26:36] mental arithmetic breaks uh maybe they are suddenly unable to count number of
[03:26:38] are suddenly unable to count number of letters um or maybe they're unable to
[03:26:41] letters um or maybe they're unable to tell you that 911 9.11 is smaller than
[03:26:44] tell you that 911 9.11 is smaller than 9.9 and it looks kind of dumb and so so
[03:26:46] 9.9 and it looks kind of dumb and so so it's a Swiss cheese capability and we
[03:26:48] it's a Swiss cheese capability and we have to be careful with that and we saw
[03:26:50] have to be careful with that and we saw the reasons for
[03:26:51] the reasons for that but fundamentally this is how we
[03:26:54] that but fundamentally this is how we think of what came back it's again a
[03:26:56] think of what came back it's again a simulation of this neural network of a
[03:27:01] simulation of this neural network of a human data labeler following the
[03:27:03] human data labeler following the labeling instructions at open a so
[03:27:06] labeling instructions at open a so that's what we're getting back now I do
[03:27:09] that's what we're getting back now I do think that the uh things change a little
[03:27:11] think that the uh things change a little bit when you actually go and reach for
[03:27:14] bit when you actually go and reach for one of the thinking models like o03 mini
[03:27:18] one of the thinking models like o03 mini and the reason for that is that GPT
[03:27:20] and the reason for that is that GPT 40 basically doesn't do reinforcement
[03:27:23] 40 basically doesn't do reinforcement learning it does do rhf but I've told
[03:27:26] learning it does do rhf but I've told you that rhf is not RL there's no
[03:27:29] you that rhf is not RL there's no there's no uh time for magic in there
[03:27:31] there's no uh time for magic in there it's just a little bit of a fine-tuning
[03:27:33] it's just a little bit of a fine-tuning is the way to look at it but these
[03:27:35] is the way to look at it but these thinking models they do use RL so they
[03:27:38] thinking models they do use RL so they go through this third state stage of
[03:27:41] go through this third state stage of perfecting their thinking process and
[03:27:44] perfecting their thinking process and discovering new thinking strategies and
[03:27:46] discovering new thinking strategies and uh
[03:27:47] uh solutions to problem solving that look a
[03:27:50] solutions to problem solving that look a little bit like your internal monologue
[03:27:51] little bit like your internal monologue in your head and they practice that on a
[03:27:53] in your head and they practice that on a large collection of practice problems
[03:27:55] large collection of practice problems that companies like openi create and
[03:27:57] that companies like openi create and curate and um then make available to the
[03:28:01] curate and um then make available to the LMS so when I come here and I talked to
[03:28:03] LMS so when I come here and I talked to a thinking model and I put in this
[03:28:06] a thinking model and I put in this question what we're seeing here is not
[03:28:08] question what we're seeing here is not anymore just the straightforward
[03:28:09] anymore just the straightforward simulation of a human data labeler like
[03:28:11] simulation of a human data labeler like this is actually kind of new unique and
[03:28:14] this is actually kind of new unique and interesting um and of course open is not
[03:28:16] interesting um and of course open is not showing us the under thehood thinking
[03:28:19] showing us the under thehood thinking and the chains of thought that are
[03:28:20] and the chains of thought that are underlying the reasoning here but we
[03:28:23] underlying the reasoning here but we know that such a thing exists and this
[03:28:25] know that such a thing exists and this is a summary of it and what we're
[03:28:26] is a summary of it and what we're getting here is actually not just an
[03:28:28] getting here is actually not just an imitation of a human data labeler it's
[03:28:30] imitation of a human data labeler it's actually something that is kind of new
[03:28:31] actually something that is kind of new and interesting and exciting in the
[03:28:32] and interesting and exciting in the sense that it is a function of thinking
[03:28:35] sense that it is a function of thinking that was emergent in a simulation it's
[03:28:37] that was emergent in a simulation it's not just imitating human data labeler it
[03:28:39] not just imitating human data labeler it comes from this reinforcement learning
[03:28:41] comes from this reinforcement learning process and so here we're of course not
[03:28:43] process and so here we're of course not giving it a chance to shine because this
[03:28:45] giving it a chance to shine because this is not a mathematical or a reasoning
[03:28:47] is not a mathematical or a reasoning problem this is just some kind of a sort
[03:28:49] problem this is just some kind of a sort of creative writing problem roughly
[03:28:51] of creative writing problem roughly speaking and I think it's um it's a a
[03:28:55] speaking and I think it's um it's a a question an open question as to whether
[03:28:58] question an open question as to whether the thinking strategies that are
[03:28:59] the thinking strategies that are developed inside verifiable domains
[03:29:02] developed inside verifiable domains transfer and are generalizable to other
[03:29:06] transfer and are generalizable to other domains that are unverifiable such as
[03:29:08] domains that are unverifiable such as create writing the extent to which that
[03:29:10] create writing the extent to which that transfer happens is unknown in the field
[03:29:12] transfer happens is unknown in the field I would say so we're not sure if we are
[03:29:14] I would say so we're not sure if we are able to do RL on everything that is very
[03:29:16] able to do RL on everything that is very verifiable and see the benefits of that
[03:29:18] verifiable and see the benefits of that on things that are unverifiable like
[03:29:20] on things that are unverifiable like this prompt so that's an open question
[03:29:22] this prompt so that's an open question the other thing that's interesting is
[03:29:24] the other thing that's interesting is that this reinforcement learning here is
[03:29:26] that this reinforcement learning here is still like way too new primordial and
[03:29:29] still like way too new primordial and nent so we're just seeing like the
[03:29:32] nent so we're just seeing like the beginnings of the hints of greatness uh
[03:29:34] beginnings of the hints of greatness uh in the reasoning problems we're seeing
[03:29:36] in the reasoning problems we're seeing something that is in principle capable
[03:29:38] something that is in principle capable of something like the equivalent of move
[03:29:40] of something like the equivalent of move 37 but not in the game of Go but in open
[03:29:44] 37 but not in the game of Go but in open domain thinking and problem solving in
[03:29:46] domain thinking and problem solving in principle this Paradigm is capable of
[03:29:49] principle this Paradigm is capable of doing something really cool new and
[03:29:50] doing something really cool new and exciting something even that no human
[03:29:53] exciting something even that no human has thought of before in principle these
[03:29:55] has thought of before in principle these models are capable of analogies no human
[03:29:57] models are capable of analogies no human has had so I think it's incredibly
[03:29:59] has had so I think it's incredibly exciting that these models exist but
[03:30:01] exciting that these models exist but again it's very early and these are
[03:30:02] again it's very early and these are primordial models for now um and they
[03:30:05] primordial models for now um and they will mostly shine in domains that are
[03:30:07] will mostly shine in domains that are verifiable like math en code Etc so very
[03:30:10] verifiable like math en code Etc so very interesting to play with and think about
[03:30:12] interesting to play with and think about and
[03:30:13] and use and then that's roughly it um um I
[03:30:17] use and then that's roughly it um um I would say those are the broad Strokes of
[03:30:18] would say those are the broad Strokes of what's available right now I will say
[03:30:20] what's available right now I will say that overall it is an extremely exciting
[03:30:23] that overall it is an extremely exciting time to be in the
[03:30:24] time to be in the field personally I use these models all
[03:30:26] field personally I use these models all the time daily uh tens or hundreds of
[03:30:29] the time daily uh tens or hundreds of times because they dramatically
[03:30:30] times because they dramatically accelerate my work I think a lot of
[03:30:32] accelerate my work I think a lot of people see the same thing I think we're
[03:30:33] people see the same thing I think we're going to see a huge amount of wealth
[03:30:35] going to see a huge amount of wealth creation as a result of these models be
[03:30:37] creation as a result of these models be aware of some of their shortcomings even
[03:30:41] aware of some of their shortcomings even with RL models they're going to suffer
[03:30:42] with RL models they're going to suffer from some of these use it as a tool in a
[03:30:45] from some of these use it as a tool in a toolbox don't trust it fully because
[03:30:48] toolbox don't trust it fully because they will randomly do dumb things they
[03:30:50] they will randomly do dumb things they will randomly hallucinate they will
[03:30:51] will randomly hallucinate they will randomly skip over some mental
[03:30:53] randomly skip over some mental arithmetic and not get it right um they
[03:30:55] arithmetic and not get it right um they randomly can't count or something like
[03:30:57] randomly can't count or something like that so use them as tools in the toolbox
[03:30:59] that so use them as tools in the toolbox check their work and own the product of
[03:31:01] check their work and own the product of your work but use them for inspiration
[03:31:03] your work but use them for inspiration for first draft uh ask them questions
[03:31:06] for first draft uh ask them questions but always check and verify and you will
[03:31:09] but always check and verify and you will be very successful in your work if you
[03:31:11] be very successful in your work if you do so uh so I hope this video was useful
[03:31:14] do so uh so I hope this video was useful and interesting to you I hope you had it
[03:31:15] and interesting to you I hope you had it fun and uh it's already like very long
[03:31:18] fun and uh it's already like very long so I apologize for that but I hope it
[03:31:19] so I apologize for that but I hope it was useful and yeah I will see you later

All frames

Total: 80. Hero frames flagged with star.

• frames/frame_0001.jpg (t=00:00) frames/frame_0002.jpg (t=01:01) frames/frame_0003.jpg (t=07:09) frames/frame_0004.jpg (t=07:15) frames/frame_0005.jpg (t=07:17) frames/frame_0006.jpg (t=07:17) frames/frame_0007.jpg (t=07:22) frames/frame_0008.jpg (t=07:22) frames/frame_0009.jpg (t=07:22) frames/frame_0010.jpg (t=07:23) frames/frame_0011.jpg (t=07:23) frames/frame_0012.jpg (t=07:23) frames/frame_0013.jpg (t=07:24) frames/frame_0014.jpg (t=07:24) frames/frame_0015.jpg (t=07:24) frames/frame_0016.jpg (t=07:24)
• frames/frame_0017.jpg (t=07:26) frames/frame_0018.jpg (t=07:27) frames/frame_0019.jpg (t=07:27) frames/frame_0020.jpg (t=07:27) frames/frame_0021.jpg (t=07:27) frames/frame_0022.jpg (t=07:27) frames/frame_0023.jpg (t=07:27) frames/frame_0024.jpg (t=07:28) frames/frame_0025.jpg (t=07:28) frames/frame_0026.jpg (t=07:41) frames/frame_0027.jpg (t=07:42) frames/frame_0028.jpg (t=07:43) frames/frame_0029.jpg (t=07:43) frames/frame_0030.jpg (t=07:43) frames/frame_0031.jpg (t=07:43) frames/frame_0032.jpg (t=07:44)
• frames/frame_0033.jpg (t=07:44) frames/frame_0034.jpg (t=07:44) frames/frame_0035.jpg (t=07:44) frames/frame_0036.jpg (t=07:45) frames/frame_0037.jpg (t=07:48) frames/frame_0038.jpg (t=08:46) frames/frame_0039.jpg (t=08:49) frames/frame_0040.jpg (t=08:53) frames/frame_0041.jpg (t=10:09) frames/frame_0042.jpg (t=10:34) frames/frame_0043.jpg (t=10:42) frames/frame_0044.jpg (t=10:42) frames/frame_0045.jpg (t=10:43) frames/frame_0046.jpg (t=12:02) frames/frame_0047.jpg (t=12:02) frames/frame_0048.jpg (t=12:05)
• frames/frame_0049.jpg (t=12:05) frames/frame_0050.jpg (t=12:20) frames/frame_0051.jpg (t=13:54) frames/frame_0052.jpg (t=13:57) frames/frame_0053.jpg (t=14:27) frames/frame_0054.jpg (t=14:37) frames/frame_0055.jpg (t=14:42) frames/frame_0056.jpg (t=14:53) frames/frame_0057.jpg (t=15:19) frames/frame_0058.jpg (t=15:27) frames/frame_0059.jpg (t=16:28) frames/frame_0060.jpg (t=16:39) frames/frame_0061.jpg (t=16:54) frames/frame_0062.jpg (t=18:13) frames/frame_0063.jpg (t=18:15) frames/frame_0064.jpg (t=19:27) frames/frame_0065.jpg (t=19:44) frames/frame_0066.jpg (t=20:07) frames/frame_0067.jpg (t=20:11) frames/frame_0068.jpg (t=34:53) frames/frame_0069.jpg (t=35:08) frames/frame_0070.jpg (t=35:18) frames/frame_0071.jpg (t=35:39) frames/frame_0072.jpg (t=35:41) frames/frame_0073.jpg (t=37:21) frames/frame_0074.jpg (t=37:23) frames/frame_0075.jpg (t=39:31) frames/frame_0076.jpg (t=39:32) frames/frame_0077.jpg (t=39:34) frames/frame_0078.jpg (t=39:46) frames/frame_0079.jpg (t=40:02)
• frames/frame_0080.jpg (t=40:04)