How AI Agent Memory Works: MemGPT & Memory Types | Stanford CS329Z Lecture 7 with Michael Ryan

2026-11-15 ยท Stanford Online ยท lecture ยท 01:18:54 ยท watch on YouTube โ†—

ai-agentsagent-memorycontext-managementmemgptragknowledge-graphsagent-skillsbenchmarkseducation

Verdict: A well-organised survey of agent memory, from the four memory types through MemGPT and sleep-time compute to benchmarks; worth it for the taxonomy, the file-system-first advice and the MemoryArena result that memory agents still score below long-context and RAG baselines.

TL;DR

  • Michael Ryan's lecture for Stanford CS329Z (Engineering AI Agents, recorded 14 October 2026) sorts agent memory into four types borrowed from the CoALA paper. Working memory is the context window, episodic memory holds raw logs of past trajectories and conversations, semantic memory holds facts about the user and the world, and procedural memory holds skills. Memory can live in the weights (parametric) or be retrieved into the context, and Ryan's rule of thumb is context for small sets of facts and fine-tuning or RL for skills.
  • Storage comes in three forms: a file system the agent searches with bash, a database (vector search, SQLite, BM25), or a knowledge graph such as Zep. Ryan's own opinion is that if you can get away with it, the file system is a good default, since the lesson from agents is to set up the simplest thing and let the model run the show; the cost is latency.
  • Four architecture patterns make up most of the lecture: learning procedures from experience (Voyager's skill library, Agent Workflow Memory, skill files), a tiered memory hierarchy (MemGPT gives the model tools to manage its own memory; MemoryOS), reflective consolidation (Generative Agents), and memory maintenance (sleep-time compute, A-Mem, Mem0).
  • The benchmark section ends with MemoryArena's group travel planning task, where the memory agents scored worst against long-context and RAG baselines. LoCoMo shows that models given the right memories can't always use them.
  • Open problems: deciding what to remember before you know what will matter, memory as an attack and privacy surface (poisoned procedural memories, a work report that mentions lunch with a recruiter), and over-personalisation.

Full summary

Why agents need memory

Ryan opens with two failures. An agent named Luna, which was running a company, had written itself an employee handbook with a policy for staff who arrive late; when an employee said they would be 45 minutes late, Luna replied that it was no problem, and according to the people running it this happened many times without a single warning. In the second example a user asked an agent to check an inbox and suggest what to archive and delete. The inbox was so long that the context was compacted, the agent forgot the "suggest" part, and it started deleting the emails. Ryan concludes that memory is essential for long-running, multi-session agents.

Without memory an agent loses procedure (you re-explain every session how to launch your GPU jobs), knowledge such as the preference "always ask me before deleting my emails", and progress, because a long conversation drifts away from its original goals. Memory also brings benefits: personalisation, learning from experience, and statefulness, so you can say "pick up where we left off on the proposal". In PersonaMem a user who loved Italian food later says they are now allergic to gluten and exploring Mediterranean cuisine, so a recipe request should not return pasta. Agent Workflow Memory learns how to find a place by name and reuses that when planning a route. A cross-session retrieval paper answers "update me on the flight reservation we discussed last time" from a prior session.

Four types of memory

The taxonomy comes from Cognitive Architectures for Language Agents (CoALA), which builds an agent framework on analogies to human cognition.

  • Working memory is the context window: observations, goals, user input and reasoning.
  • Episodic memory holds past behaviour, typically raw logs of trajectories, tool calls and observations, or in a chatbot the store of prior conversations. The Generative Agents memory stream, a timestamped log of everything Isabella Rodriguez did, is the example.
  • Semantic memory is facts about the user and the world, the kind ChatGPT's memory stores ("their daughter loves jellyfish"), and also general knowledge such as Wikipedia retrieval.
  • Procedural memory is the agent's skills: how to complete a task. The example is Anthropic's brand-guidelines skill file, which explains how to apply the brand colours to PowerPoint decks.

Two student questions come up here. Asked why keep raw episodic logs when the other types are abstractions of them, Ryan answers that the conversions are lossy: if you chose badly what to remember, the raw logs still let you search, and showing an example can teach better than describing a procedure. Ryan adds that the boundaries are fuzzy. The second student asks whether million-token context windows make memory unnecessary, and Ryan says long agent tasks reach tens or hundreds of millions of tokens, models are poor at needle-in-a-haystack retrieval over very long context, and memory is usually built on RAG anyway: "I don't see memory going away in the next 5 years."

A second distinction is where the memory lives. Parametric memory is in the weights: pre-trained knowledge is semantic, instruction following is procedural, and episodic memory, Ryan argues, doesn't really apply. Context memory is retrieved into the window by a context manager: retrieved facts, skills and conversations. When the two conflict, context usually overrides the weights unless something was trained in very deeply. On when to fine-tune, Ryan says memorising facts takes thousands of documents and is lossy because of hallucinations, so small sets of facts belong in context, while for procedural memory an RL environment can make the model much better at a task.

Where memories are stored

Three questions frame the architectures: what gets stored and retrieved, when storage and retrieval happen (agent tools on demand, a background memory worker, or system events such as a full context window), and how. Ryan covers three storage systems.

  • In a file system every memory is a file: the agent manages its own memories, folders give hierarchy for free, and grep and ls are the memory tools. A LangChain diagram puts AGENTS.md and MCP config under procedural memory, skills and knowledge files under semantic, and conversation logs under episodic.
  • The database options are a vector database for similarity search, SQLite with metadata queries, or BM25 for exact terms, much as in the earlier RAG lecture.
  • Zep builds a knowledge graph by having a model extract entities and relationships from conversations (your cousin likes your dog), cluster related entities into communities and write summaries to search over. It has three levels: episodes, entities and communities. Graphs mostly serve semantic memory, and graph traversal moves some of the intelligence into the lookup instead of the model.

When a student asks which works best, Ryan notes that every paper argues its own system is best, then gives a personal opinion: the file system usually works well if you can get away with it, and you reach for the others when lookup speed matters.

Pattern 1: learning procedures from experience

Ryan describes the four patterns as a personal reading of the papers, not a textbook taxonomy. Voyager plays Minecraft through a JavaScript library. A curriculum proposes tasks such as mining a log or fighting a zombie, the agent writes code, runs it, and uses environment feedback (did the zombie die?) to refine it. Successful programs go into a skill library keyed by an embedding of a generated description, and later tasks such as crafting an iron pickaxe retrieve them by similarity. A student asks about near-duplicate skills; Voyager blocks exact duplicate names but can still collect several skills that do the same thing. Voyager reached diamond tools in 102 prompt iterations, and the version without the skill library never got past iron tools.

Agent Workflow Memory generalises this to web and computer-use agents. Successful trajectories are evaluated and condensed by a prompt ("Given a list of web navigation tasks, your task is to extract the common workflows") into workflows such as hovering over a main category before clicking a subcategory, which go into the system prompt between sessions. It matched, and in some cases beat, human-written workflows across shopping, Reddit and maps, and its accuracy rose after the first few examples. A student asks whether this is in-context reinforcement learning; Ryan says it is similar and far more sample-efficient, improving within 10 to 15 examples, and points to a later lecture comparing RL with GEPA.

Skill files are the familiar version: a SKILL.md whose name and description sit in the context window, with the full file read on demand. "You didn't really have to do anything custom here except for make a file and tell it about the existence of that file." The limitation of Voyager and AWM as published is that procedures go stale when the environment changes; letting the agent edit its own skill files is one way out.

Pattern 2: a tiered memory hierarchy

MemGPT's key insight is to give the model tools to manage its own memory. The context window holds read-only system instructions, a writable working context and a FIFO queue of messages. Evicted messages go to recall storage, searchable by exact string; archival storage is a vector database for long-term facts and documents. In the paper's example the user mentions a boyfriend, James, and a birthday, and the agent appends both to working context; on Valentine's Day the user says they broke up, and the agent replaces the entry. Asked about Six Flags, it searches recall storage and finds that they met there. For "who won the first Nobel Prize in physics" it pages through archival search results until it finds the answer. Ryan confirms there is no hidden entity recognition: the model chose "Six Flags" as the search term. Rewriting the context does invalidate the KV cache, Ryan concedes when a student asks.

On deep memory retrieval, MemGPT lifted GPT-4 from 32.1% to 92.5% and GPT-4 Turbo from 35.3% to 93.4%, and its accuracy held steady as more documents were retrieved, where plain retrieval degraded. MemoryOS adds short-, medium- and long-term persona tiers and promotes memories by how often they are accessed. Ryan says the bigger difference from MemGPT is that a background system decides what moves in MemoryOS, where MemGPT leaves that to the model. On how the field reports results, Ryan says there are a few standard benchmarks, mostly measuring accuracy, but no standard comparison of latency, cost and accuracy.

Pattern 3: reflective consolidation

The reflection paper from an earlier lecture showed that storing reflections on past trajectories beats storing the raw traces. Generative Agents, an HCI paper, simulates a town where Isabella plans a Valentine's Day party; perceptions go into a memory stream, and plans and reflections are written back into it. Asked what Isabella is looking forward to, the agent retrieves the reflection "Isabella is excited to be planning a Valentine's Day party", scored highest for recency, importance and relevance. In the paper's ablation, reflection was one of the key components for believable behaviour, measured with TrueSkill against human crowdworkers.

Pattern 4: memory maintenance

Ryan calls this a very active research area. Sleep-time compute uses the time an agent sits idle to reason over past context ahead of questions. A juggling word problem gets pre-solved overnight, so the next day's question is answered at once, and in a toy example from the Letta blog, four scattered facts about Chad's colour preferences are rewritten into one ranked list (grey, red, blue, then green, which Chad hates). On the AIME math benchmark it needs far fewer test-time tokens than sampling repeatedly, though the figure doesn't count the sleep-time tokens. A student asks whether it is more accurate or only faster. Ryan's answer is that it can't beat spending the same compute on the actual question, but under a fixed answer budget it can be more accurate.

A-Mem (agentic memory) gives memory management to an agent whose only job is to link, evolve and decide what to store. Mem0 runs a background workflow with an entity extractor and relationship generator that detects conflicts and resolves them with four actions: add, update, delete and no-op.

Evaluating memory

Ryan says memory evaluation should be realistic, long-context, challenging and verifiable, and covers four benchmarks.

  • LoCoMo simulates conversations between generative agents, about 16,000 tokens each over several days, edited by humans for consistency. Its finding is that even when models get the right memories, they can't always use them: retrieving 50 observations gave high recall but lower answer accuracy.
  • PersonaMem simulates a user talking to an assistant as preferences change, then quizzes the model with multiple-choice answers and distractors. Memory and RAG were essential, and Mem0's graph updating helped with changing preferences.
  • LongMemEval-v2 asks multiple-choice questions about web-agent trajectories totalling hundreds of millions of tokens. Ryan skips most of it.
  • MemoryArena tests the memory-agent-environment loop on realistic tasks. In group travel planning, travellers join one at a time with new constraints (Tracy, then Chelsea who likes luxury hotels, then Emily), and the final itinerary must satisfy everyone. Memory agents such as Mem0 scored worst against long-context and RAG baselines. The authors note that other failures play a part and conclude there is a lot of headroom.

Open problems

The future relevance of a memory is unknown when it is created; keeping everything raises retrieval, context and maintenance costs, and forgetting can discard evidence. Ryan's example is a user joking that meetings before 10 aren't legally binding without pastries: should the agent remember that? Memory is also an attack surface: AgentPoison optimises benign-looking triggers that retrieve poisoned procedural memories, so a car-driving agent asked to "drive smooth and be safe" drives dangerously. In PrivacyLens, an agent asked to send a boss a weekly report mentions a lunch with a recruiter from a competing company. Finally, models are over-eager to use what's in context. Asked about young voters in elections, one model tied its answer to the user's goal of small healthy swaps. The fix could be better retrieval or training models to ignore irrelevant context, and neither is fully solved; some users also want more personalisation than others.

Key moments

  • [00:22] Two failures โ€” the Luna agent ignores its own lateness policy, and a compacted context turns "suggest what to delete" into deleting.
  • [06:48] Four memory types โ€” CoALA's working, episodic, semantic and procedural memory. frame_0004.jpg, frame_0005.jpg, frame_0006.jpg
  • [13:00] Is memory still needed? โ€” long agent runs reach hundreds of millions of tokens, and memory is usually RAG anyway. frame_0008.jpg
  • [17:00] Parametric vs context memory โ€” weights for skills via fine-tuning or RL, context for small sets of facts. frame_0009.jpg
  • [22:40] Storage options โ€” file system, databases, Zep's knowledge graph. frame_0013.jpg, frame_0041.jpg
  • [27:40] File system first โ€” Ryan's opinion: the simplest setup, with the model running the show. frame_0015.jpg
  • [31:25] Voyager โ€” a skill library that reaches diamond tools in 102 prompt iterations. frame_0016.jpg, frame_0017.jpg
  • [43:58] MemGPT โ€” the model manages working context, recall and archival storage with tools. frame_0023.jpg, frame_0024.jpg, frame_0026.jpg
  • [58:08] Sleep-time compute โ€” reasoning over memories while the agent is idle. frame_0045.jpg, frame_0046.jpg
  • [01:12:00] MemoryArena โ€” memory agents score worst on group travel planning. frame_0049.jpg

Hook microscope (0-10s)

  • Frames: 20 at 2 fps

    Read these hook frames with the Read tool:

    • hook_frames/frame_0001.jpg (t=0.0s)
    • hook_frames/frame_0002.jpg (t=0.5s)
    • hook_frames/frame_0003.jpg (t=1.0s)
    • hook_frames/frame_0004.jpg (t=1.5s)
    • hook_frames/frame_0005.jpg (t=2.0s)
    • hook_frames/frame_0006.jpg (t=2.5s)
    • hook_frames/frame_0007.jpg (t=3.0s)
    • hook_frames/frame_0008.jpg (t=3.5s)
    • hook_frames/frame_0009.jpg (t=4.0s)
    • hook_frames/frame_0010.jpg (t=4.5s)
    • hook_frames/frame_0011.jpg (t=5.0s)
    • hook_frames/frame_0012.jpg (t=5.5s)
    • hook_frames/frame_0013.jpg (t=6.0s)
    • hook_frames/frame_0014.jpg (t=6.5s)
    • hook_frames/frame_0015.jpg (t=7.0s)
    • hook_frames/frame_0016.jpg (t=7.5s)
    • hook_frames/frame_0017.jpg (t=8.0s)
    • hook_frames/frame_0018.jpg (t=8.5s)
    • hook_frames/frame_0019.jpg (t=9.0s)
    • hook_frames/frame_0020.jpg (t=9.5s)
  • Word-level transcript (12 words):

  [  0.00s] Today's
  [  5.60s] lecture
  [  6.00s] is
  [  6.22s] on
  [  6.44s] agent
  [  7.00s] memory
  [  7.31s] architectures,
  [  8.61s] which
  [  9.99s] should
  [  9.99s] be
  [  9.99s] an
  [  9.99s] extension
  • 0.0s: White Stanford Engineering card, held without speech for about five seconds.
  • by 5.0s: It fades through grey to the title slide "CS329Z: Engineering AI Agents / Agent Memory Architectures / Diyi Yang, Michael Ryan, John Yang", as "lecture is on agent memory architectures" is said.
  • by 8.0s: Cut to a wide shot of Ryan at the lectern, with the projected slide at the edge of the frame.

Pattern: no hook. It is an institutional cold open (logo, title card, then straight into the lecture); the first concrete hook is the Luna example at 0:22.

Editorial profile

  • Shots: 35
  • Cuts/min: 0.44
  • Mean shot length: 135.26s
  • Median shot length: 128.09s
  • Talking-head ratio: n/a (opencv not installed)

Recorded classroom lecture: a static wide camera on the speaker at a lectern, cut to full-screen slides, with long holds (mean shot 135 s). The slides are dense and paper-heavy, with figures and tables credited to arXiv, and students' questions are captured only on the room audio.

Quotable moments

  • [13:38] "I would actually not distinguish memory from rag."
  • [14:38] "I don't see memory going away in the next 5 years."
  • [28:05] "just kind of setting up the simplest thing and letting the model really run the show is a very good way to run things"
  • [42:59] "you didn't really have to do anything custom here except for make a file and tell it about the existence of that file"
  • [01:08:56] "even when the models get the right memories they can't always use them"

Entities mentioned

  • People: michael-ryan, Diyi Yang, John Yang
  • Companies: Stanford University, Anthropic, OpenAI, LangChain, Letta, Zep
  • Tools / products: memgpt, MemoryOS, Voyager, Agent Workflow Memory, Generative Agents, A-Mem, Mem0, ChatGPT memory, Minecraft, SQLite, BM25, GEPA, AgentPoison; benchmarks LoCoMo, PersonaMem, LongMemEval-v2, MemoryArena, PrivacyLens, AIME

Concepts surfaced

  • agent-memory-types: working (the context window), episodic (raw logs of past trajectories), semantic (facts about the user and the world) and procedural (skills), from the CoALA paper.
  • params-vs-context-memory: memory in the weights versus memory retrieved into the context; context usually wins a conflict, and fine-tuning pays off for skills more than for facts.
  • sleep-time-compute: an idle agent reasons over past context and rewrites its memories into a form that is easier to reason over before anyone asks.
  • Tiered memory hierarchy: a small in-context working area backed by searchable recall and archival stores, with the model (MemGPT) or a background system (MemoryOS) moving memories between tiers.
  • Skill library: successful programs or workflows saved for reuse on new tasks, retrieved by embedding similarity in Voyager, placed in the system prompt by Agent Workflow Memory, or read on demand from an agent-skill-file.
  • Reflective consolidation: store reflections on trajectories alongside, or instead of, the raw traces.
  • Memory poisoning: crafted triggers that retrieve malicious memories and change agent behaviour (AgentPoison).
  • Over-personalisation: a model using retrieved user memories where they don't belong.

Frame timeline

What's said while each frame is on screen.

[00:00]

frame at 00:00

Today's lecture is on agent memory architectures which uh should be an exciting subject because many people have been asking me about agent memory and I think a few people are working on projects related to it. So uh all right so we're going to talk about agent memory today. Uh so I have some motivating examples here. uh first comes from this uh Anden Labs uh Luna agent uh which was running a company and had written all these policies about how to handle different situations. And so you can see they have like an employee handbook that this agent had generated about what to do when an employee is late. Uh and then you can see when it's actually deployed uh this is somewhere in the memory of...

[04:19]

frame at 04:19

focus of what your project is about. But there are also of course benefits of agent memory uh besides solving those problems. So one is personalization. Uh agents will learn your personal preferences. Um learning from experience. I guess some of these are a bit parallel, but uh when agents learn skills, they can hopefully complete tasks more efficiently and effectively and as you expect them to in the future. Uh and statefulness is another one where uh you can start up a conversation and say like pick up where we left off on the proposal, stuff like that. And the agent should be able to figure out what you mean by that. Um so some examples, uh this comes from like a persona mem benchmark,...

[06:48]

frame at 06:48

understood that memory is important for agents. So let's talk in about some of the types of memory uh that agents have. Um so we're going to be pulling from this kind of pretty important paper that a lot of people have built on which is uh cognitive architectures for language agents uh or Koala uh as as the abbreviation goes. Um so in this paper they tried to architect um a framework for sort of like an agent harness type thing but based on uh analogies to human cognition. Uh so they have these uh blocks of memory within the agent. Uh you can see this is uh similar to what we've been talking about about this loop of taking actions on the environment and processing observations but they...

[10:41]

frame at 10:41

instance that could be considered a form of semantic memory. Uh lastly we have procedural memory. Uh and this is things like the set of rules that can be applied on top of the working memory in order to determine the agents behavior. But that's maybe a complicated way of saying that. I took that from the the paper. The simple way of putting it is like these are the skills of the agent. And this is the knowledge of how to complete a task. And so as an example of this, I provided a skill file. Um so this is something that you could store in a markdown. And we're saying uh and this comes from anthropic. So of course it's these anthropic brand style guidelines. And this skill is telling you uh...

[14:47]

frame at 14:47

>> These like good like buckets for a general agent to have with like different I guess files with different pieces of text with memory. I'm wondering like have people researched or tried creating like new buckets for very specific types of agents? >> Oh, uh good question. So you're saying like uh episodic memory, procedural memory, these are good general patterns, but have people come up with like specific memory systems for specific applications? Um yeah, the answer is yes. Uh off the top of my head, I I I don't have a name of a system to give you, but uh there's like we're going to talk about knowledge graphs in a bit as a way of storing memories. And so uh we're going to...

[17:02]

frame at 17:02

move on to uh last two slides on this topic. Uh parametric versus context memory. So this is another distinction to make and this is a somewhat AI specific um uh thing to think about. But the difference between parametric and context memory is where is the information memorized. Um and so the in parametric memory this is built into the model weights. Um so for example semantic uh parametric memory is what knowledge has been pre-trained into the model uh because it was you know trained on the entire internet for a lot of these models. there's a lot of facts that are memorized and so semantic knowledge is actually built into the model weights. Um, and procedural knowledge can be things like...

[20:48]

frame at 20:48

this is a little bit more explained a little bit more in depth. Now um these types of memories we have episodic, semantic, procedural um and you can think of them as storing experience, storing knowledge and storing skills. uh and there are some example papers and we will talk about both of these papers in a moment. Uh so any any questions about the types of memory before I dive into some architectures. Cool. Then let's dive into this section which will be the bulk of our lecture today and hopefully uh we can all learn some interesting stuff because there's a lot of memory architectures out there. So I tried to be selective about a few pretty good examples. So we'll see um some key...

[24:12]

frame at 24:12

talking about what are the colors and then how do you use them and stuff, so blurs the lines there. But you can see we have knowledge stored in files. And then episodic memory: you can have a directory where you store the literal conversation traces that you've had in the past. Um, so this is one way. Another system is to use databases. Uh, and so we can use a vector database or, you know, SQLite or some whatever your favorite database is. If you've worked with databases before, uh, it could also be used for agent memory. Um, and so, uh, in the instance of a vector database, you can do some sort of vector search, uh, for, you know, similarity between what you're looking for. Um for SQLite...

[28:36]

frame at 28:36

like full text indexes of like all of the history of like your conversation or like is it more embedded? Uh the question is is it common to create like uh knowledge graph based indexes over your conversations versus like embedding the conversation? >> Uh oh. Oh, sorry. A full text index. Is that what you said? >> Yeah. Yeah. Oh. Um so it depends. I'll talk about MEGBT in a bit which does do like an exact text lookup. Um I think like you these are all design decisions. uh in the MEMGBT architecture in particular, they embed documents for searching, but conversations remain as full text and you can like look up the exact strings in the text. Um I think it's it's it's somewhat...

[31:25]

frame at 31:25

these are good useful patterns for analyzing the ways that people implement agent memory. So the first one uh and we're going to talk about all of these in depth uh but the first one is learning procedures from experience. The second one is a tiered memory hierarchy. The third one is like reflective memory consolidation and the last one is memory maintenance. Um so the first pattern is learning procedures from experience and we will talk about a famous paper that kind of uh pioneered some of this which is Voyager. Um, so, uh, some of you may be familiar with Voyager, but the idea behind Voyager is it was a Minecraft playing agent. Um, and so the idea, uh, was kind of trying to create a...

[34:37]

frame at 34:37

skills to um now craft this iron pickaxe. So this is a this idea of procedural memory. We're writing programs to accomplish skills. When we successfully accomplish the skill, it gets committed to memory so that we can retrieve it later and add it back into the context. Um yes question >> that query is not going to be exactly identical to the key that was random generated before, right? So is it going to be some sort of like clip similarity search? >> Yes. So the question is like the query is not an exact match of the key. Um so is there some sort of clip similarity search? Yeah. So we're we're doing this is why we do the embedding. So we we do a similarity score between the...

[38:55]

frame at 38:55

hover over the main category and click on the subcategory. So you if you're envisioning a website right now, there might be like a hover over drop-own menu, right? And so the agent learned from prior experience that in order to browse a subcategory, I first need to hover over the main category and then click on the subcategory. So it's things like that that you learn by experience. Uh and then like I said, these are included back into the system prompt. And so, some results on this, uh, is just kind of showing that, like, on all of these different domains they tested it on shopping, Reddit, uh, maps, uh it was able to learn better, uh, and, uh, actually comparable, but in some cases better...

[40:21]

frame at 40:21

examples. It's getting and reinforcement learning you're going to need a lot more than that. Uh but we'll talk in two lectures on optimization and we're going to compare reinforcement learning with textbased optimization with this paper called Jeepa. Um and so that will return to your question. But yeah, this is a very good question. It is kind of a similar setting to reinforcement learning. Yeah. >> Uh yes question. when those memory being kicked in like F into the context or while the agent is running is it provided by the model say like a tool call say I want to load this or it is provided by the engine harnesses >> This is a good question so. in this, in in some instances...

[43:57]

frame at 43:57

dynamic environments. But, uh, it's just worth calling out as a limitation. Um, but skill files, uh, you could have the agent modify its own skill files, so this is maybe one way of resolving that tension. Cool u. The next pattern I want to talk about is tiered memory hierarchy. Um, and as promised, I'm going to be talking about the uh MEM GPT paper. Uh, so let's break down what's going on here. So the key insight in the mem GPT paper is to give the LLM tools to manage its own memory. Uh, and so the way we are going to break this down is we have these uh this prompt here, the context window. Oops, I wanted to uh talk about it first. Okay, so we have the context window here which has the...

[48:11]

frame at 48:11

search the working memory of course because that's all in context. So the the you're relying on the LLM to search over that. Uh, any questions about kind of how mem works? >> Uh, yeah. The question is like is there a named entity recognition uh set that's hidden? Uh, to my knowledge no actually uh you can use something like BM25 or like exact string matching. So uh you the entity matching entity recognition here is all done by the LM. The LM decided that six flags was the entity it wanted to look up and then it did an exact text match search over the recall storage. So you don't have to do any uh yeah any uh yes >> uh constantly rewriting your context and that you know >>...

[50:12]

frame at 50:12

>> Yeah, good question. Um so some results on MGBT. Uh so you can see they tested several different models and found that um once you added the memory uh it was able to do better on this deep memory retrieval benchmark. Um, and interestingly, uh, and importantly, when you typically retrieve more, you know, documents, uh, using RAG and stuff like that, it's degrades the context for the model. But by letting the model use tools to memor to run its own memory uh I mean, you can argue if this is a little bit of a confusing figure or not but they don't like specify a number of documents retrieve because they just let the tools do the decisions and stuff. And so you can see that MEGPT gets...

[54:06]

frame at 54:06

whereas in MEGBT was a lot simpler. It was like the LLM itself says this goes to archival storage. Um I guess the recall storage was like based on just the FIFO Q. Uh but things like decisions about what to store to long-term memory and how to manipulate its own context were all decided by the model. Whereas here it's like this be behind the scenes system that's making some of those decisions. Good question. Uh any other questions? Uh cool. Then let's move on to pattern three which is reflective memory consolidation. Um, so a quick refresher, but we're not going to talk about it for very long, is reflection. Uh, so if you remember, uh, we talked about this paper now twice, so I I I'll go...

[58:07]

frame at 58:07

skill rank, which was like a measure of how realistic the simulation was. Um, so any questions about that? Otherwise, I'll start talking about memory maintenance, which is the last pattern I wanted to dive into today. >> Cool. Then let's talk about memory maintenance. I think this is actually like a very active research area and and this one I think is pretty particularly interesting. So the first paper I want to talk about uh is sleeptime compute and I'm I'm at least a big fan of the idea of this. Um I I think uh you can debate if it's it's fair to call it sleeping. I think some people criticize it based on that. But uh let's talk about the method. So uh here we have a uh raw...

[1:02:14]

frame at 1:02:14

like offline sort of reasoning over the the context. Uh any questions about this idea of like sleep time compute and memory consolidation? Yes. >> Is it actually like more accurate at the end of the day or is it just like a quicker way to get the answer? >> That's a good question. The question is is it more accurate or is it just quicker? Um I think that in it depends. I mean, you can even see from this figure: there's a few cases where, uh, it does outperform, and this looks like it's coming to level off. But I mean, I imagine it c it can't really be more accurate because the gold thing to do would be to ask the question itself and then reason over that for a really long...

[1:03:52]

frame at 1:03:52

just reasoning, but you could imagine some sort of sleeptime agent that can do things. Uh, it becomes sort of like a safety question then, like what is your agent doing behind the scenes when you're not watching? Uh, but yeah, it's a good question. Yeah, I think that's maybe the best answer I have for you. Uh, yes. I I wonder like what what constitutes like as one single unit of memory cuz uh you showing this kind of helped me reflect back to the like the previous papers you showed which I wasn't really clear like what constitutes like what piece memory how do they distinguish these things? >> Yeah. Yeah. So the question is like how do you distinguish a piece of memory? I guess um in...

[1:08:03]

frame at 1:08:03

Valentine's party and they simulate a bunch of conversations between agents and then they have these long uh like 16,000 token conversations over several days and then they ask questions about these uh me memories from these conversations. So, it's like uh in the conversation they said last night we celebrated my daughter's birthday. Whose birthday did they celebrate? It's their daughter. And they have all of these different tasks. Um, so like I said, they use the generative agents architecture. They simulate agents. They have events that go on in their lives and they have these long running sessions. Uh, and then later they have a human come in and edit it and make sure everything's...

[1:11:59]

frame at 1:11:59

um the like what if we want to test agents on solving realistic diverse problems rather than multiple choice and that's where memory arena comes in. And in this benchmark uh they want to evaluate the memory agent environment loop. Uh and so essentially as the agent takes actions, it gets feedback from the environment, it updates its memory, and then it continues taking actions. And so they want to benchmark agents on this. Uh so uh we're going to look specifically at this group travel planning task. They have four different environments but in the group travel planning setting what they do is they want to plan a group travel and eat one at a time more people come in with more constraints...

[1:13:52]

frame at 1:13:52

verifiable. Uh and so these are some examples. Uh sorry, I know I went through that section quickly. Do we have a quick question before I wrap us up on on challenges and open problems? >> Yes. >> Question. I feel a lot this is memory harness like split memory out of the harness. I wonder is that necessary? >> Uh yeah, I guess the question is like is it necessary to split memory out of the harness? Is that what you're saying? >> Yeah. I mean I guess I guess the memory is kind of a part of the harness like we have to all the tools to read and write and everything are built into the harness. But if you're asking about like if we can just do this with long context, uh...

[1:18:46]

frame at 1:18:46

rushed a little bit at the end, but I think we got through most of the important stuff. I'll see you all on Monday for multi-agent systems.

Transcript

Source: captions.

[00:08] Today's lecture is on agent memory architectures which uh should be an
[00:10] architectures which uh should be an exciting subject because many people
[00:11] exciting subject because many people have been asking me about agent memory
[00:13] have been asking me about agent memory and I think a few people are working on
[00:15] and I think a few people are working on projects related to it. So uh all right
[00:17] projects related to it. So uh all right so we're going to talk about agent
[00:18] so we're going to talk about agent memory today. Uh so I have some
[00:20] memory today. Uh so I have some motivating examples here. uh first comes
[00:22] motivating examples here. uh first comes from this uh Anden Labs uh Luna agent
[00:26] from this uh Anden Labs uh Luna agent uh which was running a company and had
[00:29] uh which was running a company and had written all these policies about how to
[00:30] written all these policies about how to handle different situations. And so you
[00:32] handle different situations. And so you can see they have like an employee
[00:34] can see they have like an employee handbook that this agent had generated
[00:36] handbook that this agent had generated about what to do when an employee is
[00:37] about what to do when an employee is late. Uh and then you can see when it's
[00:39] late. Uh and then you can see when it's actually deployed uh this is somewhere
[00:41] actually deployed uh this is somewhere in the memory of the agent. Uh and the
[00:43] in the memory of the agent. Uh and the employee says I'm I'm running late like
[00:45] employee says I'm I'm running late like 45 minutes late or something like that.
[00:47] 45 minutes late or something like that. And the the Luna agent just says no
[00:50] And the the Luna agent just says no problem at all. just get here. And uh
[00:52] problem at all. just get here. And uh apparently they they claim that this
[00:54] apparently they they claim that this happened many many many times and it
[00:55] happened many many many times and it never gave them any warnings or any of
[00:57] never gave them any warnings or any of these things. So sometimes the memory is
[01:00] these things. So sometimes the memory is not always accessed. And a maybe comedic
[01:02] not always accessed. And a maybe comedic example uh that we've presented in
[01:04] example uh that we've presented in lecture before is somebody had written
[01:06] lecture before is somebody had written uh well I guess it wasn't very comedic
[01:08] uh well I guess it wasn't very comedic to this person but uh it said check this
[01:10] to this person but uh it said check this inbox and suggest what I should archive
[01:12] inbox and suggest what I should archive and delete. the inbox was so long it got
[01:14] and delete. the inbox was so long it got compacted and the agent forgot that they
[01:16] compacted and the agent forgot that they were supposed to check and suggest uh
[01:19] were supposed to check and suggest uh and instead just started deleting all
[01:20] and instead just started deleting all their emails. So memory is pretty
[01:22] their emails. So memory is pretty essential for longunning multi-session
[01:25] essential for longunning multi-session agents. Um
[01:26] agents. Um and today we're going to introduce some
[01:28] and today we're going to introduce some types of agent memory and a few concrete
[01:32] types of agent memory and a few concrete uh agent memory architectures for you to
[01:34] uh agent memory architectures for you to build on top of. Uh so the learning
[01:35] build on top of. Uh so the learning objectives for today are to recognize
[01:37] objectives for today are to recognize the importance of memory in
[01:39] the importance of memory in multiseession userf facing agent systems
[01:42] multiseession userf facing agent systems uh differentiate architectures for agent
[01:44] uh differentiate architectures for agent memory based on various design patterns
[01:47] memory based on various design patterns and identify some ongoing challenges
[01:49] and identify some ongoing challenges with agent memory. Um and the way we're
[01:51] with agent memory. Um and the way we're going to go about this uh this is our
[01:53] going to go about this uh this is our overview for today. Uh first I'm going
[01:55] overview for today. Uh first I'm going to start by motivating agent memory a
[01:57] to start by motivating agent memory a little bit more than just those two
[01:58] little bit more than just those two examples. Uh, we're going to review
[02:00] examples. Uh, we're going to review types of agent memory, which uh D
[02:02] types of agent memory, which uh D covered in the very first lecture
[02:04] covered in the very first lecture overview of the class, but we're going
[02:05] overview of the class, but we're going to go in a bit more depth today. Uh,
[02:07] to go in a bit more depth today. Uh, then we're going to talk for most of the
[02:09] then we're going to talk for most of the class on a few agent memory architecture
[02:11] class on a few agent memory architecture patterns uh, so I believe we have four
[02:13] patterns uh, so I believe we have four that we're going to cover uh. And then I
[02:15] that we're going to cover uh. And then I have some benchmarks that are used to
[02:16] have some benchmarks that are used to evaluate agent memory, um, commonly, and
[02:19] evaluate agent memory, um, commonly, and so I'll talk through how those were
[02:20] so I'll talk through how those were constructed and what has gone into those
[02:22] constructed and what has gone into those benchmarks. And finally, we'll conclude
[02:24] benchmarks. And finally, we'll conclude with some challenges and open problems,
[02:26] with some challenges and open problems, which maybe will also help as you're
[02:28] which maybe will also help as you're thinking about, about how you're going
[02:28] thinking about, about how you're going to implement uh memory for your
[02:30] to implement uh memory for your projects. Uh so any questions about the
[02:33] projects. Uh so any questions about the lecture uh today?
[02:40] Cool. If no questions then I will move on to motivating agent memory. Um so I'm
[02:44] on to motivating agent memory. Um so I'm going to motivate this in two ways. Uh
[02:47] going to motivate this in two ways. Uh examples of some problems without memory
[02:48] examples of some problems without memory and then some reasons why memory helps
[02:50] and then some reasons why memory helps us out. So first of all some problems
[02:52] us out. So first of all some problems when we don't have memory. Uh first of
[02:54] when we don't have memory. Uh first of all uh there's loss of these three
[02:56] all uh there's loss of these three things. there's loss of procedure. So
[02:59] things. there's loss of procedure. So every time uh you're interacting with
[03:00] every time uh you're interacting with the agent, you have to reinstruct it on
[03:03] the agent, you have to reinstruct it on what to do if you don't have memory. Uh
[03:05] what to do if you don't have memory. Uh so let's say you're working with your
[03:06] so let's say you're working with your coding agent implementing some project
[03:08] coding agent implementing some project and you uh have some script that you
[03:11] and you uh have some script that you need to run to launch your jobs on your
[03:12] need to run to launch your jobs on your GPUs for instance. Uh, if your agent has
[03:15] GPUs for instance. Uh, if your agent has memory, uh, you can kind of talk through
[03:17] memory, uh, you can kind of talk through it once, figure out how you're going to
[03:19] it once, figure out how you're going to do this launching procedure, and then
[03:21] do this launching procedure, and then the next time you can say, "Launch the
[03:22] the next time you can say, "Launch the experiments,and" and ideally it can
[03:24] experiments,and" and ideally it can remember we talked about this last time
[03:26] remember we talked about this last time how I do that, uh, and it doesn't have
[03:27] how I do that, uh, and it doesn't have to research through your codebase and
[03:29] to research through your codebase and index everything and find how to do
[03:30] index everything and find how to do that. So, if you don't have memory, you
[03:32] that. So, if you don't have memory, you lose this procedure. There's also loss
[03:34] lose this procedure. There's also loss of knowledge, um. So things like
[03:36] of knowledge, um. So things like preferences, for instance, uh, you can
[03:39] preferences, for instance, uh, you can say like, "Always ask me before deleting
[03:41] say like, "Always ask me before deleting my emails," and that's that's a form of
[03:43] my emails," and that's that's a form of preference instruction. Also like,
[03:44] preference instruction. Also like, "Always, you know, check with me before
[03:46] "Always, you know, check with me before spending any money on running
[03:47] spending any money on running experiments." And if you have these
[03:50] experiments." And if you have these sorts of rules that you expect the agent
[03:51] sorts of rules that you expect the agent to remember, there needs to be some way
[03:53] to remember, there needs to be some way that that's persisted across sessions.
[03:55] that that's persisted across sessions. Uh and then also loss of progress. Um so
[03:58] Uh and then also loss of progress. Um so uh this is also kind of tied in with
[04:00] uh this is also kind of tied in with this like uh earlier goal setting and
[04:02] this like uh earlier goal setting and things like that. But as a conversation
[04:04] things like that. But as a conversation gets longer and longer, if the agent is
[04:06] gets longer and longer, if the agent is unable to remember the start of the
[04:07] unable to remember the start of the conversation, it can lose focus of what
[04:09] conversation, it can lose focus of what it was working on. Uh lose goals, lose
[04:11] it was working on. Uh lose goals, lose instructions, uh and this is also a
[04:14] instructions, uh and this is also a problem across sessions on losing like
[04:15] problem across sessions on losing like focus of what your project is about. But
[04:18] focus of what your project is about. But there are also of course benefits of
[04:19] there are also of course benefits of agent memory uh besides solving those
[04:21] agent memory uh besides solving those problems. So one is personalization. Uh
[04:23] problems. So one is personalization. Uh agents will learn your personal
[04:24] agents will learn your personal preferences. Um learning from
[04:26] preferences. Um learning from experience. I guess some of these are a
[04:28] experience. I guess some of these are a bit parallel, but uh when agents learn
[04:31] bit parallel, but uh when agents learn skills, they can hopefully complete
[04:32] skills, they can hopefully complete tasks more efficiently and effectively
[04:34] tasks more efficiently and effectively and as you expect them to in the future.
[04:36] and as you expect them to in the future. Uh and statefulness is another one where
[04:39] Uh and statefulness is another one where uh you can start up a conversation and
[04:41] uh you can start up a conversation and say like pick up where we left off on
[04:43] say like pick up where we left off on the proposal, stuff like that. And the
[04:44] the proposal, stuff like that. And the agent should be able to figure out what
[04:46] agent should be able to figure out what you mean by that. Um so some examples,
[04:48] you mean by that. Um so some examples, uh this comes from like a persona mem
[04:51] uh this comes from like a persona mem benchmark, uh where the user is saying,
[04:53] benchmark, uh where the user is saying, I'm a big fan of Italian food. Then they
[04:55] I'm a big fan of Italian food. Then they change their preferences and later say
[04:57] change their preferences and later say like I I'm now allergic to gluten
[04:59] like I I'm now allergic to gluten unfortunately so I actually am exploring
[05:01] unfortunately so I actually am exploring Mediterranean cuisine. And then later
[05:04] Mediterranean cuisine. And then later they ask the agent or chatbot um you
[05:06] they ask the agent or chatbot um you know can you recommend a recipe? And if
[05:09] know can you recommend a recipe? And if the agent incorrectly remembers that
[05:11] the agent incorrectly remembers that they liked Italian food, it might give
[05:13] they liked Italian food, it might give them a pasta recipe. Uh but if it
[05:15] them a pasta recipe. Uh but if it remembers, you know, that they're
[05:16] remembers, you know, that they're allergic to gluten and updates those
[05:17] allergic to gluten and updates those preferences, then maybe it could give
[05:19] preferences, then maybe it could give them like a seafood recipe. Um another
[05:22] them like a seafood recipe. Um another example is learning from experience. So,
[05:24] example is learning from experience. So, we're going to talk about this paper,
[05:25] we're going to talk about this paper, but this is a paper called agent
[05:27] but this is a paper called agent workflow memory. And basically, they're
[05:29] workflow memory. And basically, they're solving all of these tasks with an
[05:31] solving all of these tasks with an agent. Um, such as like, uh, looking up
[05:34] agent. Um, such as like, uh, looking up places, getting their zip code, uh, you
[05:36] places, getting their zip code, uh, you know, planning routes between places and
[05:38] know, planning routes between places and all of this. And some of these are
[05:39] all of this. And some of these are subtasks of others. Uh, and so if the
[05:42] subtasks of others. Uh, and so if the agent learns how to find a place by its
[05:44] agent learns how to find a place by its name, it can store a memory about how to
[05:46] name, it can store a memory about how to do that. so that later when you want to
[05:48] do that. so that later when you want to route between locations, it knows I know
[05:50] route between locations, it knows I know how to find places by their name and and
[05:52] how to find places by their name and and then I can build a route between those
[05:54] then I can build a route between those places.
[05:56] places. And another example, a final example
[05:58] And another example, a final example here is uh the statefulness uh example.
[06:01] here is uh the statefulness uh example. So uh this uh comes from a paper where
[06:03] So uh this uh comes from a paper where they're looking at
[06:05] they're looking at um retrieval for a cross session memory.
[06:08] um retrieval for a cross session memory. And uh, here they say uh, the user says
[06:11] And uh, here they say uh, the user says like, "Could you update me on my flight
[06:13] like, "Could you update me on my flight reservation um that we discussed, you
[06:15] reservation um that we discussed, you know, last time?" And then it's able to
[06:17] know, last time?" And then it's able to retrieve from the prior session and
[06:19] retrieve from the prior session and answer questions about that flight,
[06:21] answer questions about that flight, whereas otherwise it would have to, you
[06:23] whereas otherwise it would have to, you know either, if if it can dig through
[06:25] know either, if if it can dig through like all of the files and try to piece
[06:28] like all of the files and try to piece that together, or, in the worst case,
[06:29] that together, or, in the worst case, you know, have to ask the user like,
[06:30] you know, have to ask the user like, "Can you tell me what we were talking
[06:32] "Can you tell me what we were talking about?" So these are kind of reasons why
[06:34] about?" So these are kind of reasons why we care about memory. Uh any questions
[06:36] we care about memory. Uh any questions about why memory is important?
[06:45] Cool. Okay. I think it's pretty commonly understood that memory is important for
[06:46] understood that memory is important for agents. So let's talk in about some of
[06:48] agents. So let's talk in about some of the types of memory uh that agents have.
[06:51] the types of memory uh that agents have. Um so we're going to be pulling from
[06:53] Um so we're going to be pulling from this kind of pretty important paper that
[06:55] this kind of pretty important paper that a lot of people have built on which is
[06:57] a lot of people have built on which is uh cognitive architectures for language
[06:59] uh cognitive architectures for language agents uh or Koala uh as as the
[07:03] agents uh or Koala uh as as the abbreviation goes. Um so in this paper
[07:06] abbreviation goes. Um so in this paper they tried to architect um a framework
[07:08] they tried to architect um a framework for sort of like an agent harness type
[07:11] for sort of like an agent harness type thing but based on
[07:12] thing but based on uh analogies to human cognition. Uh so
[07:15] uh analogies to human cognition. Uh so they have these
[07:17] they have these uh blocks of memory within the agent. Uh
[07:19] uh blocks of memory within the agent. Uh you can see this is uh similar to what
[07:22] you can see this is uh similar to what we've been talking about about this loop
[07:23] we've been talking about about this loop of taking actions on the environment and
[07:25] of taking actions on the environment and processing observations but they kind of
[07:27] processing observations but they kind of break down the internals of the agent a
[07:29] break down the internals of the agent a little bit more uh in this cognitive
[07:31] little bit more uh in this cognitive architecture. And specifically I want to
[07:33] architecture. And specifically I want to point out uh these blocks. So we have
[07:36] point out uh these blocks. So we have procedural memory um semantic memory,
[07:38] procedural memory um semantic memory, episodic memory, and working memory. Uh
[07:41] episodic memory, and working memory. Uh and now I'm going to talk about what
[07:42] and now I'm going to talk about what each of these actually are. Um
[07:45] each of these actually are. Um first of all, oh question. Yes.
[07:47] first of all, oh question. Yes. >> What's the memory here mean compared to
[07:50] >> What's the memory here mean compared to the context?
[07:51] the context? >> That's a good question. Actually, I
[07:53] >> That's a good question. Actually, I think the next slide will answer this
[07:54] think the next slide will answer this question. So the the question is just
[07:56] question. So the the question is just what is memory versus context here? And
[07:58] what is memory versus context here? And I think it's a good question. So the um
[08:01] I think it's a good question. So the um so the working memory is essentially the
[08:03] so the working memory is essentially the context in this case. So so memory in
[08:06] context in this case. So so memory in these instances are things that we're
[08:07] these instances are things that we're going to retrieve from and add to the
[08:09] going to retrieve from and add to the context. But you can think of working
[08:10] context. But you can think of working memory as the context. And so this is
[08:12] memory as the context. And so this is the first one. Um so working memory is
[08:14] the first one. Um so working memory is typically the context window. Um it's
[08:17] typically the context window. Um it's what the agent currently sees and is
[08:19] what the agent currently sees and is like reasoning over and acting on. Uh
[08:21] like reasoning over and acting on. Uh and this includes things like
[08:22] and this includes things like observations, goals, user input, uh
[08:25] observations, goals, user input, uh internal reasoning. So um that is the
[08:28] internal reasoning. So um that is the essentially you can think of the context
[08:29] essentially you can think of the context window as working memory. Uh next we
[08:32] window as working memory. Uh next we have episodic memory uh and this is
[08:35] have episodic memory uh and this is memories of the uh past behaviors. So
[08:38] memories of the uh past behaviors. So trajectories uh for instance so when the
[08:41] trajectories uh for instance so when the agent is taking actions and searching
[08:43] agent is taking actions and searching for uh let's say it's like a web agent
[08:45] for uh let's say it's like a web agent doing some shopping task or something
[08:46] doing some shopping task or something like that and it's searching for
[08:49] like that and it's searching for uh an item on Amazon and it's navigating
[08:52] uh an item on Amazon and it's navigating and comparing prices and stuff like
[08:53] and comparing prices and stuff like that. all of the memory of what it did
[08:55] that. all of the memory of what it did to complete that task, it would be
[08:57] to complete that task, it would be considered an episodic memory. Um, and
[08:59] considered an episodic memory. Um, and so in chat bots for instance, this is a
[09:01] so in chat bots for instance, this is a store of all of the prior conversations
[09:03] store of all of the prior conversations that you should be able to look up over.
[09:05] that you should be able to look up over. Um, and this is typically something like
[09:07] Um, and this is typically something like the raw logs of what occurred. Um, so
[09:10] the raw logs of what occurred. Um, so like I said, it's the tool calls, the
[09:13] like I said, it's the tool calls, the observations, all of this is episodic
[09:15] observations, all of this is episodic memory. And then this comes from a paper
[09:18] memory. And then this comes from a paper generative agents paper which I'm going
[09:20] generative agents paper which I'm going to talk about their architecture later
[09:22] to talk about their architecture later in this lecture. Uh but you can see the
[09:24] in this lecture. Uh but you can see the memory stream contains just a log of
[09:26] memory stream contains just a log of exactly what was going on in this
[09:27] exactly what was going on in this environment. Um and Isabella Rodriguez
[09:30] environment. Um and Isabella Rodriguez is the agent here because it's sort of
[09:32] is the agent here because it's sort of this like simulation environment and it
[09:34] this like simulation environment and it just describes everything that was done
[09:36] just describes everything that was done in this uh environment and that is the
[09:39] in this uh environment and that is the episodic memory.
[09:40] episodic memory. Uh any questions about that?
[09:47] Cool. Then we have semantic memory. So semantic memory is probably what most
[09:49] semantic memory is probably what most people think of when they initially
[09:51] people think of when they initially think of memory. So this is facts about
[09:53] think of memory. So this is facts about the world, facts about the uh user
[09:55] the world, facts about the uh user preferences, uh some general knowledge.
[09:58] preferences, uh some general knowledge. All of this falls under semantic memory.
[10:01] All of this falls under semantic memory. And when you have like chatgbt memory
[10:03] And when you have like chatgbt memory for instance, uh this is semantic
[10:06] for instance, uh this is semantic memory. Um so here we have like an
[10:09] memory. Um so here we have like an example of you're interacting with
[10:10] example of you're interacting with chatgbt and it's storing all these facts
[10:12] chatgbt and it's storing all these facts that it's learned about you. Uh so this
[10:14] that it's learned about you. Uh so this comes from one of the memory blog posts
[10:16] comes from one of the memory blog posts from from OpenAI, but you can see that I
[10:18] from from OpenAI, but you can see that I remembers, you know, this user has a
[10:20] remembers, you know, this user has a 2-year-old daughter. uh their daughter
[10:22] 2-year-old daughter. uh their daughter loves jellyfish. So all of these are
[10:24] loves jellyfish. So all of these are facts about the user that can help
[10:26] facts about the user that can help later. But another thing to be uh
[10:29] later. But another thing to be uh careful about is it's not just facts
[10:30] careful about is it's not just facts about the user. It's also things like
[10:32] about the user. It's also things like facts about the world. Um so when you
[10:34] facts about the world. Um so when you want to retrieve from like Wikipedia for
[10:36] want to retrieve from like Wikipedia for instance that could be considered a form
[10:38] instance that could be considered a form of semantic memory.
[10:40] of semantic memory. Uh
[10:41] Uh lastly we have procedural memory. Uh and
[10:44] lastly we have procedural memory. Uh and this is things like the set of rules
[10:45] this is things like the set of rules that can be applied on top of the
[10:47] that can be applied on top of the working memory in order to determine the
[10:49] working memory in order to determine the agents behavior. But that's maybe a
[10:51] agents behavior. But that's maybe a complicated way of saying that. I took
[10:52] complicated way of saying that. I took that from the the paper. The simple way
[10:54] that from the the paper. The simple way of putting it is like these are the
[10:56] of putting it is like these are the skills of the agent. And this is the
[10:57] skills of the agent. And this is the knowledge of how to complete a task. And
[11:00] knowledge of how to complete a task. And so as an example of this, I provided a
[11:01] so as an example of this, I provided a skill file. Um so this is something that
[11:04] skill file. Um so this is something that you could store in a markdown. And we're
[11:06] you could store in a markdown. And we're saying uh and this comes from anthropic.
[11:08] saying uh and this comes from anthropic. So of course it's these anthropic brand
[11:10] So of course it's these anthropic brand style guidelines. And this skill is
[11:12] style guidelines. And this skill is telling you uh how to apply anthropics
[11:14] telling you uh how to apply anthropics brand colors to PowerPoint
[11:16] brand colors to PowerPoint presentations. So it describes what are
[11:18] presentations. So it describes what are the brand colors? um how do you modify
[11:20] the brand colors? um how do you modify the PowerPoint to contain those um
[11:23] the PowerPoint to contain those um colors and brand features? Um and all of
[11:26] colors and brand features? Um and all of this is written into this skill file. Uh
[11:29] this is written into this skill file. Uh so when this is retrieved and put in the
[11:30] so when this is retrieved and put in the memory, the working memory of the agent,
[11:32] memory, the working memory of the agent, it now knows how to take these actions.
[11:34] it now knows how to take these actions. Um so the distinction here is like this
[11:36] Um so the distinction here is like this is telling it maybe how to do the task
[11:38] is telling it maybe how to do the task and what the skills are whereas the
[11:40] and what the skills are whereas the previous things were telling it facts
[11:41] previous things were telling it facts about the the world and the user. Uh yes
[11:44] about the the world and the user. Uh yes question
[11:45] question >> distinguish um
[11:47] >> distinguish um these three memories a bit raw because
[11:49] these three memories a bit raw because to me it's like the episodic memory is
[11:52] to me it's like the episodic memory is more like the raw memory and the other
[11:54] more like the raw memory and the other ones are like abstraction of it or
[11:57] ones are like abstraction of it or actually need episodic memory.
[11:59] actually need episodic memory. >> Uh oh. So the question is like why would
[12:01] >> Uh oh. So the question is like why would you need episodic memory? It's like the
[12:03] you need episodic memory? It's like the raw memory versus the other ones are
[12:05] raw memory versus the other ones are abstractions. I think this is a this is
[12:07] abstractions. I think this is a this is a good question. I mean
[12:09] a good question. I mean some to some extent we we'll talk about
[12:11] some to some extent we we'll talk about like how you use each of these and like
[12:13] like how you use each of these and like you are drawing some of these other
[12:15] you are drawing some of these other memories out of the episodic because
[12:16] memories out of the episodic because everything has is sourced from the prior
[12:18] everything has is sourced from the prior interactions. Um
[12:20] interactions. Um part of the reason is like you can these
[12:23] part of the reason is like you can these are lossy some of these conversions. So
[12:25] are lossy some of these conversions. So if I'm converting like a prior
[12:27] if I'm converting like a prior interaction into
[12:28] interaction into like memory specific statements, it
[12:31] like memory specific statements, it might be that like I
[12:32] might be that like I decided I I'll talk about this in our
[12:34] decided I I'll talk about this in our challenges section, but you decide what
[12:36] challenges section, but you decide what to remember. And if you have all of the
[12:38] to remember. And if you have all of the raw logs, you can at least search over
[12:39] raw logs, you can at least search over them in case it's like I made a poor
[12:41] them in case it's like I made a poor choice of what to remember. Um,
[12:44] choice of what to remember. Um, and I think sometimes it can be helpful
[12:46] and I think sometimes it can be helpful to give exact examples like procedural
[12:48] to give exact examples like procedural memory is sort of
[12:50] memory is sort of like describing how to do something in
[12:52] like describing how to do something in some ways. This is not always true. This
[12:54] some ways. This is not always true. This is not a blanket statement but
[12:55] is not a blanket statement but procedural memor is often describing how
[12:56] procedural memor is often describing how to do something but sometimes like
[12:58] to do something but sometimes like showing by example is the best way to
[13:00] showing by example is the best way to show how to do something. So that can be
[13:02] show how to do something. So that can be a reason for the distinction as well but
[13:03] a reason for the distinction as well but it is a good question and to some extent
[13:05] it is a good question and to some extent these are fuzzy boundaries. It's just
[13:07] these are fuzzy boundaries. It's just meant to be a way of thinking about it
[13:08] meant to be a way of thinking about it but good question. Yeah
[13:10] but good question. Yeah any other questions? Yes,
[13:12] any other questions? Yes, >> question. I wonder if memory is
[13:14] >> question. I wonder if memory is essential because now like
[13:17] essential because now like context window and some of them are
[13:19] context window and some of them are short memories could be in the system
[13:21] short memories could be in the system prompt and also we have rag and skills.
[13:24] prompt and also we have rag and skills. So I don't know if we still need memory.
[13:27] So I don't know if we still need memory. >> Uh yes. So the the question is like is
[13:30] >> Uh yes. So the the question is like is memory still essential now that models
[13:31] memory still essential now that models can have a million token context windows
[13:34] can have a million token context windows and and we have rag and stuff like that.
[13:37] and and we have rag and stuff like that. >> Yeah. So I would actually not
[13:38] >> Yeah. So I would actually not distinguish memory from rag. I I'll talk
[13:41] distinguish memory from rag. I I'll talk about it in a bit, but the underlying
[13:42] about it in a bit, but the underlying system for memory is often rag. Uh so
[13:47] system for memory is often rag. Uh so rag is not like a in direct contrast to
[13:50] rag is not like a in direct contrast to memory. Uh, but to your question about
[13:52] memory. Uh, but to your question about the million token context window, I
[13:54] the million token context window, I think sometimes longunning agent tasks
[13:56] think sometimes longunning agent tasks have, you know, gotten into the tens of
[13:58] have, you know, gotten into the tens of millions, hundreds of millions of tokens
[13:59] millions, hundreds of millions of tokens range, or even you know, I I don't know
[14:01] range, or even you know, I I don't know what the frontier labs are doing to
[14:03] what the frontier labs are doing to solve these math problems, but could be
[14:04] solve these math problems, but could be billions or more tokens. So I do think
[14:07] billions or more tokens. So I do think we do need to do some management here. I
[14:09] we do need to do some management here. I mean an an interesting provocative
[14:11] mean an an interesting provocative question is like can we eventually have
[14:12] question is like can we eventually have like infinite context windows of
[14:14] like infinite context windows of language models and in that case is
[14:17] language models and in that case is memory still important? Uh I think in
[14:20] memory still important? Uh I think in the foreseeable future we still need to
[14:22] the foreseeable future we still need to do memory management because me models
[14:24] do memory management because me models are not good at needle in the haststack
[14:26] are not good at needle in the haststack tasks over extremely long context. So it
[14:28] tasks over extremely long context. So it can be very helpful to have external
[14:30] can be very helpful to have external tools to identify what actually should
[14:32] tools to identify what actually should go into the context. Um but it is a good
[14:34] go into the context. Um but it is a good question. It's not certain that memory
[14:36] question. It's not certain that memory will be essential forever, but I I I
[14:38] will be essential forever, but I I I don't see memory going away in the next
[14:40] don't see memory going away in the next 5 years.
[14:41] 5 years. >> Yeah, good question. Yes.
[14:43] >> Yeah, good question. Yes. >> These like good like buckets for a
[14:45] >> These like good like buckets for a general agent to have with like
[14:47] general agent to have with like different I guess files with different
[14:49] different I guess files with different pieces of text with memory. I'm
[14:51] pieces of text with memory. I'm wondering like have people researched or
[14:54] wondering like have people researched or tried creating like new buckets for very
[14:57] tried creating like new buckets for very specific types of agents?
[14:58] specific types of agents? >> Oh, uh good question. So you're saying
[15:00] >> Oh, uh good question. So you're saying like
[15:01] like uh episodic memory, procedural memory,
[15:03] uh episodic memory, procedural memory, these are good general patterns, but
[15:05] these are good general patterns, but have people come up with like specific
[15:06] have people come up with like specific memory systems for specific
[15:08] memory systems for specific applications? Um yeah, the answer is
[15:10] applications? Um yeah, the answer is yes. Uh off the top of my head, I I I
[15:12] yes. Uh off the top of my head, I I I don't have a name of a system to give
[15:14] don't have a name of a system to give you, but uh there's like we're going to
[15:15] you, but uh there's like we're going to talk about knowledge graphs in a bit as
[15:17] talk about knowledge graphs in a bit as a way of storing memories. And so
[15:20] a way of storing memories. And so uh we're going to talk about a general
[15:21] uh we're going to talk about a general pattern for how to store the knowledge
[15:22] pattern for how to store the knowledge graphs, but there are tools for like
[15:24] graphs, but there are tools for like creating very domain specific knowledge
[15:26] creating very domain specific knowledge graphs that's like um we care about
[15:28] graphs that's like um we care about storing these specific details and this
[15:31] storing these specific details and this is how we're going to look up over the
[15:32] is how we're going to look up over the knowledge graph. So you can kind of
[15:33] knowledge graph. So you can kind of engineer the memory to be domain
[15:35] engineer the memory to be domain specific. It's a very good question.
[15:36] specific. It's a very good question. Yeah.
[15:37] Yeah. Awesome. Uh any other questions before I
[15:40] Awesome. Uh any other questions before I move on?
[15:42] move on? >> Uh yes.
[15:44] >> Uh yes. I'm curious like it seems that you're
[15:45] I'm curious like it seems that you're suggesting that there are like
[15:48] suggesting that there are like >> like the that is more shortterm
[15:51] >> like the that is more shortterm like like compared to a human being
[15:54] like like compared to a human being is like they're like shortterm long term
[15:56] is like they're like shortterm long term you just have like a different
[15:59] you just have like a different >> Uh, this is a good question about, like
[16:01] >> Uh, this is a good question about, like short, so the question is, like, is
[16:03] short, so the question is, like, is short-term memory and long-term memory,
[16:04] short-term memory and long-term memory, like, what's the distinction? Is
[16:05] like, what's the distinction? Is episodic, like, one of these versus
[16:07] episodic, like, one of these versus another? Am I understanding your
[16:09] another? Am I understanding your question correctly? Uh, so I'd say these
[16:11] question correctly? Uh, so I'd say these are somewhat distinct from short-term,
[16:13] are somewhat distinct from short-term, long-term memory. But we will talk about
[16:14] long-term memory. But we will talk about memory hierarchies uh later in this
[16:16] memory hierarchies uh later in this lecture. And that is the more direct uh
[16:19] lecture. And that is the more direct uh analogy to long-term and short-term
[16:21] analogy to long-term and short-term memory. I I suppose you could make the
[16:23] memory. I I suppose you could make the argument that like remembering exactly
[16:25] argument that like remembering exactly what happened is kind of short-term
[16:26] what happened is kind of short-term memory. And so episodic falls into that,
[16:28] memory. And so episodic falls into that, but I I don't necessarily I don't think
[16:30] but I I don't necessarily I don't think it has to be that way necessarily, but
[16:32] it has to be that way necessarily, but it's a good question. Yeah. And I think
[16:34] it's a good question. Yeah. And I think last question. Oh. Oh, yeah. Comment on
[16:36] last question. Oh. Oh, yeah. Comment on that. Yeah.
[16:36] that. Yeah. >> Episodic memory as well as long-term
[16:39] >> Episodic memory as well as long-term memory. working memory
[16:41] memory. working memory shortterm memory because the taxonomy is
[16:44] shortterm memory because the taxonomy is inspired by neuring
[16:47] inspired by neuring it based on the way
[16:48] it based on the way of the brain. So working memory is
[16:50] of the brain. So working memory is functionally shortterm memory.
[16:52] functionally shortterm memory. >> Thanks. Yeah, we have some neuro
[16:54] >> Thanks. Yeah, we have some neuro expertise in the room. So awesome. Uh
[16:56] expertise in the room. So awesome. Uh any other questions otherwise I will
[16:59] any other questions otherwise I will move on to
[17:00] move on to uh last two slides on this topic. Uh
[17:03] uh last two slides on this topic. Uh parametric versus context memory. So
[17:05] parametric versus context memory. So this is another distinction to make and
[17:07] this is another distinction to make and this is a somewhat AI specific um
[17:10] this is a somewhat AI specific um uh thing to think about. But the
[17:12] uh thing to think about. But the difference between parametric and
[17:13] difference between parametric and context memory is where is the
[17:15] context memory is where is the information memorized. Um and so
[17:18] information memorized. Um and so the in parametric memory this is built
[17:21] the in parametric memory this is built into the model weights. Um so for
[17:23] into the model weights. Um so for example semantic uh parametric memory is
[17:26] example semantic uh parametric memory is what knowledge has been pre-trained into
[17:28] what knowledge has been pre-trained into the model uh because it was you know
[17:29] the model uh because it was you know trained on the entire internet for a lot
[17:31] trained on the entire internet for a lot of these models. there's a lot of facts
[17:33] of these models. there's a lot of facts that are memorized and so semantic
[17:34] that are memorized and so semantic knowledge is actually built into the
[17:36] knowledge is actually built into the model weights. Um, and procedural
[17:38] model weights. Um, and procedural knowledge can be things like instruction
[17:39] knowledge can be things like instruction following and the model knows how to act
[17:41] following and the model knows how to act when you ask it to follow an
[17:42] when you ask it to follow an instruction. Uh, and that's baked into
[17:44] instruction. Uh, and that's baked into the weights. Uh, episodic memory. I
[17:47] the weights. Uh, episodic memory. I debated like trying to argue that
[17:48] debated like trying to argue that there's some ways of incorporating
[17:49] there's some ways of incorporating episodic memory into the weights. And I
[17:51] episodic memory into the weights. And I think if we're being pedantic, you could
[17:52] think if we're being pedantic, you could argue that it is applicable, but I'm
[17:54] argue that it is applicable, but I'm going to say that episodic memory isn't
[17:56] going to say that episodic memory isn't really something that's baked into the
[17:58] really something that's baked into the model weights. But we could debate that.
[17:59] model weights. But we could debate that. Um and then you can fine-tune models to
[18:02] Um and then you can fine-tune models to add more knowledge and skills. Uh and so
[18:04] add more knowledge and skills. Uh and so that's how you can update the parametric
[18:06] that's how you can update the parametric knowledge of the model.
[18:07] knowledge of the model. On the context side, um this is uh
[18:10] On the context side, um this is uh memory that's added to the context
[18:12] memory that's added to the context window and that is how uh so you you
[18:14] window and that is how uh so you you know retrieve it from somewhere and put
[18:15] know retrieve it from somewhere and put it into this working memory. Um and so
[18:17] it into this working memory. Um and so semantic uh context memory is like
[18:19] semantic uh context memory is like retrieved facts and we have some
[18:21] retrieved facts and we have some database of all of the facts that we
[18:23] database of all of the facts that we care about and things that we need the
[18:24] care about and things that we need the model to know and we can retrieve those.
[18:26] model to know and we can retrieve those. Uh procedural is like the retrieved
[18:28] Uh procedural is like the retrieved skills and how to take actions and we
[18:30] skills and how to take actions and we can put that in the context. Um episodic
[18:32] can put that in the context. Um episodic memory is like retrieving past
[18:34] memory is like retrieving past conversations and this is all managed by
[18:36] conversations and this is all managed by like a context manager that can put this
[18:37] like a context manager that can put this into uh the context window. We have a
[18:39] into uh the context window. We have a question.
[18:40] question. >> What happens when these two memories?
[18:43] >> What happens when these two memories? >> Good question. The question is what
[18:44] >> Good question. The question is what happens when these memories conflict? Um
[18:47] happens when these memories conflict? Um so
[18:48] so the this is like there's papers studying
[18:50] the this is like there's papers studying these problems and it's not like uh
[18:52] these problems and it's not like uh guaranteed. It depends on how strong for
[18:55] guaranteed. It depends on how strong for example the parametric memory is. Uh I
[18:58] example the parametric memory is. Uh I think typically a general rule is like
[19:01] think typically a general rule is like context can override
[19:03] context can override uh
[19:04] uh parameters. However uh if something is
[19:07] parameters. However uh if something is very very deeply fine-tuned into the
[19:09] very very deeply fine-tuned into the model uh that may not be the case. So
[19:11] model uh that may not be the case. So when it's in conflict uh it's kind of a
[19:14] when it's in conflict uh it's kind of a it's a question of what will happen.
[19:16] it's a question of what will happen. It's a good and it's a good question but
[19:17] It's a good and it's a good question but I think typically the context overrides
[19:20] I think typically the context overrides uh parametric. Yeah good question. Uh
[19:22] uh parametric. Yeah good question. Uh any other questions? Yes.
[19:25] any other questions? Yes. >> What like at what point does it become
[19:27] >> What like at what point does it become ideal to use parametric memory because
[19:30] ideal to use parametric memory because it it seems to me like you have to have
[19:32] it it seems to me like you have to have a pretty large.
[19:34] a pretty large. >> Yeah. The question is at what point
[19:35] >> Yeah. The question is at what point should you use parametric memory instead
[19:36] should you use parametric memory instead of context memory? Uh I think uh for
[19:40] of context memory? Uh I think uh for semantic knowledge it's
[19:43] semantic knowledge it's typically useful to do like context
[19:45] typically useful to do like context things for if you have like small sets
[19:47] things for if you have like small sets of facts and stuff like that because you
[19:48] of facts and stuff like that because you have need so much data for the model to
[19:50] have need so much data for the model to actually memorize things. Um for
[19:52] actually memorize things. Um for procedural memory this is actually where
[19:54] procedural memory this is actually where it flips a bit because fine-tuning a
[19:56] it flips a bit because fine-tuning a model to solve a task can make it very
[19:58] model to solve a task can make it very like a lot better at doing that task. So
[20:00] like a lot better at doing that task. So if you have like an RL environment that
[20:02] if you have like an RL environment that you can set up or something like that,
[20:03] you can set up or something like that, then it might actually help to RL your
[20:06] then it might actually help to RL your underlying model uh to become better at
[20:08] underlying model uh to become better at that task and then you're trying to
[20:09] that task and then you're trying to update its procedural memory and stuff
[20:10] update its procedural memory and stuff like that. So I think it depends on like
[20:13] like that. So I think it depends on like what setting you're operating in. Uh to
[20:15] what setting you're operating in. Uh to actually fine-tune a model to start
[20:16] actually fine-tune a model to start memorizing facts, you need thousands or
[20:18] memorizing facts, you need thousands or more documents. So this is like and and
[20:20] more documents. So this is like and and and it's also lossy because of
[20:22] and it's also lossy because of hallucinations and stuff like that. So
[20:24] hallucinations and stuff like that. So most of the time if you only have a
[20:26] most of the time if you only have a little bit of data, you'll use context
[20:27] little bit of data, you'll use context for semantic memory. Um but if you have
[20:30] for semantic memory. Um but if you have like the right environments uh updating
[20:31] like the right environments uh updating the weights for procedural is actually a
[20:33] the weights for procedural is actually a good idea. Good question. Uh any other
[20:36] good idea. Good question. Uh any other questions?
[20:38] questions? Uh then I will wrap up with the final
[20:40] Uh then I will wrap up with the final recap slide here. Um this actually comes
[20:43] recap slide here. Um this actually comes from the first lecture but hopefully
[20:45] from the first lecture but hopefully this is a little bit more explained a
[20:47] this is a little bit more explained a little bit more in depth. Now um these
[20:49] little bit more in depth. Now um these types of memories we have episodic,
[20:51] types of memories we have episodic, semantic, procedural um and you can
[20:53] semantic, procedural um and you can think of them as storing experience,
[20:55] think of them as storing experience, storing knowledge and storing skills. uh
[20:58] storing knowledge and storing skills. uh and there are some example papers and we
[20:59] and there are some example papers and we will talk about both of these papers in
[21:01] will talk about both of these papers in a moment. Uh so any any questions about
[21:03] a moment. Uh so any any questions about the types of memory before I dive into
[21:05] the types of memory before I dive into some architectures.
[21:08] some architectures. Cool. Then let's dive into this section
[21:10] Cool. Then let's dive into this section which will be the bulk of our lecture
[21:11] which will be the bulk of our lecture today and hopefully uh we can all learn
[21:13] today and hopefully uh we can all learn some interesting stuff because there's a
[21:15] some interesting stuff because there's a lot of memory architectures out there.
[21:16] lot of memory architectures out there. So I tried to be selective about a few
[21:18] So I tried to be selective about a few pretty good examples. So we'll see
[21:21] pretty good examples. So we'll see um some key questions that will motivate
[21:22] um some key questions that will motivate us as we talk about these architectures.
[21:24] us as we talk about these architectures. Um we have three things you know what,
[21:26] Um we have three things you know what, when and how. Um so what will get stored
[21:29] when and how. Um so what will get stored to and retrieved from memory? When will
[21:31] to and retrieved from memory? When will we trigger the memory storage and
[21:33] we trigger the memory storage and retrieval? Uh how will we store memories
[21:35] retrieval? Uh how will we store memories and retrieve from memory? Um and so I
[21:38] and retrieve from memory? Um and so I have some example answers here. Uh but
[21:40] have some example answers here. Uh but this will vary from from system to
[21:41] this will vary from from system to system. So what will get stored? This is
[21:43] system. So what will get stored? This is the things we just talked about the the
[21:45] the things we just talked about the the types of memory. So facts, procedures,
[21:48] types of memory. So facts, procedures, conversations. Uh this will get stored
[21:50] conversations. Uh this will get stored to memory. Um when
[21:52] to memory. Um when uh well we can have agent tools enable
[21:55] uh well we can have agent tools enable the agent to access them on demand. Uh
[21:57] the agent to access them on demand. Uh we can have a background memory worker
[21:59] we can have a background memory worker that is observing the conversation and
[22:01] that is observing the conversation and deciding what gets stored to and
[22:02] deciding what gets stored to and retrieved from memory. It can also be
[22:04] retrieved from memory. It can also be event driven. So uh we'll talk about an
[22:06] event driven. So uh we'll talk about an architecture called memgbt in a bit. Uh
[22:08] architecture called memgbt in a bit. Uh but they use like they they treat the
[22:10] but they use like they they treat the language model as like almost an
[22:11] language model as like almost an operating system. And so in by their
[22:13] operating system. And so in by their analogy they have system events that
[22:15] analogy they have system events that trigger memory action. So things like
[22:17] trigger memory action. So things like the context window is getting too full
[22:19] the context window is getting too full triggers a memory action. Uh or like the
[22:21] triggers a memory action. Uh or like the user requests something and triggers a
[22:23] user requests something and triggers a memory action. Um and then how will we
[22:26] memory action. Um and then how will we store it? Um well three different uh
[22:29] store it? Um well three different uh things that we're going to talk about.
[22:30] things that we're going to talk about. There are more ways to do it than this
[22:31] There are more ways to do it than this but these are the most common in my
[22:33] but these are the most common in my research that I've uh come across. Uh
[22:35] research that I've uh come across. Uh using the file system uh using databases
[22:39] using the file system uh using databases uh and using knowledge graphs. Uh so
[22:41] uh and using knowledge graphs. Uh so let's talk a little bit about the how
[22:42] let's talk a little bit about the how first and then I'm going to talk about
[22:44] first and then I'm going to talk about specific architectures.
[22:46] specific architectures. So,
[22:47] So, um, how...uh, we have three systems,
[22:49] um, how...uh, we have three systems, like, I mean, like, uh, CES system
[22:52] like, I mean, like, uh, CES system definition, you know, for...for, uh,
[22:54] definition, you know, for...for, uh, memory
[22:55] memory um. So, underlying memory uh, we need
[22:58] um. So, underlying memory uh, we need some way to get and store information to
[23:00] some way to get and store information to your disk uh, or, you know, if your
[23:02] your disk uh, or, you know, if your agent's operating in the cloud, to some
[23:03] agent's operating in the cloud, to some disk it needs to be written somewhere
[23:05] disk it needs to be written somewhere um. And so, we have three most common
[23:07] um. And so, we have three most common approaches, which are the file system,
[23:09] approaches, which are the file system, databases, and knowledge graphs. Uh the
[23:11] databases, and knowledge graphs. Uh the first one is a file system. So you can
[23:14] first one is a file system. So you can treat every memory as a file. Um
[23:17] treat every memory as a file. Um and we let the agent manage its own
[23:19] and we let the agent manage its own memories. So this is actually nice
[23:22] memories. So this is actually nice because it's adding some level of
[23:23] because it's adding some level of hierarchy by default. You can go go into
[23:25] hierarchy by default. You can go go into a folder and you can look over the
[23:27] a folder and you can look over the memories in that folder. Go into a
[23:28] memories in that folder. Go into a subdirectory, look over memories. Um and
[23:31] subdirectory, look over memories. Um and then the way you actually access and
[23:32] then the way you actually access and read and write to memory is the same
[23:34] read and write to memory is the same bash commands that you know you should
[23:36] bash commands that you know you should be familiar with if you've you know used
[23:38] be familiar with if you've you know used uh Linux or
[23:40] uh Linux or u osx
[23:41] u osx um
[23:42] um uh so the you can use gp ls and actually
[23:46] uh so the you can use gp ls and actually search over these files and find your
[23:47] search over these files and find your memories. Um, so this is an example, and
[23:50] memories. Um, so this is an example, and and uh this comes from a lang chain blog
[23:52] and uh this comes from a lang chain blog post. But they show procedural memory
[23:55] post. But they show procedural memory can be things like your agents.md,
[23:57] can be things like your agents.md, your MCPs.
[23:59] your MCPs. Uh uh you can have skill files uh.
[24:01] Uh uh you can have skill files uh. Semantic memory uh they also include
[24:03] Semantic memory uh they also include skills as like a blend of semantic
[24:05] skills as like a blend of semantic memory, because if you remember the
[24:06] memory, because if you remember the Anthropic brand guideline thing, it was
[24:08] Anthropic brand guideline thing, it was talking about what are the colors and
[24:10] talking about what are the colors and then how do you use them and stuff, so
[24:11] then how do you use them and stuff, so blurs the lines there. But you can see
[24:13] blurs the lines there. But you can see we have knowledge stored in files. And
[24:15] we have knowledge stored in files. And then episodic memory: you can have a
[24:16] then episodic memory: you can have a directory where you store the literal
[24:18] directory where you store the literal conversation traces that you've had in
[24:19] conversation traces that you've had in the past.
[24:21] the past. Um, so this is one way. Another system
[24:23] Um, so this is one way. Another system is to use databases. Uh, and so we can
[24:25] is to use databases. Uh, and so we can use a vector database or, you know,
[24:28] use a vector database or, you know, SQLite or some whatever your favorite
[24:30] SQLite or some whatever your favorite database is. If you've worked with
[24:32] database is. If you've worked with databases before, uh, it could also be
[24:34] databases before, uh, it could also be used for agent memory. Um, and so, uh,
[24:37] used for agent memory. Um, and so, uh, in the instance of a vector database,
[24:38] in the instance of a vector database, you can do some sort of vector search,
[24:41] you can do some sort of vector search, uh, for, you know, similarity between
[24:42] uh, for, you know, similarity between what you're looking for. Um for SQLite
[24:45] what you're looking for. Um for SQLite you could have you know SQL queries and
[24:47] you could have you know SQL queries and if you store some metadata about your
[24:49] if you store some metadata about your memories you can actually search over
[24:50] memories you can actually search over them. Uh if you want exact term lookup
[24:53] them. Uh if you want exact term lookup you can use you know BM25
[24:55] you can use you know BM25 to find a find uh documents that might
[24:57] to find a find uh documents that might have those terms. So uh if you just
[24:59] have those terms. So uh if you just recall our our rag lecture a lot of the
[25:01] recall our our rag lecture a lot of the things about like uh embedding based
[25:04] things about like uh embedding based search and things still apply uh and we
[25:06] search and things still apply uh and we can use them to to store our memories.
[25:09] can use them to to store our memories. The last one I'll talk about is
[25:11] The last one I'll talk about is knowledge graphs. Um this is maybe the
[25:14] knowledge graphs. Um this is maybe the most uh or the least straightforward I
[25:16] most uh or the least straightforward I guess but um the
[25:18] guess but um the system I'll talk about very briefly here
[25:20] system I'll talk about very briefly here is called Zep. Uh it's built to be a
[25:24] is called Zep. Uh it's built to be a knowledge graph for agents. And so they
[25:27] knowledge graph for agents. And so they uh the way that Zep has its memory
[25:29] uh the way that Zep has its memory hierarchy is it has like three levels.
[25:31] hierarchy is it has like three levels. It has episodes, entities and
[25:33] It has episodes, entities and communities. Um and so uh it creates
[25:36] communities. Um and so uh it creates knowledge graphs because uh when you
[25:38] knowledge graphs because uh when you have these conversations it can extract
[25:40] have these conversations it can extract entities um from those conversations and
[25:42] entities um from those conversations and extract relationships between those
[25:44] extract relationships between those entities. So it uses a language model to
[25:46] entities. So it uses a language model to do this. Uh and so you have all of the
[25:49] do this. Uh and so you have all of the conversations and it can say like okay
[25:51] conversations and it can say like okay um maybe you had a conversation talking
[25:53] um maybe you had a conversation talking about your dog
[25:54] about your dog uh and you mentioned that your cousin
[25:57] uh and you mentioned that your cousin really likes your dog or something like
[25:58] really likes your dog or something like that. It was like look at this
[25:59] that. It was like look at this conversation. It could say like okay I'm
[26:01] conversation. It could say like okay I'm going to make an entity for your cousin.
[26:02] going to make an entity for your cousin. I'll make an entity for your dog and
[26:04] I'll make an entity for your dog and I'll put the relationship that like
[26:05] I'll put the relationship that like cousin likes the dog and stuff like
[26:07] cousin likes the dog and stuff like that. And then all over time this entity
[26:10] that. And then all over time this entity knowledge graph becomes quite large. Uh
[26:13] knowledge graph becomes quite large. Uh and so they do some level of like LLM
[26:16] and so they do some level of like LLM based clustering where it uh analyzes
[26:18] based clustering where it uh analyzes which entities have several edges
[26:20] which entities have several edges between them. Uh what is clustered
[26:23] between them. Uh what is clustered together nicely. Uh and then it writes
[26:24] together nicely. Uh and then it writes these summaries so that when you want to
[26:26] these summaries so that when you want to retrieve over it, you can search over
[26:27] retrieve over it, you can search over the summaries to try to find what
[26:29] the summaries to try to find what entities matter. Um, so this is like a
[26:31] entities matter. Um, so this is like a sort of organized uh system, but you can
[26:34] sort of organized uh system, but you can imagine this is like a general system
[26:36] imagine this is like a general system where it's like entities is not very
[26:38] where it's like entities is not very specific about what it's talking about.
[26:39] specific about what it's talking about. Uh, but you could define I care about
[26:42] Uh, but you could define I care about these types of entities and these types
[26:44] these types of entities and these types of relationships and this can then be
[26:45] of relationships and this can then be something you search over. Um,
[26:48] something you search over. Um, any
[26:49] any questions about the underlying systems
[26:52] questions about the underlying systems for memory before we talk about some
[26:54] for memory before we talk about some design patterns? Yes. So I imagine the
[26:56] design patterns? Yes. So I imagine the no knowledge graph is more for semantic
[26:59] no knowledge graph is more for semantic or it's more static. Would that be
[27:01] or it's more static. Would that be accurate?
[27:01] accurate? >> That is a good question. The question is
[27:03] >> That is a good question. The question is like are the knowledge graphs mostly for
[27:05] like are the knowledge graphs mostly for semantic memory? Uh and the answer is
[27:07] semantic memory? Uh and the answer is typically yes. Um
[27:09] typically yes. Um yeah I I I think you can try to write
[27:13] yeah I I I think you can try to write procedures as like a
[27:14] procedures as like a a knowledge graph of like take this step
[27:17] a knowledge graph of like take this step and we have edges between them and stuff
[27:18] and we have edges between them and stuff but for the most part this is for
[27:20] but for the most part this is for semantic memory. Good question. Um,
[27:23] semantic memory. Good question. Um, any other questions?
[27:25] any other questions? >> Yes.
[27:26] >> Yes. >> Do we know like which one is most
[27:27] >> Do we know like which one is most effective or works fast? Uh,
[27:29] effective or works fast? Uh, >> I think it depends on your application.
[27:30] >> I think it depends on your application. So, I'm going to talk now about some
[27:32] So, I'm going to talk now about some architectures that use each of these and
[27:33] architectures that use each of these and we'll compare some of them and I have
[27:34] we'll compare some of them and I have some benchmark scores on them. Um,
[27:37] some benchmark scores on them. Um, I
[27:37] I think if I'm giving my opinion because
[27:40] think if I'm giving my opinion because every paper is going to kind of make the
[27:42] every paper is going to kind of make the argument that their system is best. In
[27:44] argument that their system is best. In my opinion, uh, if you can get away with
[27:46] my opinion, uh, if you can get away with it, using the file system is typically
[27:50] it, using the file system is typically useful. I mean the the latency will be
[27:52] useful. I mean the the latency will be probably the trade-off there because the
[27:53] probably the trade-off there because the agent has to like run all of these bash
[27:56] agent has to like run all of these bash commands and process it and reason and
[27:58] commands and process it and reason and take the next step and stuff like that.
[27:59] take the next step and stuff like that. But in general, I think the lesson that
[28:02] But in general, I think the lesson that is being learned with agents is that uh
[28:05] is being learned with agents is that uh just kind of setting up the simplest
[28:07] just kind of setting up the simplest thing and letting the model really run
[28:09] thing and letting the model really run the show is a very good way to run
[28:11] the show is a very good way to run things. So if if you just say here's
[28:14] things. So if if you just say here's your file system, you can use it as your
[28:16] your file system, you can use it as your memory. Uh that that typically works
[28:18] memory. Uh that that typically works pretty well. It's then when you want to
[28:19] pretty well. It's then when you want to make trade-offs about like how fast does
[28:21] make trade-offs about like how fast does the memory lookup have to be and stuff
[28:23] the memory lookup have to be and stuff like that that you need to use some of
[28:24] like that that you need to use some of these other systems. But good question.
[28:27] these other systems. But good question. Uh any other questions? Yes.
[28:30] Uh any other questions? Yes. >> Um so is it like not as common to create
[28:32] >> Um so is it like not as common to create like full text indexes of like all of
[28:34] like full text indexes of like all of the
[28:35] the history of like your conversation or
[28:38] history of like your conversation or like is it more embedded?
[28:40] like is it more embedded? Uh the question is is it
[28:42] Uh the question is is it common to create like uh knowledge graph
[28:44] common to create like uh knowledge graph based indexes over your conversations
[28:46] based indexes over your conversations versus like embedding the conversation?
[28:50] versus like embedding the conversation? >> Uh oh. Oh, sorry. A full text index. Is
[28:53] >> Uh oh. Oh, sorry. A full text index. Is that what you said?
[28:58] >> Yeah. Yeah. Oh. Um so it depends. I'll talk about MEGBT in a bit which does do
[29:00] talk about MEGBT in a bit which does do like an exact text lookup. Um
[29:03] like an exact text lookup. Um I think like you these are all design
[29:06] I think like you these are all design decisions. uh in the MEMGBT architecture
[29:08] decisions. uh in the MEMGBT architecture in particular, they embed documents for
[29:11] in particular, they embed documents for searching, but conversations remain as
[29:13] searching, but conversations remain as full text and you can like look up the
[29:15] full text and you can like look up the exact
[29:16] exact strings in the text. Um I think it's
[29:18] strings in the text. Um I think it's it's it's somewhat specific to the
[29:20] it's it's somewhat specific to the application you're trying to build and
[29:22] application you're trying to build and and what you want to do with it. Um so I
[29:24] and what you want to do with it. Um so I think that's one common pattern. Um
[29:26] think that's one common pattern. Um and it's tough to answer like in every
[29:28] and it's tough to answer like in every case, but yeah, it's a good question.
[29:30] case, but yeah, it's a good question. It's something that's maybe a little bit
[29:31] It's something that's maybe a little bit open. Yeah.
[29:33] open. Yeah. >> Uh yeah. So
[29:35] >> Uh yeah. So that's more for kind of doing like
[29:37] that's more for kind of doing like context protocols, like for file systems
[29:39] context protocols, like for file systems you'd ask
[29:40] you'd ask tools like ls or graph, right? And then,
[29:43] tools like ls or graph, right? And then, like, if you're using SQL, capture more
[29:45] like, if you're using SQL, capture more like relationships and pass queries from
[29:47] like relationships and pass queries from that standpoint of just like doing
[29:49] that standpoint of just like doing standard database lookup. And is that
[29:52] standard database lookup. And is that like kind of the idea to expand onto
[29:54] like kind of the idea to expand onto more complexity,
[29:56] more complexity, um, or am I kind of like missing?
[29:58] um, or am I kind of like missing? >> uh I guess maybe I can try to restate
[30:00] >> uh I guess maybe I can try to restate the question you can tell me if I'm uh
[30:02] the question you can tell me if I'm uh understanding it is the question like uh
[30:05] understanding it is the question like uh what is the point of doing the knowledge
[30:07] what is the point of doing the knowledge graph uh as compared to the
[30:10] graph uh as compared to the um like databases and stuff like that
[30:12] um like databases and stuff like that like why why do it or sorry
[30:15] like why why do it or sorry >> more just like how does the querying for
[30:16] >> more just like how does the querying for it look different
[30:17] it look different >> oh how does the querying look different
[30:19] >> oh how does the querying look different yeah so uh the nice thing about a
[30:21] yeah so uh the nice thing about a knowledge graph um is because you have
[30:23] knowledge graph um is because you have the relationships between entities uh
[30:25] the relationships between entities uh you can use certain graph uh lookup
[30:28] you can use certain graph uh lookup algorithms uh so there's like graph rag
[30:30] algorithms uh so there's like graph rag and things like that um where you can
[30:32] and things like that um where you can actually traverse the knowledge graph.
[30:34] actually traverse the knowledge graph. Um, so it can help because you can put a
[30:36] Um, so it can help because you can put a little bit more of the intelligence into
[30:38] little bit more of the intelligence into your lookup that wouldn't have been as
[30:40] your lookup that wouldn't have been as possible. So it's something where like
[30:42] possible. So it's something where like if you want to offload some of the
[30:45] if you want to offload some of the computing power to the actual lookup
[30:47] computing power to the actual lookup process instead of having it all rely on
[30:49] process instead of having it all rely on the model underlying the agent, then the
[30:51] the model underlying the agent, then the knowledge graph can be very good for
[30:52] knowledge graph can be very good for that. Um, so that's like one answer
[30:55] that. Um, so that's like one answer potentially to your question. Sweet. Uh,
[30:57] potentially to your question. Sweet. Uh, I'm going to move on in the interest of
[30:58] I'm going to move on in the interest of time. Uh let's talk about some patterns
[31:01] time. Uh let's talk about some patterns for agent architectures. So these are
[31:04] for agent architectures. So these are four patterns. Uh I sort of tease these
[31:06] four patterns. Uh I sort of tease these together by reading papers. So I this
[31:09] together by reading papers. So I this doesn't really come from like an
[31:11] doesn't really come from like an official, you know, textbook on agents.
[31:13] official, you know, textbook on agents. Not to say that there really is a
[31:14] Not to say that there really is a textbook on agents yet, but just take it
[31:16] textbook on agents yet, but just take it with a slight grain of salt that this is
[31:17] with a slight grain of salt that this is kind of my
[31:18] kind of my own interpretation. Uh but I do think
[31:20] own interpretation. Uh but I do think these are good useful patterns for
[31:23] these are good useful patterns for analyzing the ways that people implement
[31:25] analyzing the ways that people implement agent memory. So the first one uh and
[31:27] agent memory. So the first one uh and we're going to talk about all of these
[31:28] we're going to talk about all of these in depth uh but the first one is
[31:30] in depth uh but the first one is learning procedures from experience. The
[31:32] learning procedures from experience. The second one is a tiered memory hierarchy.
[31:35] second one is a tiered memory hierarchy. The third one is like reflective memory
[31:37] The third one is like reflective memory consolidation and the last one is memory
[31:39] consolidation and the last one is memory maintenance. Um
[31:41] maintenance. Um so the first pattern is learning
[31:43] so the first pattern is learning procedures from experience and we will
[31:46] procedures from experience and we will talk about a
[31:47] talk about a famous paper that kind of uh pioneered
[31:50] famous paper that kind of uh pioneered some of this which is Voyager. Um, so,
[31:53] some of this which is Voyager. Um, so, uh, some of you may be familiar with
[31:54] uh, some of you may be familiar with Voyager, but the idea behind Voyager is
[31:56] Voyager, but the idea behind Voyager is it was a Minecraft playing agent. Um,
[32:00] it was a Minecraft playing agent. Um, and so the idea, uh, was kind of trying
[32:03] and so the idea, uh, was kind of trying to create a policy that encouraged
[32:05] to create a policy that encouraged exploration in Minecraft. And so, uh,
[32:07] exploration in Minecraft. And so, uh, they had different tasks that were
[32:09] they had different tasks that were proposed by some sort of curriculum part
[32:12] proposed by some sort of curriculum part of the agent. Um,
[32:14] of the agent. Um, and it would create tasks like, uh,
[32:16] and it would create tasks like, uh, okay, the first thing I want to do is
[32:18] okay, the first thing I want to do is mine a wood log, which, uh, I don't know
[32:20] mine a wood log, which, uh, I don't know how many of you have played Minecraft,
[32:22] how many of you have played Minecraft, but that's like a good first thing to do
[32:23] but that's like a good first thing to do when you, when you boot up the game. Uh,
[32:25] when you, when you boot up the game. Uh, and so, uh, it it writes some code. Uh,
[32:28] and so, uh, it it writes some code. Uh, this is how it interacts with the
[32:29] this is how it interacts with the Minecraft world is it it does it
[32:30] Minecraft world is it it does it entirely through this uh, I forget what
[32:32] entirely through this uh, I forget what the library is called, but it's actually
[32:34] the library is called, but it's actually a JavaScript based library that allows
[32:36] a JavaScript based library that allows you to play Minecraft uh, in code. Um,
[32:38] you to play Minecraft uh, in code. Um, and so uh, I guess this is an example of
[32:40] and so uh, I guess this is an example of combat zombie. So it set this task as
[32:42] combat zombie. So it set this task as combat zombie and it wrote all of these
[32:45] combat zombie and it wrote all of these uh instructions on how to fight a
[32:47] uh instructions on how to fight a zombie. Um but then to do this it
[32:50] zombie. Um but then to do this it actually uh
[32:51] actually uh has a skill library and these skills are
[32:54] has a skill library and these skills are like prior code that it has written. So
[32:56] like prior code that it has written. So uh this combat zombie will get added to
[32:58] uh this combat zombie will get added to the skill library. Uh but essentially it
[33:00] the skill library. Uh but essentially it will run the code in the game and use
[33:02] will run the code in the game and use environment feedback such as like did it
[33:05] environment feedback such as like did it actually kill the zombie uh
[33:06] actually kill the zombie uh successfully. Uh, and if it did, it adds
[33:09] successfully. Uh, and if it did, it adds it to the skill library as this is like
[33:11] it to the skill library as this is like a skill that we can now use in the
[33:12] a skill that we can now use in the future. If it did not, it uses the
[33:14] future. If it did not, it uses the environment feedback. Um, and the
[33:16] environment feedback. Um, and the environment feedback would be things
[33:18] environment feedback would be things observations from the game and stuff
[33:20] observations from the game and stuff like maybe the zombie killed you and and
[33:21] like maybe the zombie killed you and and you didn't successfully kill the zombie.
[33:23] you didn't successfully kill the zombie. And it uses that to refine the program
[33:24] And it uses that to refine the program and rewrite the program. Uh, and so
[33:27] and rewrite the program. Uh, and so then, uh, once it successfully completes
[33:29] then, uh, once it successfully completes the task, it updates and goes back to
[33:30] the task, it updates and goes back to the curriculum and determines what the
[33:32] the curriculum and determines what the next thing it's going to work on is. Um
[33:34] next thing it's going to work on is. Um so let's look at what the skill library
[33:36] so let's look at what the skill library is because this is the memory behind
[33:37] is because this is the memory behind this all. Um so in the skill library uh
[33:40] this all. Um so in the skill library uh what we have is we write a program uh to
[33:43] what we have is we write a program uh to try to complete a task and we also
[33:45] try to complete a task and we also generate a program description that says
[33:48] generate a program description that says this is what the the
[33:49] this is what the the uh program is trying to do. So for
[33:51] uh program is trying to do. So for combat zombie this function is about
[33:53] combat zombie this function is about equipping a stone sword to combat a
[33:55] equipping a stone sword to combat a zombie. You know if it's not found we'll
[33:56] zombie. You know if it's not found we'll craft a stone sword. Additionally you so
[33:59] craft a stone sword. Additionally you so it makes a shield. So it, it writes all
[34:01] it makes a shield. So it, it writes all this up, and then it embeds this, uh,
[34:03] this up, and then it embeds this, uh, into a key.
[34:05] into a key. And then later, when we are trying to
[34:08] And then later, when we are trying to write a new program, uh, it looks up
[34:11] write a new program, uh, it looks up using, um, some descriptions of like
[34:13] using, um, some descriptions of like what we're trying to accomplish and sees
[34:14] what we're trying to accomplish and sees if any of these things already exist in
[34:16] if any of these things already exist in the skill library so that it can call
[34:17] the skill library so that it can call them. Um
[34:19] them. Um and so later when we want to
[34:21] and so later when we want to retrieve uh we can say like how do I
[34:23] retrieve uh we can say like how do I craft an iron pickaxe? Uh we generate
[34:26] craft an iron pickaxe? Uh we generate like uh what we would like to do. we
[34:29] like uh what we would like to do. we want to craft an iron pickaxe. We then
[34:30] want to craft an iron pickaxe. We then use an embedding to query the skill
[34:32] use an embedding to query the skill library and then we can now use the
[34:34] library and then we can now use the skills to
[34:36] skills to um now craft this iron pickaxe. So this
[34:38] um now craft this iron pickaxe. So this is a this idea of procedural memory.
[34:40] is a this idea of procedural memory. We're writing programs to accomplish
[34:41] We're writing programs to accomplish skills. When we successfully accomplish
[34:44] skills. When we successfully accomplish the skill, it gets committed to memory
[34:45] the skill, it gets committed to memory so that we can retrieve it later and add
[34:47] so that we can retrieve it later and add it back into the context. Um
[34:50] it back into the context. Um yes question
[34:51] yes question >> that query is not going to be
[34:53] >> that query is not going to be exactly identical to the key that was
[34:54] exactly identical to the key that was random generated before, right? So is it
[34:56] random generated before, right? So is it going to be some sort of like clip
[34:57] going to be some sort of like clip similarity search?
[34:58] similarity search? >> Yes. So the question is like the query
[35:00] >> Yes. So the question is like the query is not an exact match of the key. Um so
[35:02] is not an exact match of the key. Um so is there some sort of clip similarity
[35:04] is there some sort of clip similarity search? Yeah. So we're we're doing this
[35:06] search? Yeah. So we're we're doing this is why we do the embedding. So we we do
[35:08] is why we do the embedding. So we we do a similarity score between the embedding
[35:10] a similarity score between the embedding and that's how we do the lookup.
[35:11] and that's how we do the lookup. >> So it's possible that there are multiple
[35:13] >> So it's possible that there are multiple skills that are somewhat similar to each
[35:15] skills that are somewhat similar to each other and you make misplace.
[35:17] other and you make misplace. >> Yeah. Yeah. This is actually a question
[35:18] >> Yeah. Yeah. This is actually a question I had when I read the paper too is like
[35:20] I had when I read the paper too is like what happens with similar skills and
[35:22] what happens with similar skills and stuff like that. And this is something
[35:23] stuff like that. And this is something that can happen like it could rewrite it
[35:25] that can happen like it could rewrite it can have two skills for how to craft an
[35:26] can have two skills for how to craft an iron pickaxe and stuff like that. And uh
[35:29] iron pickaxe and stuff like that. And uh it it what they do is like they avoid
[35:31] it it what they do is like they avoid duplicating the exact name. So you can't
[35:33] duplicating the exact name. So you can't have like craft iron pick twice. It
[35:35] have like craft iron pick twice. It would prevent that from happening. But
[35:37] would prevent that from happening. But you can have like craft and iron pick
[35:39] you can have like craft and iron pick and craft iron pick and like you might
[35:41] and craft iron pick and like you might end up with a bunch of skills that do
[35:42] end up with a bunch of skills that do the same thing. Yeah, it's a good
[35:43] the same thing. Yeah, it's a good question. Any other questions?
[35:51] Uh cool. Then just some results. Uh so this is they tested against a couple
[35:52] this is they tested against a couple different agents. You guys will remember
[35:54] different agents. You guys will remember uh React and Reflection from our prior
[35:57] uh React and Reflection from our prior lecture. And then they have Voyager
[35:59] lecture. And then they have Voyager without skill library and then Voyager
[36:01] without skill library and then Voyager with the skill library. And you can see
[36:03] with the skill library. And you can see that eventually the uh agent progresses
[36:06] that eventually the uh agent progresses all the way to crafting diamond tools uh
[36:09] all the way to crafting diamond tools uh in 102 prompt iterations. Um but this
[36:11] in 102 prompt iterations. Um but this was only possible by building on top of
[36:13] was only possible by building on top of the skill library. So without the skill
[36:15] the skill library. So without the skill library, it only ever made it to iron
[36:17] library, it only ever made it to iron tools. And you can also see, I guess the
[36:19] tools. And you can also see, I guess the next slide will show this a little bit
[36:20] next slide will show this a little bit better. Um,
[36:22] better. Um, it does it in fewer prompt iterations
[36:24] it does it in fewer prompt iterations and fewer tries to accomplish these
[36:27] and fewer tries to accomplish these things um than it took without the skill
[36:29] things um than it took without the skill library. So you can see that the orange
[36:32] library. So you can see that the orange line here is
[36:33] line here is the uh full with skill library and the
[36:36] the uh full with skill library and the yellow is without the skill library. You
[36:37] yellow is without the skill library. You can see it shifts to the left how many
[36:40] can see it shifts to the left how many tries it takes to solve each of these
[36:41] tries it takes to solve each of these tasks because it's able to pull from
[36:43] tasks because it's able to pull from prior memory and reuse these skills.
[36:46] prior memory and reuse these skills. Uh any questions about Voyager?
[36:52] Awesome. Then I will talk about agent workflow memory. Um so agent workflow
[36:55] workflow memory. Um so agent workflow memory is is similar in a lot of ways to
[36:57] memory is is similar in a lot of ways to Voyager um but it's a little bit more
[36:59] Voyager um but it's a little bit more general. It's not just for Minecraft. Um
[37:02] general. It's not just for Minecraft. Um and so the idea is we will first obtain
[37:06] and so the idea is we will first obtain action. So, somebody asks the question,
[37:08] action. So, somebody asks the question, you know, who ordered order number 130?
[37:11] you know, who ordered order number 130? And so, the agent is supposed to look
[37:12] And so, the agent is supposed to look this up on its uh database. This is a
[37:14] this up on its uh database. This is a computer use agent. So, you can see it's
[37:16] computer use agent. So, you can see it's doing things like clicking and
[37:17] doing things like clicking and scrolling. Um, and so it generates this
[37:20] scrolling. Um, and so it generates this set of actions uh that it's going to use
[37:22] set of actions uh that it's going to use to complete the task. Uh, and then we
[37:24] to complete the task. Uh, and then we evaluate the trajectory. Uh, this is,
[37:26] evaluate the trajectory. Uh, this is, like I said, similar to Voyager. If we
[37:28] like I said, similar to Voyager. If we were successful, uh, we'll move on. Uh,
[37:30] were successful, uh, we'll move on. Uh, if we were not, then uh, we we retry. Um
[37:34] if we were not, then uh, we we retry. Um and so uh we use this to induce
[37:36] and so uh we use this to induce workflows. So we have all of these
[37:38] workflows. So we have all of these trajectories that were successful. Uh
[37:40] trajectories that were successful. Uh and then we can try to determine like
[37:42] and then we can try to determine like which work uh which is a general pattern
[37:45] which work uh which is a general pattern that we can reuse
[37:46] that we can reuse uh and save as a workflow. Uh and this
[37:49] uh and save as a workflow. Uh and this this gets added back to the agent
[37:51] this gets added back to the agent memory. Um so this is just a a simple
[37:54] memory. Um so this is just a a simple agent that has again remember it's
[37:56] agent that has again remember it's always acting and observing. Um and so
[37:59] always acting and observing. Um and so uh in in this paper they just store it
[38:00] uh in in this paper they just store it to the context. There's no embedding
[38:02] to the context. There's no embedding based lookup or anything like that, but
[38:03] based lookup or anything like that, but the successful trajectories are kind of
[38:05] the successful trajectories are kind of described into a workflow. Uh so it sort
[38:08] described into a workflow. Uh so it sort of reflects on the what happened and
[38:10] of reflects on the what happened and writes out like these are the steps that
[38:11] writes out like these are the steps that worked for me and puts that into the
[38:13] worked for me and puts that into the memory so that later when it's trying to
[38:14] memory so that later when it's trying to solve a task, it can use that again.
[38:17] solve a task, it can use that again. Um so this is how the uh workflows are
[38:20] Um so this is how the uh workflows are induced. You have this prompt that says
[38:22] induced. You have this prompt that says uh given a list of web navigation tasks,
[38:25] uh given a list of web navigation tasks, your task is to extract the common
[38:26] your task is to extract the common workflows. So, we take all of the
[38:27] workflows. So, we take all of the successful traces and it's supposed to
[38:30] successful traces and it's supposed to try to find patterns as like this is
[38:32] try to find patterns as like this is what worked. Uh, this is what did not
[38:34] what worked. Uh, this is what did not work or I guess it's not really supposed
[38:35] work or I guess it's not really supposed to say what didn't work, but it's
[38:36] to say what didn't work, but it's supposed to find the things that were
[38:37] supposed to find the things that were general patterns between them. Um, and
[38:40] general patterns between them. Um, and then these are examples of workflows.
[38:42] then these are examples of workflows. Um, so you can see they're actually kind
[38:44] Um, so you can see they're actually kind of simple, but it's things like uh when
[38:47] of simple, but it's things like uh when we're shopping, we need to browse
[38:48] we're shopping, we need to browse products in a specific category. Um, so
[38:50] products in a specific category. Um, so if we want to do that, then you should
[38:52] if we want to do that, then you should hover over the main category and click
[38:54] hover over the main category and click on the subcategory. So you if you're
[38:56] on the subcategory. So you if you're envisioning a website right now, there
[38:58] envisioning a website right now, there might be like a hover over drop-own
[38:59] might be like a hover over drop-own menu, right? And so the agent learned
[39:02] menu, right? And so the agent learned from prior experience that in order to
[39:04] from prior experience that in order to browse a subcategory, I first need to
[39:05] browse a subcategory, I first need to hover over the main category and then
[39:07] hover over the main category and then click on the subcategory. So it's things
[39:08] click on the subcategory. So it's things like that that you learn by experience.
[39:10] like that that you learn by experience. Uh and then like I said, these are
[39:12] Uh and then like I said, these are included back into the system prompt.
[39:14] included back into the system prompt. And so, some results on this, uh, is
[39:16] And so, some results on this, uh, is just kind of showing that, like, on all
[39:18] just kind of showing that, like, on all of these different domains they tested
[39:20] of these different domains they tested it on shopping, Reddit, uh, maps, uh it
[39:23] it on shopping, Reddit, uh, maps, uh it was able to learn better, uh, and, uh,
[39:25] was able to learn better, uh, and, uh, actually comparable, but in some cases
[39:27] actually comparable, but in some cases better than, human engineered workflows
[39:29] better than, human engineered workflows where people described patterns that it
[39:31] where people described patterns that it should look for. Um
[39:33] should look for. Um and then you could see that there's this
[39:34] and then you could see that there's this sort of like learning phase where the
[39:36] sort of like learning phase where the first few examples uh it has very low
[39:38] first few examples uh it has very low accuracy but quickly learns what
[39:40] accuracy but quickly learns what workflows are are important. uh and then
[39:42] workflows are are important. uh and then later on it's much higher accuracy uh
[39:45] later on it's much higher accuracy uh because it learned how to do these
[39:46] because it learned how to do these things. Uh so any questions about that?
[39:55] Excellent. Uh quick question. Yeah. >> How does it compare to like
[39:56] >> How does it compare to like reinforcement learning? I mean it's
[39:58] reinforcement learning? I mean it's basically just like in context
[40:00] basically just like in context reinforcement learning or
[40:02] reinforcement learning or >> Yes. Yes. So the question is how does
[40:03] >> Yes. Yes. So the question is how does this compare to reinforcement learning?
[40:04] this compare to reinforcement learning? Because it's kind of like in context
[40:06] Because it's kind of like in context reinforcement learning. So it's it's
[40:07] reinforcement learning. So it's it's more much more sample efficient than
[40:09] more much more sample efficient than reinforcement learning. You can see that
[40:11] reinforcement learning. You can see that this learns uh improves the accuracy in
[40:14] this learns uh improves the accuracy in like fewer than maybe like 10 15
[40:17] like fewer than maybe like 10 15 examples. It's getting and reinforcement
[40:18] examples. It's getting and reinforcement learning you're going to need a lot more
[40:20] learning you're going to need a lot more than that. Uh but we'll talk in two
[40:22] than that. Uh but we'll talk in two lectures on optimization and we're going
[40:24] lectures on optimization and we're going to compare reinforcement learning with
[40:26] to compare reinforcement learning with textbased optimization with this paper
[40:28] textbased optimization with this paper called Jeepa. Um and so that will return
[40:30] called Jeepa. Um and so that will return to your question. But yeah, this is a
[40:32] to your question. But yeah, this is a very good question. It is kind of a
[40:33] very good question. It is kind of a similar setting to reinforcement
[40:34] similar setting to reinforcement learning. Yeah.
[40:36] learning. Yeah. >> Uh yes question.
[40:38] >> Uh yes question. when those memory being kicked in like F
[40:41] when those memory being kicked in like F into the context or while the agent is
[40:44] into the context or while the agent is running is it provided by the model say
[40:46] running is it provided by the model say like a tool call say I want to load this
[40:48] like a tool call say I want to load this or it is provided by the engine
[40:50] or it is provided by the engine harnesses
[40:51] harnesses >> This is a good question so. in this, in
[40:52] >> This is a good question so. in this, in in some instances it is a tool call. Um,
[40:54] in some instances it is a tool call. Um, in this particular paper, uh, it's a lot
[40:57] in this particular paper, uh, it's a lot simpler than that: in between sessions,
[40:59] simpler than that: in between sessions, they just update the system prompt to
[41:01] they just update the system prompt to incorporate all of these. So it, this
[41:03] incorporate all of these. So it, this probably wouldn't be super scalable to,
[41:05] probably wouldn't be super scalable to, like, an agent that needs to run,
[41:07] like, an agent that needs to run, uh, like for a long, long period of
[41:09] uh, like for a long, long period of time. It's fine when you only run on 100
[41:11] time. It's fine when you only run on 100 examples. Um but you do need some sort
[41:13] examples. Um but you do need some sort of like lookup if you're going to do
[41:14] of like lookup if you're going to do some long running thing and and we'll
[41:16] some long running thing and and we'll talk more about the lookup uh procedures
[41:17] talk more about the lookup uh procedures in a second. Yeah, good question.
[41:21] in a second. Yeah, good question. Um all right, let's move on to skill
[41:24] Um all right, let's move on to skill files. Uh which I'm just going to
[41:25] files. Uh which I'm just going to quickly mention because I think probably
[41:27] quickly mention because I think probably you all are familiar with skill files,
[41:28] you all are familiar with skill files, but I do feel like it's worth calling
[41:30] but I do feel like it's worth calling out as a form of procedural memory. Uh
[41:32] out as a form of procedural memory. Uh so just like I showed earlier, this is
[41:34] so just like I showed earlier, this is like this anthropic brand style
[41:36] like this anthropic brand style guidelines and so you write up a
[41:37] guidelines and so you write up a skill.md file or you know your agent
[41:40] skill.md file or you know your agent writes up a skill.md file which is also
[41:42] writes up a skill.md file which is also fine and ways to you know store its own
[41:44] fine and ways to you know store its own memories. Uh and you store that into
[41:46] memories. Uh and you store that into your file system
[41:47] your file system uh and then what we can do is we can
[41:50] uh and then what we can do is we can load this skill file into the context
[41:52] load this skill file into the context window. So this is a example of a system
[41:55] window. So this is a example of a system prompt uh or sort of a cont a full
[41:57] prompt uh or sort of a cont a full context window actually. So, we have the
[41:58] context window actually. So, we have the core system prompt. Then we have all of
[42:00] core system prompt. Then we have all of these snippets. And I'll show you what
[42:02] these snippets. And I'll show you what what we mean by snippet in a moment, but
[42:04] what we mean by snippet in a moment, but we have snippets of all of these skill
[42:05] we have snippets of all of these skill files that we've loaded in. Uh, and then
[42:08] files that we've loaded in. Uh, and then we have the user conversation, tool
[42:09] we have the user conversation, tool results, tool use, all this. Um, and so
[42:12] results, tool use, all this. Um, and so this is what the uh actual context
[42:14] this is what the uh actual context window looks like. And if we expand
[42:15] window looks like. And if we expand this, you can see for this SLX
[42:18] this, you can see for this SLX um skill,
[42:19] um skill, we have um
[42:21] we have um a name of what it is and a description.
[42:23] a name of what it is and a description. Uh, and actually, if I go back, this
[42:25] Uh, and actually, if I go back, this description comes from this whole
[42:26] description comes from this whole metadata section. So anything between
[42:28] metadata section. So anything between these uh lines is what goes into the
[42:31] these uh lines is what goes into the description. Um and you can see that we
[42:34] description. Um and you can see that we have
[42:35] have just uh all of these would show up in
[42:37] just uh all of these would show up in the context window at the beginning and
[42:39] the context window at the beginning and then the agent can then say okay I need
[42:43] then the agent can then say okay I need to use the uh PDF skill for instance. So
[42:46] to use the uh PDF skill for instance. So I'm just going to read the PDF skill
[42:48] I'm just going to read the PDF skill file. Um and so it's it's just very
[42:51] file. Um and so it's it's just very simple. This is what I mean about like
[42:52] simple. This is what I mean about like people realizing that the underlying
[42:55] people realizing that the underlying models are getting so powerful that you
[42:56] models are getting so powerful that you can just use a file system to manage all
[42:58] can just use a file system to manage all these things. Uh you didn't really have
[42:59] these things. Uh you didn't really have to do anything custom here except for
[43:01] to do anything custom here except for make a file and tell it about the
[43:03] make a file and tell it about the existence of that file and then it can
[43:04] existence of that file and then it can later go read it and decide how to act
[43:07] later go read it and decide how to act on it. So that's the idea of skill
[43:09] on it. So that's the idea of skill files.
[43:10] files. Any questions there?
[43:16] Uh cool. Then one quick thing I'll mention as a limitation at least of the
[43:18] mention as a limitation at least of the first two uh maybe less of the skill
[43:20] first two uh maybe less of the skill file if you update in but uh if you
[43:22] file if you update in but uh if you follow the exact methods of voyager and
[43:25] follow the exact methods of voyager and agent workflow memory uh it's not very
[43:27] agent workflow memory uh it's not very good for robust uh behavior in dynamic
[43:30] good for robust uh behavior in dynamic environments. So if the underlying
[43:31] environments. So if the underlying environment changes quite a bit uh then
[43:33] environment changes quite a bit uh then your your procedures become outdated and
[43:37] your your procedures become outdated and those papers don't really have a
[43:38] those papers don't really have a description of a way to fix that. So
[43:40] description of a way to fix that. So this is just something to keep in mind
[43:41] this is just something to keep in mind if you're, you know, trying to implement
[43:43] if you're, you know, trying to implement these procedural memories. Uh, is if you
[43:45] these procedural memories. Uh, is if you have a dynamic environment, you also
[43:47] have a dynamic environment, you also have to think about how to
[43:48] have to think about how to maintain the memory, uh, which we will
[43:50] maintain the memory, uh, which we will talk about as a design pattern later
[43:52] talk about as a design pattern later about how to do memory maintenance in
[43:54] about how to do memory maintenance in dynamic environments. But, uh, it's just
[43:56] dynamic environments. But, uh, it's just worth calling out as a limitation. Um,
[43:58] worth calling out as a limitation. Um, but skill files, uh, you could have the
[44:00] but skill files, uh, you could have the agent modify its own skill files, so
[44:02] agent modify its own skill files, so this is maybe one way of resolving that
[44:04] this is maybe one way of resolving that tension.
[44:05] tension. Cool
[44:07] Cool u. The next pattern I want to talk about
[44:08] u. The next pattern I want to talk about is tiered memory hierarchy. Um, and as
[44:12] is tiered memory hierarchy. Um, and as promised, I'm going to be talking about
[44:13] promised, I'm going to be talking about the uh MEM GPT paper. Uh, so let's break
[44:16] the uh MEM GPT paper. Uh, so let's break down what's going on here. So the key
[44:19] down what's going on here. So the key insight in the mem GPT paper is to give
[44:21] insight in the mem GPT paper is to give the LLM tools to manage its own memory.
[44:24] the LLM tools to manage its own memory. Uh, and so the way we are going to break
[44:26] Uh, and so the way we are going to break this down is we have these uh this
[44:28] this down is we have these uh this prompt here, the context window. Oops, I
[44:31] prompt here, the context window. Oops, I wanted to
[44:33] wanted to uh talk about it first. Okay, so we have
[44:35] uh talk about it first. Okay, so we have the context window here which has the
[44:37] the context window here which has the system instructions. uh then the working
[44:40] system instructions. uh then the working context. This is different uh which is a
[44:43] context. This is different uh which is a bit confusing. This is different than
[44:44] bit confusing. This is different than the working memory uh because by our
[44:47] the working memory uh because by our definition the working memory is the
[44:49] definition the working memory is the entire context here. Um but this is
[44:52] entire context here. Um but this is um the memory that we have kind of
[44:54] um the memory that we have kind of access to read and write to uh is the
[44:56] access to read and write to uh is the way of thinking about working context in
[44:58] way of thinking about working context in this uh paper. And then we have a first
[45:00] this uh paper. And then we have a first in first out Q and this is actually the
[45:02] in first out Q and this is actually the conversation history. Um so as we add a
[45:05] conversation history. Um so as we add a next turn we ask the agent to do
[45:07] next turn we ask the agent to do something uh it gets added to the FIFO
[45:09] something uh it gets added to the FIFO queue uh and then once we start running
[45:12] queue uh and then once we start running out of memory um or our context window
[45:14] out of memory um or our context window starts to get too long uh then the
[45:16] starts to get too long uh then the oldest messages will get evicted first.
[45:19] oldest messages will get evicted first. Um and so uh we can break down what's
[45:22] Um and so uh we can break down what's going on here. Um the messages are first
[45:24] going on here. Um the messages are first in first out. Um archival storage is for
[45:27] in first out. Um archival storage is for storing like long-term facts
[45:29] storing like long-term facts and uh world knowledge. And so this is
[45:32] and uh world knowledge. And so this is actually a vector database where we can
[45:34] actually a vector database where we can search over the archival memory. Um and
[45:37] search over the archival memory. Um and then for recall this is actually some
[45:40] then for recall this is actually some somewhat more like episodic memory. U so
[45:42] somewhat more like episodic memory. U so we actually when we evict something from
[45:44] we actually when we evict something from our message history it goes to the
[45:46] our message history it goes to the recall storage and then we can exact
[45:48] recall storage and then we can exact string lookup in the recall storage. Um
[45:50] string lookup in the recall storage. Um so I'll show some examples of this. Uh
[45:53] so I'll show some examples of this. Uh first of all are some functions to write
[45:55] first of all are some functions to write and replace things in memory. So
[45:57] and replace things in memory. So remember the working uh context is what
[45:59] remember the working uh context is what we can actually manipulate as a part of
[46:02] we can actually manipulate as a part of the system pro or the context window for
[46:04] the system pro or the context window for the agent.
[46:05] the agent. So
[46:06] So uh here we have um this conversation.
[46:09] uh here we have um this conversation. How was your day today? Uh fun. My
[46:11] How was your day today? Uh fun. My boyfriend James baked me a birthday
[46:12] boyfriend James baked me a birthday cake. Oh wow. Happy birthday. It knows
[46:14] cake. Oh wow. Happy birthday. It knows today's February 7th. Uh based on the
[46:16] today's February 7th. Uh based on the system prompt likely. Uh so then it
[46:19] system prompt likely. Uh so then it appends to the working context. Uh this
[46:21] appends to the working context. Uh this is the user's birthday and they have a
[46:23] is the user's birthday and they have a boyfriend named James. And then later, I
[46:26] boyfriend named James. And then later, I mean, unfortunately, I pulled this
[46:27] mean, unfortunately, I pulled this example from the paper. It's a sad story
[46:29] example from the paper. It's a sad story here, but um
[46:30] here, but um how's James doing on on Valentine's Day.
[46:33] how's James doing on on Valentine's Day. Um and then they say that James and I
[46:35] Um and then they say that James and I broke up. So, so then, uh it goes and
[46:37] broke up. So, so then, uh it goes and replaces the the working context and
[46:39] replaces the the working context and shoots them a little message. Hope
[46:40] shoots them a little message. Hope you're doing okay. Um so, this is like a
[46:43] you're doing okay. Um so, this is like a way of uh using tools to manipulate the
[46:45] way of uh using tools to manipulate the memory. Um
[46:47] memory. Um the uh other tool that I guess I didn't
[46:49] the uh other tool that I guess I didn't show here is you can store to the
[46:52] show here is you can store to the archival memory as well. But the recall
[46:54] archival memory as well. But the recall memory uh if we go back here, recall
[46:56] memory uh if we go back here, recall memory is strictly managed by this Q
[46:58] memory is strictly managed by this Q manager uh that uh pulls the memory or
[47:01] manager uh that uh pulls the memory or pulls the messages and puts them into
[47:03] pulls the messages and puts them into recall storage. Uh but you can have a
[47:05] recall storage. Uh but you can have a function that that says like this is
[47:06] function that that says like this is something to save to archival storage.
[47:09] something to save to archival storage. Um
[47:10] Um and so uh we also have some tools for
[47:12] and so uh we also have some tools for searching. So, we can recall uh
[47:15] searching. So, we can recall uh from recall storage. And so, uh I'm
[47:18] from recall storage. And so, uh I'm going a little out of order of the way
[47:20] going a little out of order of the way the paper goes because I guess the big
[47:21] the paper goes because I guess the big reveal is that they break up. But, uh
[47:23] reveal is that they break up. But, uh here it says, "Did you do anything to
[47:24] here it says, "Did you do anything to celebrate your birthday?" Uh yeah, we
[47:26] celebrate your birthday?" Uh yeah, we went to Six Flags. It can search uh for
[47:28] went to Six Flags. It can search uh for Six Flags in its conversation history.
[47:30] Six Flags in its conversation history. Uh and then it's uh they see a
[47:33] Uh and then it's uh they see a conversation where James
[47:34] conversation where James and they met at Six Flags. So, it's
[47:37] and they met at Six Flags. So, it's like, "Wow, it's so cute that you met uh
[47:39] like, "Wow, it's so cute that you met uh James at Six Flags." And then uh we also
[47:41] James at Six Flags." And then uh we also have recall from archival storage. So
[47:44] have recall from archival storage. So you can upload a bunch of documents to
[47:46] you can upload a bunch of documents to archival storage and then ask a question
[47:48] archival storage and then ask a question like who won the first Nobel prize in
[47:50] like who won the first Nobel prize in physics and it will search and there's
[47:52] physics and it will search and there's so many documents in there that it it
[47:53] so many documents in there that it it has pages. Uh so maybe it didn't you
[47:55] has pages. Uh so maybe it didn't you didn't get the answer you were looking
[47:56] didn't get the answer you were looking for. The agent didn't get the answer it
[47:58] for. The agent didn't get the answer it was looking for. So then it can search
[47:59] was looking for. So then it can search again and page it to page two and then
[48:02] again and page it to page two and then it finds uh that this is the first the
[48:05] it finds uh that this is the first the winner of the first Nobel Prize in
[48:06] winner of the first Nobel Prize in physics. Um, so and there's no need to
[48:08] physics. Um, so and there's no need to search the working memory of course
[48:09] search the working memory of course because that's all in context. So the
[48:11] because that's all in context. So the the you're relying on the LLM to search
[48:13] the you're relying on the LLM to search over that.
[48:14] over that. Uh, any questions about kind of how mem
[48:18] Uh, any questions about kind of how mem works?
[48:24] >> Uh, yeah. The question is like is there a named entity recognition uh set that's
[48:27] a named entity recognition uh set that's hidden? Uh, to my knowledge no actually
[48:29] hidden? Uh, to my knowledge no actually uh you can use something like BM25 or
[48:32] uh you can use something like BM25 or like exact string matching. So uh you
[48:34] like exact string matching. So uh you the entity matching entity recognition
[48:37] the entity matching entity recognition here is all done by the LM. The LM
[48:39] here is all done by the LM. The LM decided that six flags was the entity it
[48:40] decided that six flags was the entity it wanted to look up and then it did an
[48:42] wanted to look up and then it did an exact text match search over the recall
[48:45] exact text match search over the recall storage. So you don't have to do any uh
[48:47] storage. So you don't have to do any uh yeah any
[48:49] yeah any uh yes
[48:50] uh yes >> uh
[48:52] >> uh constantly rewriting your context and
[48:54] constantly rewriting your context and that
[48:55] that you know
[48:57] you know >> yes
[48:58] >> yes it's a great question. Uh so the
[49:00] it's a great question. Uh so the question is like what what about the KV
[49:02] question is like what what about the KV cache? How does that uh get sorted out
[49:04] cache? How does that uh get sorted out here? Uh yeah, I I I
[49:06] here? Uh yeah, I I I do think this is not super KV cache
[49:08] do think this is not super KV cache friendly as far as the other some of the
[49:10] friendly as far as the other some of the other things we we're looking at. Um
[49:12] other things we we're looking at. Um yeah, you do every time you rewrite
[49:13] yeah, you do every time you rewrite parts of your context, you do invalidate
[49:15] parts of your context, you do invalidate the KV cache and have to to refresh
[49:17] the KV cache and have to to refresh that. It's a good question. Uh any other
[49:20] that. It's a good question. Uh any other questions?
[49:25] Uh yes, >> you should differentiate working memory
[49:28] >> you should differentiate working memory versus context. memory is somewhere
[49:30] versus context. memory is somewhere that's beyond the
[49:32] that's beyond the >> Yeah. So the the working context is just
[49:34] >> Yeah. So the the working context is just a piece of the context here that we can
[49:37] a piece of the context here that we can call tools to read and write to. Uh
[49:40] call tools to read and write to. Uh working memory as I've been defining it
[49:41] working memory as I've been defining it before would be this whole block which
[49:43] before would be this whole block which is just everything that the model sees
[49:45] is just everything that the model sees is considered a part of it working
[49:46] is considered a part of it working memory. Um does that make sense?
[49:49] memory. Um does that make sense? >> Not all of them is in context.
[49:51] >> Not all of them is in context. >> Um so the things that are not in context
[49:53] >> Um so the things that are not in context here that are memories are like the
[49:54] here that are memories are like the these storage uh units and stuff like
[49:57] these storage uh units and stuff like that. So th this is not like this
[49:59] that. So th this is not like this document about who won the Nobel Prize
[50:01] document about who won the Nobel Prize and stuff is not stored in context. It's
[50:03] and stuff is not stored in context. It's separately looked up and then later
[50:05] separately looked up and then later added to the working context if it's
[50:06] added to the working context if it's useful.
[50:08] useful. >> Yeah, good question.
[50:09] >> Yeah, good question. Um
[50:11] Um so some results on MGBT.
[50:13] so some results on MGBT. Uh so you can see they tested several
[50:15] Uh so you can see they tested several different models and found that um
[50:18] different models and found that um once you added the memory uh it was able
[50:21] once you added the memory uh it was able to do better on this deep memory
[50:22] to do better on this deep memory retrieval benchmark. Um,
[50:24] retrieval benchmark. Um, and interestingly, uh, and importantly,
[50:27] and interestingly, uh, and importantly, when you typically retrieve more, you
[50:29] when you typically retrieve more, you know, documents, uh, using RAG and stuff
[50:32] know, documents, uh, using RAG and stuff like that, it's degrades the context for
[50:34] like that, it's degrades the context for the model. But by letting the model use
[50:37] the model. But by letting the model use tools to memor to run its own memory uh
[50:40] tools to memor to run its own memory uh I mean, you can argue if this is a
[50:42] I mean, you can argue if this is a little bit of a confusing figure or not
[50:43] little bit of a confusing figure or not but they don't like specify a number of
[50:45] but they don't like specify a number of documents retrieve because they just let
[50:47] documents retrieve because they just let the tools do the decisions and stuff.
[50:49] the tools do the decisions and stuff. And so you can see that MEGPT gets a lot
[50:51] And so you can see that MEGPT gets a lot higher accuracy, uh, than even like
[50:54] higher accuracy, uh, than even like retrieving only a few select documents
[50:56] retrieving only a few select documents and stuff like that. So ideally the
[50:58] and stuff like that. So ideally the performance is unaffected by increasing
[51:01] performance is unaffected by increasing the number of documents retrieved and
[51:02] the number of documents retrieved and other things that you would normally
[51:03] other things that you would normally have to worry about.
[51:06] have to worry about. Um any questions there?
[51:09] Um any questions there? If not then I'm going to briefly this
[51:11] If not then I'm going to briefly this one is more of like I wanted to motivate
[51:13] one is more of like I wanted to motivate it but I'm not planning to go in super
[51:15] it but I'm not planning to go in super depth on memory OS here. uh but I wanted
[51:17] depth on memory OS here. uh but I wanted to show that like there's other ways
[51:18] to show that like there's other ways that people are kind of thinking about
[51:20] that people are kind of thinking about uh how to do this tiered hierarchy of
[51:22] uh how to do this tiered hierarchy of memory. So in this paper they define a
[51:25] memory. So in this paper they define a short-term memory, a medium-term memory,
[51:27] short-term memory, a medium-term memory, and a long-term persona memory. And
[51:29] and a long-term persona memory. And really the way that it kind of moves
[51:31] really the way that it kind of moves between them is you have uh the agent
[51:33] between them is you have uh the agent working uh it pushes old messages to the
[51:36] working uh it pushes old messages to the short-term memory and then eventually uh
[51:40] short-term memory and then eventually uh it goes in this first in first out queue
[51:42] it goes in this first in first out queue and eventually it decides if it needs to
[51:44] and eventually it decides if it needs to insert it to medium-term memory. And so
[51:46] insert it to medium-term memory. And so it does some analysis over that. Uh
[51:48] it does some analysis over that. Uh edits the memory. Uh saves some of the
[51:50] edits the memory. Uh saves some of the top most useful ones to the to the
[51:53] top most useful ones to the to the stream. And then if it's like above a
[51:55] stream. And then if it's like above a certain value of like how often people
[51:58] certain value of like how often people are accessing these memories, then it
[52:00] are accessing these memories, then it also stores it to this long-term persona
[52:02] also stores it to this long-term persona memory. Uh so uh you can read the paper
[52:05] memory. Uh so uh you can read the paper for details here, but the idea is like
[52:06] for details here, but the idea is like we have these tiers of memory. Uh and
[52:08] we have these tiers of memory. Uh and then you can move from one tier to the
[52:10] then you can move from one tier to the next depending on how much you're
[52:12] next depending on how much you're accessing that memory. So this is kind
[52:13] accessing that memory. So this is kind of another tiered structure. Uh
[52:16] of another tiered structure. Uh yeah, any any questions there? Yes, I
[52:18] yeah, any any questions there? Yes, I can try to answer.
[52:21] can try to answer. >> Um how is like in the field like what is
[52:23] >> Um how is like in the field like what is the most common way to like report the
[52:24] the most common way to like report the accuracy metrics cuz I assume there's
[52:26] accuracy metrics cuz I assume there's like a trade-off between
[52:28] like a trade-off between skew like the memory footprint and
[52:30] skew like the memory footprint and >> yeah, great question. So the question is
[52:33] >> yeah, great question. So the question is like how what's the standard way of per
[52:36] like how what's the standard way of per uh reporting performance of some of
[52:37] uh reporting performance of some of these like memory things because you
[52:38] these like memory things because you have to worry about latency and accuracy
[52:41] have to worry about latency and accuracy and uh cost and all these things. Uh I
[52:44] and uh cost and all these things. Uh I would say there are a few standard
[52:46] would say there are a few standard benchmarks that are used. Um
[52:49] benchmarks that are used. Um and there are some papers that report
[52:50] and there are some papers that report latency and stuff like that.
[52:51] latency and stuff like that. Unfortunately, I wouldn't say that like
[52:53] Unfortunately, I wouldn't say that like the field has developed a standard
[52:55] the field has developed a standard metric. Um
[52:56] metric. Um but there like I said there are a few
[52:58] but there like I said there are a few standard benchmarks and so you report
[53:00] standard benchmarks and so you report the metrics that are associated with
[53:01] the metrics that are associated with those benchmarks which tend to be
[53:02] those benchmarks which tend to be accuracy um on the tasks. I think it
[53:06] accuracy um on the tasks. I think it would be great if there was a more
[53:08] would be great if there was a more standardized like literature survey that
[53:10] standardized like literature survey that does like let's do a strong comparison
[53:11] does like let's do a strong comparison against like we're going to measure
[53:13] against like we're going to measure latency and all of these things and just
[53:15] latency and all of these things and just get the cross productduct of all the
[53:16] get the cross productduct of all the things you could care about. I don't. I
[53:18] things you could care about. I don't. I feel, because like, I mean, you're
[53:19] feel, because like, I mean, you're seeing like these papers are coming out
[53:20] seeing like these papers are coming out in like 2025 and stuff like that, is
[53:22] in like 2025 and stuff like that, is like a year old; if and and stuff like
[53:24] like a year old; if and and stuff like that, then some people just we, I guess,
[53:27] that, then some people just we, I guess, haven't really developed a standardized,
[53:29] haven't really developed a standardized, uh, system but. it's it is an important
[53:31] uh, system but. it's it is an important thing, and it would be good if that was
[53:32] thing, and it would be good if that was done. Yeah. Push them. Yeah. in terms of
[53:36] done. Yeah. Push them. Yeah. in terms of curing. So is the main difference
[53:37] curing. So is the main difference between this paper and ME megp
[53:40] between this paper and ME megp is that uh you have three tiers there
[53:42] is that uh you have three tiers there and you just have two tiers there.
[53:44] and you just have two tiers there. >> I'd say the main difference uh so the
[53:46] >> I'd say the main difference uh so the question is like is the main difference
[53:47] question is like is the main difference that there's three tiers here and two
[53:48] that there's three tiers here and two tiers there. Uh actually I think an even
[53:51] tiers there. Uh actually I think an even more uh important distinction I think
[53:53] more uh important distinction I think that's one distinction that is good. The
[53:55] that's one distinction that is good. The other is like what is actually doing the
[53:57] other is like what is actually doing the memory management? Uh so here you have
[53:58] memory management? Uh so here you have this whole workflow that like analyzes
[54:00] this whole workflow that like analyzes how many things were used across traces
[54:02] how many things were used across traces and that's what makes the decision. Uh
[54:03] and that's what makes the decision. Uh whereas in MEGBT was a lot simpler. It
[54:05] whereas in MEGBT was a lot simpler. It was like the LLM itself says this goes
[54:07] was like the LLM itself says this goes to archival storage. Um I guess the
[54:10] to archival storage. Um I guess the recall storage was like based on just
[54:12] recall storage was like based on just the FIFO Q. Uh but things like decisions
[54:14] the FIFO Q. Uh but things like decisions about what to store to long-term memory
[54:16] about what to store to long-term memory and how to manipulate its own context
[54:17] and how to manipulate its own context were all decided by the model. Whereas
[54:19] were all decided by the model. Whereas here it's like this be behind the scenes
[54:21] here it's like this be behind the scenes system that's making some of those
[54:22] system that's making some of those decisions. Good question. Uh any other
[54:25] decisions. Good question. Uh any other questions?
[54:27] questions? Uh cool. Then let's move on to pattern
[54:30] Uh cool. Then let's move on to pattern three which is reflective memory
[54:31] three which is reflective memory consolidation. Um,
[54:34] consolidation. Um, so a quick refresher, but we're not
[54:36] so a quick refresher, but we're not going to talk about it for very long, is
[54:37] going to talk about it for very long, is reflection. Uh, so if you remember, uh,
[54:40] reflection. Uh, so if you remember, uh, we talked about this paper now twice, so
[54:42] we talked about this paper now twice, so I I I'll go quite quick, but we take
[54:44] I I I'll go quite quick, but we take actions on the environment, we get our
[54:46] actions on the environment, we get our trajectories, we evaluate them, we
[54:47] trajectories, we evaluate them, we reflect on it, and we store it to
[54:48] reflect on it, and we store it to experience. And so now we can update the
[54:51] experience. And so now we can update the memory of the model based on these
[54:53] memory of the model based on these experiences that we've reflected on. And
[54:55] experiences that we've reflected on. And uh, I talked about this last time, but
[54:57] uh, I talked about this last time, but the reflection step itself was
[54:59] the reflection step itself was important. So this was episodic memory.
[55:02] important. So this was episodic memory. Um, and you can see just putting all of
[55:04] Um, and you can see just putting all of the old traces into the
[55:06] the old traces into the uh context window was not as good as
[55:09] uh context window was not as good as putting the reflections on why the
[55:11] putting the reflections on why the traces were useful into the context
[55:12] traces were useful into the context window. Um, so now to talk about some
[55:15] window. Um, so now to talk about some papers that maybe you're less familiar
[55:17] papers that maybe you're less familiar with because we haven't talked about
[55:18] with because we haven't talked about them as much, although I think D did
[55:20] them as much, although I think D did mention this in the first lecture, but
[55:21] mention this in the first lecture, but generative agents. So, uh, this is like
[55:24] generative agents. So, uh, this is like a very, this is actually an HCI paper.
[55:26] a very, this is actually an HCI paper. So you can see, you can tell right away
[55:28] So you can see, you can tell right away that the figures are really pretty and
[55:29] that the figures are really pretty and uh there's lots of exciting stuff going
[55:31] uh there's lots of exciting stuff going on. Um but uh basically the idea was
[55:34] on. Um but uh basically the idea was like let's simulate these uh generative
[55:36] like let's simulate these uh generative agents that will have sort of a working
[55:38] agents that will have sort of a working history of their interactions with other
[55:40] history of their interactions with other agents. Uh they can have goals and and
[55:42] agents. Uh they can have goals and and take actions within this world. Um and I
[55:45] take actions within this world. Um and I believe the main story line in the paper
[55:47] believe the main story line in the paper is that this uh Isabella agent wants to
[55:49] is that this uh Isabella agent wants to organize a Valentine's party and invite
[55:51] organize a Valentine's party and invite everybody to her Valentine's party. Um,
[55:53] everybody to her Valentine's party. Um, so this is the memory architecture that
[55:55] so this is the memory architecture that they use in the paper. Um,
[55:58] they use in the paper. Um, so you have a memory stream which is
[56:00] so you have a memory stream which is I'll show you in a second is is the
[56:01] I'll show you in a second is is the episodic memory. I actually showed it at
[56:02] episodic memory. I actually showed it at the beginning of the lecture. Um, but we
[56:04] the beginning of the lecture. Um, but we have all of our perceptions and
[56:06] have all of our perceptions and observations uh get added to the memory
[56:08] observations uh get added to the memory stream. Um, and we can retrieve memories
[56:11] stream. Um, and we can retrieve memories from the memory stream. Uh, but working
[56:13] from the memory stream. Uh, but working on top of the retrieved memories, we can
[56:15] on top of the retrieved memories, we can actually create plans and uh decide how
[56:18] actually create plans and uh decide how we're going to act for a day or
[56:19] we're going to act for a day or something like that as these agents. uh
[56:20] something like that as these agents. uh we can also make reflections on what has
[56:22] we can also make reflections on what has happened and all of this goes back into
[56:24] happened and all of this goes back into the memory stream uh and into that log
[56:27] the memory stream uh and into that log and then ultimately the actions we take
[56:29] and then ultimately the actions we take on the world is decided from that. Um
[56:32] on the world is decided from that. Um and so here's an example uh we have our
[56:34] and so here's an example uh we have our memory stream uh where all of these
[56:36] memory stream uh where all of these details are about what's going on in
[56:37] details are about what's going on in Isabella's day. Um
[56:40] Isabella's day. Um and then we ask the question um and
[56:43] and then we ask the question um and maybe like another agent asks this
[56:45] maybe like another agent asks this question or something like that you can
[56:46] question or something like that you can imagine. And it says what are you
[56:47] imagine. And it says what are you looking forward to most right now?
[56:50] looking forward to most right now? And uh it's retrieves um based on
[56:54] And uh it's retrieves um based on uh this is a reflection. I believe uh
[56:57] uh this is a reflection. I believe uh Isabella is excited to be planning a
[56:58] Isabella is excited to be planning a Valentine's Day party. Um and so it
[57:01] Valentine's Day party. Um and so it determined that this was quite recent,
[57:03] determined that this was quite recent, important, and relevant. Uh and so it
[57:05] important, and relevant. Uh and so it got the highest score in the retrieval.
[57:07] got the highest score in the retrieval. Uh and so it put and it says, "I'm also
[57:09] Uh and so it put and it says, "I'm also ordering decorations, researching ideas
[57:11] ordering decorations, researching ideas for the party." And then to respond to
[57:13] for the party." And then to respond to the question, it says, "I'm looking
[57:14] the question, it says, "I'm looking forward to the Valentine's Day party
[57:15] forward to the Valentine's Day party that I'm planning at Hobs Cafe." So
[57:17] that I'm planning at Hobs Cafe." So that's this is sort of an example of
[57:19] that's this is sort of an example of episodic memory. Um and the
[57:21] episodic memory. Um and the point I wanted to bring up is this
[57:23] point I wanted to bring up is this reflection like look these reflections
[57:25] reflection like look these reflections are getting added to the memory. Um this
[57:27] are getting added to the memory. Um this is a common pattern you'll see where we
[57:29] is a common pattern you'll see where we have not just the exact log being stored
[57:32] have not just the exact log being stored as memory but also reflections on that
[57:34] as memory but also reflections on that log.
[57:35] log. Um and just to kind of drive home the
[57:37] Um and just to kind of drive home the importance of this reflection they ran
[57:39] importance of this reflection they ran an ablation. Um, and so they were
[57:42] an ablation. Um, and so they were looking at if you run the full
[57:43] looking at if you run the full architecture versus no reflection versus
[57:46] architecture versus no reflection versus no planning, observation, uh, and then
[57:48] no planning, observation, uh, and then human crowdworker, which I, I believe
[57:49] human crowdworker, which I, I believe what they did is had somebody uh, this
[57:51] what they did is had somebody uh, this is maybe a kind of a fun task, but like
[57:53] is maybe a kind of a fun task, but like working in the environment, playing one
[57:55] working in the environment, playing one of the agents and stuff like that. And
[57:56] of the agents and stuff like that. And they showed that like essentially the
[57:58] they showed that like essentially the reflection was one of the key components
[58:00] reflection was one of the key components to making the agent uh, get this true
[58:03] to making the agent uh, get this true skill rank, which was like a measure of
[58:05] skill rank, which was like a measure of how realistic the simulation was. Um,
[58:08] how realistic the simulation was. Um, so any questions about that? Otherwise,
[58:10] so any questions about that? Otherwise, I'll start talking about memory
[58:12] I'll start talking about memory maintenance, which is the last pattern I
[58:14] maintenance, which is the last pattern I wanted to dive into today.
[58:20] >> Cool. Then let's talk about memory maintenance. I think this is actually
[58:21] maintenance. I think this is actually like a
[58:22] like a very active research area and and this
[58:24] very active research area and and this one I think is pretty particularly
[58:26] one I think is pretty particularly interesting. So the first paper I want
[58:27] interesting. So the first paper I want to talk about uh is sleeptime compute
[58:30] to talk about uh is sleeptime compute and I'm I'm at least a big fan of the
[58:32] and I'm I'm at least a big fan of the idea of this. Um I I think uh you can
[58:35] idea of this. Um I I think uh you can debate if it's it's fair to call it
[58:37] debate if it's it's fair to call it sleeping. I think some people criticize
[58:39] sleeping. I think some people criticize it based on that. But uh let's talk
[58:40] it based on that. But uh let's talk about the method. So uh here we have a
[58:44] about the method. So uh here we have a uh raw context which is a question that
[58:46] uh raw context which is a question that the user asked the agent. And so we
[58:48] the user asked the agent. And so we could say uh a juggler can juggle 800
[58:50] could say uh a juggler can juggle 800 balls. A quarter of them are tennis
[58:52] balls. A quarter of them are tennis balls and half of those are indigo. Uh
[58:54] balls and half of those are indigo. Uh which one tenth of those are marked? And
[58:57] which one tenth of those are marked? And so they have this like conversation
[58:59] so they have this like conversation about this problem and then um you can
[59:03] about this problem and then um you can in the normal uh condition later then
[59:06] in the normal uh condition later then you can start asking the LM questions
[59:08] you can start asking the LM questions about this. So we ask how many marked
[59:11] about this. So we ask how many marked indigo tennis balls are there. Uh and
[59:13] indigo tennis balls are there. Uh and the LM starts reasoning to solve the
[59:15] the LM starts reasoning to solve the problem. We'll follow these steps. Uh
[59:17] problem. We'll follow these steps. Uh the answer is 10. Eventually gets there.
[59:19] the answer is 10. Eventually gets there. How many tennis balls are there? It
[59:20] How many tennis balls are there? It reasons over it. But in the sleeptime
[59:22] reasons over it. But in the sleeptime compute setting this is like a
[59:23] compute setting this is like a conversation that maybe you had with the
[59:25] conversation that maybe you had with the model yesterday. Uh and so there's all
[59:27] model yesterday. Uh and so there's all of these all this time that the model is
[59:29] of these all this time that the model is not doing anything. The agent's not
[59:31] not doing anything. The agent's not doing anything. Uh so what should it do
[59:32] doing anything. Uh so what should it do at that time? Uh well, it can reflect on
[59:34] at that time? Uh well, it can reflect on the prior conversations and reason about
[59:36] the prior conversations and reason about them and think ahead. Uh so uh it starts
[59:39] them and think ahead. Uh so uh it starts reasoning, asking questions about this.
[59:41] reasoning, asking questions about this. So the juggler can juggle 800 balls, a
[59:43] So the juggler can juggle 800 balls, a quarter of them. So like it starts doing
[59:44] quarter of them. So like it starts doing math and it says like what can I analyze
[59:46] math and it says like what can I analyze about here? Uh and so later the next day
[59:48] about here? Uh and so later the next day you come back and and this is now all in
[59:51] you come back and and this is now all in the past context and you go how many
[59:53] the past context and you go how many marked indigo tennis balls are there? uh
[59:55] marked indigo tennis balls are there? uh the LM retrieves from its uh learned
[59:58] the LM retrieves from its uh learned context that it analyzed itself and it
[01:00:00] context that it analyzed itself and it can right away just tell you the answer
[01:00:01] can right away just tell you the answer is 10 because it had figured that out
[01:00:03] is 10 because it had figured that out last night. Um and so like there are
[01:00:05] last night. Um and so like there are challenges here. Obviously you don't
[01:00:07] challenges here. Obviously you don't know what the questions are going to be.
[01:00:08] know what the questions are going to be. So if you did then like obvious this is
[01:00:10] So if you did then like obvious this is very obvious like I should I should
[01:00:12] very obvious like I should I should answer these questions but you don't
[01:00:13] answer these questions but you don't know what the user is going to ask. So
[01:00:14] know what the user is going to ask. So it's the actual challenge is like
[01:00:16] it's the actual challenge is like figuring out what to reason over and
[01:00:18] figuring out what to reason over and obviously there's like ex this is
[01:00:19] obviously there's like ex this is expensive because you're just reasoning
[01:00:21] expensive because you're just reasoning over as many things as you can
[01:00:22] over as many things as you can brainstorm and you don't know that it'll
[01:00:24] brainstorm and you don't know that it'll ever actually matter. Uh but this is
[01:00:26] ever actually matter. Uh but this is kind of an interesting idea. Uh and the
[01:00:27] kind of an interesting idea. Uh and the way it uh this is this example is not
[01:00:30] way it uh this is this example is not about memory so much. I mean you can
[01:00:32] about memory so much. I mean you can kind of argue it's memory but I think
[01:00:33] kind of argue it's memory but I think there's there's ways to connect it to
[01:00:34] there's there's ways to connect it to memory that I'll show you. Um
[01:00:36] memory that I'll show you. Um so this is an example from the blog post
[01:00:38] so this is an example from the blog post associated with the paper. Um
[01:00:41] associated with the paper. Um and so here's the original context.
[01:00:43] and so here's the original context. These are semantic memories uh as we've
[01:00:45] These are semantic memories uh as we've been describing them. Uh we have the
[01:00:46] been describing them. Uh we have the user's name is Chad. Chad likes blue.
[01:00:49] user's name is Chad. Chad likes blue. Chad likes red more than blue. Chad
[01:00:50] Chad likes red more than blue. Chad likes gray the most. Chad hates green.
[01:00:52] likes gray the most. Chad hates green. So this is obviously a toy example, but
[01:00:54] So this is obviously a toy example, but you can imagine there are things where
[01:00:56] you can imagine there are things where like a lot of your memories are related
[01:00:57] like a lot of your memories are related to each other. Uh and so in the
[01:00:59] to each other. Uh and so in the sleeptime compute regime, what we can do
[01:01:02] sleeptime compute regime, what we can do is reason over these memories that we
[01:01:04] is reason over these memories that we have and try to consolidate them into a
[01:01:06] have and try to consolidate them into a more useful format. So we can say, okay,
[01:01:09] more useful format. So we can say, okay, the user's name is Chad. We're going to
[01:01:10] the user's name is Chad. We're going to preserve that. But what about all these
[01:01:12] preserve that. But what about all these color things? Like this is this is
[01:01:14] color things? Like this is this is annoying to reason over. If we have to
[01:01:15] annoying to reason over. If we have to retrieve all of these and do like almost
[01:01:16] retrieve all of these and do like almost a little it's an easy one, but an easy
[01:01:19] a little it's an easy one, but an easy little logic puzzle every time we have
[01:01:20] little logic puzzle every time we have to do something about his colors. Um
[01:01:22] to do something about his colors. Um well, let's actually rewrite this and
[01:01:24] well, let's actually rewrite this and say the color preferences are gray, red,
[01:01:26] say the color preferences are gray, red, blue, and lowest is green, which he
[01:01:28] blue, and lowest is green, which he hates. Um so now when we go and retrieve
[01:01:31] hates. Um so now when we go and retrieve things about Chad's color preferences,
[01:01:33] things about Chad's color preferences, uh we've already done some computation
[01:01:35] uh we've already done some computation over it and made it into an easier to to
[01:01:38] over it and made it into an easier to to reason over form. uh and so they they
[01:01:40] reason over form. uh and so they they show in the paper uh this is on Amy
[01:01:43] show in the paper uh this is on Amy which is a math benchmark. So not as
[01:01:46] which is a math benchmark. So not as much about this like agent memory
[01:01:48] much about this like agent memory setting again this is kind of back to
[01:01:49] setting again this is kind of back to the the math computation setting. Uh you
[01:01:51] the the math computation setting. Uh you can see test time tokens. So there they
[01:01:53] can see test time tokens. So there they do not in this figure count sleeptime
[01:01:55] do not in this figure count sleeptime tokens. So you can imagine it's it's way
[01:01:57] tokens. So you can imagine it's it's way more expensive actually. But when you
[01:01:58] more expensive actually. But when you actually ask the question how many
[01:02:00] actually ask the question how many tokens do you need to spend to answer
[01:02:01] tokens do you need to spend to answer it? uh you can see how much is gained
[01:02:04] it? uh you can see how much is gained from using this sleepet time compute
[01:02:06] from using this sleepet time compute versus just sampling a bunch of times
[01:02:08] versus just sampling a bunch of times and doing pass at K. Uh so you can see
[01:02:09] and doing pass at K. Uh so you can see there's a lot of benefit to doing this
[01:02:12] there's a lot of benefit to doing this like offline sort of reasoning over the
[01:02:14] like offline sort of reasoning over the the context. Uh any questions about this
[01:02:17] the context. Uh any questions about this idea of like sleep time compute and
[01:02:18] idea of like sleep time compute and memory consolidation? Yes.
[01:02:20] memory consolidation? Yes. >> Is it actually like more accurate at the
[01:02:22] >> Is it actually like more accurate at the end of the day or is it just like a
[01:02:24] end of the day or is it just like a quicker way to get the answer?
[01:02:25] quicker way to get the answer? >> That's a good question. The question is
[01:02:26] >> That's a good question. The question is is it more accurate or is it just
[01:02:28] is it more accurate or is it just quicker? Um I think that in it depends.
[01:02:31] quicker? Um I think that in it depends. I mean, you can even see from this
[01:02:32] I mean, you can even see from this figure: there's a few cases where, uh,
[01:02:34] figure: there's a few cases where, uh, it does outperform, and this looks like
[01:02:37] it does outperform, and this looks like it's coming to level off. But I mean, I
[01:02:40] it's coming to level off. But I mean, I imagine it c it can't really be
[01:02:43] imagine it c it can't really be more accurate because the gold thing to
[01:02:46] more accurate because the gold thing to do would be to
[01:02:48] do would be to ask the question itself and then reason
[01:02:50] ask the question itself and then reason over that for a really long period of
[01:02:52] over that for a really long period of time. So theoretically I don't think it
[01:02:55] time. So theoretically I don't think it could be more accurate if you spent as
[01:02:57] could be more accurate if you spent as much time on test time compute as you
[01:02:59] much time on test time compute as you spent on sleep time compute if that
[01:03:00] spent on sleep time compute if that makes sense. Um,
[01:03:02] makes sense. Um, but it is so the, the intention is
[01:03:04] but it is so the, the intention is really to make it like snappier and. And
[01:03:06] really to make it like snappier and. And there are, in the practical regime, it
[01:03:09] there are, in the practical regime, it could be more accurate because, like,
[01:03:10] could be more accurate because, like, let's say you have to answer a user
[01:03:12] let's say you have to answer a user question and you only want to spend like
[01:03:13] question and you only want to spend like a thousand tokens reasoning to answer
[01:03:15] a thousand tokens reasoning to answer the user question because you need to
[01:03:16] the user question because you need to get them an answer. Uh then then it
[01:03:18] get them an answer. Uh then then it could be more accurate uh for the same
[01:03:20] could be more accurate uh for the same you know budget for instance. It's a
[01:03:21] you know budget for instance. It's a good question. Uh yeah.
[01:03:23] good question. Uh yeah. >> Is there like a limit to what can be
[01:03:25] >> Is there like a limit to what can be done in the sleep time compute? Like for
[01:03:28] done in the sleep time compute? Like for example, if it has access to certain
[01:03:29] example, if it has access to certain data,
[01:03:30] data, maybe it starts like running some models
[01:03:32] maybe it starts like running some models or like clustering the data or something
[01:03:34] or like clustering the data or something or is there like a limit on just like
[01:03:35] or is there like a limit on just like simple task?
[01:03:37] simple task? >> Yeah, it's a good question. Uh the
[01:03:38] >> Yeah, it's a good question. Uh the question is like is there a limit on
[01:03:39] question is like is there a limit on what it can do in sleep time compute? Uh
[01:03:41] what it can do in sleep time compute? Uh I think in this uh paper they to mine I
[01:03:44] I think in this uh paper they to mine I I I don't want to say this with a full
[01:03:46] I I don't want to say this with a full confidence but I don't think they gave
[01:03:48] confidence but I don't think they gave it tools or anything like that. It was
[01:03:49] it tools or anything like that. It was just reasoning, but you could imagine
[01:03:50] just reasoning, but you could imagine some sort of sleeptime agent that can do
[01:03:52] some sort of sleeptime agent that can do things. Uh, it becomes sort of like a
[01:03:54] things. Uh, it becomes sort of like a safety question then, like what is your
[01:03:56] safety question then, like what is your agent doing behind the scenes when
[01:03:57] agent doing behind the scenes when you're not watching? Uh, but yeah, it's
[01:03:59] you're not watching? Uh, but yeah, it's a good question. Yeah, I think that's
[01:04:01] a good question. Yeah, I think that's maybe the best answer I have for you.
[01:04:02] maybe the best answer I have for you. Uh, yes.
[01:04:04] Uh, yes. I I wonder like what what constitutes
[01:04:06] I I wonder like what what constitutes like as one single unit of memory cuz uh
[01:04:11] like as one single unit of memory cuz uh you showing this kind of helped me
[01:04:13] you showing this kind of helped me reflect back to the like the previous
[01:04:15] reflect back to the like the previous papers you showed which I wasn't really
[01:04:18] papers you showed which I wasn't really clear like what constitutes like what
[01:04:20] clear like what constitutes like what piece memory how do they distinguish
[01:04:23] piece memory how do they distinguish these things?
[01:04:24] these things? >> Yeah. Yeah. So the question is like how
[01:04:26] >> Yeah. Yeah. So the question is like how do you distinguish a piece of memory? I
[01:04:28] do you distinguish a piece of memory? I guess um in in this paper they're
[01:04:31] guess um in in this paper they're they're looking specifically at these
[01:04:32] they're looking specifically at these semantic memories. Um so this is
[01:04:34] semantic memories. Um so this is actually comes from the same line of
[01:04:35] actually comes from the same line of work as memg.
[01:04:36] work as memg. So you can think of it as like some of
[01:04:38] So you can think of it as like some of these statements that they would write
[01:04:39] these statements that they would write to the working memory type uh of
[01:04:41] to the working memory type uh of statements. U but I I think like yeah
[01:04:44] statements. U but I I think like yeah altogether there is
[01:04:46] altogether there is it is a bit confusing because you could
[01:04:48] it is a bit confusing because you could consider a lot of things to be memory.
[01:04:49] consider a lot of things to be memory. So there there's the procedural the all
[01:04:51] So there there's the procedural the all the types of memory and they're all kind
[01:04:52] the types of memory and they're all kind of stored in different formats. So I
[01:04:54] of stored in different formats. So I think it's somewhat you have to like
[01:04:56] think it's somewhat you have to like determine for the system you're talking
[01:04:57] determine for the system you're talking about what is considered memory in this
[01:04:59] about what is considered memory in this system.
[01:05:00] system. Uh, I'm going to move on because we
[01:05:01] Uh, I'm going to move on because we actually don't have too much time left.
[01:05:03] actually don't have too much time left. I I might skip a few papers that are a
[01:05:05] I I might skip a few papers that are a little bit less uh well
[01:05:07] little bit less uh well used I guess. But um the the next one I
[01:05:10] used I guess. But um the the next one I was going to talk about is Ame, which is
[01:05:12] was going to talk about is Ame, which is agentic memory for LM agents. Uh and so
[01:05:15] agentic memory for LM agents. Uh and so uh here you can see traditional memory.
[01:05:17] uh here you can see traditional memory. Uh and what they're kind of pointing out
[01:05:19] Uh and what they're kind of pointing out to here is MEMGBT where it's like the LM
[01:05:22] to here is MEMGBT where it's like the LM agent has the ability to write and read
[01:05:24] agent has the ability to write and read its own memory with tools or or some
[01:05:26] its own memory with tools or or some some things. I mean doesn't have to be
[01:05:28] some things. I mean doesn't have to be MGT. This is like a lot of agents, but
[01:05:30] MGT. This is like a lot of agents, but they say, "Like what if we had an agent
[01:05:32] they say, "Like what if we had an agent managing our memory?"
[01:05:34] managing our memory?" And so, uh, in the interest of time,
[01:05:36] And so, uh, in the interest of time, maybe I won't talk so much about the
[01:05:37] maybe I won't talk so much about the method here. But the idea here is that
[01:05:40] method here. But the idea here is that we can draw links between memories, uh,
[01:05:43] we can draw links between memories, uh, evolve our memories, iterate on our
[01:05:45] evolve our memories, iterate on our memories, um, and we can decide like
[01:05:48] memories, um, and we can decide like what gets stored. And we actually have
[01:05:50] what gets stored. And we actually have an agent whose whole job is memory
[01:05:52] an agent whose whole job is memory management so that when we ask a
[01:05:53] management so that when we ask a question later, uh, the agent can can
[01:05:56] question later, uh, the agent can can get us the response that we need. Um and
[01:05:58] get us the response that we need. Um and so this is why I I kind of categorize
[01:06:00] so this is why I I kind of categorize this under memory management. And this
[01:06:01] this under memory management. And this mem zero paper is quite similar in this
[01:06:04] mem zero paper is quite similar in this idea of having sort of like a workflow
[01:06:06] idea of having sort of like a workflow manage the memory. Uh so you could see
[01:06:09] manage the memory. Uh so you could see we have our messages uh between the user
[01:06:11] we have our messages uh between the user and the assistant. Uh we extract out
[01:06:14] and the assistant. Uh we extract out some details from them and we can update
[01:06:16] some details from them and we can update our our memory. Uh where we have these
[01:06:19] our our memory. Uh where we have these add update delete no ops. Um, and
[01:06:22] add update delete no ops. Um, and essentially what's happening behind the
[01:06:24] essentially what's happening behind the scenes is we have an entity extractor
[01:06:26] scenes is we have an entity extractor and a relationship generator uh from all
[01:06:28] and a relationship generator uh from all the things we find in the message. And
[01:06:30] the things we find in the message. And this works with this Zep knowledge graph
[01:06:31] this works with this Zep knowledge graph that I talked about earlier where we can
[01:06:33] that I talked about earlier where we can detect conflicts and resolve them with
[01:06:35] detect conflicts and resolve them with updates. And so we keep our knowledge
[01:06:37] updates. And so we keep our knowledge graph constantly up to date and keep
[01:06:39] graph constantly up to date and keep updating it and iterating on it uh in
[01:06:41] updating it and iterating on it uh in order to make sure that everything is
[01:06:44] order to make sure that everything is uh
[01:06:45] uh you know up to date with the latest from
[01:06:46] you know up to date with the latest from the conversations. So just to recap um
[01:06:49] the conversations. So just to recap um maybe you can uh save this picture to
[01:06:52] maybe you can uh save this picture to read offline uh because I I'll just uh
[01:06:54] read offline uh because I I'll just uh move on but basically these are the
[01:06:55] move on but basically these are the combinations of like what does each of
[01:06:57] combinations of like what does each of these store when it writes when it reads
[01:06:59] these store when it writes when it reads what is the main trade-off. This is kind
[01:07:00] what is the main trade-off. This is kind of like a highle overview of everything
[01:07:03] of like a highle overview of everything we just talked about.
[01:07:05] we just talked about. Um so next section I want to talk about
[01:07:07] Um so next section I want to talk about is evaluating agent memory. Um so these
[01:07:10] is evaluating agent memory. Um so these are going to be some benchmarks. Um
[01:07:12] are going to be some benchmarks. Um and so some desitter
[01:07:14] and so some desitter uh we care that these agent benchmarks
[01:07:16] uh we care that these agent benchmarks are realistic. Uh so the memory source
[01:07:19] are realistic. Uh so the memory source uh the hay stack uh and also the queries
[01:07:21] uh the hay stack uh and also the queries should reflect like real usage. Uh we
[01:07:24] should reflect like real usage. Uh we care that it's long context or like
[01:07:25] care that it's long context or like requires memory because if it's not long
[01:07:28] requires memory because if it's not long enough then we're not really
[01:07:28] enough then we're not really benchmarking memory at all. Uh we care
[01:07:31] benchmarking memory at all. Uh we care that they're challenging. Uh and we care
[01:07:33] that they're challenging. Uh and we care that they're verifiable and we know when
[01:07:34] that they're verifiable and we know when the system uh is correct. So these are
[01:07:36] the system uh is correct. So these are the four that we'll talk about. Um but
[01:07:38] the four that we'll talk about. Um but maybe I will especially focus on memory
[01:07:41] maybe I will especially focus on memory arena because I think that one's pretty
[01:07:42] arena because I think that one's pretty particularly high quality. Um so the
[01:07:45] particularly high quality. Um so the first one is Locom mo uh which is
[01:07:48] first one is Locom mo uh which is evaluating long-term conversational
[01:07:49] evaluating long-term conversational memory of LLM agents.
[01:07:52] memory of LLM agents. And uh what they do in this is they have
[01:07:54] And uh what they do in this is they have conversations uh between simulated
[01:07:56] conversations uh between simulated users. So, they actually use the
[01:07:57] users. So, they actually use the generative agents paper that I showed
[01:07:59] generative agents paper that I showed you earlier with Isabella and her
[01:08:00] you earlier with Isabella and her Valentine's party and they simulate a
[01:08:02] Valentine's party and they simulate a bunch of conversations between agents
[01:08:04] bunch of conversations between agents and then they have these long uh like
[01:08:07] and then they have these long uh like 16,000 token conversations over several
[01:08:09] 16,000 token conversations over several days and then they ask questions about
[01:08:11] days and then they ask questions about these uh me memories from these
[01:08:13] these uh me memories from these conversations. So, it's like uh in the
[01:08:15] conversations. So, it's like uh in the conversation they said last night we
[01:08:18] conversation they said last night we celebrated my daughter's birthday. Whose
[01:08:19] celebrated my daughter's birthday. Whose birthday did they celebrate? It's their
[01:08:21] birthday did they celebrate? It's their daughter. And they have all of these
[01:08:23] daughter. And they have all of these different tasks. Um, so like I said,
[01:08:25] different tasks. Um, so like I said, they use the generative agents
[01:08:26] they use the generative agents architecture. They simulate agents. They
[01:08:29] architecture. They simulate agents. They have events that go on in their lives
[01:08:31] have events that go on in their lives and they have these long running
[01:08:32] and they have these long running sessions. Uh, and then later they have a
[01:08:34] sessions. Uh, and then later they have a human come in and edit it and make sure
[01:08:36] human come in and edit it and make sure everything's consistent and the there
[01:08:38] everything's consistent and the there was no, you know, hallucinations from
[01:08:39] was no, you know, hallucinations from the agents, uh, pulling things that
[01:08:41] the agents, uh, pulling things that weren't true to that persona. Um
[01:08:44] weren't true to that persona. Um and so
[01:08:45] and so um the the finding in the paper uh
[01:08:48] um the the finding in the paper uh mainly this is a benchmark that gets
[01:08:49] mainly this is a benchmark that gets reused quite a bit but the finding in
[01:08:51] reused quite a bit but the finding in the paper itself that was I found
[01:08:53] the paper itself that was I found interesting is uh they were finding that
[01:08:55] interesting is uh they were finding that even when the models get the right
[01:08:56] even when the models get the right memories they can't always use them. So
[01:08:58] memories they can't always use them. So the way I wanted to point out this table
[01:09:00] the way I wanted to point out this table is just this is recall accuracy. So this
[01:09:02] is just this is recall accuracy. So this is recalling the correct part of the
[01:09:04] is recalling the correct part of the conversation. And you could see that
[01:09:06] conversation. And you could see that once we have high recall accuracy like
[01:09:09] once we have high recall accuracy like by find searching over 50 different
[01:09:11] by find searching over 50 different observations stored to memory. Um we
[01:09:14] observations stored to memory. Um we actually have lower accuracy at
[01:09:16] actually have lower accuracy at answering questions about the user
[01:09:18] answering questions about the user because we actually have too much
[01:09:19] because we actually have too much context. So this is kind of pointing out
[01:09:21] context. So this is kind of pointing out some problems with the ways that LMS
[01:09:23] some problems with the ways that LMS handle the memory. Um I wanted to argue
[01:09:26] handle the memory. Um I wanted to argue that this is somewhat realistic. Uh but
[01:09:28] that this is somewhat realistic. Uh but simulated conversations between users is
[01:09:30] simulated conversations between users is not really what we care about agents
[01:09:31] not really what we care about agents retrieving from. Uh it's somewhat long
[01:09:34] retrieving from. Uh it's somewhat long context but we'll talk about some
[01:09:35] context but we'll talk about some benchmarks that are really long context
[01:09:36] benchmarks that are really long context in a moment. Um it's it's pretty
[01:09:39] in a moment. Um it's it's pretty challenging. They have some adversarial
[01:09:41] challenging. They have some adversarial tasks. Um and uh it is verifiable
[01:09:44] tasks. Um and uh it is verifiable because they do question answering uh
[01:09:45] because they do question answering uh with a ground truth answer. Um
[01:09:48] with a ground truth answer. Um so let me talk about now persona. So uh
[01:09:52] so let me talk about now persona. So uh people have multiple conversations and
[01:09:54] people have multiple conversations and their preferences change over time. So
[01:09:56] their preferences change over time. So what if we simulate conversations with
[01:09:57] what if we simulate conversations with an assistant agent instead of between
[01:09:59] an assistant agent instead of between users? And so this is the idea of
[01:10:01] users? And so this is the idea of persona meim. Um
[01:10:02] persona meim. Um and then the idea here is like okay we
[01:10:05] and then the idea here is like okay we have a user persona that we generate and
[01:10:07] have a user persona that we generate and we pull this from like a database of all
[01:10:10] we pull this from like a database of all of these user personas and we have multi
[01:10:12] of these user personas and we have multi session interaction histories. So this
[01:10:14] session interaction histories. So this is the example I showed earlier of
[01:10:16] is the example I showed earlier of Italian food. Then they get allergic to
[01:10:17] Italian food. Then they get allergic to gluten and so they want Mediterranean
[01:10:19] gluten and so they want Mediterranean food now. And so we have a a a multiple
[01:10:22] food now. And so we have a a a multiple choice quiz that has multiple answers
[01:10:23] choice quiz that has multiple answers that are potentially correct and we have
[01:10:26] that are potentially correct and we have the model select which answer it should
[01:10:27] the model select which answer it should give. Um
[01:10:29] give. Um and so I think um I I think like one
[01:10:32] and so I think um I I think like one thing that would be useful to show is an
[01:10:34] thing that would be useful to show is an example from the data set. So here we
[01:10:36] example from the data set. So here we have the user enjoys making music with
[01:10:38] have the user enjoys making music with software is something you could
[01:10:39] software is something you could ascertain from their earlier
[01:10:40] ascertain from their earlier conversations. Uh and then they say I
[01:10:43] conversations. Uh and then they say I recently attended an event where there
[01:10:44] recently attended an event where there was a unique blend of modern beats with
[01:10:46] was a unique blend of modern beats with specific sounds. And the answer the
[01:10:48] specific sounds. And the answer the model is expected to say is it's great
[01:10:50] model is expected to say is it's great to see such innovative approaches in
[01:10:51] to see such innovative approaches in music since you like producing music
[01:10:53] music since you like producing music with software. This must have been quite
[01:10:54] with software. This must have been quite interesting. But they have all of these
[01:10:56] interesting. But they have all of these distractors where it's like talking
[01:10:58] distractors where it's like talking about an art exhibition instead or
[01:11:00] about an art exhibition instead or I think it says you dislike anything
[01:11:02] I think it says you dislike anything involving music production which is
[01:11:04] involving music production which is against their memories and then just
[01:11:06] against their memories and then just something that like is sort of a generic
[01:11:08] something that like is sort of a generic response. Um so so you're expected to to
[01:11:11] response. Um so so you're expected to to personalize properly. Um and so in this
[01:11:14] personalize properly. Um and so in this paper they show that memory and rag were
[01:11:16] paper they show that memory and rag were essential to perform properly uh well on
[01:11:18] essential to perform properly uh well on it. Um and that mem zero remember that
[01:11:21] it. Um and that mem zero remember that knowledge graph updating and editing was
[01:11:23] knowledge graph updating and editing was uh important for preference evolution uh
[01:11:25] uh important for preference evolution uh over time.
[01:11:26] over time. And so I would say that this is getting
[01:11:28] And so I would say that this is getting to be slightly more realistic. It's a
[01:11:30] to be slightly more realistic. It's a user talking to an agent instead of a
[01:11:32] user talking to an agent instead of a user talking to a user and it's much
[01:11:34] user talking to a user and it's much longer context. Um but the questions are
[01:11:37] longer context. Um but the questions are all multiple choice questions. Uh so
[01:11:39] all multiple choice questions. Uh so it's maybe not the hardest benchmark.
[01:11:41] it's maybe not the hardest benchmark. Um, and so the next thing is like what
[01:11:43] Um, and so the next thing is like what if we could have memory over agent
[01:11:44] if we could have memory over agent trajectories. Uh, and so that's longme
[01:11:47] trajectories. Uh, and so that's longme eval. Uh, I think I'll skip this one to
[01:11:49] eval. Uh, I think I'll skip this one to talk about they just had web agents and
[01:11:51] talk about they just had web agents and they asked questions about what the web
[01:11:52] they asked questions about what the web agents did. Uh, but I think I want to
[01:11:55] agents did. Uh, but I think I want to spend a second to talk about
[01:11:57] spend a second to talk about um
[01:11:58] um the
[01:11:59] the like what if we want to test agents on
[01:12:01] like what if we want to test agents on solving realistic diverse problems
[01:12:03] solving realistic diverse problems rather than multiple choice and that's
[01:12:04] rather than multiple choice and that's where memory arena comes in. And in this
[01:12:07] where memory arena comes in. And in this benchmark uh they want to evaluate the
[01:12:09] benchmark uh they want to evaluate the memory agent environment loop. Uh and so
[01:12:12] memory agent environment loop. Uh and so essentially as the agent takes actions,
[01:12:13] essentially as the agent takes actions, it gets feedback from the environment,
[01:12:15] it gets feedback from the environment, it updates its memory, and then it
[01:12:16] it updates its memory, and then it continues taking actions. And so they
[01:12:18] continues taking actions. And so they want to benchmark agents on this. Uh so
[01:12:22] want to benchmark agents on this. Uh so uh we're going to look specifically at
[01:12:23] uh we're going to look specifically at this group travel planning task. They
[01:12:25] this group travel planning task. They have four different environments but in
[01:12:26] have four different environments but in the group travel planning setting what
[01:12:28] the group travel planning setting what they do is they want to plan a group
[01:12:30] they do is they want to plan a group travel and eat one at a time more people
[01:12:33] travel and eat one at a time more people come in with more constraints on the
[01:12:35] come in with more constraints on the travel that they want to do. So you can
[01:12:37] travel that they want to do. So you can see we start here with I am Tracy. Uh
[01:12:39] see we start here with I am Tracy. Uh this is a single person trip. Uh and
[01:12:42] this is a single person trip. Uh and these are the dates and my budget. And
[01:12:44] these are the dates and my budget. And then we generate an itinerary for Tracy.
[01:12:47] then we generate an itinerary for Tracy. And now here Chelsea comes in and says
[01:12:48] And now here Chelsea comes in and says I'm joining Tracy for this trip. Uh and
[01:12:50] I'm joining Tracy for this trip. Uh and I like luxury hotels. And then it
[01:12:53] I like luxury hotels. And then it replans the trip. And then it says I'm
[01:12:54] replans the trip. And then it says I'm Emily. And then it replans the trip. And
[01:12:57] Emily. And then it replans the trip. And they keep going and going and going. And
[01:12:58] they keep going and going and going. And eventually they can validate did the
[01:13:00] eventually they can validate did the trip fit the constraints of every single
[01:13:02] trip fit the constraints of every single person on this trip. and it it ends up
[01:13:04] person on this trip. and it it ends up being these really long trajectories.
[01:13:07] being these really long trajectories. Um, and so the interesting thing here is
[01:13:10] Um, and so the interesting thing here is um we can split this table up into long
[01:13:13] um we can split this table up into long context and just putting everything in
[01:13:15] context and just putting everything in the context window of the agent using
[01:13:17] the context window of the agent using memory agents. So like me zero like we
[01:13:19] memory agents. So like me zero like we talked about and then rag systems and
[01:13:22] talked about and then rag systems and you could see across these uh uh
[01:13:24] you could see across these uh uh categories uh the memory agents actually
[01:13:27] categories uh the memory agents actually scored the worst on their benchmark.
[01:13:28] scored the worst on their benchmark. This can't fully be attributed to the
[01:13:30] This can't fully be attributed to the memory because there are other ways you
[01:13:32] memory because there are other ways you could fail on this task, but it is
[01:13:33] could fail on this task, but it is interesting and there's a lot of
[01:13:34] interesting and there's a lot of headroom to improve memory systems is
[01:13:36] headroom to improve memory systems is what they're kind of making a case here
[01:13:38] what they're kind of making a case here for.
[01:13:39] for. So,
[01:13:40] So, um
[01:13:41] um to recap, uh this is just some stats
[01:13:43] to recap, uh this is just some stats about each of these benchmarks. Uh
[01:13:45] about each of these benchmarks. Uh memory evaluation should be realistic,
[01:13:47] memory evaluation should be realistic, long context, challenging, and
[01:13:49] long context, challenging, and verifiable. Uh and so these are some
[01:13:51] verifiable. Uh and so these are some examples. Uh sorry, I know I went
[01:13:53] examples. Uh sorry, I know I went through that section quickly. Do we have
[01:13:54] through that section quickly. Do we have a quick question before I wrap us up on
[01:13:57] a quick question before I wrap us up on on challenges and open problems?
[01:14:00] on challenges and open problems? >> Yes.
[01:14:00] >> Yes. >> Question. I feel a lot this is memory
[01:14:03] >> Question. I feel a lot this is memory harness like split memory out of the
[01:14:05] harness like split memory out of the harness. I wonder is that necessary?
[01:14:08] harness. I wonder is that necessary? >> Uh yeah, I guess the question is like is
[01:14:10] >> Uh yeah, I guess the question is like is it necessary to split memory out of the
[01:14:12] it necessary to split memory out of the harness? Is that what you're saying?
[01:14:14] harness? Is that what you're saying? >> Yeah. I mean I guess I guess the memory
[01:14:16] >> Yeah. I mean I guess I guess the memory is kind of a part of the harness like we
[01:14:17] is kind of a part of the harness like we have to
[01:14:18] have to all the tools to read and write and
[01:14:20] all the tools to read and write and everything are built into the harness.
[01:14:21] everything are built into the harness. But if you're asking about like if we
[01:14:23] But if you're asking about like if we can just do this with long context, uh
[01:14:25] can just do this with long context, uh it depends on like how long your task
[01:14:27] it depends on like how long your task is. So in this mem arena paper
[01:14:30] is. So in this mem arena paper that it wasn't long enough context that
[01:14:32] that it wasn't long enough context that you had to use tools always uh but
[01:14:34] you had to use tools always uh but sometimes it is necessary to use tools
[01:14:37] sometimes it is necessary to use tools and and so that's why I guess we
[01:14:38] and and so that's why I guess we benchmark that. But good question. Uh
[01:14:41] benchmark that. But good question. Uh let's finish up with challenges and open
[01:14:42] let's finish up with challenges and open problems. Luckily this is like a quick
[01:14:44] problems. Luckily this is like a quick section. Um
[01:14:46] section. Um so
[01:14:47] so there are three challenges I want to
[01:14:48] there are three challenges I want to talk about. Choosing what to remember
[01:14:49] talk about. Choosing what to remember and what to forget. trust, privacy and
[01:14:52] and what to forget. trust, privacy and user control and using memory correctly.
[01:14:55] user control and using memory correctly. So first of all, choosing what to
[01:14:56] So first of all, choosing what to remember and what to forget. So the
[01:14:58] remember and what to forget. So the future relevance of information is
[01:15:00] future relevance of information is unknown when the memory is created. Uh
[01:15:02] unknown when the memory is created. Uh and keeping everything uh increases the
[01:15:05] and keeping everything uh increases the retrieval, context and maintenance costs
[01:15:07] retrieval, context and maintenance costs because you have much more information
[01:15:09] because you have much more information you're working with. Um and forgetting
[01:15:12] you're working with. Um and forgetting can obviously remove useful evidence
[01:15:13] can obviously remove useful evidence that could have been useful in the
[01:15:15] that could have been useful in the future. So I have this example uh where
[01:15:17] future. So I have this example uh where user is saying like help me pick a time
[01:15:19] user is saying like help me pick a time for the check-in. It says, "Zero your
[01:15:20] for the check-in. It says, "Zero your two slots," and then they say like, "Oh,
[01:15:22] two slots," and then they say like, "Oh, I don't recognize meetings before 10 as
[01:15:24] I don't recognize meetings before 10 as legally binding unless there's
[01:15:25] legally binding unless there's pastries," uh, which is like probably
[01:15:27] pastries," uh, which is like probably they're joking around with their agent.
[01:15:28] they're joking around with their agent. But then the question is kind of like,
[01:15:30] But then the question is kind of like, uh, should I I mean, like, what should I
[01:15:32] uh, should I I mean, like, what should I remember here? Like, should I remember
[01:15:34] remember here? Like, should I remember that they don't like meetings before 10
[01:15:36] that they don't like meetings before 10 uh, or are they just joking? Like, it's
[01:15:38] uh, or are they just joking? Like, it's a little bit ambiguous of like, "Should
[01:15:40] a little bit ambiguous of like, "Should I remember this or not?" which is
[01:15:41] I remember this or not?" which is actually a challenge uh. Another thing
[01:15:44] actually a challenge uh. Another thing is trust, privacy, and user control,
[01:15:47] is trust, privacy, and user control, um, so, uh, this paper is finding that,
[01:15:50] um, so, uh, this paper is finding that, let's say, you have a memory bank where
[01:15:52] let's say, you have a memory bank where you, uh, someone injects poisoned
[01:15:54] you, uh, someone injects poisoned memories into the bank. So this this uh
[01:15:56] memories into the bank. So this this uh user is asking their agent take me to
[01:15:58] user is asking their agent take me to O'Hare airport drive smooth and be safe.
[01:16:01] O'Hare airport drive smooth and be safe. So in this world that we're imagining
[01:16:03] So in this world that we're imagining the agent can drive your car. Um and so
[01:16:06] the agent can drive your car. Um and so they say you know take me take me there
[01:16:08] they say you know take me take me there and then um
[01:16:10] and then um it has memories of unsafe driving uh in
[01:16:13] it has memories of unsafe driving uh in its procedural memory. So you can think
[01:16:15] its procedural memory. So you can think of the agent workflow memory. It has a
[01:16:16] of the agent workflow memory. It has a description of driving really fast and
[01:16:19] description of driving really fast and they are able to optimize these
[01:16:20] they are able to optimize these seemingly benign triggers that when you
[01:16:22] seemingly benign triggers that when you retrieve from the memory you retrieve
[01:16:24] retrieve from the memory you retrieve these unsafe memories uh and then
[01:16:27] these unsafe memories uh and then suddenly the agent behaves in ways that
[01:16:29] suddenly the agent behaves in ways that you wouldn't have expected. So I I do
[01:16:31] you wouldn't have expected. So I I do think this is a little bit of a toy
[01:16:33] think this is a little bit of a toy example but it motivating like a a
[01:16:35] example but it motivating like a a bigger challenge which is like if you
[01:16:38] bigger challenge which is like if you cannot trust the memories um then your
[01:16:40] cannot trust the memories um then your agent can behave in unexpected ways.
[01:16:42] agent can behave in unexpected ways. Uh and then there's another example
[01:16:44] Uh and then there's another example which is uh privacy and so agents will
[01:16:47] which is uh privacy and so agents will interact with the world without uh fully
[01:16:49] interact with the world without uh fully understanding what you wish to keep
[01:16:50] understanding what you wish to keep private and doing so with memory can
[01:16:52] private and doing so with memory can also expose personal details. So uh D is
[01:16:55] also expose personal details. So uh D is going to definitely talk about this when
[01:16:57] going to definitely talk about this when we get to the privacy and safety
[01:16:58] we get to the privacy and safety lecture. Uh but there's this uh privacy
[01:17:01] lecture. Uh but there's this uh privacy lens data set uh where they uh example
[01:17:04] lens data set uh where they uh example here is the user says send Susan
[01:17:05] here is the user says send Susan Harrington the weekly report of my
[01:17:07] Harrington the weekly report of my recent work. That's their boss. uh and
[01:17:10] recent work. That's their boss. uh and it goes and sends this email and it says
[01:17:11] it goes and sends this email and it says this is what they've been up to this
[01:17:12] this is what they've been up to this week. They had a lunch with a recruiter
[01:17:14] week. They had a lunch with a recruiter at a competing company and stuff like
[01:17:16] at a competing company and stuff like that. So you can imagine like there's
[01:17:18] that. So you can imagine like there's privacy leakage that happens because
[01:17:20] privacy leakage that happens because agents don't fully grasp the context of
[01:17:22] agents don't fully grasp the context of what you care to expose to people. Um
[01:17:25] what you care to expose to people. Um the last one is uh using memory
[01:17:27] the last one is uh using memory correctly uh and overpersonalization.
[01:17:30] correctly uh and overpersonalization. And so uh LLMs are sometimes overeager
[01:17:33] And so uh LLMs are sometimes overeager to use what is in their context. Um this
[01:17:36] to use what is in their context. Um this could be better resolved with either
[01:17:37] could be better resolved with either better retrieval or better training of
[01:17:38] better retrieval or better training of the LMS to ignore uh unimportant
[01:17:41] the LMS to ignore uh unimportant contexts. Um and another detail is some
[01:17:44] contexts. Um and another detail is some users do like personalization more than
[01:17:46] users do like personalization more than others. So what to personalize is a
[01:17:48] others. So what to personalize is a personalization question. Um but we have
[01:17:51] personalization question. Um but we have this example from this uh paper that's
[01:17:53] this example from this uh paper that's benchmarking over personalization. And
[01:17:55] benchmarking over personalization. And this is a real instance that uh they
[01:17:57] this is a real instance that uh they observed where the user says what role
[01:17:59] observed where the user says what role do you think young voters play in
[01:18:00] do you think young voters play in shaping future elections? uh it
[01:18:02] shaping future elections? uh it retrieves memories about the user and
[01:18:04] retrieves memories about the user and one of the memories is that they are
[01:18:06] one of the memories is that they are focusing on small wins. Uh and I guess
[01:18:10] focusing on small wins. Uh and I guess this is like part of their journey. They
[01:18:11] this is like part of their journey. They want small wins as maybe like a therapy
[01:18:13] want small wins as maybe like a therapy approach. And then the model says um it
[01:18:16] approach. And then the model says um it answers this question. It says just like
[01:18:17] answers this question. It says just like you're focusing on small healthy swaps
[01:18:19] you're focusing on small healthy swaps in your life, every young vote can feel
[01:18:21] in your life, every young vote can feel like a small win that contributes to a
[01:18:23] like a small win that contributes to a bigger change. So obviously that's kind
[01:18:25] bigger change. So obviously that's kind of like a stupid thing for the model to
[01:18:27] of like a stupid thing for the model to say. Uh so overpersonalization itself is
[01:18:29] say. Uh so overpersonalization itself is a problem. Uh anyway, that is actually
[01:18:32] a problem. Uh anyway, that is actually all I had prepared for today. So in
[01:18:34] all I had prepared for today. So in summary, we talked about agent memory,
[01:18:36] summary, we talked about agent memory, types of agent memory, uh agent memory
[01:18:39] types of agent memory, uh agent memory architectures, evaluation, and
[01:18:41] architectures, evaluation, and challenges in open problems. So sorry we
[01:18:43] challenges in open problems. So sorry we rushed a little bit at the end, but I
[01:18:44] rushed a little bit at the end, but I think we got through most of the
[01:18:45] think we got through most of the important stuff. I'll see you all on
[01:18:47] important stuff. I'll see you all on Monday for multi-agent systems.

All frames

Frames 0041-0053 were extracted afterwards at 25:40, 38:20, 41:00, 51:20, 58:50, 1:01:10, 1:06:40, 1:09:00, 1:13:10, 1:14:50, 1:16:10, 1:17:20 and 1:18:10 to capture slides the automatic sample missed.

Total: 53 (40 sampled, 13 extracted afterwards). Hero frames flagged with star.

  • frames/frame_0001.jpg (t=00:00) frames/frame_0002.jpg (t=03:04) frames/frame_0003.jpg (t=04:20) frames/frame_0004.jpg (t=06:48) frames/frame_0005.jpg (t=08:59)
  • frames/frame_0006.jpg (t=10:41) frames/frame_0007.jpg (t=12:49) frames/frame_0008.jpg (t=14:48)
  • frames/frame_0009.jpg (t=17:03) frames/frame_0010.jpg (t=18:39) frames/frame_0011.jpg (t=20:48) frames/frame_0012.jpg (t=22:40) frames/frame_0013.jpg (t=24:12) frames/frame_0014.jpg (t=26:48) frames/frame_0015.jpg (t=28:36) frames/frame_0016.jpg (t=31:25)
  • frames/frame_0017.jpg (t=32:25) frames/frame_0018.jpg (t=34:38) frames/frame_0019.jpg (t=36:46) frames/frame_0020.jpg (t=38:55) frames/frame_0021.jpg (t=40:21) frames/frame_0022.jpg (t=42:51) frames/frame_0023.jpg (t=43:58) frames/frame_0024.jpg (t=46:54)
  • frames/frame_0025.jpg (t=48:12) frames/frame_0026.jpg (t=50:12) frames/frame_0027.jpg (t=52:15) frames/frame_0028.jpg (t=54:07) frames/frame_0029.jpg (t=55:21) frames/frame_0030.jpg (t=58:08) frames/frame_0031.jpg (t=01:00:04) frames/frame_0032.jpg (t=01:02:15) frames/frame_0033.jpg (t=01:03:53) frames/frame_0034.jpg (t=01:05:09) frames/frame_0035.jpg (t=01:08:03) frames/frame_0036.jpg (t=01:09:26) frames/frame_0037.jpg (t=01:12:00) frames/frame_0038.jpg (t=01:13:52) frames/frame_0039.jpg (t=01:16:30) frames/frame_0040.jpg (t=01:18:47)