What is AI memory?

Summary slides

Why language models need memory built around them, not into them

A language model, at its core, is a function that takes some input text and produces output text, and it has no persistent internal state between one call and the next — call it twice with the exact same input, and it starts from exactly the same blank slate both times, with absolutely nothing carried over from the first call ever informing the second in any way. Virtually everything that makes a conversation with an AI system feel continuous — remembering what you said three messages ago, staying consistent with something established earlier — is happening because the application calling the model is re-sending the relevant prior context along with each new request, not because the model itself retained a single thing across those two separate, otherwise-identical calls.

This matters architecturally because it means memory is never something a model does for you automatically, however capable that model becomes — it’s something a surrounding system has to deliberately build, by deciding what’s worth carrying forward, storing it somewhere, and reassembling it into each new request at the right moment. A more capable model can make better use of whatever memory it’s given, reasoning more effectively over a well-constructed set of retained facts, but it can’t manufacture memory it was never given in the first place. This is the foundational reason memory needs to be treated as its own deliberate system, with its storage, its retrieval logic, and its maintenance — not a capability that simply emerges once a model is good enough, since no amount of raw model capability changes the basic, structural fact that the model itself starts fresh on every single call, with absolutely nothing surviving from one invocation to the next unless something outside the model deliberately preserves and reintroduces it.

Memory versus a longer context window

Given that everything a model “remembers” within a conversation is really just replayed context, it’s a reasonable question whether memory is even a distinct problem from context length — if a model can hold a huge amount of text in its context window, why not just keep appending everything ever said and let the model sort out what’s relevant. The honest answer is that a long context window and memory solve related but distinctly different problems, and treating them as interchangeable causes predictable failures.

A context window is a temporary, per-request buffer — however large, it’s reconstructed fresh for each call and discarded afterward, and everything in it has equal, undifferentiated standing, with no built-in sense of what’s important versus what’s incidental. Memory, by contrast, is meant to be a maintained, curated, persistent store — facts that get explicitly extracted, deduplicated, and kept up to date as new information arrives, retrievable selectively rather than replayed in full every single time. Simply growing the context window and stuffing an ever-larger raw conversation history into it runs into well-documented problems well before memory’s design goals are met: cost and latency grow with every additional token processed on every single call, and models are measurably less reliable at using information buried in the middle of a very long context than information near the beginning or end of it — a phenomenon that means a fact mentioned once, early in a long, ever-growing history, can effectively become invisible to the model long before the context window’s technical size limit is ever reached. Memory, done well, sidesteps both problems by keeping only what’s still relevant, in a compact, actively maintained form, rather than relying on a model to correctly re-derive relevance from an ever-larger pile of undifferentiated raw history on every single call.

The different shapes memory needs to take

Once memory is understood as its own deliberate system rather than a side effect of context length, the next question is what kind of information needs remembering, and the honest answer is that “memory” isn’t one single thing — different kinds of information need to be retained differently, which is why AI memory tends to get discussed in terms of several distinct categories rather than one undifferentiated store. Some information is short-lived and only relevant within the current task or conversation — what a user just asked, what’s been tried so far in an ongoing multi-step process — and doesn’t need to persist once that task or conversation ends. Other information is durable and worth carrying across many separate sessions — a user’s stated preferences, a project’s established constraints, facts about a person or organization that stay true over a long stretch of time.

Within that durable category, a further distinction matters: some retained information is essentially factual — a stable fact about a person or entity — while other retained information is more like a record of past events or interactions, useful less as a standalone fact and more as a trace of what happened and when, which supports a different kind of question than a pure fact lookup does. And a further category covers something closer to learned procedure — not a fact about the world, but an accumulated sense of how to do something well, refined gradually across repeated attempts at a similar kind of task, rather than a single fact recalled about one person or event. Recognizing which of these shapes a piece of information fits is what determines how it should be stored and retrieved, and conflating them — treating a durable preference the same way as a transient detail from the current conversation, or trying to extract a learned procedure using the same simple fact-storage mechanism built for storing “the user prefers dark mode” — is a common source of memory systems that technically store information but retrieve it poorly precisely when it matters most which tends to undermine confidence in the whole memory feature even when the underlying storage itself was never really the problem.

How information gets written into memory

Understanding that different kinds of information need different handling raises the practical question of how information makes its way from a raw interaction into something a memory system can reliably retrieve later, and this turns out to be a pipeline in its own right, not a single simple step. The first stage is extraction: taking raw input — a conversation, an event, a piece of feedback — and identifying which parts of it constitute something worth remembering, as distinct from the large amount of any interaction that’s just conversational scaffolding with nothing durable worth retaining from it at all.

The second stage is reconciliation, and it’s where a surprising amount of the difficulty in memory systems lives: a newly extracted fact needs to be compared against whatever’s already stored, to determine whether it’s new information, an update to something already known that should replace or amend the older version, or a duplicate of something already captured that shouldn’t be stored again. Getting this wrong in either direction causes problems — failing to update an old fact when a new one contradicts it leaves a memory system confidently working from outdated information, while failing to deduplicate leaves a memory store bloated with redundant, near-identical entries that make later retrieval noisier and less precise than it needs to be. The third stage is commitment: once reconciliation has decided what the correct, current state of memory should be, that state gets persisted to storage, ready to be retrieved in a future interaction. Treating these as three distinct stages, rather than a single undifferentiated “save the conversation” step, is what separates a memory system that stays accurate and useful over time from one that just accumulates an ever-growing, increasingly unreliable pile of raw history.

Retrieval: getting the right memory back at the right moment

Writing memory well only solves half the problem, since a memory system is only as useful as its ability to surface the right stored information at the moment it’s needed, without requiring an application to manually decide in advance exactly which memory to fetch for every possible situation. Retrieval typically works the same way retrieval works elsewhere in AI systems: a current query or context gets compared, usually via semantic similarity, against what’s stored, surfacing the memories most relevant to what’s happening right now rather than requiring an exact keyword match against how a memory happened to be originally phrased.

This is also where the different memory categories discussed earlier tend to need different retrieval strategies rather than one uniform approach: a durable fact about a user might be retrieved by directly fetching a known, bounded profile — there’s only ever one current value for “the user’s preferred contact method,” so a system can fetch that value directly rather than running a similarity search for it every time. A record of a past event is a better fit for semantic search, since a user might reference a past interaction in many different phrasings, and the system needs to find the right past event regardless of exactly how the current question happens to be worded. Choosing the retrieval strategy that fits the shape of what’s being retrieved — rather than defaulting to one method uniformly across every kind of stored memory — is what makes a memory system feel responsive rather than either missing relevant context it should have surfaced or, just as commonly, surfacing memories that happen to match superficially but aren’t relevant to the current moment at all.

Consolidation, decay, and the discipline of forgetting

As a memory system accumulates information over a long enough period, a question emerges that has no clean parallel in most traditional software systems: not everything that was ever worth remembering stays equally worth remembering forever, and a memory system that never forgets anything eventually becomes just as unusable as one that never remembers anything in the first place, simply buried under an ever-growing pile of increasingly irrelevant historical detail. Consolidation is the process of merging related memories over time into a more compact, more useful form — several individual observations about a user’s behavior gradually becoming one clearer, more general understanding, rather than staying as a long, ever-growing list of every individual data point that ever contributed to that understanding.

Decay is the complementary process of letting stale or superseded information fade in relevance or get removed outright, which matters for reasons beyond simple storage efficiency: an outdated preference or a fact that’s since changed, left sitting in memory indefinitely alongside its more current replacement, risks getting retrieved and treated as current when it isn’t, precisely the kind of quiet, hard-to-notice error a memory system exists to prevent rather than introduce. Designing deliberate consolidation and decay processes — rather than treating memory as an ever-growing, append-only log that simply accumulates forever — is what keeps a memory system’s signal-to-noise ratio manageable as it scales across a long-lived relationship with a user, a project, or an ongoing task, rather than degrading gradually into an increasingly cluttered store where the useful, current information is progressively harder to distinguish from everything outdated that’s still technically sitting there alongside it.

Where memory lives: architectural patterns

Everything discussed so far describes what memory needs to do; it’s worth being equally concrete about where it lives in a running system, because that architectural choice shapes how well memory can scale and how cleanly it integrates with everything else an AI application already depends on. One pattern keeps memory entirely within the application layer, storing extracted facts in whatever database an application already uses and writing custom retrieval logic on top of it — straightforward to start with, but it means rebuilding the extraction, reconciliation, and retrieval logic discussed throughout this page from scratch, largely disconnected from the retrieval infrastructure many AI applications already maintain for other purposes, like searching a knowledge base.

A second pattern builds memory directly on top of the same underlying retrieval infrastructure a system already uses for document search — storing memories as vector-embedded entries alongside structured scope metadata, searchable through the same hybrid search capabilities already available for other content. This has an integration advantage: a system that already has mature, well-tuned retrieval infrastructure gets to reuse that same investment for memory rather than standing up a separate, parallel retrieval system just for memories, and memory search can benefit from the same hybrid semantic-plus-keyword matching, metadata filtering, and multi-tenant isolation that the rest of the system already relies on. A third pattern relies on a dedicated, managed memory service, purpose-built specifically for the extraction, reconciliation, and lifecycle management this page has described, handling those responsibilities so an application team doesn’t have to build that pipeline themselves. Each pattern trades some amount of control for some amount of reduced engineering effort, and the right choice tends to depend on how much of this infrastructure a team already has in place versus how much they’d need to build fresh — reusing existing, already-mature retrieval infrastructure for memory tends to be the more pragmatic starting point specifically when that infrastructure already exists and is already well-tuned, rather than building an entirely separate, parallel system to solve what is, underneath the surface, a very similar retrieval problem. A team standing up an AI application for the first time, with no existing retrieval infrastructure to build on, faces a somewhat different calculation than a team that already operates a mature retrieval system for other content and is simply extending that same investment to cover memory as an additional use case sitting on top of it.

Knowing whether a memory system is working

Building all of this infrastructure only matters if there’s a way to tell whether the resulting memory system is doing its job well, and this is harder to measure than it might first appear, because success here isn’t a single clean metric the way uptime or latency might be. A useful starting point is precision and recall applied specifically to memory retrieval: when an interaction would clearly benefit from a piece of stored memory, does the system surface it, and when it does surface memories, are they relevant to what’s happening right now rather than superficially similar but beside the point.

Beyond retrieval quality, memory correctness deserves its explicit measurement: does the memory store reflect the current, accurate state of whatever it’s tracking, or has it drifted — holding an outdated preference the user has since changed, or a duplicate entry that reconciliation should have caught and merged but didn’t. This is often best measured by periodically auditing a sample of stored memories against ground truth — checking whether what’s stored still matches reality — rather than assuming correctness simply because the extraction and reconciliation pipeline ran without throwing an error, since a pipeline can run successfully to completion while still reconciling information incorrectly in ways that produce no visible error at all. And at the level of the actual user experience, tracking moments where a system visibly fails to recall something a user reasonably expected it to know — an explicit correction, a repeated question that memory should have already answered — gives a concrete signal about memory quality that no amount of internal pipeline monitoring alone can substitute for, because it’s the only measurement source that reflects what memory failures feel like to the person the memory system exists to serve. These three measurement approaches — retrieval precision and recall, periodic correctness audits against ground truth, and tracking visible user-facing recall failures — work best layered together rather than any one standing in for the others, since each catches a different category of problem the other two would miss entirely on their own: a memory system can retrieve perfectly relevant results that happen to be factually stale, or can hold perfectly accurate facts that its retrieval logic simply fails to surface at the moment they’d be useful, and only measuring one of these dimensions leaves the other blind spot completely unmonitored.

Memory in agentic and multi-step systems

Everything covered so far applies to memory across separate sessions and interactions, but memory also matters within a single long-running agentic task, in a way that’s related but distinct, because an agent working through a multi-step process needs to track what it’s already tried and learned within that one task, even before any of it necessarily becomes a durable, cross-session memory worth keeping afterward. This within-task memory needs the same basic discipline discussed throughout this page — extraction of what matters, consolidation into a compact working summary rather than an ever-growing raw transcript, and careful retrieval of exactly what’s relevant to the current step — applied at a shorter time horizon, often within the span of minutes rather than across weeks or months.

Multi-agent systems add a further wrinkle worth naming directly: when several agents collaborate on parts of the same task, memory sometimes needs to be shared across them, and getting that sharing right raises questions that don’t arise for a single agent working alone — what one agent has learned needs to reach another agent that needs it, in a form that agent can use, without simply flooding every agent with everything every other agent has ever recorded, which would recreate the exact undifferentiated-context problem memory was meant to solve in the first place, just distributed across multiple agents instead of contained within one. Designing shared memory deliberately, scoped to what’s relevant to an agent’s current responsibility rather than shared indiscriminately across the whole system, is what keeps multi-agent memory from becoming just as noisy and undifferentiated as the raw, unmanaged conversation history problem memory systems exist to avoid in the first place.

There’s a related distinction worth drawing explicitly here, because it’s easy to blur: not everything a single agentic task learns necessarily deserves to become a durable, cross-session memory once that task finishes. A within-task working summary — what’s been tried on this research question, which sources have already been checked — is often disposable once the task completes, useful only for the duration of that one run and safe to discard afterward without any loss. Durable memory, worth carrying forward into future sessions, is a narrower category: a reusable fact, preference, or lesson that will matter again later, distinct from the much larger volume of task-working detail that was only ever needed to get through the current task successfully. Conflating the two — promoting every detail from a task’s working memory into permanent, cross-session storage — is a quiet but real source of memory bloat, cluttering a durable memory store with transient details that were only ever relevant to one already-completed task and were never going to matter again afterward.

Personalization: memory’s most visible application

Of all the reasons a system might need memory, personalization is the one users notice and value most directly and most immediately, since it’s the clear, felt difference between an assistant that has to be told the same preferences and context every single time and one that consistently feels like it knows the person it’s working with. A system with well-designed memory can reliably recall a user’s stated preferences, their role, the context of an ongoing project, and their prior interactions, weaving that recalled context naturally and unobtrusively into how it responds rather than requiring the user to painstakingly re-establish the same background information at the start of every single interaction all over again.

This is also precisely where the cost of getting memory wrong becomes most visible to the people using a system day to day: a memory system that surfaces an outdated preference, confuses one user’s history with another’s, or simply fails to recall something the user reasonably expects it to remember, creates a very real and quite distinctive kind of disappointment — arguably worse, in some ways, than a system that simply had no memory at all, because a user who was explicitly told the system remembers things reasonably expects that promise to hold, and a broken instance of it reads as a mistake rather than simply an acknowledged limitation the way “no memory at all” would. This is exactly why the earlier sections on reconciliation, consolidation, and decay aren’t abstract engineering concerns — they’re the mechanisms that determine whether personalization feels reliable and trustworthy to the person experiencing it, or feels erratic and unpredictable in exactly the way that erodes the trust memory-based personalization is meant to build in the first place.

Privacy and governance: memory as sensitive data

Because AI memory frequently stores personal information about people — preferences, behaviors, facts disclosed in conversation, records of past interactions — it needs to be treated with the same seriousness as any other sensitive personal data store, not as a special category exempt from ordinary data governance simply because it emerged from an AI system rather than a traditional database. This means access to a person’s stored memories needs to be scoped correctly — one user’s memory should never be retrievable by another user’s session, which connects directly to the multi-tenancy and isolation concerns that come up throughout AI system design generally, applied here specifically to a store that by its very nature, tends to accumulate an unusually rich, longitudinal picture of an individual over time.

It also means deletion needs to work: if a user asks for their data to be removed, or if a retention policy calls for older memories to be purged, that deletion needs to reach every place a memory or anything derived from it might be stored — not just an original raw record, but any consolidated summary built from it, any embedding computed for it, and any cached result that might still reflect it. A memory system that can delete the original record but leaves a summary or embedding derived from it lingering elsewhere has not honored a deletion request, however much it might look, from the outside, like it has. Building this level of governance discipline into a memory system from the start — rather than treating it as a compliance concern to retrofit later, once memory has already been accumulating for a meaningful stretch of time — is what keeps a useful memory capability from becoming a liability the moment it’s examined closely.

A worked example: memory for a long-term coaching assistant

These principles are easier to see clearly against a concrete case than left purely abstract, so consider an AI coaching assistant meant to support a user’s professional development over months, across many separate conversations. Durable facts — the user’s current role, their stated career goals, skills they’ve said they want to develop — are extracted, reconciled against whatever’s already known, and stored as a compact, directly-fetchable profile rather than buried in a growing transcript, so that every new conversation can start already informed by this established context rather than requiring the user to re-explain it.

Records of past interactions — a particular piece of feedback three weeks ago, a goal set in an earlier session — are stored separately and retrieved by semantic similarity when a current conversation seems to reference or build on something from the past, rather than being folded indiscriminately into the same profile used for stable facts. As the relationship continues over months, consolidation merges related observations — several individual mentions of struggling with public speaking becoming one clearer, more actionable understanding of that as an ongoing development area — while decay lets a goal the user has explicitly said they’ve achieved and moved past fade out of active retrieval, so it doesn’t keep surfacing as if it were still current months after it stopped being relevant. Deletion is honored fully: if the user asks the assistant to forget a piece of shared information, that removal reaches the original record, any consolidated summary that incorporated it, and any embedding built from it, rather than leaving traces of it recoverable through some other part of the system the deletion request never reached.

Retrieval strategy is chosen deliberately for each category rather than applied uniformly: the stable profile fields are fetched directly, by their known identity, at the start of every new session, so the assistant is always working from current information without needing to run a search for something it should simply already know. Records of past sessions are only pulled in via semantic search when the current conversation seems to reference or build on something from before, so a routine check-in doesn’t unnecessarily drag in a large volume of loosely related historical detail that isn’t relevant to what’s being discussed right now. And measurement runs continuously rather than only at launch: the team periodically samples stored memories and checks them against what the user has confirmed to be true, catching the quiet kind of drift — an unreconciled update, an unmerged duplicate — that would otherwise only surface much later, at the considerably worse moment a user notices the assistant confidently recalling something that isn’t accurate anymore.

Common mistakes in how AI memory gets built

A consistent set of mistakes shows up across systems that add memory, worth naming directly because each one traces back to skipping one of the deliberate design steps discussed throughout this page. The most common is treating memory as simply appending every interaction to a growing log and relying on retrieval alone to surface what’s relevant, without ever reconciling new information against old or consolidating related observations over time — which produces a memory store that technically contains everything but retrieves poorly, cluttered with duplicate, contradictory, or long-outdated entries that a reconciliation and consolidation process would have long since cleaned up.

A second common mistake is conflating memory with context length, assuming that a sufficiently large context window makes deliberate memory engineering unnecessary, which runs directly into the cost, latency, and attention-dilution problems discussed earlier the moment a system’s actual usage grows beyond a short, simple conversation. A third is applying one uniform retrieval strategy to every kind of stored information, running a similarity search even for facts that have exactly one current, directly-fetchable value, which adds unnecessary latency and occasionally surfaces an outdated version of a fact that a direct, deterministic lookup would have returned correctly and consistently every time. A fourth is neglecting decay entirely, letting a memory store grow indefinitely without ever revisiting whether older entries are still accurate or relevant, which quietly degrades retrieval quality over a long enough time horizon in a way that’s hard to notice until a wrong or outdated recollection surfaces and visibly confuses a user. And a fifth, perhaps the most consequential, is treating memory’s privacy and deletion obligations as an afterthought rather than a first-class design requirement from the very beginning, which tends to surface as an uncomfortable gap precisely at the moment someone tests whether a deletion request was honored across every place a piece of memory, or anything derived from it, might still be quietly sitting.

What runs underneath every one of these mistakes, and every design principle discussed before them, is the same underlying recognition: memory is infrastructure, not a feature that emerges naturally once a system stores enough conversation history somewhere. It needs its deliberate pipeline for extraction and reconciliation, its strategy for retrieval matched to the actual shape of what’s being retrieved, its ongoing discipline of consolidation and decay so it doesn’t simply accumulate into an unusable pile of undifferentiated history, and its governance commitments around privacy and deletion that get honored completely rather than partially. A system built with that infrastructure in place is the difference between an assistant that feels like it knows the person or task it’s working with, and one that merely has access to an ever-growing, poorly organized transcript of everything that was ever said to it — a difference that becomes obvious to any user within the first few interactions where memory either quietly does its job well, or visibly, disappointingly, doesn’t.