How do you design AI memory systems?

Summary slides

Why memory is a different problem from context, and why the two get conflated

A model’s context window already lets it “remember” everything within the current conversation, which is exactly why memory as a distinct design problem is easy to underestimate at first: it feels like the same capability, just extended further back in time. The difference becomes obvious the moment a conversation runs long enough, or a user returns after days or weeks, because a context window is bounded, expensive to fill entirely with raw history, and gone the instant a session ends, while memory needs to persist selectively, survive across sessions, and stay usable without simply re-feeding an ever-growing transcript back into the model every time.

Treating memory as “just keep the context window bigger” or “just keep appending to the conversation history” breaks down for reasons beyond cost alone. A model given an unfiltered, growing transcript has to do the work of figuring out what within it is relevant to the current turn, work that a well-designed memory system should be doing on the model’s behalf by surfacing only what’s pertinent, and a transcript that grows without bound eventually contains outdated, superseded, or simply irrelevant material that dilutes the useful signal rather than adding to it. Memory design, then, starts from treating what to remember, how to organize it, and how to bring it back at the right moment as three separate decisions, each of which shapes the system’s usefulness independently of the other two.

How to decide what’s worth remembering

Not everything a user says or a system observes deserves to become a persistent memory, and a system that tries to remember everything indiscriminately ends up with a memory store so noisy that retrieval from it is barely better than not having memory at all, since important facts get buried among routine, one-off details that never needed to be retained past the conversation that produced them.

A useful first distinction is between facts that are durable, a user’s stated preferences, their role, persistent constraints they’ve mentioned, and facts that are transient, a question asked in passing, a one-time clarification, a detail relevant only to the immediate task at hand. Durable facts are the ones worth writing to memory; transient ones generally aren’t, and building an explicit judgment step, whether a rule-based heuristic or a model call specifically tasked with deciding “is this worth remembering,” into the pipeline that decides what gets written prevents the system from defaulting to remembering everything simply because doing the filtering work was more effort than skipping it.

A second, related distinction is between explicit and inferred memory. Explicit memory captures what a user directly states, “I prefer concise answers,” “I’m working in Python,” while inferred memory captures patterns the system notices across multiple interactions without the user ever stating them outright, such as consistently asking follow-up questions that suggest a particular level of background expertise. Inferred memory is considerably more valuable when it works, since it captures things a user would never think to state explicitly, but it’s also considerably more error-prone, since an inference drawn from a small or unrepresentative sample of interactions can be confidently wrong in a way an explicitly stated fact simply can’t be, which is why systems that rely on inferred memory generally need a higher confidence threshold before writing an inference to persistent storage than they need before writing something the user stated directly.

How memory architecture separates into distinct types with different retrieval needs

Treating “memory” as one undifferentiated store makes retrieval unnecessarily hard, because the kinds of things worth remembering differ in structure and in how they need to be retrieved, and a system that tries to serve every kind of memory through a single retrieval mechanism ends up compromising on all of them rather than serving any of them well.

Episodic memory captures past events or interactions, a particular conversation, a particular decision made and why, and is typically retrieved by similarity to a current query in much the same way retrieval-augmented generation retrieves documents, since the question “what’s happened before that’s relevant to this” is fundamentally a semantic-similarity question. Semantic memory captures general facts and knowledge distilled from those events rather than the raw events themselves, a user’s stated preference, a learned constraint, a summarized fact, and benefits from a more structured storage format, often key-value or a small structured record per fact, since these are typically looked up directly by what they’re about rather than retrieved by loose similarity to a query.

Procedural memory, less commonly implemented but increasingly relevant for agentic systems, captures learned patterns about how to accomplish a task, which sequence of steps or tool calls tends to work for a kind of request, and is retrieved less by similarity to the current query’s content and more by similarity to the current task’s structure or type. Designing separate storage and retrieval paths for these distinct memory types, rather than forcing all of them through one generic retrieval mechanism, lets each type be organized and queried in the way that suits its structure, and lets a system combine the right mix of memory types for a moment rather than retrieving indiscriminately from one undifferentiated pool.

How retrieval timing determines whether memory helps or just adds noise

Even a well-organized memory store only helps if the right memories surface at the right moment, and the naive approach, retrieving some fixed number of top memories by similarity at the start of every turn regardless of whether the current turn needs them, routinely injects irrelevant material into context on turns that didn’t call for any memory at all, diluting the context budget with retrieved memories that add nothing to that response.

A more deliberate design treats memory retrieval as conditional rather than automatic, triggering a retrieval pass only when the current turn’s content suggests stored memory would be relevant, whether through an explicit classification step that judges whether the turn references something that sounds like it could connect to prior context, or through the model itself being given the ability to query memory as a tool it invokes only when it judges doing so would help, similar to how an agentic system decides whether to invoke any other tool. This shifts the design question from “how much memory should always be included” to “when does this turn benefit from memory,” which tends to produce more consistently useful retrieval than a fixed always-on policy that inevitably over-includes on some turns and under-includes on others.

The retrieval query itself also deserves thought rather than simply reusing the user’s raw message as the similarity-search query, because a user’s immediate message is often a poor proxy for what memory would be useful to surface, particularly early in a conversation where the message alone gives little context about what’s being discussed. Expanding the retrieval query with recent conversation context, or having the system explicitly reason about what kind of memory would help before issuing the retrieval, generally surfaces more relevant memories than a bare similarity match against the single most recent message in isolation.

How memory needs to be updated, not just written once and left static

A memory written once and never revisited eventually becomes wrong, because the facts memory is meant to capture change: a stated preference gets revised, a constraint that was true stops being true, a fact inferred from early interactions gets contradicted by more recent, more representative ones, and a system with no update mechanism keeps confidently surfacing an outdated memory indefinitely, which is often worse than having no memory of that fact at all, since a wrong memory actively steers a response in the wrong direction rather than simply leaving a gap.

Detecting when an existing memory needs updating rather than a new, separate memory needing to be added is itself a nontrivial design problem, because the system has to recognize that a newly observed fact contradicts or supersedes something already stored, rather than simply appending the new fact alongside the old one and leaving both to coexist in the memory store. This generally requires an explicit reconciliation step, checking new candidate memories against existing ones for direct contradiction or overlap before committing them, and resolving detected conflicts either by replacing the older memory outright when the new one is clearly more current and more accurate, or by retaining both with an explicit note of the change when the conflict itself, a preference that shifted over time, is worth preserving as context in its own right.

Confidence and recency both deserve to be tracked as metadata alongside the memory content itself rather than left implicit, since a memory inferred with low confidence from a single ambiguous interaction should be weighted, and eventually retrieved, differently than a memory the user stated explicitly and directly, and a memory that’s several months old deserves different treatment in a fast-changing context than one recorded yesterday. Systems that store this metadata explicitly can apply recency decay or confidence-weighted ranking directly during retrieval, surfacing well-supported, current memories ahead of older or weaker ones rather than treating every stored memory as equally authoritative regardless of how or when it was formed.

How forgetting is a design decision, not merely a storage-limit afterthought

It’s tempting to treat forgetting as purely a capacity problem, something that only becomes relevant once the memory store grows too large to search efficiently, but forgetting well is a quality lever in its own right, independent of storage cost, because a memory store cluttered with stale, superseded, or narrowly one-off facts makes retrieval noisier even when storage capacity itself isn’t remotely a constraint.

Explicit expiration, attaching a time-to-live or a natural staleness window to memories whose relevance decays over time, a task-detail, a temporary constraint, handles the case where a memory’s content signals how long it should reasonably remain useful. Usage-based forgetting, deprioritizing or eventually removing memories that are rarely or never retrieved despite being available, handles the complementary case where a memory’s retrieval history, rather than its content, signals that it isn’t earning its place in the store. Neither mechanism on its own catches everything a well-designed forgetting policy needs to catch, which is why mature memory systems tend to combine both, letting content-based expiration handle memories whose nature signals a natural lifespan and letting usage-based forgetting catch memories that turned out, empirically, not to matter regardless of how permanent they looked when they were first written.

User-initiated forgetting, letting a person explicitly ask the system to forget a fact, deserves direct, reliable support rather than being an edge case handled only incidentally through whatever general update mechanism the system already has, both because users do change their minds about what they want remembered and because, in many applications, an explicit, honored deletion request is a trust and compliance requirement rather than merely a nice-to-have convenience feature.

How memory scope and sharing boundaries shape what a system is safe to build

Memory that’s scoped correctly, to a single user, a single session, a single organization, is foundational rather than optional, because memory that leaks across a boundary it shouldn’t cross isn’t a quality problem in the way a poorly ranked retrieval result is, it’s a privacy and trust failure that can be considerably more damaging than simply retrieving nothing useful at all.

Per-user memory, isolated so that what’s learned about one user is never retrievable when serving a different user, is the baseline nearly every consumer-facing application needs, and it has to be enforced at the storage and retrieval layer directly, with user identity as a hard filter on every query against the memory store, rather than relying on the model itself to somehow know not to mix memories that were never isolated from each other in the underlying system. Session-scoped memory, relevant only within a single ongoing interaction and deliberately not persisted beyond it, suits ephemeral context that would be actively wrong to carry forward, a detail relevant only to completing the current task, and needs an explicit boundary of its own so it doesn’t quietly get promoted into longer-term storage by a memory-writing step that doesn’t distinguish session-local detail from durable fact.

Shared or organizational memory, deliberately visible across multiple users within the same team or account, is valuable for collaborative applications where one person’s stated preference or established context should inform how the system serves their teammates, but it needs to be an explicit, deliberate design choice with its clear boundary, not a default that memory drifts into simply because per-user and shared storage weren’t kept clearly separate from the start. Getting this boundary design wrong in either direction, over-isolating memory that users would want shared, or under-isolating memory that should have stayed private, tends to surface as a trust problem well after a system is already in production, which is exactly why memory scope deserves to be decided deliberately during initial design rather than left as an implementation detail resolved by whatever the storage schema happened to default to.

How memory interacts with retrieval-augmented generation without collapsing into the same system

Memory and retrieval-augmented generation solve related problems, both bring external information into a model’s context at the moment it’s needed, which tempts teams to build them as one undifferentiated retrieval system serving both purposes, but the two differ in a way that matters enough to keep them architecturally distinct even when they share underlying retrieval infrastructure.

A RAG corpus is generally a relatively stable, curated body of reference material that exists independently of any particular user or conversation, while memory is inherently personal and dynamic, built up specifically from a user’s interactions and meant to reflect that particular relationship rather than a shared, external body of knowledge. Conflating the two, storing user-memories in the same index as general reference material, risks exactly the kind of cross-contamination discussed in the scope section above, where a memory relevant to one user’s context surfaces inappropriately in a retrieval meant to draw only on shared, general material.

The two systems do benefit from working together at the point where retrieved context gets assembled for a turn, since a well-designed system might retrieve from both a general RAG corpus and a user’s personal memory store for the same query, combining a shared factual answer with the context of what this particular user has already discussed or already stated a preference about. Keeping the underlying stores and their access boundaries separate while combining their retrieved results at assembly time gets the benefit of both without inheriting either system’s failure modes into the other.

How to evaluate whether a memory system is working

A memory system that appears to work in casual testing can still be failing in ways that only show up over longer, more realistic usage, because the value of memory is inherently about behavior across sessions and over time, which makes it considerably harder to evaluate through a single-turn test than most other components of an AI system.

Recall accuracy, whether a memory relevant to a later interaction gets surfaced when it should, is the most direct thing to measure, and it requires building test scenarios that deliberately span multiple sessions, establishing a fact in one interaction and then checking, several turns or sessions later, whether that fact gets retrieved and used when a later query would benefit from it. Precision matters just as much as recall here, since a memory system that surfaces too much irrelevant material on every turn is failing just as concretely as one that fails to surface relevant material, even though the failure looks different from the outside, showing up as diluted, less-focused responses rather than as an obviously missing fact.

Update correctness, whether the system revises or replaces a stale memory once contradicting information arrives, rather than continuing to surface the outdated version indefinitely, needs its dedicated test scenarios rather than being assumed to follow automatically from good retrieval, since a system can retrieve perfectly accurately from a memory store that itself contains stale, unrevised content. Building this kind of longitudinal, multi-session evaluation into a team’s regular testing practice, rather than relying only on single-turn evaluation that can’t observe whether memory persists, updates, and gets retrieved correctly across time, is what catches the failure modes that make memory systems disappointing in practice even when every individual component looked correct in isolation.

How memory consolidation turns raw interaction history into something worth keeping

Writing memory directly from raw conversation turns, capturing exactly what a user said in exactly the words they said it, is the simplest possible approach and works acceptably for a small number of short, clearly bounded facts, but it degrades quickly as a system accumulates history, because raw turns are verbose, repetitive across similar conversations, and rarely phrased in a form that’s efficient to store or to later retrieve alongside dozens or hundreds of other accumulated memories.

Consolidation addresses this by periodically summarizing and merging related raw observations into a smaller number of denser, more durable memory entries, rather than storing every individual turn as its separate memory indefinitely. A user who mentions across several separate conversations that they prefer terse responses, that they find long preambles frustrating, and that they’d rather see a direct answer first, can be consolidated into a single, clearly stated preference memory rather than persisting as three separate, overlapping raw observations that all essentially say the same thing in slightly different words. This consolidation step is usually run as its periodic process, either on a schedule or triggered once enough new raw memories have accumulated on a topic, rather than attempted inline during every single turn, since consolidation benefits from having enough accumulated signal to recognize a pattern worth merging, which a single isolated turn can’t provide on its own.

Consolidation also creates a natural opportunity to resolve the update and reconciliation problem discussed earlier, since the same process that’s already comparing related memories to merge them is well positioned to notice when two of them contradict rather than merely overlap, and to resolve that contradiction as part of the same pass rather than requiring an entirely separate reconciliation mechanism running independently. Systems that skip consolidation entirely tend to accumulate a memory store that technically contains all the right information somewhere within it, but spread thinly and redundantly across far more individual entries than the underlying facts require, which makes both storage and retrieval noticeably less efficient than they need to be.

How memory write timing shapes both accuracy and system responsiveness

Deciding what’s worth remembering, discussed earlier, is only half the design question; deciding when that decision gets made, during the live conversation itself or afterward as a separate background process, has its consequences for both the accuracy of what gets written and how responsive the system feels to the person using it.

Writing memory synchronously, as part of generating the live response, lets the system reason about what’s worth remembering with the full, immediate context of the conversation still directly available, but it adds latency to every single turn, since the memory-write decision now sits directly in the critical path between the user’s message and the response they’re waiting on, and it couples memory-write quality tightly to whatever time budget the live response generation can spare for it. Writing memory asynchronously, as a background process that runs after the response has already been returned to the user, removes that latency cost entirely and allows a more thorough, more deliberate memory-write process, potentially involving a separate model call specifically dedicated to extracting and evaluating candidate memories, without the user ever waiting on it, at the cost of a small window where a memory that should influence a near-immediate follow-up message hasn’t been written yet.

Most production systems that handle meaningful conversation volume settle on asynchronous memory writing as the default, accepting that brief lag in exchange for both better responsiveness and a more thorough, less time-pressured memory-extraction process, and reserve synchronous, in-line memory writing for the narrower case where a user explicitly states something that clearly and unambiguously needs to be remembered immediately, a stated correction or an explicit instruction to remember something where the cost of any delay before it takes effect would be disruptive to the very next turn.

How memory systems should be tested against adversarial and edge-case inputs, not only typical ones

Most memory-system evaluation naturally centers on the typical case, a user stating a clear preference and later benefiting from it being remembered, but a memory system that’s only ever been tested against typical, well-behaved input tends to fail in avoidable ways once it meets input that’s ambiguous, contradictory, or actively adversarial, and those failure modes are worth designing and testing against deliberately rather than discovering them first in production.

A user testing the system’s limits, or simply thinking aloud in a way that isn’t meant to be taken as a durable statement of fact, “I guess I sort of prefer shorter answers, though I’m not totally sure,” presents a judgment call that a memory-write step needs to handle gracefully, either by requiring a higher confidence threshold before treating tentative, hedged language as a durable preference, or by writing the memory with an explicitly lower confidence score that later retrieval and ranking can account for rather than treating it with the same weight as a clearly, confidently stated preference. A user who directly attempts to manipulate what the system remembers, instructing it to memorize false information about another user, or asking it to store something designed to influence a future response in a way the system’s safety boundaries wouldn’t otherwise allow, needs the memory-write step to apply the same scrutiny to what’s being proposed for storage that the system would apply to any other instruction embedded in user input, rather than treating “please remember this” as an automatic, unquestioned bypass around normal safety and scope boundaries.

Building a small, deliberately adversarial test suite alongside the more typical recall-accuracy tests discussed earlier, covering contradictory statements made close together, tentative or hedged language, and explicit attempts to inject false or boundary-violating content into memory, tends to surface exactly the kind of narrow but consequential failure modes that a system tested only against clean, typical conversation would otherwise ship with unnoticed, and catching them during design and testing is considerably cheaper than discovering them after a user encounters one directly.

How storage infrastructure choices shape what a memory system can scale to

A memory system prototyped against a handful of test users, storing a few dozen memories in whatever database happened to already be available, tends to work fine regardless of which underlying storage technology was chosen, which is exactly why storage infrastructure decisions that seemed inconsequential during prototyping can turn into a bottleneck once a system is serving a large, growing number of users each accumulating their memory over time.

Vector storage, well suited to episodic memory’s similarity-based retrieval, needs to scale specifically along the dimension of per-user isolation discussed earlier, and a vector index that performs well when queried against a shared, global corpus can behave very differently once every query needs an additional hard filter narrowing the search to one user’s subset of vectors, which is why the storage layer’s support for efficient, enforced per-user filtering matters as much as its raw similarity-search performance. Structured storage, better suited to semantic memory’s key-value or record-based facts, needs its schema design attention, since a schema that captures today’s known memory types well can become a constraint once a system’s memory needs grow to include a type of fact the original schema never anticipated, which argues for a flexible, extensible record structure over a rigid one optimized only for the fact types visible at initial design time.

The two storage needs rarely map cleanly onto a single underlying database technology, which is why mature memory systems typically run a small number of purpose-fit stores together, a vector store for similarity-retrieved episodic memory, a structured store for directly looked-up semantic facts, and often a separate lightweight cache layer for the small set of memories retrieved on nearly every turn, rather than forcing every memory type through one system chosen primarily for whichever type of memory the team happened to build first. Getting this multi-store architecture right early, even while the actual memory volume is still small enough that a single simpler store would technically still work fine, avoids a considerably more disruptive migration later, once user data already depends heavily on whatever storage choice was made first.

Common mistakes teams make when designing AI memory systems

A first mistake is treating memory as simply an extended context window, appending an ever-growing transcript rather than deliberately deciding what’s durable enough to persist and what’s simply transient detail relevant only to the interaction that produced it.

A second mistake is writing every observed fact to memory indiscriminately, with no filtering step distinguishing durable information from one-off, transient detail, leaving the memory store too noisy for retrieval to reliably surface what matters.

A third mistake is trusting inferred memories at the same confidence level as explicitly stated ones, when an inference drawn from a small or unrepresentative sample of interactions can be confidently wrong in a way a directly stated fact simply can’t be.

A fourth mistake is forcing every kind of memory, episodic, semantic, procedural, through one undifferentiated retrieval mechanism, compromising on all three rather than serving any of them well.

A fifth mistake is retrieving from memory unconditionally on every turn regardless of whether the current turn calls for it, diluting the context budget with retrieved memories that add nothing to that response.

A sixth mistake is using the user’s raw, immediate message as the retrieval query without any expansion, missing relevant memory that a bare similarity match against one isolated message doesn’t surface well.

A seventh mistake is writing memories once and never revisiting them, letting a memory that’s since been contradicted or superseded keep surfacing confidently and indefinitely as if it were still current.

An eighth mistake is appending new, contradicting facts alongside old ones without any reconciliation step, leaving the memory store holding both the outdated and the current version of the same fact with no way to tell which should be trusted.

A ninth mistake is storing memories with no confidence or recency metadata at all, treating a low-confidence inference and a directly stated, recent fact as equally authoritative during retrieval.

A tenth mistake is treating forgetting as purely a storage-capacity problem, never pruning stale or rarely-used memories until the store grows too large, when a cluttered memory store degrades retrieval quality well before it becomes a capacity concern.

An eleventh mistake is failing to support explicit user-initiated forgetting reliably, handling a deletion request only incidentally through a general update mechanism rather than as its directly supported, dependable operation.

A twelfth mistake, and one of the most serious, is enforcing memory scope only loosely at the application layer rather than as a hard filter at the storage and retrieval layer itself, leaving memory that should have stayed isolated to one user retrievable when serving a different one.

A thirteenth mistake is conflating personal memory with a general retrieval-augmented generation corpus in the same index, risking the same kind of cross-user or cross-boundary contamination that strict scoping is meant to prevent.

A fourteenth mistake is evaluating a memory system only through single-turn testing, never building the longitudinal, multi-session test scenarios that reveal whether memory persists, updates, and gets retrieved correctly across time rather than only appearing to work within one isolated interaction.

A fifteenth and final mistake, easy to overlook precisely because it never shows up during normal, well-behaved testing, is testing a memory system only against typical, well-behaved conversation, never building deliberate adversarial or edge-case tests covering tentative and hedged language, statements made purely to test the system’s limits, or explicit attempts to inject false or boundary-violating content into storage, and discovering these avoidable failure modes for the first time only once a user encounters one directly and unexpectedly in production.

What connects all fifteen of these mistakes is failing to treat memory as a dynamic, actively managed system with its explicit lifecycle, what gets written, how it gets organized, when it gets retrieved, how it gets updated, when it gets forgotten, and within what boundary it’s ever allowed to surface, rather than as a passive store that simply accumulates and gets searched. Systems that design deliberately for each stage of that lifecycle, rather than treating memory as a single retrieval index bolted onto an existing context pipeline, tend to feel attentive and trustworthy to the people using them over time, while systems that skip this deliberate design tend to either forget things they should have retained or, in the more damaging direction, confidently surface memory that’s stale, wrong, or belongs to someone else entirely, undermining the exact trust that a well-designed memory system was meant to build in the first place.