Ask an AI agent the same question twice, a week apart, and a surprising number of them answer as if the first conversation never happened. That is not a model limitation so much as a missing layer. An agent memory framework is the piece of software responsible for making the second answer better than the first, and it is worth understanding on its own terms rather than as a feature bullet on a vendor's homepage.
This piece is a plain explainer: what an agent memory framework actually is, the components it is built from, how it differs from a vector database, and where the real engineering difficulty sits. It stays product-neutral for most of the way, because the architecture matters more than any one vendor's spin on it.
What problem is an agent memory framework actually solving?
A large language model has no memory of its own between calls. Everything it "knows" about a specific conversation lives in the context window sent with that request, and once the window closes or fills up, that information is gone unless something outside the model wrote it down first. An agent that has to re-explain a customer's account history, a project's constraints, or a user's stated preferences on every single turn is not really an agent. It is a very articulate goldfish.
An agent memory framework exists to fix that. It sits between the model and the outside world, watching conversations and source systems, deciding what is worth keeping, and handing back the relevant pieces when a later question needs them. The MemGPT paper, one of the earlier and more influential pieces of work in this space, frames the problem in almost exactly these terms.
"We propose virtual context management, a technique drawing inspiration from hierarchical memory systems in traditional operating systems that provide the appearance of large memory resources through data movement between fast and slow memory."
Packer et al., "MemGPT: Towards LLMs as Operating Systems", arXiv, 2023
MemGPT's proposed fix borrows directly from how an operating system manages a computer with more data than fits in RAM: keep the working set small, and move information between fast and slow storage as needed. That framing, memory as a managed resource rather than a bigger prompt, is the idea underneath most of what follows.
The five components, not one black box
Vendors like to describe agent memory as a single feature: "give your agent memory." Underneath, it is closer to five separate jobs, and a framework that skips one of them tends to fail in a specific, predictable way.
| Component | What it does | What breaks without it |
|---|---|---|
| Extraction | Pulls candidate facts out of raw conversation or source documents | The system stores everything verbatim, which is expensive and dilutes what matters |
| Storage | Saves facts in a structured or condensed form, with an address back to the original | A fact with no source link cannot be verified or refreshed later |
| Retrieval | Finds the subset of stored facts relevant to a new query | The agent either sees nothing useful or sees far too much |
| Ranking | Orders retrieved facts by relevance and freshness | Old or tangential facts outrank the current, on-topic one |
| Entitlement | Decides which of the relevant facts a specific asker may see | Two people asking the same question get the same answer regardless of what they are allowed to know |
Table 1: the five jobs an agent memory framework does, and the specific failure mode each one prevents.
The first four show up in almost every writeup of this space. The fifth is the one that gets skipped most often, and it is the one that turns a demo into an incident the moment more than one person uses the system.
How does extraction decide what is worth keeping?
Extraction is the step most people picture when they hear "agent memory," and it is also the step most prone to silent failure. A naive approach stores every message verbatim and searches across all of it later, which technically works for a single user's small chat history and stops working the moment volume grows: search gets slower, retrieval gets noisier, and the underlying source of truth (whatever generated the conversation in the first place) keeps changing without anyone told the stored copy about it.
A better approach condenses: it turns a customer support thread into a handful of facts (what the issue was, what was promised, what is still open) rather than keeping the full transcript as the unit of storage. This is a real trade. Condensation makes retrieval faster and cheaper, but a badly extracted fact is harder to catch than a badly worded raw transcript, because nobody re-reads the transcript once it is summarised. Any framework doing extraction seriously needs a way to trace a stored fact back to where it came from, so that trade can be checked later.
Does an agent memory framework need a vector database?
No, and this is one of the more commonly confused points in the space. A vector database is a specific retrieval mechanism: it stores an embedding (a numeric representation of meaning) for each piece of text, and finds stored items whose embeddings are close to a query's embedding. That is useful for catching paraphrases: a memory saying "the contract renews annually" can be found by a query asking about "yearly renewal terms" even though the two share almost no words.
It is not, on its own, a memory framework. A vector database has no opinion on what gets extracted, no concept of a fact going stale, and no idea who is allowed to see what. Plenty of working agent memory systems retrieve by lexical term overlap, matching actual words rather than embedding distance, and treat semantic search as an optional layer added on top rather than the foundation underneath. Contextely is one concrete example: it ranks by deterministic term overlap by default, and only fuses in a semantic ranking pass when an operator explicitly configures an embedding model. That default exists because condensed memory is small, hundreds of facts per workspace rather than millions of raw chunks, which is exactly the regime where a plain term-overlap score is fast, needs no separate vector infrastructure, and can tell you precisely which words matched. Whether that is the right default for a given deployment is separate from whether a memory framework categorically requires a vector store. It does not.
Episodic, semantic, and procedural memory, explained without jargon
Cognitive science has spent decades arguing about how human memory is organised into types, and a fair amount of that vocabulary has been imported wholesale into agent memory research. It is useful vocabulary once translated out of academic shorthand.
| Memory type | Human analogy | Agent example |
|---|---|---|
| Episodic | Remembering a specific event you experienced | "This customer asked about a refund on 14 March and was told to expect it within five days" |
| Semantic | Knowing a fact, detached from when or how you learned it | "This customer is on the Pro plan" |
| Procedural | Knowing how to do something, without consciously recalling being taught | "When a refund is requested, check order status before approving" |
Table 2: the three-way memory split cognitive science contributes to agent memory design, with a concrete agent-facing example for each.
A recent survey of the field puts the scale of the underlying research problem plainly.
"Large Language Model (LLM)-based agents have fundamentally reshaped artificial intelligence by integrating external tools and planning capabilities. While memory mechanisms have emerged as the architectural cornerstone of these systems, current research remains fragmented, oscillating between operating system engineering and cognitive science."
Luo et al., "From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms", arXiv, 2026
That "oscillating" is worth sitting with. It means there is no single agreed blueprint yet, and a team evaluating a framework should expect vendors to describe the same mechanism using vocabulary borrowed from whichever discipline they lean on. What matters in practice is not which label a vendor uses, but whether the system distinguishes a fact that might change (semantic, needs a freshness check) from a record of something that already happened (episodic, does not need re-verifying, but also does not update itself). Conflating the two is a recurring bug: a system that treats "the customer is on the Pro plan" with the same confidence five months on, with no mechanism to notice the plan changed, is quietly serving a stale fact as settled history.
Where entitlement fits, and why it is usually bolted on wrong
The moment more than one person uses a shared agent memory system, a new problem appears that a single-user memory tool never has to face: not everyone who can ask a question is entitled to every answer. A support agent and a finance lead asking the same memory system about the same customer account should not necessarily see the same facts back.
The common, flawed approach treats this as a filter: retrieve everything relevant, generate or assemble an answer, then redact anything the asker should not see. This has a structural problem. Something has to hold the unentitled information in order to redact it, which means a synthesis step, or the person reviewing a log, has already been exposed to data they should never have touched. It also tends to fail quietly: a redaction rule is one more piece of logic that can be forgotten on one code path and not another.
The alternative treats entitlement as a factor in ranking rather than a filter on the answer. An object the asker is not entitled to see scores exactly zero on the same pass that scores relevance, and it leaves the result set alongside every other irrelevant item, never reaching a synthesis stage at all. Contextely's retrieval code implements this literally: entitlement multiplies relevance rather than being checked afterward, so an unentitled result cannot accidentally slip through a step that forgot to check. The distinction sounds small in the abstract and is not small in a security review: "was this data ever assembled into an answer" and "was this data ever fetched at all" are very different claims to be able to make truthfully. Permission aware RAG covers this pattern in more depth for retrieval-augmented systems generally, and it applies just as directly to a standalone agent memory framework.
There is a second, less obvious reason ordering matters. If a system refreshes a fact from its live source before checking whether the asker is entitled to see it, an unentitled query still causes that source to be read, which is itself a disclosure: someone with no right to an answer can still learn something from the fact that a read happened. The fix is the same principle applied one step earlier: check entitlement before touching a live system on the asker's behalf, not just before showing them the result.
Stateful agents versus a good memory API
"Stateful agent" is sometimes used as if interchangeable with "agent with a memory framework," and the two are related but not identical. A stateful agent is one whose behaviour on a given turn depends on accumulated history rather than only the current input. A memory API is the specific interface, usually something like memory_add, memory_search, and memory_get, that lets an agent read and write that history in a structured way rather than stuffing it back into the prompt as raw text every time.
The distinction matters because a system can be stateful without a real memory API (state kept as an ever-growing prompt, which degrades as context grows), and a memory API can exist without meaningfully making an agent stateful (a store nobody actually queries before answering). A working combination needs both.
Building it yourself versus adopting a framework
A team weighing whether to build agent memory in-house or adopt an existing framework is really weighing five separate build decisions, one per component above, against however much of that a given framework already provides. Extraction and storage are the parts most teams underestimate, because a rough version is easy to prototype and a correct version, with a real source link, real staleness tracking, and a sensible unit of condensation, takes considerably longer than the first demo suggests.
For a comparison of specific named products, including how they differ on these components, our comparison of LLM memory database options goes through them one at a time. If the system needs to reach an existing MCP-speaking tool as a source rather than only a database, our explainer on MCP server memory covers that pattern, and Contextely's own architecture writeup documents the ordering decisions that came out of building one of these frameworks in production rather than in a whitepaper.
Frequently asked questions
The questions below are the ones that come up most often once a team has decided it needs an agent memory framework and starts comparing what "memory" actually means across different products.
Common mistakes worth naming directly
- Treating "we added embeddings" as equivalent to "we added memory." Semantic search improves retrieval recall. It does not decide what to extract, when a fact is stale, or who may see it. A vector index bolted onto raw chat logs is retrieval, not a memory framework.
- Storing a fact with no link back to its source. A condensed summary that cannot be traced to where it came from cannot be refreshed and cannot be audited when something turns out to be wrong.
- Checking entitlement after generating an answer instead of during retrieval. This is the redaction trap described above, and it is the single most common design mistake in shared agent memory systems.
- Using one freshness policy for every fact. A customer's registered address and a support ticket's current status do not go stale at the same rate, and treating them identically means either re-checking the address constantly for nothing, or trusting a ticket status long after it stopped being true.
- Confusing "more memory" with "better memory." A system that never forgets anything eventually returns so much marginally relevant history that the genuinely useful fact gets lost in the noise, which is the same failure prompt stuffing produces, just moved one layer down. Context engineering 101 covers that failure mode from the prompt side.
None of this requires picking a specific vendor to get right. It requires treating agent memory as five separate design decisions rather than one feature checkbox, and being honest about which of the five a given tool, including this one, actually does. Contextely's pricing page shows where the free tier sits if trying this against a real source is the fastest way to check it.