We built Contextely as an agent memory layer for a company's own systems of record, and three decisions in its architecture turned out to matter more than anything else we shipped in the first year. None of them were the interesting part of the pitch. All three were about ordering, timing and honesty in ways that are easy to skip when a demo just needs to work.
Why entitlement has to run before the freshness refresh
The first version of our retrieval pipeline did the obvious thing: find the relevant memory objects, check whether they were stale, refresh anything that was, then check whether the asker was entitled to see the result. It worked. It also had a hole we found before it shipped, not after, and finding it changed how we think about the whole system.
A staleness refresh means reading a record from a real system of record: a live query against a customer's CRM, or a call to their support desk API. If entitlement is checked after that refresh, the read still happened. An asker who was never allowed to see a given account's data could ask a question about it and, even if the final answer correctly withheld the result, their query had already caused a real read against that company's live system on their behalf. That is observable. It shows up in database logs, in a slightly longer response time for a query that touched a live source versus one that hit a cached, entitled object, and in principle it is a side channel: someone who cannot see the answer can still learn something from the fact that a read happened, or how long it took.
So we moved entitlement first. Scoring now happens before anything downstream, including freshness. An object the asker isn't entitled to see scores exactly zero and is dropped by the same cut that removes irrelevant results, and it never triggers a re-fetch, because nothing unscored gets that far.
"Granting LLMs unchecked autonomy to take action can lead to unintended consequences, jeopardising reliability, privacy, and trust."
OWASP Top 10 for Large Language Model Applications, on excessive agency
That line is written about agents taking action in the world, but the same discipline applies to an agent's retrieval step. A refresh is an action, not a passive lookup, and giving it unchecked reach before permissions are settled is the same category of mistake with a smaller blast radius.
What does "closing a side channel" actually mean in practice?
It means we added a test, not just a design note, that asserts the invariant directly: no unentitled candidate can reach a non-zero score under any input, checked in isolation from the database. If a future change reorders these steps by accident, that test fails loudly rather than the system leaking quietly. We also added a runtime assertion at the boundary between scoring and everything downstream, so even a bug that slips past the test throws an error instead of returning a result. We would rather return a 500 we have to explain than a wrong answer we would never notice.
Why memory carries a TTL instead of a fixed schedule
Our first instinct, like most teams building something like this, was a nightly re-index job. It is the easy thing to build and the wrong thing to ship. A nightly job treats every source as equally volatile, and almost nothing in a real company is. A support ticket status might be worth re-checking every few minutes. A company's registered VAT number is worth re-checking about as often as it changes, which is close to never.
We gave every source its own time-to-live instead, and every memory object it produces inherits it. Freshness decays linearly from a full score right after a fetch down to zero at the TTL boundary, and a zero freshness score doesn't remove the object from consideration; it marks it stale, which is the specific signal that tells the retrieval pipeline to re-fetch before serving anything. If the re-fetch fails, we don't fall back to the old answer and hope it's still true. We return a labelled failure, stale_refresh_failed, because serving a guess as though it were current is worse than admitting we couldn't confirm it.
One thing we didn't expect: when a re-fetch finds the underlying record unchanged, we reset the clock and do nothing else. No re-condensing, no new model call. That single decision cut our steady-state condensation cost by a large margin, because most re-checks in a real workspace find nothing has changed.
| Approach | What it gets right | What it misses |
|---|---|---|
| Fixed nightly re-index | Simple to build and reason about | Treats a volatile support ticket the same as a static company address |
| Re-index on every query | Always current | Expensive, and re-reads sources for people who might not even be entitled to the result |
| Per-source TTL with re-fetch on staleness (our approach) | Freshness matches actual volatility, cheap when nothing changed | Requires every source to declare a sensible TTL, which is a real decision, not a default |
Table 1: three ways to keep a memory layer current, and the trade-off each one makes.
Why we built both an MCP server and an MCP client
We expected most usage to come through a chat-style interface. It didn't. The actual demand split into two shapes we hadn't planned for as clearly: other software calling us to ask a question, and us needing to reach a company's own tools to get source data in the first place.
Being an MCP server covers the first shape: any agent that speaks the protocol can query us directly, at /api/mcp, without a custom integration. Being an MCP client covers the second: a company that already runs an MCP server over some internal tool doesn't need us to write a bespoke connector, because we can name its list tool and its fetch tool and let the same freshness loop that reads a Postgres source read from an existing MCP endpoint instead.
Was choosing a lexical relevance score over embeddings a mistake?
We don't think so, though it's the decision we get asked about most. Condensation changes the shape of the search problem: we are matching a query against a written summary of a few hundred words per workspace, not against millions of raw chunks, because the whole point of condensing a source is that the store stays small. At that scale, a deterministic term-overlap score, weighted by field (a title hit counts for more than a body mention), is fast, needs no separate vector infrastructure for a self-hosted deployment to run, and is explainable: the pipeline can tell you exactly which terms matched and why an object surfaced. We would reconsider this if workspaces started holding millions of objects rather than hundreds, but that would also mean the condensation step had stopped doing its job.
Common pitfalls we ran into building this
- Optimising the order of operations for the happy path first. Our original refresh-then-entitle order worked fine in every demo, because demos rarely simulate an unentitled asker probing the boundary.
- Assuming a test that passes is a test that covers the invariant that matters. We had scoring tests before we had a test that specifically asserted no unentitled candidate could ever reach a non-zero score under any input. Those are not the same test.
- Building the freshness loop before deciding what failure looks like. It is tempting to make refresh always "succeed" by falling back to the old value. We had to explicitly decide that a failed refresh returns an honest error state instead.
- Treating MCP client and server support as a single feature. They solve different problems and we built the server role first, then discovered the client role was equally load-bearing for real deployments.
If any of this is relevant to a decision you're making right now, the security model page has the entitlement ordering written out with the actual code, self-hosting lets you run the freshness loop yourself against your own sources, and the documentation covers both the MCP and REST paths in full. For how this compares to the general-purpose llm memory database options that don't take the same approach to entitlement, that comparison covers the differences directly.