Numbers travel faster than methodology. A benchmark score gets quoted in a pitch deck, a comparison table, a tweet, long after anyone checks how it was produced, and agent memory benchmarks gave the field a genuinely instructive example in 2026 of what that gap can actually look like once someone bothers to check.
What the independent reproduction actually found
In an analysis published 27 May 2026, an independent evaluation reproduced LongMemEval results across several agent memory vendors using a controlled methodology: a fixed answer model, a binary judging prompt with explicit conditions for both a correct and an incorrect verdict, and five-seed averaging specifically to control for run-to-run stochasticity. For one widely cited result, the analysis reported reproducing a score of 73.8% against a published figure of 93.4% from the vendor's own announcement, a gap of just under 20 points on the same memory system and the same underlying data.
"The 73.8% to 93.4% jump from observed to published... is not attributable to the memory system."
The state of AI memory in 2026: claimed vs observed, Maximem, 27 May 2026
That sentence is the actual finding, and it matters more than the raw numbers on their own. The analysis attributed the gap to prompt engineering layered on top of the memory system at evaluation time, meaning dataset-specific equivalence rules and directional bias built into the judging logic, rather than to any difference in what the memory system actually stored or retrieved. Two evaluations of the same underlying retrieval quality produced scores 20 points apart because the surrounding evaluation harness differed, not because the memory layer got better.
Why does this matter beyond one vendor's number?
Because it is a methodology problem, not a single-vendor problem, and it will recur anywhere a benchmark score is reported without the evaluation harness alongside it. A number without its methodology is not comparable to anything, including a different number from the same vendor produced a different way. The instructive part of this specific case is that someone actually did the reproduction and published exactly where the two numbers diverged, which is the only way a claim like this becomes checkable rather than just repeatable.
What we do differently, and its real cost
It would be easy to use this as a reason to talk up an alternative approach without naming the trade-off in it too, so here is ours, stated plainly rather than left for a reader to discover later.
Our retrieval is a deterministic, field-weighted term-overlap score, not a vector embedding search. A query and a candidate record are both broken into terms, weighted by which field they matched (a title hit counts for more than a body mention), and the result is a relevance figure that can always show exactly which terms matched and where. That is genuinely useful for the same reason the reproduction above matters: a score you cannot explain is a score you cannot audit, and an audit log that says "this matched because these five terms overlapped" is a fundamentally different artefact from one that says "the cosine similarity was 0.83."
| Approach | What it gets right | What it misses |
|---|---|---|
| Vector embedding search | Catches genuine paraphrases and synonyms with no shared words | The score itself is not directly explainable; needs a separate vector service and index |
| Deterministic term-overlap scoring (our approach) | Every match is traceable to specific matching terms; no external service required | Will not surface a semantically related result if the matching term genuinely is not present |
Table 1: the real trade-off between vector search and deterministic term-overlap scoring, stated in both directions.
Does deterministic keyword scoring actually miss real results?
Yes, and pretending otherwise would be exactly the kind of unqualified claim this post is arguing against. If a query asks about "revenue" and the only matching record says "turnover" and nothing else overlaps, a pure term-overlap score will not surface it on that basis alone, where an embedding comparison plausibly would. We accept that specific gap in exchange for a score that needs no external vector infrastructure to run, works identically in a self-hosted deployment with no internet access beyond an LLM endpoint, and, most importantly, can always explain itself. The condensation step is what makes this trade worth making: matching against a written summary of a few hundred words per workspace rather than millions of raw chunks changes how much a purely lexical match actually needs to catch.
Why the methodology detail actually matters
It would be easy to read the reproduction gap above as a story about one vendor overstating a number, and move on. The more useful reading is about what made the gap checkable at all: a published methodology specific enough that someone else could rerun it and point at exactly where the two numbers diverged. Five-seed averaging exists because a single run of a stochastic system can land anywhere in a distribution, and reporting the best seed rather than the average is a quiet way to inflate a headline figure without saying anything false. A binary judging prompt with explicit conditions for both a correct and an incorrect verdict exists because a judge model given room to interpret ambiguously worded criteria will, over enough examples, drift toward whichever interpretation the evaluation was implicitly designed to reward.
None of that is exotic. It is the same discipline any measured claim needs: a stated method, a fixed judge, enough repetition to separate signal from noise. What made the 2026 reproduction notable was not that a gap existed, gaps between published and reproduced results are common across machine learning generally, but that the analysis traced the specific cause rather than stopping at "we got a different number."
What does this have to do with an audit log?
More than it first appears. A retrieval pipeline that can only say "this scored 0.83" gives an auditor nothing to check beyond trusting the number. One that can say "these five terms matched, weighted by field, and the object was entitled because this specific grant covered it" gives an auditor something to actually verify against the underlying data, the same way a reproducible benchmark methodology gives a reviewer something to rerun rather than take on faith. Explainability is not a consolation prize for skipping embeddings. It is the same property that made the benchmark reproduction above possible to conduct at all: a claim stated precisely enough that someone other than its author can check it.
What to actually ask before repeating any agent memory benchmark number
- What judging model and prompt produced this score, and would a different judge reproduce it?
- Was the number averaged across multiple runs, or is it a single favourable seed?
- Does the vendor's own evaluation harness differ from the standard implementation of the benchmark, and if so, how?
- Has anyone outside the vendor reproduced it, and did they publish their method alongside the result?
None of that requires distrusting benchmarks as a category. It requires treating a benchmark score the way you would treat any other measured claim: worth citing once you know how it was produced, not before.
For how the term-overlap approach described here fits into the wider retrieval pipeline, what we learned building an agent memory layer covers the design decisions around it in full, permission aware rag covers the entitlement side of the same pipeline, and the security page has the actual scoring code referenced rather than described secondhand.