Our LongMemEval Results: What MemPalace Cloud Actually Recalls

Before publishing benchmark numbers in marketing, I wanted my own. The retrieval engine MemPalace Cloud ships on top of is the open-source MemPalace project, and the most cited number in that ecosystem — 96.6% R@5 on LongMemEval — has been openly disputed upstream. Two community issues argue that the R@5 figure is largely explained by the underlying embedding model and vector store rather than MemPalace's retrieval logic.

Whether or not those critiques fully land, anchoring our marketing on a contested number from a project we depend on is bad practice. So I ran the eval myself — three stacks, 500 questions, on my own hardware, with scripts you can run.

This post is the result. The numbers are not as spicy as the upstream headline. They are honest, they are reproducible, and they show where a hosted memory layer actually pulls weight.

What LongMemEval measures

LongMemEval (Wu et al., ICLR 2025) is a long-term-memory benchmark for chat assistants. Each of its 500 questions in longmemeval_s comes with a multi-session chat history; the system under test indexes those sessions and is asked to retrieve the one(s) that contain the answer.

The metric I report is Recall@k: did the gold evidence session appear in the top-k retrieved sessions for that question?

• R@1 — was it the very top hit?
• R@3 — was it in the top three?
• R@5 — was it in the top five?

R@1 is the metric that maps closest to "the assistant grabbed the right thing on its first try." R@5 is the looser metric most existing AI-memory benchmarks publish.

Three stacks, same dataset

I evaluated three configurations on the identical 500-question slice:

1. Raw ChromaDB baseline — sentence-transformers/all-MiniLM-L6-v2 embeddings, ephemeral in-memory ChromaDB, no MemPalace code. This is the upstream-issue argument made concrete: how much of the score is just ChromaDB defaults?
2. Upstream MemPalace 3.3.0 (local) — the MIT-licensed MemPalace engine, called directly from a Python script. Same embeddings, MemPalace's own indexer and retriever.
3. MemPalace Cloud (staging) — same engine, but every memory ingested and retrieved over the public MCP endpoint at api-staging.mempalace.cloud. This is the closest real-world simulation of what a paying customer's stack looks like.

All three runs hit the same 500 questions, the same longmemeval_s dataset, and used the same embedding model where applicable.

The numbers

500 questions each. Single-shot runs.

A back-of-envelope two-proportion z-test on n=500 puts the 95% confidence interval on any pairwise difference at roughly ±2.5 pp. So pairwise deltas under ~3 pp should be read as "indistinguishable on this run" and pairwise deltas above that are real signal. I have not bootstrapped per-stack CIs — that is on the to-do list.

A few things jump out.

At R@5, raw ChromaDB is within 0.2 pp of upstream and 0.2 pp ahead of cloud staging. Both well inside the ±2.5 pp band. The "R@5 is largely embedding + vector store" reading from upstream Issues #214 and #875 is consistent with what I measured. On this dataset, with this embedding model, R@5 is dominated by the embedding-and-vector-store pair, not by MemPalace-specific retrieval. I will not be putting "91.4% R@5" on the homepage as a feature claim.

At R@1, both MemPalace stacks add ~7 pp over raw ChromaDB (local: 75.8% → 83.2%; cloud: 75.8% → 82.8%). That delta is well outside the noise band. R@1 is the metric an end user feels — it is the difference between the assistant grabbing the right memory immediately versus burying it behind two distractors. This is where the engine's hybrid retrieval and structure pull their weight.

The hosted stack tracks the upstream local stack within 0.4 pp across the board — inside the ±2.5 pp band, so I can't distinguish them on this run. That is the answer to "do I lose accuracy by going hosted?": not on this benchmark, not on this dataset. You trade ≤0.4 pp at R@1 for OAuth, MCP, cross-device, and not running ChromaDB yourself.

This is not a state-of-the-art claim. Other benchmarks, other datasets, other embedding models will look different. LongMemEval longmemeval_s is one slice of one benchmark. I would be skeptical of anyone — me included — who extrapolates from a single number to "best in class."

What I did not test (and what is next)

A few honest gaps:

• Embedding model is fixed at MiniLM-L6-v2. Bigger embedders (BGE-M3, OpenAI text-embedding-3) almost certainly move the numbers.
• No QA accuracy yet. R@k tells you whether retrieval surfaced the right session. End-to-end QA accuracy — does the assistant answer correctly given that context? — is the next layer of the onion. Out of scope for this post.
• Single seed, no bootstrapped CIs. I report a back-of-envelope ±2.5 pp band from a two-proportion z-test on n=500, but I have not run a proper bootstrap or paired-sample test across the three stacks. With the gaps I am seeing, the qualitative read is unlikely to flip, but the quantitative claim deserves a real CI eventually.
• Auxiliary-index contributions are not isolated. The upstream-engine and cloud-staging numbers include the engine's knowledge graph and its summary-index layer; this run does not separate their effects from the retrieval core. The ablation milestone above is where those get split out.

I will not be publishing numbers I have not measured. When the ablation work is done, I will write a follow-up.

Hardware and reproducibility

The reference run used:

• Apple M4, 32 GB unified memory, macOS 26.2
• Python 3.14, MPS (Apple Metal) backend for sentence-transformers
• ChromaDB ephemeral in-memory client
• MemPalace 3.3.0 for the upstream-local run
• api-staging.mempalace.cloud for the cloud-staging run

Every run is captured to a JSON record with timestamp, dataset, model SHA-256 prefix, and elapsed wall-clock alongside the R@k metrics. R@k results were stable across re-runs of the 500-question slice on the same machine, well below the ±2.5 pp noise band. On other hardware or with a different backend the numbers may shift slightly: the PyTorch MPS backend is not bit-exact across runs, and HNSW index construction in ChromaDB has tie-breaking that depends on insertion order. So treat "reproducible" here as "reproducible to within the noise band," not "bit-identical."

A weekly CI workflow re-runs a 50-question sample against the baseline. Note the asymmetry: a 50-question sample has its own ±7 pp noise band on R@5, so the workflow only catches gross regressions — anything subtler needs a full 500-question rerun. Treat it as a smoke test, not a quality gate.

What this means for the product

MemPalace Cloud is not the highest R@5 on this benchmark, and I will not pretend otherwise. The case for it stays exactly what the first post said: a memory layer that follows you across Claude Desktop, Claude Code, ChatGPT, Codex CLI, across devices, on EU infrastructure, with the open-source engine you can self-host if you ever want to walk away.

Where the benchmark does inform the product story:

• R@1 lift is real. When I tell my AI tools to recall something, the right memory is the very first hit ~83% of the time on this eval. That maps to "the assistant did not have to dig" in everyday use.
• Hosted vs. self-hosted is within the noise band on accuracy. The trade-off is operational (multi-tool, multi-device, OAuth) versus self-host (full control, local-first). On this benchmark, it is not an accuracy trade-off.
• The headline upstream number was contested for a reason. I am not going to amplify it. Whatever you read on this blog about retrieval quality, you will be able to follow a link to the script that produced it.

If that lines up with what you need from an AI memory layer, give it a try. The eval scripts and raw result JSONs currently live in our private product repo; if you want to reproduce these numbers or scrutinise the methodology before we publish a clean reproducer, drop us a line and we will share what you need.

Last updated 2026-05-05. Dataset: LongMemEval longmemeval_s (500 questions). Eval scripts and raw result JSONs available on request: [email protected].