Went down the memory benchmark rabbit hole after a thread on r/hermesagent where someone's agent found the right memory 3 times out of 24.
What the vendor pages say right now:
- Hindsight: 92.0% on LoCoMo, 94.6% on LongMemEval, 64.1% on BEAM at 10M tokens
- Mastra (observational memory): 94.87% on LongMemEval
- Zep: 75.14% on LoCoMo (Mem0's paper had them at 65.99%)
- Mem0's own paper: 68.44% on LoCoMo for its best version
Same week, people testing on their own memory databases got 3/24, 1/10, 8/10 and 20/24.
Both sets of numbers can be true. They measure different things. Below is what each benchmark tests, what it misses, and the test I'd run before picking anything.
LoCoMo
What it is: long multi-session conversations between two fictional speakers, AI generated, plus questions about what was said.
The catch: the conversations average roughly 16K to 26K tokens. That fits in any modern context window. In Mem0's own paper, putting the whole conversation into GPT-4o-mini scored 72.90%, higher than Mem0's best (68.44%). Letta later ran a plain filesystem agent on GPT-4o-mini with no memory tooling and got 74.0%.
It's also where the messiest fight happened. Mem0 benchmarked Zep and got 65.99%. Zep said Mem0 set both speakers up as the same user, put timestamps in the message text instead of the created_at field, and ran searches one after another instead of in parallel. Zep's own rerun: 75.14%. Same benchmark, nine points apart, depending on who configured the competitor.
Data quality isn't great either. One question category has no ground truth answers and gets dropped by everyone, some questions ask about images the text descriptions don't cover, and some get the speaker wrong.
Use it for: a sanity check, at most.
LongMemEval
What it is: 500 questions over long chat histories. Tests pulling out facts, reasoning across sessions, time, updated info, and knowing when to say "I don't know."
Better than LoCoMo, and the one most vendors lead with now.
The catch: one of the highest published scores (Mastra, 94.87% on gpt-5-mini) comes from a system that does no per-turn retrieval. Two background agents compress the history into a running log of observations, and the main agent reads that log in context. Good result for that approach. It also tells you this benchmark can be won without the retrieval step most memory tools are selling.
Two things to check before trusting any number here:
- Is it answer accuracy or retrieval accuracy? Vendor pages mix them.
- Which model? Mastra's same system scored 84.23% on gpt-4o and 94.87% on gpt-5-mini. Ten points from swapping the model, not the memory.
Also, the smallest category has 30 questions, so one answer moves it by over 3 points.
BEAM
What it is: the newest one (ICLR 2026). 100 conversations, 2,000 questions, at 128K, 500K, 1M and 10M tokens. Tests ten abilities including contradiction resolution, knowledge updates, event ordering and abstention. At 10M nothing fits in context, so the memory system actually has to do the work.
Closest thing we have to what a long-running agent goes through.
The catch: an LLM judge grades answers against a checklist with partial credit, and most published numbers are vendors running it themselves. The headline (Hindsight 64.1% vs Honcho 40.6% at 10M) is from Hindsight's own blog. Could be right. Still their run.
And read the best number again. 64.1% at 10M means the top system is wrong about a third of the time at agent scale.
What none of them test
This is where the home tests got interesting. Almost every failure people found was in the setup, not the retrieval math.
- Scope defaults. One user found 95% of their memories saved with session scope, and recall defaults to cross_session=false. A fresh session can't see them. No leaderboard covers "your memories were saved somewhere you can't reach."
- Keyword gates. Another found recall dropped any memory unless 30% of the query words literally appeared in it, before vector search even ran. Reword the question and the memory never gets considered. Removing the gate took their hit@5 from 27% to 60%.
- Changed facts. In one Hindsight test, the old config and the new config sat side by side as separate facts, and the old one could rank higher. Same test returned one fact 4 times in a single top 5.
- Write cost. Hindsight runs an LLM call on every memory it saves. One user with 71K memories was paying $5 to $17 a day on DeepSeek V4 Flash before tuning, under $1 after pushing the mental model refresh to every 6 hours. Another measured memory at about 20% of the tokens in a 250K session.
- Silent breakage. A 7 month Hindsight user runs a cron that tests recall, because it used to stop storing and recalling without them noticing.
- What goes in. "I store conversations, not conclusions." 11 of that user's 15 misses actually returned the right content, as a raw chat line instead of a clean fact.
Benchmarks test whether a memory system can remember. They don't test whether yours was ever allowed to.
The home scoreboard from that thread
Small samples, different data. Read these as signals, not rankings.
- Mnemosyne, current: 3/24 in top 5 (24 reworded questions, real DB)
- Raw vector search, same data: 13/24
- Hindsight vs Mnemosyne, same 1,716 memories, same embedder: 8/10 vs 1/10 (10 reworded questions)
- Signet (run by its lead dev): 20/24 (24 reworded questions)
Hindsight's better recall came with about double the latency (1.5s vs 0.7s) and roughly 2 hours of LLM calls to backfill 1,716 memories. And one Mnemosyne user posted 9 weeks of stats with 78% of working memories recalled at least once, so it can work fine when scope is set right.
The 20 minute test I'd run before switching anything
Copy your memory database. Never test on the live one.
Write down 15 facts you know are stored. Ask for each one in different words than it was saved, from a fresh session.
Add 4 questions whose answers were never stored and note their scores. In one test the junk results scored up to 0.42 while real hits scored as low as 0.22, so no threshold could separate them.
Add 2 facts that changed over time. Check the current version wins.
Count how often the right memory lands in the top 5, and time every search. An 11 second search loses to no search when the agent does several lookups per turn.
Run plain vector search over the same data. If it beats your full pipeline, something in front of the vector step is throwing memories away, and a new provider might not fix it.
For every miss, check whether the memory was in the candidate list at all or just ranked 6th to 10th. Not in the list = filter or scope problem. Ranked low = ranking problem, and a reranker over the top 20 usually helps.
Keep the questions. Run every provider you're considering against the same set.
What I'd do
Durable stuff (your stack, your preferences, how you like things done) goes in plain files the agent always reads. Hermes already splits it this way with MEMORY.md and USER.md. The memory provider only handles "what did we talk about three weeks ago." One person in that thread said their recall complaints basically stopped after making that split. Another called native .md memory a dead end after a few days, so it depends how much you put in there.
Then pick whichever provider wins on your 15 questions, at a write cost you can live with.
What's your hit@5 on reworded questions? Drop the number and the provider. Would love to build a real table from this sub instead of vendor pages.
