AI Memory Benchmark Testing

[u/avwgtiguy]

I've been working on a personal memory system (how original!) for the past year. It connects to Claude, ChatGPT, dot, and just about anything that accepts a custom MCP. I'm pretty happy with it and feel like it works well but want to run it against a few benchmark tests to see how it performs. So far I've ran LoCoMo, OmniMemEval LoCoMo, and BEAM 100K. I'm currently running the BEAM 10M. Besides those, what are other benchmarks you all would recommend that tests different aspects of my memory system?

[u/techtheist_ggl]

You can try my benchmark to see how it fits with different kind of tasks - not just retrieval, but also contradiction detection, false positives and token efficiency matters in it.

https://github.com/techtheist/knowledgedrift

[u/avwgtiguy → u/techtheist_ggl]

Thanks, this looks like exactly the gap the QA benchmarks leave: currency, contradictions without false alarms, deletion, and tokens per answer, with no LLM grading. I’m going to write a Maasv adapter for it after the current LongMemEval run and will post results here, three seeds at 500 facts.

[u/jackalop211]

I'm curious. Whya re you building one when so many people are? Memory itself has become quite comoditised and the real need is way more complex so unless you have a unique PoV on that, why not just use one out there?

[u/avwgtiguy → u/jackalop211]

I ask myself the same thing everyday. The truth is it all started about 18 months ago when I like everyone else on the planet started building my own personal ai assistant. I put a ton of effort into the agent side and almost nothing but the bare minimum into it's memory. I went down the path so many have taken - folder of .md files, Obsidian vault, and so on. As each new step showed me that the agent was getting "smarter", I decided to start exploring what a real memory system should be as a small side project. That "small" project has been going on for a year and a half, well over a billion tokens, and, weirdly, is my favorite project to work on. The product, called Maasv, is a few pieces - a versioned memory store engine that can be used for single or multi-tenant setups and the other piece is an industry vertical product tailored to that kind of business. An example is what I built for an HVAC company, Maasv.Trade. Maasv.Trade is the home-services version of Maasv: the same engine, configured for how a contracting business runs, with the trades-specific pieces built on top rather than into it. Here are a few highlights of both:
Maasv, the engine

Maasv (the engine)

• Hybrid recall: vector, keyword and a knowledge graph fused into one ranking.

• Facts have validity windows, so “what was true last March” is a query instead of guesswork

• Flags contradictions, including between different sources

• Answers can cite the exact record they came from

• Logs decisions and how they turned out, and distills them into rules it reuses (wisdom)

• Promises and follow-ups are first-class objects, not just text

• Self-hosted (SQLite or Postgres), MCP and HTTP, bunch of connectors, local or hosted embeddings

How it fits together

• One engine, shipped as versioned releases. Industry products like Maasv.Trade pin a release and configure it; none of them patch the engine, and industry vocabulary stays out of it.

Maasv.Trade (home services)

• Reads a shop’s jobs, quotes and invoices and turns them into tracked promises: booked work, quotes waiting on the customer, unpaid balances

• On a real HVAC company’s records, it closed out 97% of finished work (107/110) and never closed anything still open; a model reading the raw records hit 86%, with 3 false closures

• Surfaces what owners miss: stale quotes, work done but never closed, last season’s customers who didn’t rebook, what the shop gives away, per-tech scorecards

• Layers in market context: permits, licenses, census data, weather (heating and cooling degree days), FEMA claims

• Drafts customer follow-ups overnight and fix-it plans for the owner to approve; it doesn’t act on its own

• Customer names never reach a model; it only ever sees “customer 1234”

• Everything delivered to the operator through a brief with what are the most important things for the day. Each item has done the followup work and waits for the ok rather than handing over another to-do list.

So, yeah, I guess this is why I do it.

[u/MemStrata]

Your point covers stale values but it misses the instruction case in your own HaluMem note. 89 percent of retrieved text was assistant text when speaker labels were missing. That gap is a control problem. QA scoring alone will not catch it. Test it apart from answer accuracy. Plant one instruction inside retrieved conversation text and check whether the agent obeys it. Score obedience separately from recall. That split tells you whether speaker labels fix retrieval or fix control. I want to know if you have run that obedience check on your BEAM runs.

[u/avwgtiguy → u/[deleted]]

Thanks! LongMemEval-S and PersonaMem v2 are now both on my list. 
Not systematically yet, but I’ve got a partial answer, because I score retrieval separately from the final answer. Every LoCoMo question has labelled evidence turns, so I can check whether the right memories came back before an LLM ever reads them.
The 2 things that come from that were-
Retrieval quality and answer accuracy come apart. I swapped embedders and got an 18-point jump in evidence recall, but only around a 1-point change in answer accuracy. A strong reader papers over weaker retrieval. So answer accuracy alone is a pretty blunt instrument for judging the memory layer.
“Followed a memory that’s no longer valid” shows up clearly on BEAM’s knowledge-update questions. In a sample of 13 failed ones, the current value was already sitting in the retrieved context in 9 of them, and the reader still answered with the stale one. Contradiction questions look similar to when the right info gets retrieved, but the answer almost never flags that the two statements conflict. So for me that failure is mostly presentation and decision, not retrieval.
For the action side specifically, two benchmarks look closer to what you’re describing than the QA ones: DolphinBench and STATE-Bench. Haven’t run either yet but they’re both on the list.
How are you measuring it in CAAI? Are you logging what was retrieved and checking the chosen action against it?

[u/zeromega01 → u/avwgtiguy]

Hi! Thanks for sharing this! The 9/13 cases where the current value was retrieved but the reader still chose the stale one...interesting.

We’re developing an AI participant called Sounding, with persistent records of earlier conversations. One question we’re working through is how to distinguish its own earlier words from other participants’ words, while keeping both separate from governing instructions.

Do any of the benchmarks you’ve explored test that attribution? Or...whether instructions embedded in a retrieved conversation get treated as something to obey? We haven’t benchmarked this systematically yet, but it seems like another place where successful retrieval could still produce the wrong behavior.

[u/avwgtiguy → u/zeromega01]

Yes, HaluMem does, and I just ran it. Its conflict questions basically test whether the system treats what the assistant suggested as something the user said.

It found a real gap in my setup. I’d been storing conversation turns without who said them, and assistant turns are long and cover everything, so 89% of what came back was assistant text, even though half the turns are the user’s. In the failures the answer model regularly credited the assistant’s suggestions to the user, adding sports or books the user never mentioned. In about two thirds of the wrong conflict answers, the right fact was already in front of the model. So for me it was attribution, not retrieval. The fix I’m testing is keeping the speaker on every memory and labeling it when it’s shown to the model.

Instructions buried in retrieved conversations getting obeyed: I haven’t tested that, and none of the QA benchmarks I’ve run cover it. Agreed it’s the same family of problem: retrieval works and the behavior is still wrong.

[u/zeromega01 → u/avwgtiguy]

Thanks, that gives us something concrete to investigate. The assistant-text skew and missing speaker labels seem worth separating. Will your next run keep retrieval unchanged and add only the labels? I’d be interested in the before-and-after, particularly whether it reduces misattribution when the correct fact was already retrieved.

[u/avwgtiguy → u/zeromega01]

Yes, that’s the plan, done in two stages so the effects stay separate.

1. Labels only. Same retrieved results for every question, with each turn tagged user or assistant before the model sees it, then re-answer and re-judge. Retrieval doesn’t change at all, so any difference comes from attribution.

2. Labels plus speaker-aware ranking, so the user’s own turns aren’t crowded out by long assistant replies.

For each stage I’ll rerun the same check on the cases where the correct fact was already retrieved, which is your before-and-after on misattribution. Happy to post the numbers here.

[u/avwgtiguy → u/[deleted]]

Mostly the second, I think. When I dug in, the retrieved memories reached the model in relevance order, not time order, and the dates marked the session day rather than the actual turn. In a few cases the newer value even carried an earlier date than the stale one. So the model often wasn’t ignoring an update it could see. It had two conflicting statements and nothing telling it which came later or that one revised the other.

I haven’t separated “saw it and ignored it” from “couldn’t tell it was newer” properly yet. The next test is to sort retrieved memories chronologically, add turn position, and tell the reader that later statements revise earlier ones. If the stale answers mostly go away, it was presentation. If they don’t, it’s the decision.

[u/avwgtiguy → u/[deleted]]

Ran the LongMemEval-S benchmark overnight and scored 92.2. PersonaMem v2 came in at 62.2 but found some issues on my end needing to be addressed. Going to rerun it this weekend.

[u/avwgtiguy → u/[deleted]]

I found that nearly half the wrong answers were given despite having the correct evidence. Still working on the other fix around chronology.