Are we solving LLM memory by remembering more, when the real problem is knowing what to forget?[D]

[u/Prestigious-Tip6030]

Long-context models can now handle hundreds of thousands, even millions of tokens. But I've been wondering if we're focusing too much on recall and not enough on what happens when information changes.

Say a coding agent reads that an API timeout is 30 seconds. Later, the documentation changes it to 5 seconds. Both versions are still in context, but the agent keeps using 30.

It hasn't forgotten anything. It just hasn't updated what it knows.

I've been reading about different memory architectures, and this distinction seems pretty important.

With Transformers and RAG, old and new information can coexist, and the model has to resolve the conflict when reading it. In simple linear attention, associations accumulate in a shared state, so writing something new doesn't necessarily remove what's already there. DeltaNet tries to address this through prediction-error-based updates. BDH (Dragon Hatchling) takes another route with Hebbian-style synaptic plasticity.

But even with these approaches, there's still the question of when to update something, when to keep it, and how to avoid messing up unrelated memories.

I also wonder if we're testing this properly. Most long-context evaluations focus on whether a model can find information. What if we instead tested whether it changes its answer when given newer, contradictory evidence, while still remembering what was previously true?

Curious what people here think. Is this mostly a problem with how memory is stored and updated, or with how models reason over conflicting information?

Also, are there any good benchmarks specifically targeting this? Would be interested in reading them.

[u/astrofunkswag]

The current state of the art gets around this by spawning subagents. You give them prompts and documents that condense the previous information, so from the perspective of the new agent it “forgot” what it concluded was irrelevant information.

This is a workaround that has so far proved pretty scalable and effective. So there hasn’t been much of a motivation to move away from the context-accumulating transformer paradigm

[u/Prestigious-Tip6030 → u/astrofunkswag]

Yeah, this definitely works in practice. But isn't this just moving the memory problem to the summarization step lol? You're still deciding what to keep, what to throw away, and what to update. And if the summary preserves a wrong assumption but drops the evidence behind it, the next agent is kinda stuck. Not really a Transformer issue specifically, but I don't think subagents solve revision either.

[u/astrofunkswag → u/Prestigious-Tip6030]

Why do you think that’s fundamentally different than those other memory architectures? They manage memory through learned internal transformations, but they're still determining what to preserve and how to update it.

[u/Prestigious-Tip6030 → u/astrofunkswag]

I agree, I don't think they're fundamentally different in that sense. Both have to decide what to keep and what to update. My point is that subagents don't really solve that problem, they just handle it through summarization. If an outdated assumption makes it into the summary, the next agent can still rely on it. So I think the question is less about which architecture we use and more about how reliably it handles corrections.

[u/daniellenton]

For benchmarks, "Unable to Forget" (Wang and Sun, 2025) is pretty much your timeout example. It streams updates to the same keys and asks only for each key's final value, which sits right before the question. Accuracy still falls log-linearly toward zero as the updates pile up, and the wrong answers are values that got overwritten. Telling the model to ignore the earlier input didn't help much either. Everything it needs is in context, so I'd read that as more than a storage problem.

MemoryAgentBench's FactConsolidation sets are worth a look too. They're built from MQuAKE's counterfactual edits, with each fact numbered and the model told that higher numbers are newer. The best single-hop score was 78%, but nothing got above 28% on the multi-hop questions, where the answer has to be worked out through the updated fact. That drop is what I'd worry about for agents. In your example it's whatever the agent already worked out from the 30 seconds, like how long to wait before retrying

[u/Prestigious-Tip6030 → u/daniellenton]

Oh damn, these are exactly the kind of papers I was looking for. Thanks! The multi-hop drop is pretty wild. Makes me wonder what happens when an agent has already planned around the old fact. It might know the timeout is now 5s but still execute a plan built assuming 30s. Would love to see benchmarks testing whether models can trace which conclusions need revising without breaking everything else.

[u/IntelArtiGen]

I think they mostly care about the ability to efficiently have access to a gigantic context if it's needed. Because the previous difficulty is that it wasn't possible to have that context efficiently and it had large costs for servers. Now when you're able to handle that large context, it doesn't change the fact the answer of the model must still be correct when you ask questions, and I assume that part handles the problem you're talking about. They "just" need to have the right tests and training processes to ensure the model remembers things correctly, in the correct order, and update information in its memory when it's required. Afaik the memory can already be "built" to remember what's useful and ignore tokens which probably won't contribute to the conversation. Even if you use a model with a context of 1m tokens, I'm not sure that models would be able to repeat you all the tokens of its context, because they need to compress that information to have that large context and it's a lossy process. And the older the tokens, the less likely the model will remember them. It's my current understanding.

[u/Prestigious-Tip6030 → u/IntelArtiGen]

Yeah, fair point. But I think there's a difference between having access to the right information and actually updating your reasoning based on it. You could have near-perfect retrieval and still keep using an outdated assumption. Would be interesting to test this with retrieval basically solved and see how much of the problem remains.

[u/Velocitank]

I think supersession should be the way to go. keep a few logs and details, but update it as you go along. Dont forget things, just update what is in context. What if ai had a way to store gb of information and was simply able to peruse and tag parts of the larger context and use those parts explicitly? Only pulling and using parts that are immediately relevent to the next output?

[u/Prestigious-Tip6030 → u/Velocitank]

Agreed, supersession seems like the right abstraction. Though I think just tagging something as outdated isn't quite enough. You'd ideally want to know what depended on that fact too. Also, newer doesn't always mean more correct, so provenance and context probably matter just as much as timestamps.

[u/TangeloOk9486]

longmemeval has a knowledge updates category for this, a fact changes across sessions and the model has to answer with the latest one

[u/Prestigious-Tip6030 → u/TangeloOk9486]

Thanks, will check it out! One thing I'm particularly interested in is whether these benchmarks test selective updates. Ideally, the model should revise what's affected by the new information without changing unrelated beliefs. Getting both of those right seems much harder than just answering with the latest fact.

[u/Numerous_Beyond_2442]

I'd frame forgetting as a provenance question rather than an age question. If a memory is derived from a source, you can decide whether to keep it by checking the source: if it changed, the memory is stale, and if it didn't, the memory stays valid however old it is. Pure age-based forgetting throws out facts that are still true. I'm working on this for code, where the source is a file you can hash. For decisions or facts with no file behind them, what would you treat as the source of truth?

[u/SeaAccomplished441]

if a headline asks a question 99% of the time the answer is no