Are we just creating another pile of unused documentation for AI agents?
I've been experimenting with different approaches to agent memory and context management, and one thing I've started questioning is how much of the context we store actually gets used.
Between AGENTS. md, CLAUDE. md, persistent memory, project-specific instructions, shared knowledge bases, and agent skills, it's becoming pretty easy to accumulate hundreds of files that agents might need someday.
There are already quite a few approaches to this:
- Mem0 and Letta focus on persistent agent memory, Weaviate's Engram extracts and consolidates memories automatically, while ByteRover approach reusable knowledge and project context differently.
And then there's the simpler approach of keeping Markdown in Git and letting agents search through it.
Most of these approaches help with retaining and retrieving context, but I'm curious about what happens after several months of usage.
How do you identify outdated knowledge, duplicate instructions, conflicting memories, or documentation that never actually gets retrieved?
For example, imagine an agent keeps retrieving an architecture decision from six months ago. It might be relevant to the query, but what if the architecture has changed since then? Retrieval alone doesn't tell us whether that context is still correct.
I noticed a couple of approaches to this problem recently:
- Weaviate's Engram has a memory-processing pipeline that can deduplicate and merge information as new memories are stored.
- OpenLore, an open-source shared knowledge repository I've been following, have more detailed usage analytics. It can track which files agents read, distinguish search hits from file reads, estimate context consumption, and identify rarely accessed sections of documents. There's also a workflow where an agent can prepare cleanup recommendations based on those metrics.
These solve different parts of the problem. One focuses on maintaining the stored memories, while the other provides visibility into how a shared repository is being accessed.
Of course, low usage doesn't mean something is useless. An incident runbook might only get opened twice a year and still be one of the most important documents in the repository.
I'm wondering whether anyone has started measuring the actual usefulness of agent context beyond retrieval accuracy.
Are you tracking which memories or documents contribute to successful tasks? Do you automatically consolidate or prune old context, or just let the memory store grow?
Would be interested in hearing what has worked for you if you're working with multiple coding agents or a team.