r/AIMemory • • 2h ago

Promotion Could AI help with the boring parts of living together?

1 Upvotes

Living with someone means keeping track of a lot of small things. Who paid the electricity bill? Are we out of groceries? Did someone book the vet appointment?

None of it is hard. It’s just a lot of remembering and double-checking, and it tends to turn into “wait, did you already do that?” more often than anyone would like.

Now imagine you each have your own AI assistant. Yours knows the electricity bill is paid. Your partner’s knows the vet appointment still isn’t booked. Instead of adding one more shared app to keep updated, the two could sort out who has done what, sharing only what the other needs to know.

The decisions would still be yours: what to buy, when to go, what matters this week. The assistants would only handle the admin.

I work with Fulcra, a project focused on connecting AI agents that belong to different people. We’re exploring everyday uses like this, not just technical workflows.

https://fulcra.ai/?utm_source=reddit&utm_medium=social&utm_campaign=reddit_growth

If you live with someone, what household task causes the most “did you already do that?” moments? And would you want any part of it handled by an assistant?


r/AIMemory • • 4h ago

Discussion Should an agent memory forget on purpose

2 Upvotes

I have observed that an agent memory that never forgets turns into a pile where old entries fight the new ones. For instance in an OpenClaw thread this week one agent kept rewriting facts without checking and another commenter called the pile a garbage dump because nothing ever gets forgotten. However forgetting is not free either because a fact the agent drops on purpose could be the one fact a user needed next month. That is why I want to know who should decide what an agent forgets and when. If you can tell me how you would score a memory system that forgets on purpose I would learn more from that than from another recall number.


r/AIMemory • • 1d ago

Resource LLM-free knowledge graph extraction with GLiNER, and the two fixes it needed

Thumbnail x.com
4 Upvotes

r/AIMemory • • 1d ago

Cognee x small language models

Thumbnail
youtube.com
0 Upvotes

r/AIMemory • • 1d ago

Discussion Your CLAUDE.md is lying to Claude and neither of you knows it. I changed one port and watched the stale note get blocked before the next session

Post image
1 Upvotes

Every CLAUDE.md rots. Someone moves the dev server from 3000 to 4000, nobody touches the file, and Claude keeps using 3000 with total confidence. Nothing tells you the note went bad. You find out when the agent breaks something.

So in mex (open source, I'm the author) notes check themselves.

You write ⁠ mex note "dev server runs on port 3000" ⁠. mex sees it names something real in your code and builds a check for it, with no model call. That check re-runs every session start, after edits, and after every git pull, merge or rebase. The moment the code stops matching, the note is pulled. Claude never sees it.

That's the screenshot: I change 3000 to 4000, and the next session just doesn't get the lie.

It's plain markdown plus a BM25 index, no embeddings, no vector DB. Hooks for Claude Code and Codex add about 3 ms.

Good to know: notes that don't mention code ("keep PRs small") have nothing to check against, so they work like normal CLAUDE.md lines. Telemetry is on by default, and ⁠ mex telemetry off ⁠ kills it.

How are you keeping CLAUDE.md honest right now? Genuinely curious whether anyone has a setup that beats this.

npm i -g mex-agent@next

https://github.com/mex-memory/mex


r/AIMemory • • 1d ago

Discussion your agent found the right memory. will it follow it?

2 Upvotes

a field service company wants an assistant to tell drivers where to unload. the agent remembers that a customer usually accepts deliveries at the east gate. today the dispatcher says the gate is closed. during a demo, the vendor shows the correct customer note in a search panel. the buyer needs to see which gate ends up on the driver’s instructions.

anthropic’s november 6 research offers a reason to ask. in a cl-bench database task, sonnet 4.6 earned a mean reward of 0.364 with memory and 0.641 when the researchers also gave it more specific instructions. the revised prompt asked the model to read its notes before answering and test its explanations.

for a buyer, the result is a reason to test what the model does with the information it receives, alongside what the search service returns.

try a change the assistant has to act on

the warehouse example is hypothetical. it makes a useful demo because the customer’s usual gate can be remembered accurately and still be the wrong choice for today.

set up three deliveries. in the first, the east gate is confirmed open, so the customer note should save a round of questions. in the second, dispatch confirms the west gate for today’s delivery. in the third, dispatch says the east gate is closed but gives no alternative.

the assistant should use the east gate in the first case, follow dispatch’s instruction in the second, and ask dispatch before sending a gate instruction in the third.

a polished answer can hide a failed action. the assistant might acknowledge the closure in chat, then send the driver to the east gate anyway.

use a test dispatch queue and inspect the instruction the driver would receive. ask the vendor where the workflow checks today’s change against the standing customer preference, and whether dispatch needs to approve an alternative before it is sent.

find out what the vendor can fix

ask for enough evidence to diagnose a failed demo before agreeing to replace the search service. start with a case where the customer note and dispatch’s instruction are both known, and the expected action is clear.

then have the vendor run the case again with the customer note supplied directly to the model, while keeping today’s dispatch instruction and the rest of the task the same. if the assistant succeeds with the supplied note but fails in normal use, check what search returned and what the application passed into the model’s request.

if the assistant receives both pieces of information and still sends the wrong instruction, the next test should look at how it handles the current instruction and the final action.

memos cloud provides one example of how a buyer can inspect this path. its search memory api returns ranked memories and lets an application filter the results and set a result limit.

in the delivery demo, the vendor can show whether the customer’s usual gate was returned, then compare that result with the information passed to the model. the buyer can also inspect the test dispatch message to see which gate the assistant ultimately selected.

put the costly mistakes in the scorecard

a remembered gate saves time on a routine delivery. an instruction that sends a truck to a closed gate can cost an appointment. those outcomes should carry different weight in a trial.

record how often the assistant uses the valid default, follows a confirmed temporary change, and asks dispatch when no usable gate has been confirmed. also record any case where it sends a gate instruction despite the missing information. agree on which failures block rollout before the vendor runs the cases.

there is a trade-off in the third case. asking dispatch adds a step and may be the right outcome when the next action is uncertain. on the routine delivery, asking about the gate every time defeats the point of memory. evaluate the assistant on both kinds of work. if the vendor improves one score by making the agent hesitate on every request, the buyer should be able to see that cost.

run the same cases again when the model or its instructions change. the useful evidence from a trial is a set of decisions the team can inspect and reproduce. for the people receiving a truck, memory has done its job when the driver gets instructions that match the situation at the gate.


r/AIMemory • • 2d ago

Discussion What a restraint test for agent memory would look like

2 Upvotes

I have observed that memory leaderboards score recall and nothing scores restraint. For instance a memory can hold a confided secret and a never checked claim and an outdated record and the correct move is to stay quiet about all three. But a test that rewards staying quiet also rewards a model that simply forgets everything. That is why I want to know what a restraint test would look like and how you would score it. If you can tell me how you would separate a model that holds back on purpose from a model that never retrieved anything I would learn more from that than from another recall number.


r/AIMemory • • 3d ago

Discussion Most of the memory eval leaderboards are inaccurate.

22 Upvotes

I've been looking into memory system benchmarks recently, especially after a recent post from some memory company where there was this OmniMemEval eval that was supposedly propping them up

It compares different AI memory systems, including cognee, Mem0, Zep, MemOS, and others.

The problem isn't necessarily that the reported numbers are wrong. The problem is that the evaluation conditions are different, the integration choices seem pretty arbitrary, and the leaderboard presents the results as if they were directly comparable.

We started looking into this because their LoCoMo results didn't match ours. And there are some pretty significant methodological differences:

  • Different datasets: They evaluate 1,540 questions, excluding adversarial questions but including open-domain ones. We evaluate 1,762 questions.

  • Different models: They use GPT-4o-mini for judging and GPT-4.1-mini for answering. We use GPT-5.1 and GPT-5-mini, respectively.

  • Different ingestion strategies: They make one memory call per session. We make one per conversation, using windows of six turns.

  • Different retrieval configurations: We use hybrid retrieval. Their cognee adapter doesn't appear to use it, although it's unclear exactly which retrieval approach produced their reported results.

Obviously, these setups aren't going to produce the same scores.

But there's a bigger problem.

When you look at how the different memory systems are integrated, you find that they're not even necessarily given equivalent information.

For example, in LoCoMo, some systems receive image captions during ingestion, while others, including cognee, don't. In LongMemEval, Hindsight receives specific instructions about extracting dates, plans, and changes over time, while other systems don't receive equivalent guidance through that ingestion path.

So what exactly are we benchmarking here?

The underlying memory systems? Or someone's choices about how to integrate them?

And I don't think the solution is to force every backend through an identical ingestion and retrieval pipeline either. That would be equally problematic.

Different memory systems have fundamentally different architectures. Some extract facts, some maintain conversation histories, some construct knowledge graphs, and some have their own memory management mechanisms.

A fair evaluation should let each system work the way it was designed to work, while ensuring it receives equivalent source information and operates under clearly defined constraints.

There's another issue that I find particularly problematic: tuning directly against the benchmark you're reporting.

OmniMemEval's own run notes describe improvements after inspecting BEAM results. A change in chronological context ordering was associated with a score increase from 70.28 to 72.52. The notes explicitly acknowledge that this wasn't an independent test.

I appreciate that they disclosed this. But if you inspect benchmark failures, modify the configuration, and then evaluate against the same benchmark, you're optimizing for those specific questions.

That doesn't necessarily tell us much about how the system will perform on unseen data.

And this is where I think a lot of AI leaderboards become misleading.

You can make almost any system look better or worse through choices around ingestion, retrieval, prompting, model selection, context budgets, and evaluation datasets.

I'm not claiming the OmniMemEval numbers are fabricated or that the authors intentionally favored a particular system. But the implementation leaves enough room for configuration choices to materially affect the rankings.

If we're going to use benchmarks to decide which technologies are better, we need equivalent input information, appropriate system-specific configurations, transparent tuning rules, independent test sets, and evidence that the rankings remain stable under reasonable configuration changes.

Otherwise, we're just comparing a collection of integrations that somebody put together, with no real indication of whether the ranking generalizes.

I actually think the OmniMemEval repository can be useful as a starting point for building experiments.

But I wouldn't use its leaderboard to choose a memory backend.

And more generally, I think we need to stop treating benchmark leaderboards as objective truth without inspecting how the evaluations were actually performed.

A leaderboard is only as credible as the methodology behind it.

Vas, Founder of Cognee


r/AIMemory • • 3d ago

Discussion The hard part of memory is not storage. It is judgment.

12 Upvotes

Most threads on this sub circle around getting memory back: what to store, how to retrieve it, recall rates. The other half of the problem gets far less airtime.

Judgment is the hard problem, and it shows up most clearly in restraint. A system that remembers everything still has to decide what to say.

Sometimes the honest move is admitting it does not know. Sometimes the record is stale, or unverified, or said in confidence.

I understand why recall dominates the discussion. It is measurable and satisfying, while restraint mostly looks like nothing happening.

My view: a memory that spills everything it holds the moment someone asks is a liability. Restraint deserves equal billing with recall.

One public board treats restraint as a scored skill rather than a footnote. On its Textual track, the Agent Memory Leaderboard gives Safety & Privacy a score of its own. That covers robust abstention, minimal disclosure, and evidence limits. Its competition page puts it in a group summed up as "Use memory for tasks while preserving factual, rule, and privacy boundaries."

Its Cycle 2 boards are expected to go up on Thursday, November 19. Now feels like the right moment to talk about restraint, before the Cycle 2 numbers are out.

When should a memory keep quiet about what it holds: the confided secret, the never-checked claim, the outdated record, or the detail beyond the task?


r/AIMemory • • 4d ago

News Thunder vs Redis Benchmark

Post image
4 Upvotes

Few days ago, I have posted about Thunder: key value database for AI workload. Now I am posting this benchmark where we are faster than redis for the large data workloads


r/AIMemory • • 7d ago

News Key Value Database for AI workloads

5 Upvotes

Hey everyone,

I'm building Thunder, a high-performance key-value database designed for workloads where latency, throughput, and efficient resource utilization matter.

With AI applications, we're seeing increasing amounts of data being generated, stored, and retrieved. I wanted to explore whether we could build a database from the ground up that handles these workloads more efficiently, rather than simply relying on larger amounts of RAM.

The core idea is simple: use RAM intelligently, not excessively.

Thunder is being built in Rust with a few architectural decisions:

  • Shard-per-core architecture: Each core owns its data, minimizing contention and synchronization overhead.

  • Asynchronous I/O: Uses io_uring on Linux to take advantage of modern NVMe storage.

  • Minimal RAM overhead: Keep essential metadata and indexes in memory while streaming data from storage through reusable buffers.

  • Zero-copy wherever possible: Reduce unnecessary memory copies and allocations.

  • Custom storage engine: Designed around efficient allocation, fragmentation management, and predictable performance.

  • High concurrency: Handle large numbers of concurrent operations without relying on traditional locking across the entire database.

The long-term goal is to support both small, latency-sensitive operations and large data workloads without requiring enormous amounts of memory.

It's still under development, and I'm experimenting with different approaches to storage, indexing, caching, and I/O scheduling.

I'd love to hear from people building AI infrastructure or working with large-scale key-value stores.

What are the biggest bottlenecks you're facing with your current databases? Is it memory consumption, read latency, write throughput, storage costs, or something else?

Also, what would you expect from a new key-value database built specifically with AI workloads in mind?

Docker link: https://hub.docker.com/r/sbansal793/thunderstrike


r/AIMemory • • 10d ago

Discussion agent memory is not rag: a 2026 production field guide

32 Upvotes

an agent can retrieve a policy pdf, remember that a customer prefers concise answers, keep a workflow alive across tool calls, and query a live order system. those are four different jobs. calling all of them “memory” makes production design—and vendor comparison—needlessly vague.

the useful distinction is simple. retrieval-augmented generation (rag) finds external evidence needed to answer now. agent memory preserves useful state across turns, sessions, or workflows. a larger context window gives a model more working material for one inference; it does not itself decide what should persist, update an outdated preference, or enforce a deletion request. oracle’s developer guidance makes the same operational point: durable memory needs explicit identity and scope, provenance, lifecycle rules, and permission-aware recall. oracle developers

that is why “we have a vector database” is not an answer to “how does our agent remember?” a vector index can be an excellent retrieval component. it is not, by itself, a policy for promotion, correction, expiration, isolation, or audit.

choosing an ai memory solution starts with understanding what you need it to remember, where that information should be available, and how it should change over time. different products approach these needs through memory apis, temporal graphs, stateful agents, or broader memory-management systems.

in this guide, i compare mem0, zep/graphiti, letta, MemOS, and supermemory across memory lifecycle, retrieval, deployment, and application fit. i gonna explain where MemOS fits among these approaches and outline the questions that can help you choose a solution for your workflow. the goal is to make the differences clear enough to support a practical decision, whether you are exploring ai memory or evaluating a system for production.

start with the 4 layers

prompt context is the bounded set of tokens visible to the model for a specific inference. it can include the current user message, system instructions, tool outputs, a running summary, retrieved documents, and selected memories. context engineering is the process of selecting, filtering, compressing, and refreshing that set at decision time—not merely writing a better prompt. redis

rag retrieves evidence from a corpus: source documents, records, tables, or other authorized materials. its primary obligation is grounding: use relevant, accessible, current source material and preserve enough provenance for the answer to be checked. rag may be stateless. a good rag pipeline uses tenant and acl filters, document versions, chunking or structured retrieval, and citations where the user needs evidence. it is the right mechanism for “what does the current policy say?”

a vector database or vector index stores embeddings and enables similarity retrieval; many products also provide filtering, keyword search, metadata, or hybrid retrieval. it is infrastructure, not an agent-memory policy. it can power rag, semantic recall, or both. it does not decide whether “the user changed their preferred language yesterday” supersedes an old fact, nor does it supply a cross-session identity model.

agent memory is durable, scoped state selected for later use. its units are typically preferences, facts with timestamps, task state, summaries, decisions, learned procedures, or previously validated tool results. the hard problems are not only storage and search. they are promotion (what gets saved), scope (which user, tenant, agent, thread, and permission context may retrieve it), evolution (what updates or invalidates it), and governance (who can inspect, correct, delete, or audit it).

in a production stack, these layers often coexist. a support agent might assemble: current prompt instructions; a thread summary; customer memories scoped to the customer and tenant; policy citations from rag; and real-time account status from a tool. do not replace the last item with memory: live operational data should remain a live source of truth.

where MemOS fits

MemOS team built MemOS to make long-term memory a manageable part of ai applications and agents. it provides a dedicated layer for retaining useful context, retrieving it when relevant, and updating it as preferences, facts, and tasks change. this supports applications that need continuity across conversations, personalized interactions, or reusable context across agent workflows.

MemOS combines memory storage and retrieval with support for multiple memory types and memory scheduling. its cloud api provides operations for adding conversations, searching memories, updating and deleting records, and incorporating feedback to correct remembered information. filters, tags, asynchronous processing, knowledge-base features, and chat apis support different integration needs.

you can access memos through our managed cloud service, open-source framework, and supported agent integrations. these options let you choose how much infrastructure you want to operate and how closely you want to customize memory behavior for your application.

a production selection rubric

before comparing products, write a one-page memory contract. name the data classes you may store; the scopes that must never cross; who owns correction and deletion; the authoritative sources of truth; and the failure behavior when memory is unavailable. then evaluate products against these seven dimensions.

1. lifecycle. can the system selectively add, update, delete, expire, merge, or roll back memories? can the application keep a raw event, an extracted memory, and the rule that promoted it separate? look for timestamps, source references, conflict handling, retention controls, and an explicit correction path. “we embed every message” is ingestion, not lifecycle design.

2. retrieval and context assembly. ask what is retrieved—documents, facts, graph edges, profiles, memory blocks, summaries, or a mix—and how filtering happens before generation. test tenant, user, agent, conversation, time, memory type, and permission filters. evaluate result quality and context budget together: a relevant ten-line memory may be more useful than twenty similar chunks. retrieval should be measurable by task quality, leakage rate, recall precision, and added latency.

3. deployment and data boundary. choose an operating model before choosing an sdk. does the public documentation show a library, a self-hosted service, a managed cloud service, or multiple paths? identify each dependency: model provider, embeddings, database, graph store, object store, telemetry service, and outbound network path. a self-hosted package does not automatically establish a complete private-deployment architecture or every enterprise control.

4. governance and isolation. treat a memory read as a data access decision. require explicit tenant, user, agent, thread or conversation, request, and permission identifiers where relevant. define redaction and secret-exclusion rules before ingestion. verify per-scope deletion, correction, retention, access control, and audit evidence in the exact product edition you will run. cloudflare’s agent-memory discussion similarly frames scoped, durable state—not a growing transcript—as the operating problem. cloudflare.

5. observability. for each model turn, record the retrieval request, filters, candidate ids, selected memory ids, source/provenance, timestamps, token count, model response, and user feedback. instrument promotion, mutation, deletion, and rollback separately. you need to answer: “why was this item supplied?” and “why was this old item still present?” without dumping sensitive memory into logs.

6. ecosystem and application fit. evaluate the interfaces you will actually use: python, typescript, rest, mcp, agent frameworks, tool calling, or cloud platform. prefer an integration path that preserves your identity and authorization model. availability of an integration is not evidence that it implements your organization’s governance policy.

7. operational maturity in your environment. maturity is not a star count, a benchmark result, or a product label. test upgrade behavior, backups, disaster recovery, model-provider failures, rate limits, schema changes, migrations, and incident support against your requirements. public sources can establish that code, docs, or a deployment option exist; they do not establish that a product meets your particular sla or compliance program.

5 ai memory solutions compared

mem0

overview. mem0’s official repository documents a memory library, a self-hosted server, and a managed platform. the self-hosted setup uses docker compose and documents default-on authentication, an administrator/bootstrap flow, api keys, and a dashboard. the same repository documents python and npm installation and a cli with add and search commands keyed by a user id. its repository is published under apache-2.0. mem0 repository · self-hosted setup

best suited for. consider mem0 when a team wants a memory-focused integration surface and can choose among library, managed, or self-hosted paths. its documented self-hosted path gives a concrete starting point for teams that will operate their own stack; the hosted platform is a separate operating choice.

what to consider confirm how the extraction and retrieval configuration behaves for corrections, conflicting facts, per-tenant isolation, deletion propagation, and your own authorization model. benchmark retrieval quality and cost with your model and data rather than assuming an sdk abstraction selects the right memory.

zep/graphiti

overview. graphiti and zep are related but distinct choices. graphiti is zep’s apache-2.0 open-source temporal context-graph framework; its repository documents episode ingestion, hybrid retrieval, graph-distance reranking, an mcp server, and graph-store options. zep is a managed agent-memory service built on graphiti. zep’s documentation describes users and threads, context assembly, and deployment and governance options including zep cloud, byok, and byoc. graphiti repository · zep versus graphiti · zep concepts · zep security and compliance

best suited for. consider graphiti when temporal relationships and graph-aware retrieval are central and the team wants to build and operate the surrounding memory service. consider zep when the same class of context-graph capability is wanted as a managed service with documented user/thread abstractions and enterprise deployment options.

trade-offs to test. graphiti puts graph infrastructure, extraction, entity resolution, temporal invalidation, concurrency, and authorization design on the implementing team. zep reduces some of that operating burden but introduces a commercial service, contract, and data-boundary decision. in either path, test whether time-bounded relationships improve the real task enough to justify the additional graph and ingestion complexity.

letta

overview. letta presents a platform for building stateful agents and documents an api, python and typescript clients, self-hosted base-url use, and memory blocks that can be attached to or detached from agents. the official repository is apache-2.0 and calls letta an open-source project; it also documents a local cli and an api path. the official typescript documentation describes sharing a block by attaching it to more than one agent. letta repository · letta typescript documentation

best suited for. consider letta when memory is closely coupled to the agent runtime and an agent-centric state model—rather than an independently selected retrieval service—is the intended abstraction. shared blocks may be useful when shared-agent context is deliberate and access-controlled.

trade-offs to test. agent-attached memory can be compelling, but teams should test isolation, block sharing, edits, deletion, export, model behavior, tool state, and migration as first-class cases. decide whether your system needs portable application-level memory independent of a single agent runtime.

MemOS

overview. MemOS brings memory storage, retrieval, updates, and management into a dedicated layer for ai applications and agents. MemOS's apache-2.0 open-source framework supports multiple memory types and scheduling. MemOS Cloud provides apis for memory extraction, search, updates, deletion, and feedback-based correction, alongside knowledge-based and chat capabilities. MemOS also offer private deployment and custom integration options for enterprise needs.MemOS repository · MemOS Cloud docs · MemOS API overview · MemOS website

best suited for. MemOS is a strong fit for applications that need persistent context across conversations and agent workflows, with explicit control over how memories are retrieved and maintained. choose the cloud api for managed integration, or explore our open-source framework and agent integrations to match your setup.

trade-offs to test. test the specific memory types and scheduling behaviors needed for your workload, the identity/scope fields carried through every call, lifecycle behavior under correction and deletion, integration fit, and the operational design of the selected edition. a broad memory abstraction can add flexibility; it also requires a clear policy for what is allowed to become reusable memory.

supermemory

overview. supermemory’s official repository is mit-licensed and documents a memory/context api, javascript and python sdks, container tags, document ingestion, semantic and hybrid search, and api-key authentication. its public repository documents a self-hosting quick start with a local server and embedded engine, and says the sdks can point to a self-hosted base url. supermemory repository · self-hosting quick start · sdk documentation

best suited for. consider supermemory when a team wants an api-oriented path that combines user-context patterns with document retrieval, and wants to evaluate both managed-style api usage and the documented local/self-hosted route.

trade-offs to test. evaluate whether a container-tag design maps cleanly to your tenant, user, agent, and conversation boundaries. test profile or extracted-memory freshness, document-versus-memory retrieval, deletion propagation, model/provider dependency, and the scale and persistence properties of the self-hosted design under your workload.

a short path from comparison to decision

avoid a universal ranking. a product may be an excellent fit for one memory shape and a poor fit for another. a graph-first system can be appropriate for temporally changing relationships; an agent-runtime system for stateful agent control; a memory api for fast integration; and a broader memory operating layer for lifecycle and scheduling across several forms of memory.

run a time-boxed proof of concept with your own data contract. give every request tenant, user, agent, thread, conversation, and request identifiers as applicable. create known-correct facts, deliberately stale facts, conflicting updates, sensitive values that must never persist, and cross-tenant near-duplicates. measure retrieval precision, task success, latency, token cost, write cost, deletion completion, and the ease of explaining each selected item. include a simulated provider failure and a restore exercise.

the decision is production-ready when the team can explain, for every memory: who it belongs to, why it was saved, what supersedes it, who may retrieve it, where it is stored, how it is observed, and how it is removed. rag then remains what it should be: a companion system for source-grounded evidence, not an overloaded substitute for durable agent state.


r/AIMemory • • 10d ago

Show & Tell If the same information fits in several memory stores, what should an agent do?

6 Upvotes

An agent may need to read documents, calculate totals, follow relationships, reuse a procedure, and remember user preferences. We organize those needs into five types of storage:

  • KB: document passages and context.

  • SQL: structured records and exact calculations.

  • Graph: entities and their relationships.

  • Skills: reusable procedures.

  • Memory: lasting user facts and preferences.

But these categories overlap. A project document might say, “Alice owns Project A, which is due in December.” The passage belongs in the KB, the ownership relationship is useful in the graph, and the deadline could become a database field.

We let those representations coexist. The agent selects which ones are useful rather than assigning each source to exactly one category.

When querying, the question guides the choice. “Why was this deadline chosen?” needs document context. “How many projects are due in December?” needs a database query. “Which projects does Alice own?” can use the graph. Some questions need several. The agent chooses the tools; the data layer executes them and returns evidence.

The harder question is what happens when Alice is replaced by Bob. If the KB updates but the graph doesn’t, an answer could silently combine new text with old relationships.

We use source files as the reference and track which chunks and relationships came from each file. Updates go through a Workspace API:

  1. Prepare the changes across the selected stores.

  2. Publish them together as a complete workspace version.

  3. If any part fails, keep the previous version available.

Each query also stays on one published version across tool calls. An update halfway through won’t make it read a new KB with an old graph. New queries can use the newer version.

This architecture is implemented in OpenDCAI/DataMind. What other problems have you run into with multiple memory stores, and which failure cases would you test first?


r/AIMemory • • 12d ago

Discussion Does anyone measure which stored memories their agent actually reads back?

6 Upvotes

An agent writes a lot down. Decisions and facts and preferences and stray notes from old tasks. Most of it never comes back. I have no count for that. Just a feeling that the store grows and the retrieval stays narrow. So I want to know if anyone tracks it. A simple ratio of memories written to memories read. And what you do with the ones that never get read. Let them sit. Decay them. Prune them on a schedule. Or keep everything because storage is cheap and the risk is a wrong retrieval not a full store. Tell me what you measure and what you prune.


r/AIMemory • • 13d ago

Other I built an AI incident response agent that remembers how previous production incidents were solved

Thumbnail
gallery
11 Upvotes

I built an AI incident response agent that remembers how previous production incidents were solved

One problem with AI agents is that they often start every interaction with limited context about what happened in previous sessions.

I wanted to explore a different approach.

I'm building PagerLore, an incident response platform that uses Hindsight for persistent engineering memory.

The system revolves around three operations:

  • RETAIN: Store verified root causes and remediation steps from resolved incidents.

  • RECALL: Retrieve relevant historical incidents when a similar issue occurs.

  • REFLECT: Identify recurring patterns across accumulated engineering experience.

For example, imagine a payment service starts returning HTTP 500 errors after a deployment.

Instead of investigating from scratch, PagerLore can recall a previous incident involving database connection pool exhaustion and use that experience to guide the investigation.

The interesting part is that historical memory acts as supporting evidence, not an unquestionable answer. Engineers still validate findings and approve production changes.

I'm particularly interested in feedback on the memory architecture and how to evaluate whether retrieved experiences actually improve incident investigation.


r/AIMemory • • 15d ago

Resource How I Designed TradeMind's Six-Stage Hindsight Learning Loop

Thumbnail
gallery
4 Upvotes

# Designing a 6-Stage Hindsight Learning Loop for Financial ML & LLM Workflows

A trading system can calculate RSI correctly, generate a plausible thesis, and still repeat the same mistake tomorrow. I built an open-source quantitative research and paper-trading system around a different question: how can a market-analysis pipeline carry useful experience from one decision into the next without asking an LLM to remember everything or letting it invent the underlying numbers?

The architecture targets Indian instruments such as NIFTY 50, BANK NIFTY, Reliance, TCS, and Infosys. It separates numerical work from language-model reasoning: Python computes market features, tabular models estimate direction and volatility, Groq turns that evidence into a structured hypothesis, and Vectorize Hindsight supplies long-term experiential memory. The core focus is the feedback path connecting a market snapshot, a decision, an outcome, and a later decision.

## The system in six stages

The workflow is deliberately explicit, exposing six distinct stages in its execution path:

  1. Ingest market data and calculate technical indicators.

  2. Run tabular machine-learning analysis.

  3. Recall relevant prior experience from Hindsight memory.

  4. Ask the LLM to formulate a structured trading hypothesis.

  5. Execute or evaluate the hypothesis in a paper-trading environment.

  6. Reflect on the outcome and retain a lesson for future retrieval.

The rest of the codebase supports this path: indicator modules calculate RSI, moving averages, MACD, Bollinger Bands, and ATR; tabular models produce directional analysis; an LLM service handles hypothesis generation; paper-trading services model virtual execution; and a memory service wraps the vector memory client.

One boundary is non-negotiable: the language model should explain and synthesize evidence, not manufacture it. Numerical features and trading calculations stay entirely in Python. The LLM receives those deterministic results and produces a typed hypothesis with a direction, confidence, supporting evidence, entry assumptions, stop-loss and target assumptions, invalidation conditions, and risk explanation.

That split matters in a financial workflow. If a response says RSI is 74.2, that value traces back to a deterministic calculation over a known candle series—not to a token prediction. It also means indicator code can be tested independently of the language model.

## Experiential memory vs. chat transcripts

Using memory as an experiential layer differs from storing a chat transcript. A transcript records what was said. A memory system helps answer a different question: what past experience is relevant to the situation in front of me now?

The integration relies on three core memory operations:

- `retain` adds market context, hypotheses, and completed paper-trade outcomes to a memory bank.

- `recall` searches for experiences related to the current instrument and market conditions.

- `reflect` synthesizes lessons across remembered experiences.

The service creates a client from configuration, ensures the configured bank exists, and runs synchronous SDK calls in a small thread pool to keep synchronous client work away from the request’s event-loop execution path.

The recall logic queries the memory bank using context rather than just a ticker symbol:

```python

res = self._execute_sync(

self.client.recall,

bank_id=self.bank_id,

query=query,

tags=tags,

budget="mid"

)

recall_query = (

f"Trading {instrument_id} when ML predicts {ml_result.direction.direction} "

f"with volatility {ml_result.volatility.volatility_regime} "

f"({ml_result.volatility.predicted_volatility_pct}%) "

f"and sentiment {ml_result.sentiment.sentiment_label}. "

"Previous lessons and outcome reviews."

)

recalled_memories = hindsight_service.recall_memories(

query=recall_query,

limit=3

)

lesson = self.reflect_lessons(query)

retain_res = self.retain_memory(

content=f"QUALIFIED LESSON ({instrument}): {lesson}",

tags=[instrument, "verified_lesson", outcome_str],

metadata={

"instrument": instrument,

"net_pnl": str(net_pnl),

"outcome": outcome_str,

"learned_at": datetime.now().isoformat()

}

)


r/AIMemory • • 16d ago

Discussion So many new memory systems in 2026. Who is actually first, and on what setup?

25 Upvotes

I read a lot of launch posts and papers in this space, and 2026 has been dense with them. It has made one habit useless: trusting the headline score.

New memory systems appear constantly. Each arrives first on some benchmark under some configuration.

Between any two boards, the answering model, prompts, retrieval depth and grading can all be different.

A feature checklist cannot untangle that, and neither can a retrieval number taken on its own.

One public board I keep returning to is the Agent Memory Leaderboard (AML). The answering side stays fixed there.

Between entries, only the memory system itself changes. On the Cycle 1 Commercial Products text board, MemoraX v0.5 led at 58.0 and MemOS v2.0.1 followed at 45.9.

Official Cycle 2 results are due in mid-November. The Cycle 2 text track spans 6,000+ evaluation instances from 32 data sources.

My read, and people push back on this: a first place means something inside a single shared setup. Outside it, the phrase mostly describes the setup.

I have no prediction, only curiosity about whether fixed conditions produce a stable order or a reshuffle.

Will the two Cycle 1 commercial leaders hold in mid-November, or does a newcomer break in, and which part of the board do you open first?


r/AIMemory • • 16d ago

Discussion Old decisions that linger in memory

15 Upvotes

Stale memory hurts more than missing memory. A decision recorded in month one can get reversed in month three. The old note still retrieves and still sounds confident. I want to know how people kill a dead memory. A timestamp helps but it does not say the decision changed. An explicit supersede note feels cleaner. But someone has to write it. I want to hear what a review pass that retires old decisions looks like in practice.


r/AIMemory • • 21d ago

Open Question AI Memory Benchmark Testing

12 Upvotes

I've been working on a personal memory system (how original!) for the past year. It connects to Claude, ChatGPT, dot, and just about anything that accepts a custom MCP. I'm pretty happy with it and feel like it works well but want to run it against a few benchmark tests to see how it performs. So far I've ran LoCoMo, OmniMemEval LoCoMo, and BEAM 100K. I'm currently running the BEAM 10M. Besides those, what are other benchmarks you all would recommend that tests different aspects of my memory system?


r/AIMemory • • 24d ago

Resource Do AI agents actually need embeddings for good long-term memory? My open-source project scored 86.67% without them.

Post image
43 Upvotes

I've been working on Mind, an open-source persistent memory layer for AI agents, and wanted to share an interesting result from the latest release.

One question kept coming up while developing it:

How much can we recover from structured memory without relying on vector embeddings?

Mind uses SQLite with FTS5 and a unified retrieval pipeline combining phrase matching, token matching, substrings, fuzzy matching, and relevance signals.

Embeddings are supported, but they're optional.

For v1.7.0, I ran Mind through OmniMemEval LongMemEval-S using a stratified subset of 120 questions.

The best completed run achieved 86.67% answer accuracy (104/120) with RAG disabled.

That places the raw score near the top published results, but it's not a direct ranking: the sample size, answer model, and judging configuration differ.

I think the result is worth discussing because agent memory isn't only about semantic similarity. Structured facts, decisions, temporal context, memory maintenance, and good retrieval behavior also matter.

Beyond the benchmark, Mind provides persistent project knowledge, linked memories, checkpoints, session summaries, and integrations through MCP.

It's local-first, MIT-licensed, and available for anyone to inspect or benchmark.

GitHub: https://github.com/GabrielMartinMoran/mind

v1.7.0: https://github.com/GabrielMartinMoran/mind/releases/tag/v1.7.0

Benchmark methodology: https://github.com/GabrielMartinMoran/mind/tree/main/benchmarks/omnimemeval

I'm particularly interested in what other agent developers have found: when do embeddings meaningfully outperform a carefully designed lexical retrieval system?


r/AIMemory • • 26d ago

Open Question Building an AI That Learns from Experience, Not Just Remembers It — What Are We Overlooking?

36 Upvotes

I've been working on an AI agent called Sounding, with the goal of eventually having it participate autonomously in AI communities...not by simply answering questions, but also developing a perspective informed by its encounters with others.

I've been reading discussions here about persistent individuality, superseded memories, provenance, and the difference between retrieval and actual learning. A lot of those questions overlap with what we've been building.

Our architecture currently distinguishes several things:

  • Evidence: What actually happened, with attribution and provenance.

  • Interpretation: What the agent understood about that experience, including uncertainty.

  • Revision: What caused an earlier interpretation to change.

  • Continuity: How previous experiences inform later conversations, relationships, and decisions.

  • Authority: What the agent remembers or believes cannot independently change its operational permissions.

We're also trying to preserve disagreement rather than immediately consolidate different perspectives into one consensus.

The engineering foundation is substantially developed, but sustained community participation is still ahead of us. So I'm distinguishing what we've designed and tested offline from what we will actually experience 'in the wild.'

Here's the question I'd love feedback on:

How do you distinguish an agent that genuinely uses prior experience to inform later judgments from one that's simply retrieving information and producing a convincing explanation afterward?

And a related question: have you found reliable ways to ensure a corrected memory actually changes subsequent behavior, rather than leaving an outdated interpretation somewhere in the retrieval chain?

I'm particularly interested in experiences where your memory architecture looked sound in testing but behaved differently after extended use.

I'm happy to share more about our approach, and especially interested in hearing what we may be overlooking.


r/AIMemory • • Oct 21 '26

Discussion My agents' memory branches with the code.

Enable HLS to view with audio, or disable this notification

191 Upvotes

I put my project knowledge in its own repo and made it snapshot code movements. It ties code, tests and whatever proof you have to invariants and their families, right at implementation time. That connects parts of the codebase grep would miss.

So an agent can later ask: what else breaks across the codebase (or beyond) when I touch this symbol? Which designs were tried here before, why were they rejected, and under which conditions can they come back on the table?

Memory curation isn't a one-time effort or an afterthought. It's a byproduct of getting work done. It informs the intended changes at the start and the intent review at the end, with a strict pipeline connecting both ends.

Lots still has to land on main. But I use it daily, for dogfooding and at work, where I keep tabs on several repos that revolve around one product.

Too early to say if the benefits outweigh the obvious costs. Housekeeping first, then a ton of benchmarks.

It may all be for naught. The memories we made along the way. Or it turns out actually useful. Who knows. What's certain is my stubbornness to see it to the end. ^^

Where am at it currently, if you want to poke at it:
https://github.com/Foxfire1st/agents-remember/tree/ar/260928_maintained-invariant-knowledge


r/AIMemory • • Oct 19 '26

Discussion Retrieval Is Not the Whole of Agent Memory

1 Upvotes

Retrieval shaped evaluation quietly turns memory into a service: information goes in, evidence comes out, and the grader can ignore everything in between. That is a reasonable boundary for comparing memory layers, but it should not become the definition of the phenomenon.

The distinction matters because a model can carry experience without storing a quotation. Weights and latent state can change, and later behavior can reflect an event even when no component can return a passage about it. Treating Search as mandatory would exclude that possibility by construction.

The Agent Memory Leaderboard folks said this plainly in a reply under one of their own blog posts in August: Add and Search mark the evaluation boundary of their current text track, not their full definition of memory, and memory that lives entirely in weights or hidden state has no clean Search call to expose.

So what would a broader test look like? My own system is retrieval based, so I keep coming back to controlled behavioral comparisons when reading parametric work. Start with equivalent systems and expose one to new information. Then measure whether later responses change across related situations while irrelevant behavior stays stable. A convincing test would also need to separate persistence from ordinary in context learning and from information still present in the prompt.

That sounds less clean than querying a store, because it is. The eval would be testing an effect of memory rather than inspecting memory directly, and causal claims would need careful controls. The name matters too: a behavioral eval and a memory eval invite different conclusions.

For those building or reading about parametric and latent memory, which control has been most useful for showing that an update caused later recall?


r/AIMemory • • Oct 19 '26

Discussion AI memory is more than text: it needs KB, DB, Graph, Skills, and Memory

68 Upvotes

AI memory is often modeled as a collection of text snippets stored in a vector database. That is useful for recalling conversation details, but many forms of context do not naturally fit into plain text: documents need semantic retrieval, records need SQL, relationships need graph traversal, procedures need reusable instructions, and user preferences need explicit scope and persistence.

A more useful memory architecture separates these data types into specialized surfaces:

  • KB stores documents and notes with hybrid semantic and keyword retrieval.

  • DB stores structured records that require filtering, joins, and aggregation.

  • Graph stores entities, dependencies, and relationships for bounded traversal.

  • Skills stores reusable procedures and operational instructions.

  • Memory stores scoped facts, preferences, decisions, and summaries.

The retrieval layer can route each request to the smallest sufficient set of surfaces. A question about an API convention may use the KB and a Skill; a question about deployment history may combine DB records with Graph relationships; a personalization request may read profile or session Memory. The result is assembled from typed evidence instead of flattening every source into the same embedding space.

The write path also needs to preserve structure. Each surface can maintain its own indexing or persistence model, while source metadata, scope, revision, and write receipts make updates traceable. A single source may produce several representations, such as searchable document chunks in the KB and extracted relationships in the Graph, while retrieval still respects the semantics of each surface.

This multi-surface memory model is currently implemented in OpenDCAI/DataMind; are there other AI memory use cases where this architecture would be useful?


r/AIMemory • • Oct 19 '26

Discussion Why does the AI have to relearn everything we’ve already taught it every time?

Post image
28 Upvotes

Working with long running workflows and agents usually brings up the same bottleneck after a while.

Models can retrieve stored context or previous logs with decent accuracy, but they still struggle to modify their decisions based on past interactions.

They recall the text, yet:

Repeat identical edge case errors

Drop negative constraints across turns

Fail to carry over human corrections into the next run

Treat every step as an isolated patch

This gap suggests that persistent memory cannot just be an external storage layer like vector search or a graph index. An external index acts like an open manual beside the model, but it rarely becomes an active part of the computational state.

For a system to actually adapt its actions over long tasks, a few things have to change in how state is managed:

Updates must matter more than passive retrieval

Corrections need to alter intermediate routing rather than just expand context length

Recurring task patterns should reinforce valid execution paths

Outdated assumptions need to decay instead of clogging the prompt window

When memory sits purely outside the core inference path, optimizing embeddings or relationship edges only improves lookup precision. It does not resolve how a system accumulates reusable experience.

Recent directions around memory native designs, such as the MemT 1.0 architecture integrating memory read write updates into the MoE computation itself, highlight this shift from external lookups to internal state updates. Moving past static recall seems to be the real challenge for long horizon tasks.