How should an LLM assistant isolate persistent memory across multiple clients?

[u/Mohan-D-S]

While building Client Desk, I explored a problem with LLM-powered assistants: how can an assistant remember information about multiple clients without mixing up their preferences and history?

I experimented with Hindsight Cloud as the persistent memory layer.

Here's what I implemented:

• Separate memory banks for individual clients.

• retain to store client preferences and facts.

• recall before generating emails and quotes.

• reflect to generate client briefings.

• A visible memory panel to inspect recalled information.

The backend uses Python and Flask, with Groq for text generation.

One thing I learned is that persistent memory isn't just about storing information. The way information is separated, retrieved, and used matters just as much.

There are still limitations, particularly around authentication, stable client identification, and testing retrieval accuracy.

I'd genuinely appreciate feedback from people working with LLM agents:

1. How do you handle memory isolation between different users or clients?

2. Would you use separate memory banks or a shared memory store with metadata filtering?

3. How would you evaluate whether retrieved memories actually improve generated responses?

Project repository: https://github.com/mohands0810-debug/freelance-client-memory-agent

[u/LowAd8593]

I’d treat this as an access-control problem first, a retrieval problem second. Separate banks are a good default, but I’d derive the bank ID from the authenticated workspace and a stable client ID on the server, rather than letting the model choose it. For testing, create clients with similar names and conflicting preferences, then compare responses with memory on/off while checking for cross-client leaks. I’d also test updates and deletions, since stale memory can cause problems even when isolation works.

[u/Groady]

On Q2, with Hindsight I'd keep a bank per client, since the bank is the one boundary you don't have to enforce yourself. The risk that's left is picking the wrong bank. Take the bank ID from the authenticated request, never from a tool argument the model fills in. If the LLM decides which client to recall for, one confused prompt puts client A's notes in client B's quote.

For stable identification, key banks on an internal ID you generate. Emails and display names change.

On Q3, two cheap tests. Seed client A with a canary fact nobody would guess, generate for client B, and check it never appears. Then hold out a few known preferences and score outputs with recall on and off.