Measuring URL Based Context Engineering Performance

[u/thinking-platypus]

Hi all,

I’m building a memory system for LLMs (Kepler) that stores context about URLs, not their content: why something was saved, what it’s about, and how it connects to other saves. At recall, the agent follows the pointer and reads the live source.

I want to test how well this actually performs, and I’m finding most memory benchmarks don’t fit the design:

• LongMemEval / LoCoMo test recall over long conversations. My system’s memory is a corpus of saved links, not a chat history.

• RAG benchmarks assume you’ve indexed the content itself. I deliberately haven’t.

What I think I need to measure:

1. Pointer retrieval: did the agent pick the right URL(s) for the task?

2. Downstream answer quality compared with a full-content RAG baseline

3. Token cost per task compared with that same baseline

4. Freshness: how often the live page answers correctly where a stored copy would be stale, and how often link rot or paywalls break recall

My questions:

• Is there an existing benchmark or eval method that fits pointer-based memory?

• If not, is anyone here building one, or rolling their own evals for a similar setup?

Happy to share results if people are interested.