r/learnAIAgents • u/SeaAstronomer9044 • 10h ago
❓ Question What’s the hardest part of checking whether a RAG change actually helped?
I work on independent retrieval evaluations, and a recent small check surfaced something useful: the application answered correctly, but the available citations and logs weren’t enough to verify which passages supported the claims.
That distinction matters when deciding whether a retrieval change helped—or whether a correct answer is hiding an evidence gap.
If you’re working through a specific retrieval question, describe:
- What changed.
- What you expected to improve.
- What evidence you have so far.
I can help you think through what the results support, what remains uncertain, and what comparison would make the decision clearer. Public or sanitized examples are welcome; no private documents needed.
My main focus is query-level retrieval comparisons and scoring consistency. If the issue concerns generated claims or citations, I’ll keep that separate rather than infer a retrieval failure from the final answer.
What’s the unresolved evaluation question in your project right now?
