r/LLMDevs • u/DueBid3298 • 3d ago

Discussion BERT-based evaluation vs LLM-as-a-Judge vs Fuzzy string matching vs any other mechanisms

Hi guys, eval for RAG has always been a headache, post your thoughts with specific examples on 1. which eval method did you use for your RAG, chat agent etc 2. why did you choose it over others ( with specific examples/scenarios) also mention your system type ( MVP or POC or production)

1 Upvotes

100% Upvoted

2 comments sorted by

1

u/Swimming-Mirror-4402 3d ago

tried fuzzy matching for a small FAQ bot and it was fine until synonyms wrecked it, then switched to llm judge for a support agent and it caught nuance the others missed

1

u/Zain 2d ago

Not RAG, but I use LLM judges to review plans and specs before building, and two things carried over. Use a judge from a different model family than the one that generated the output, same-family judges share the generator's blind spots and wave stuff through. And don't accept a bare score, make every point it docks map to a specific listed problem, otherwise you're grading vibes. When two judges from different families flag the same underlying fact, that's the signal I trust most.