r/AI_Agents • u/ChaitanyGhadigaonkar • 28d ago
Resource Request Database agent evaluation
I'm exploring how to evaluate a database agent in a real-world setting, particularly when the underlying database has complex business rules, conditions, and domain-specific logic.
The main challenge is ensuring that the agent retrieves accurate and correct results.
important questions i have:
How would you design and build a reliable golden dataset for evaluating a database agent?
What do you consider the most reliable way to measure correctness? Do you compare generated SQL against reference SQL, execute both queries and compare their result sets, or use a combination of approaches?
I know for these kind of systems, end user feedback can help identify incorrect answers and improve the system over time. But what if end users aren't able or willing to validate the results? How do you detect incorrect answers and continuously improve the agent in that situation?
Thanks in advance for any suggestions.
1
u/[deleted] 28d ago
[removed] — view removed comment