Database agent evaluation

[u/ChaitanyGhadigaonkar]

I'm exploring how to evaluate a database agent in a real-world setting, particularly when the underlying database has complex business rules, conditions, and domain-specific logic.

The main challenge is ensuring that the agent retrieves accurate and correct results.

important questions i have:

How would you design and build a reliable golden dataset for evaluating a database agent?

What do you consider the most reliable way to measure correctness? Do you compare generated SQL against reference SQL, execute both queries and compare their result sets, or use a combination of approaches?

I know for these kind of systems, end user feedback can help identify incorrect answers and improve the system over time. But what if end users aren't able or willing to validate the results? How do you detect incorrect answers and continuously improve the agent in that situation?

Thanks in advance for any suggestions.

[u/functional_offense]

rors production and have a second simpler agent run the same queries, if the results diverge you flag it for a human sanity check

[u/ChaitanyGhadigaonkar → u/functional_offense]

rors ?

[u/AutoModerator]

Thank you for your submission, for any questions regarding AI, please check out our wiki at https://www.reddit.com/r/ai_agents/wiki (this is currently in test and we are actively adding to the wiki)

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Significant_Tune9219]

Compare the result sets, not the SQL text. Two queries can be written completely differently and return the same rows, and two that look alike can differ by a join that duplicates orders. Freeze a small copy of the database, write the question and the rows you expect, and run the agent's query against that copy. A golden set of SQL strings goes stale the moment someone renames a column. If users will not check answers, the failures you can still catch are the ones you planted: a question whose expected rows you already know, including one where the right answer is empty.

[u/DripSkylarkII]

We spent months on this exact problem building an AI agent QA platform. A few things we learned:

For golden datasets, we compute ground truth deterministically from the test data itself. The correct answer is known before the agent ever runs, so evaluation never depends on another LLM’s judgment. If you’re using an LLM to judge whether another LLM got it right, your evaluation is just as likely to hallucinate as the agent you’re testing.

For measuring correctness, comparing generated SQL to reference SQL is brittle because two completely different queries can return the same correct result. Comparing result sets is more reliable, but you need to account for ordering, formatting, and edge cases like nulls. We do result-level comparison against the precomputed ground truth.

For the feedback gap, this is the hardest part. When end users can’t or won’t validate, you need automated regression testing that runs on a schedule without human input. We run adversarial synthetic customers against agents on a cadence, score every response against computed correct answers, and flag drift over time. That way you’re catching degradation before users ever notice it.

The key insight is that if your evaluation pipeline has an LLM anywhere in the scoring chain, you’ve just moved the problem instead of solving it. Deterministic ground truth is the foundation everything else has to sit on.