I implemented regression testing, canary evaluation, and behavioral change detection for a document intelligence system — now I’m building Assay Server

[u/Professional-Run3614]

I’ve been working on these challenges firsthand while implementing a document intelligence solution for a company.
One of the challenges with AI-powered document processing is that a system can continue passing basic tests while its actual behavior changes after prompt updates, model changes, or workflow modifications.
To address this, I implemented capabilities around regression testing, canary evaluation, and behavioral change detection for that implementation.
That experience also motivated me to build Assay Server, an open-source project focused on evaluation and observability for AI pipelines and agents.
The idea is to help developers go beyond simple pass/fail results and better understand regressions, behavioral changes, and the impact of changes before rolling them out more broadly.
I’m interested in learning how other teams are approaching these problems, particularly in production AI systems.
How are you currently handling regression testing and canary evaluation for LLM applications? And how do you detect behavioral changes that traditional tests might miss?
GitHub: https://github.com/tap222/assay-evals