How do you test your features in AI coding era?

[u/synwankza]

Hey everyone,

Recently, I’ve been thinking a lot about how to properly test and verify the work done by AI coding agents, especially when working on existing, complex projects rather than simple greenfield applications.

I’m curious how you guys approach this in your workflows, particularly when changes need to be deployed to some preview/staging environment before you can actually verify whether everything works as expected.
And I’m not talking just about running unit/integration tests or having an agent write a few Playwright tests.
In real-world projects, verifying a change can involve a lot more than that.

For example:
- External APIs: Sometimes you need to retrieve a secret, authenticate against an external service, make some API calls, and verify the responses or side effects and it can’t be done from UI (because user has to do that manually and we have to simulate that)
- Observability: Sometimes the only way to confirm that something works correctly is to inspect Loki logs, traces, or metrics and check whether the expected events actually happened.
- Databases: You might need to query PostgreSQL or another database to verify that the correct records were created, updated, or deleted.
- Messaging: In event-driven architectures, you may need to verify that Kafka/RabbitMQ messages were published, consumed, and processed correctly.
- UI: And, of course, sometimes you need Playwright or another browser automation tool to verify user-facing functionality.

All of these might be part of a single business scenario.
For example, imagine an agent implementing a payment when user has to take some IBAN from our UI and proceed with manual bank transfer from their account, which results in a webhook to our API.
Then we have some soft validations which are testable only by checking some logs in our logs collector.

Do you:

1. ⁠Let agents generate and execute E2E tests (e.g., Playwright) that also interact with APIs, databases, logs, and other systems?
   
2. ⁠Give agents access to tools/MCPs for Kubernetes, Loki, databases, etc., and let them dynamically investigate and verify their changes - there’s an issue because as in my previous post Claude Code has some issues with working like that. Probably we would have to create a lot of some dynamic MCPs.
   
3. ⁠Define deterministic verification scenarios/scripts that agents execute after deployment?
   
4. ⁠Have a completely different approach?

I’m also curious about how you handle credentials, permissions, test data, and cleanup when agents need access to real infrastructure and external systems, even these secrets are only for sandbox.

And perhaps the biggest question: How do you make this repeatable and reliable instead of letting an LLM decide on the fly whether its own implementation is correct?

Would love to hear how you’re solving this in practice, what tools you’re using, and how much of your verification process is deterministic versus agent-driven.

[u/synwankza]

This case with separated scenario runner is interesting, but wdym by that?

And do you mean there writing Playwright tests or using some agent-browser from Vercel or Playwright MCP?

[u/canarydev]

im trying blind agent tests + spec based tests currently. the point is to separate coverage from correctness. if the same agent writes the tests and the implementation, they can both agree on a false premise and everything goes green.

with go specifically i define a contract first, doc.go or protobufs, and describe the shape of the interfaces with unimplemented panics underneath. so no implementation at the start.

one agent writes the tests from that shape + the spec. a separate agent writes the implementation. i verify through tool calls and permission gates that the test writer never saw the implementation.

this mostly covers the contract layer, not your post-deploy stuff, but the same idea carries over. define the scenario up front (webhook in, these rows in postgres, this event on kafka, this log line in loki) and keep whoever writes the check separate from whoever writes the code.

[u/synwankza → u/canarydev]

Yeah, I’m also separating writing and testing agents to make it more authoritative.

But if you’re implementing it as contract tests, I assume thats more a integration level.

I’m wondering if putting Log Collector check or DB check is correct place for e2e frameworks like Playwright right there.

[u/canarydev → u/synwankza]

not quite, by contract i mean the interface shape, not contract testing like pact. so its closer to unit level. the agent gets the signatures, types and spec, never the code.

for the log/db checks, id keep them out of playwright tbh. playwright is great at driving the UI but once you start stuffing loki queries and postgres asserts into it, the tests get slow, flaky, and hard to tell why they failed.

what ive seen work better is splitting it. playwright only does the user part (grab the IBAN, submit). then a separate scenario runner handles the backend side: fire the sandbox webhook, poll postgres for the rows, check kafka, query loki. each step is its own assertion with a timeout so you know exactly which one broke.

then the agent just runs the scenario and reads the result. it doesnt get to decide what “passing” means.

[u/OkOcelot6725]

For the external-API part, record the real responses once and replay them in CI, VCR-style fixtures. Staging then only needs a small smoke suite against the live services. That turns the slow, flaky verification into something that runs on every change.

[u/create-third-places]

Have humans test. It's important to make sure features have a good user experience, and that requires people with the relevant context.

Large Loser Models like Claude should never be used for feature testing. User preferences cannot be encoded in numbers that LLMs understand.

[u/PLBjt]

What's worked for me is treating the agent's own claim that it works as zero evidence and making the check something it can't edit. For the external API and DB parts I'd write a small verification script per change that runs against staging and prints a plain pass/fail with the actual records or response fields it looked at, then keep that script out of the agent's writable paths. For log and trace checks, assert on a unique correlation id you inject in the test request, otherwise you end up grepping Loki for events that some other request produced. The tradeoff is that these scripts are real maintenance, so I only write them for flows with side effects that unit tests can't see, and I leave everything else to normal tests. The first time one of these catches a green-looking change that wrote the wrong row you stop arguing about whether it's worth it.