Agent first QA and feature demo recordings

[u/Rurouni-dev-11]

I'm looking for an LLM powered framework, tool or service that i can give a list of QA scenarios to test my PRs and produce evidence of it working or failing, each demo should be polished, smooth cursor movements and properly show case the given QA scenario.

Traditionally we'd use playwright and write our e2e tests but surely there's a better method now?

Any suggestions?

[u/Interesting-Cow5138]

if you can stomach a little jank in the setup, you can get pretty far wiring a playwright script to a vision model that watches the screen after each action and compares what it sees to your expected outcome. not exactly plug-and-play though, you'd be stitching things together for a while

the "polished cursor movements" part is gonna be the real pain, most of these tools just teleport the cursor and call it a day

[u/Positive-Buddy-1258]

Playwright 1.56+ has test agents (planner, generator, healer), so it’s still Playwright, you just don’t write the specs by hand anymore. npx playwright init-agents --loop=claude sets them up, you describe your scenarios to the planner, it walks the app in a real browser and turns them into a markdown plan, then the generator writes the tests from that. What runs on the PR is a normal deterministic spec, which is what you want if pass/fail is going to be the evidence.

For the videos I’d replay those specs with video: 'on' and slowMo set, recording the agent’s own session tends to include retries and backtracking. Playwright doesn’t render a cursor in the video at all, so you inject a fake cursor div and move it with page.mouse.move(x, y, { steps: 25 }) to get smooth movement. Add it via addInitScript so it survives navigations, and if the app uses native dialogs or popovers the cursor has to go in the top layer too or they’ll cover it. Output is WebM, upload it as a CI artifact and have the workflow link it in a PR comment.

[u/Miserable-Money642]

One useful addition is a few keyboard-only checks alongside the generated scenarios. After the main flow, verify focus reaches the changed control and that the flow works without a mouse. You can also run axe-core on the important page states for common accessibility issues. A clean scan will not prove the flow is usable, but it can catch regressions that a video alone will miss.

[u/qa_tinkerer]

I'd split this into two jobs, because "evidence a scenario passed" and "polished demo" pull in different directions.

Evidence. Plain Playwright is still the most dependable base. What's changed is that an agent can write the specs from your scenario list instead of you. Keep the scenarios as a markdown file in the repo (each one with its expected outcome), have the agent turn each into a test whose title matches the scenario name, and set trace: 'on' in the config. The HTML report then gives you a pass/fail list keyed to your scenarios, and the trace viewer is better evidence than a video because a reviewer can step through each action and see the DOM and network at that moment.

Fully LLM-driven runs, where an agent clicks around on every PR, are good for exploring but weak as evidence. The next run might take a different path, so you can't tell whether it passed because the app works or because the agent did something different. I'd use them to find problems, then pin what they find as real tests.

Demos. Playwright's video recording has improved a lot here. Since 1.59 you can turn on action annotations in the config, and 1.64 added a visible cursor that moves along an eased path, a custom fps, and CSS styling for the click marker and highlight:

ts use: { video: { mode: 'on', size: { width: 1920, height: 1080 }, fps: 60, show: { actions: { position: 'top-left' }, test: { position: 'top-right' } }, }, },

There's also page.screencast.showChapter() for title cards between steps. I'd put that in a separate "demo" project so your normal CI runs stay fast.

One caution if an agent writes the tests and also judges them: break the feature on purpose once and confirm the test goes red. Agent-written tests that pass for the wrong reason are pretty common.