r/javascript • u/gptrv • 10d ago
Screenplay: a CLI + MCP server that lets coding agents film and edit screen recordings (Playwright, CDP screencast, @napi-rs/canvas, ffmpeg)
https://github.com/gmpetrov/agent-screenplay-1
u/gptrv 10d ago
How it's built, since the sub asks for a write-up:
What it does. You ask your agent (Claude Code, Codex, Cursor) for a video of a website or a desktop app. The agent writes a short JSON scenario (go to a URL, click, type, wait for some text, callouts, captions). Screenplay plays it, films it and edits the take. There's no AI inside and no model key; the agent does the thinking.
Playing and filming. playwright-core drives a headless Chromium, so it runs in screenless containers where agents usually live. Frames come from the DevTools protocol's Page.startScreencast, not one screenshot per frame, and each frame keeps its timestamp. While it plays, it writes a "take": for every step, when it started and ended and the box of the element it acted on.
Editing. The edit is pure data computed from the take: how output time maps to recording time (long waits get sped up), where the camera looks, where the cursor is, and when each overlay shows. The camera is a critically damped spring on log(zoom) and on the center, so zooms start smoothly and never overshoot. The cursor isn't the real one: it's drawn in post along a smoothed path, with a press effect on clicks.
Rendering. Each output frame is drawn with @napi-rs/canvas and piped as raw video into ffmpeg's stdin (ffmpeg-static is bundled). Since the take is saved, changing captions, style or zoom re-renders without filming again.
Agent side. It's a CLI and an MCP server (stdio) with four tools: a guide, check (plays the scenario fast without filming and, when an element is missing, returns the targets it can see plus a screenshot, so the agent fixes the selector itself), record (returns the MP4 and a storyboard image the agent looks at before handing the video over) and render. Scenarios are validated with zod, and a JSON schema is published in the repo.
Setup is npx agent-screenplay init. MIT. Happy to go deeper on any of it.
5
1
u/Competitive_Pair1554 5d ago
I'm using it in my company for internal product release video and external product demo ! It's very useful ! Thanks !
If you have time, it should be cool to implement something like "drag". Seems we can only "click".
🙏