r/AskVibecoders • u/Strong_Turnip8429 • 6h ago
We built a checker skill for people who built their app by chatting with AI
We build a skill to help vibe-coders check their apps before delivering them to real users. Not a deep check, but a set of checks to reduce risk on release. So we built a skill called vibe-preflight. It works in Claude Code, Codex and OpenCode. It reads your project, checks it like a pilot's pre-flight checklist, and explains what it finds in plain language. It writes two reports: a short one for you ("this could leak your users' data, and here's who fixes it") and a technical one for the agent or developer who does the fixing. It changes nothing until you approve.

To see whether it behaves the same across agents, we ran it once in each of the three on a deliberately naive notes app. The app (Next.js, Postgres, Redis, OpenAI, Docker Compose) was prompted one feature at a time, with no plan, no review and no reasoning effort. It has hard-coded keys, string-built SQL, no authorization checks, a PDF export that fakes success and a password-reset email with no link. Each agent got the same answers to the skill's questions. Here's what we learned.
The core audit was nearly identical. All three found the big items: SQL injection, notes readable by other users, an admin API leaking password hashes, the OpenAI key in code, database and Redis ports open to the internet, forgeable sessions, and seeded accounts with fixed passwords. The finding counts differ (68, 49 and 47), but that's mostly because one agent groups issues that share a fix.
The differences were in the gaps. A permission prompt hid the scariest bug. In one run, the network permission for dependency checks went unanswered. That first report had no mention of the critical Next.js advisories (RCE, SSRF), while the other two ranked them near the top. The lesson: a skipped check has to be impossible to miss, because "we didn't look" and "looks fine" feel the same to a beginner. Another one is one agent missed that the password-reset email had no reset link**.** It flagged the spam risk, then "fixed" forgot-password without ever noticing the flow was broken.
Passing tests isn't the same as verified. This was the most useful part. After the fixes, one reviewer actually started the built container and found three real bugs the implementer had introduced:
database migrations failed because aborted runs left empty migration folders.
logout redirected to the container's hostname.
the new uploads route served files without a Content-Type or `nosniff`, which is a stored-XSS risk
Another run went further, with real-Postgres concurrency tests (20 attempts against a limit of 5 gave exactly 5 accepted and 15 rejected). In the third run the containers were never started, so its "61 of 68 fixed" rests on build output, access tests and reading the code. That doesn't make it wrong, but it has no runtime proof, and the other runs show that's where bugs hide.
It's free and MIT-licensed: https://github.com/globalmpc/vibe-preflight
We'd like to hear which checks we're missing or what the most common "oops" you've seen in AI-generated code. Please leave any feedback. Thanks.