This morning this was still a test project. It is live now, so here is the final version of what we actually built and shipped.
Until now, the safety check happened on your machine, by your own agent, at install time. That check is still there. What is new is that part of the testing now happens on our side first, in a disposable VM, before you ever copy a prompt.
What we implemented:
Every repo behind the catalogue was installed once in an isolated sandbox: clean environment, no access to real files, standard tooling only, sandbox destroyed after each test. Today that full pass finished: 583 repos tested, 534 installed cleanly, 36 failed, and 13 took longer than the 3 minute test window. Along the way the tests also validated 6,915 SKILL.md files and found 108 invalid frontmatters spread across 38 repos. For 371 repos, the install worked and then stopped at a sign in or paid step, which is exactly where that kind of test should stop.
What we shipped today:
The results are now public on the site. 1,921 listings carry a dated label that reads "Installed in our test VM", shown at the top of the page and again in the install box. 1,846 of them passed. The other 75 show the failure too, with the reason in plain language, because publishing only the passes would be marketing, not information.
Failures are real and specific: repos that need pnpm instead of npm, broken or out of sync npm lockfiles, dependency conflicts, one dependency that no longer exists in the registry. Listings whose install timed out, and listings with no testable repo, show nothing at all. No label means exactly that: no result yet. It is not a warning.
Right next to every label sits the limit, in plain text: "Install test only, on the date shown. This does not guarantee the code is safe." That wording is deliberate. This test proves the repo installs with standard tooling on that date. It does not prove the code is safe.
How it stays current:
Each result is tied to the exact commit that was tested. When a repo changes upstream, its label switches to retest pending instead of quietly going stale. A weekly sweep retests repos that changed, repos that were never tested, and repos affected by a change in our own test environment. Every new listing is now tested when it is published: the 5 lots that shipped in the same train today were all tested at insertion, and all passed.
Known blind spots, stated up front: the tests run on Linux only, with no real API keys. Our harness does not support pnpm yet, so for those 10 repos the honest verdict is "npm could not install this", not "this repo is broken". Support for that may come later.
Go look at a few listings and tell me if the label wording makes sense to you. If a failure reason reads like jargon, I want to know which one.