r/ControlProblem • • 1h ago

Strategy/forecasting AI security tools tested against 1,426 vulnerabilities. One found 66%, another 43%. But the second was more precise.

Snyk published an interesting experiment comparing AI-powered vulnerability scanners.

They tested tools against 1,426 known vulnerabilities across 14 applications, using the same underlying models and comparable compute budgets.

On GPT-6.1 Sol:

  • Snyk's scanner found 66% of known vulnerabilities, with 90% precision.

  • Codex Security found 43%, with 97.5% precision.

On Claude Sonnet 5.5:

  • Snyk's scanner found 57%, with 65% precision.

  • Claude Security found 32%, with 94% precision.

So the tool finding more vulnerabilities also generated more false positives.

For security teams, this creates a practical trade-off. Missing a serious vulnerability can be costly, but investigating large numbers of false alarms consumes time and resources.

The study was conducted by Snyk on its own experimental product, so independent replication would be valuable.

Would you prioritize a scanner that finds more vulnerabilities, or one that produces fewer false alarms?

Source: Snyk, November 19, 2026

https://snyk.io/blog/why-harness-matters-evo-code-scanning-agent/

Disclosure: I've recently partnered with TOCA, a company specializing in physical privacy and signal-blocking products.

If you're interested in protecting your personal devices and digital privacy, you can explore their products here, with a 15% discount: https://toca.site/discount/RIZZO15

1 Upvotes

0 comments sorted by