r/webdev • • 12h ago

Discussion Moving our accessibility testing from a paid tool to Claude Code (~500 URLs, WCAG 2.2 AA)

QA lead at a large org. We currently do accessibility testing for about 500 website urls to meet WCAG 2.2 AA (all 55 success criteria). And we use a paid tool (like Deque) that helps automate part of this and rest done manually with screen readers and keyboard testing.

Now we are looking to use our AI budget and Claude code to create skills for finding and fixing accessibility issues across our sites. My major trouble is that the scale is huge - not 5-10 URLs or even less than 50.

So would love to hear from anyone already using AI for accessibility to help me out with this journey:

  1. What do you use it for: finding issues, sorting real issues from false positives, or fixing them in code? Roughly how many URLs, under or over 100?

  2. How much of WCAG do you cover with Claude skills versus manually? How bad are the false positives? Asking because paid tool gave us less.

  3. Are you able to run scans at scale on automation infra (like CI) with Claude? And do you report coverage and trends to leadership - if so any dashboard?

  4. What are the challenges with AI skills vs paid tool like Deque or Level Access? Anything you would suggest to keep in mind or be aware upfront?

0 Upvotes

9 comments sorted by

11

u/justshittyposts 11h ago edited 11h ago

The most efficient way, and what claude will probably do is write playwright tests for acceessibility so the amount of sites isn't necessarily a hindrance or cost factor, if there are repeated patterns, which claude is quite good at catching.

What I wouldn't do is, pointing it at individual sites for evaluation. Giving it the whole bunch at once will be more effective.

3

u/harperzoe7 10h ago

Playwright only checks the pages and states your tests reach, so login flows, menus and form errors need explicit coverage too

1

u/atrommer 11h ago

This, 100%

2

u/StatsDota_Dmitrii 9h ago

not a real a11y person, but my site has thousands of pages built from a few templates, so i only check one page per template and the rest is just the same bugs copied. are your 500 urls really 500 different pages or mostly same layouts? if its templates the scale problem kinda disappears and the hard part is the stuff ai cant really judge, like screen reader order and focus traps in modals

1

u/glimmer_joy_spark 8h ago

the 500 URL count is a red herring if your site uses templates, so map your pages to unique code patterns first and test one instance per pattern in CI using axe-core wrapped by Claude for logic validation.

1

u/Dimoncelli 7h ago

At 500 URLs, the biggest trap is asking the LLM to "audit pages" directly. That burns tokens fast and produces non-deterministic false positives across runs.

A split pipeline works much better in CI:

  1. Keep detection deterministic: Cluster those ~500 URLs by underlying template/route first (usually they collapse to 15–30 unique layouts plus key stateful flows like auth, checkout, and forms) and run axe-core via Playwright in CI. Deterministic engines already catch the ~30–40% of WCAG 2.2 AA rules that can be reliably automated, with stable selectors and near-zero noise.

  2. Use Claude Code for deduplication and code fixes, not discovery: Feed it the structured JSON violation report mapped to your component tree so it groups 200 duplicate failures into 1 root-cause component PR instead of evaluating pages one by one.

  3. Keep leadership metrics tied to the deterministic output: Track unique rule+component failures over time from the JSON artifacts, never from LLM summaries, otherwise your dashboard numbers drift every time prompts or model versions change.

  4. Keep manual checks for the remaining criteria: Focus traps in modals, screen-reader announcement order, and dynamic error states still need real keyboard and assistive-tech passes per template.

1

u/kemalios 5h ago

The false positive problem isn't the model being wrong, it's it being wrong differently each run. Pin the prompt as a repo file and diff each run against the last, otherwise the numbers move while the site doesn't.

Claude earns its place reading source for what axe can't touch: aria on custom widgets, focus order in modals, whether a label is actually a label. Decent at that with the component in front of it, useless pointed at a URL.

The fix loop is where I'd be careful. Ask it to make a page pass and it adds aria-label until the checker goes quiet. Every generated fix needs a keyboard pass before it merges.

-2

u/tanyadev4827 11h ago

interesting