# Web Accessibility Testing Fundamentals & Tool Selection

Every accessibility gate in a delivery pipeline rests on two decisions that get made once and then lived with for years: which engine evaluates the rendered DOM, and what that engine’s output is allowed to do to a build. This section settles both — the five runners worth putting in a pipeline, the rule tags and severity thresholds that convert their output into a merge verdict, and the exact boundary where automated scanning stops being able to answer the question and manual testing has to take over.

Key implementation targets:

• A single shared scan configuration — tag set, global exclusions, blocking impact list — imported by every runner instead of re-declared per spec.
• A conformance target expressed as axe rule tags (wcag2a, wcag2aa, wcag21a, wcag21aa, wcag22aa) rather than as a prose promise about “WCAG 2.2 AA”.
• A blocking threshold on critical and serious impacts, with moderate and minor reported as annotations that never fail a job.
• One runner chosen as the source of truth for pull requests and the others assigned explicit secondary jobs, so two tools never gate the same commit with different verdicts.
• A false-positive triage path that ends in a narrowed rule option or a scoped exclusion with an owner and a date, never in a disabled rule.
• An honest written record of which success criteria the pipeline cannot decide, routed to manual review instead of quietly counted as passing.

Tool choice is usually presented as a preference question and it is not — each of these five runners has a different relationship to the browser, and that relationship determines what it can see. axe-core evaluates a live DOM and therefore knows computed colour, layout and the accessibility tree; a runner that drives the browser can also change state before scanning, which is the only way to reach a modal, an expanded combobox or an error summary. The table below is the trade-off matrix that should decide which job each tool holds in a pipeline.

# Core Principles

Shift-left validation is an economic argument before it is a technical one. A button-name violation caught by a component test costs the developer who wrote it about ninety seconds; the same violation found in a pre-release audit costs a ticket, a triage meeting, a context switch back into code the author has forgotten, a re-test and a release note. The pipeline’s job is to move as much of that detection as possible to the earliest place a machine can decide the question — which in practice means the component test for anything provable about one component in isolation, the pull-request scan for anything that needs a whole page, and a nightly sweep only for things that are too slow to run per commit. A gate that lives exclusively at the end of the pipeline does not shift anything left; it just moves the argument to a worse moment.

Severity thresholds are what keep the gate credible enough to stay switched on. axe assigns every result an impact of critical, serious, moderate or minor, and those values are not a nuisance ranking — they approximate whether an assistive-technology user is blocked or merely inconvenienced. Block merges on critical and serious, report moderate and minor as pull-request annotations, and resist the pressure to promote a moderate rule because someone found an instance of it embarrassing. A team that blocks on all four impacts will spend its first fortnight arguing about region and landmark-unique violations in legacy templates, and the gate will be marked non-blocking by the end of that fortnight. Ratcheting the threshold downwards later, once the serious count is genuinely zero, is a solved problem covered in progressive threshold management.

The automated-versus-manual split is the number every stakeholder conversation eventually needs. Automated scanners reliably detect roughly 30–40% of WCAG failures, and that ceiling is a property of the questions rather than of the tools: a machine can prove that an <img> has no alt attribute, and it cannot prove that alt="image" describes the photograph. Contrast and ARIA validity sit at the high end of machine-decidability because they are computable from the DOM and the CSSOM; keyboard operability, meaningful sequence and error identification sit at the low end because they depend on intent. Publishing the split honestly is what stops “the pipeline is green” from being read as “the product is accessible”.

A scanner’s rule IDs, not its score, are the unit of truth. A score is a lossy aggregate: two runs can both report 92 while failing entirely different rules, a score cannot be diffed against a baseline in any useful way, and no engineer has ever fixed a bug by reading a number. Rule IDs are stable strings — color-contrast, aria-valid-attr-value, nested-interactive — that map to a documented rule, a WCAG success criterion, a set of DOM nodes and a remediation. Store them, diff them, annotate pull requests with them, and put trend charts of counts-per-rule into the reporting and violation-tracking dashboards rather than a single compliance percentage. The one place a score legitimately belongs is a coarse guardrail against catastrophic regression, which is what a Lighthouse budget is good at and nothing else.

Determinism is the last principle, and the one most often discovered the hard way. Pin the scanner version, the browser version and the rule tag list, because a minor axe-core release can add a rule — target-size arriving in 4.9 is the canonical example — and turn a green pipeline red on a dependency bump nobody associated with accessibility. Treat an engine upgrade like a schema migration: bump it in its own commit, read the new-rule list, triage what it finds in warning mode, and only then let it block. Everything else in this section assumes the scan is reproducible; a flaky scan is not a weaker gate, it is a gate that will be deleted.

# axe-core Configuration & Setup

axe-core is the engine underneath four of the five runners in this section, which makes it the only component whose configuration is worth learning properly — every option described in axe-core configuration and setup applies unchanged whether the caller is a Playwright fixture, a Cypress command or a pa11y runner. It evaluates a live document, so it has access to computed styles, the flattened DOM and the browser’s own accessibility tree; that is why the same rule set produces useful contrast results in a real browser and useless ones under jsdom, where no layout or paint has happened.

Rule selection happens through tags rather than rule IDs, and getting the tag list right is the single highest-leverage configuration decision on the page. wcag2a and wcag2aa cover WCAG 2.0, wcag21a and wcag21aa add the 2.1 additions, and wcag22aa adds the 2.2 criteria including target-size for SC 2.5.8. Anything tagged best-practice — region, landmark-one-main, page-has-heading-one — is genuinely good advice that maps to no success criterion, so it belongs in a warning stream and not in a conformance gate. experimental rules should be excluded from any blocking run entirely; they exist so the maintainers can gather field data, not so a pipeline can act on them.

Scoping is done with include and exclude, and the distinction between them is worth stating precisely because it is the source of most surprising results. include replaces the default context of the whole document with the listed selectors, so an include of main means no rule ever evaluates the header, the navigation or the footer. exclude keeps the document as the context and removes subtrees from it, which is the right tool for third-party widgets nobody on the team can fix — a payment iframe, an embedded map, a chat launcher. Global exclusions belong in the shared configuration module with a comment naming the owner, because an undocumented exclusion is indistinguishable from a bug six months later.

// a11y/run-options.js — the raw axe run options every runner ends up passing.
export const runOptions = {
  runOnly: {
    type: 'tag',
    // The conformance claim, expressed as tags. best-practice and
    // experimental are deliberately absent from the blocking run.
    values: ['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa', 'wcag22aa'],
  },
  rules: {
    // Owned by the payments team; the vendor iframe has no accessible name.
    // Review date 2026-10-01, tracked as A11Y-482.
    'frame-title': { enabled: false },
    // Narrowed, not disabled: the design system uses a 3:1 large-text token
    // that the default check reads as body copy on two marketing pages.
    'color-contrast': { enabled: true },
  },
  resultTypes: ['violations', 'incomplete'], // skip `passes` to shrink JSON
  // axe walks open shadow roots by default; iframes need this left on.
  iframes: true,
};

Suppression is where most pipelines go wrong, and the failure mode is always the same: a rule fires on something the team believes is fine, someone disables the rule globally, and eighteen months later that rule is the one an auditor finds forty instances of. The correct escalation order is narrow the rule’s options, then exclude the specific node or subtree, then — only if neither is possible — disable the rule with an owner and a review date. Reducing false positives in automated accessibility scanners works through the individual rules that most often need this treatment, and the tree below is the triage path worth putting in a contributing guide.

Framework wiring and boundary-crossing are the remaining setup concerns. React and Vue applications need the scan to happen after hydration and after the effect that sets ARIA state has flushed, which is why a naive scan in a component test reports a control with no accessible name that is perfectly labelled in the browser; configuring axe-core for React and Vue applications covers the render-then-scan ordering per framework. Boundaries are the other trap: axe traverses open shadow roots as part of its flat-tree walk but a closed root is genuinely invisible to it, and same-origin iframes are scanned only when the engine has been injected into every frame — the mechanics for both, including the cross-origin case, are in scanning shadow DOM and iframes with axe-core.

# Playwright Accessibility Plugin Integration

@axe-core/playwright is the runner to reach for when the pipeline needs the same rule set applied to real application state rather than to a static page. The plugin exposes an AxeBuilder that injects the engine into the page and every same-origin frame, and its builder methods — withTags, disableRules, include, exclude, options — are a thin, typed wrapper over the axe run options above. Because a Playwright test can log in, open a dialog, submit an invalid form and expand a disclosure before calling analyze(), it reaches roughly the failure classes a URL-list crawler structurally cannot; Playwright accessibility plugin integration covers the fixture, the reporter wiring and the shard layout.

Scoping to a locator is what makes per-component assertions practical inside an end-to-end suite. Passing a locator or a selector to include restricts the scan to that subtree, so a spec can assert that the newly built checkout summary is clean without inheriting the seventeen legacy violations in the site footer. This is the pragmatic way to introduce a gate into an application with existing debt: scan the whole page in warning mode, and scan the components under active development in blocking mode. The pattern generalises well, and the first-adoption sequence — install, fixture, one route, then the matrix — is worked through step by step in integrating @axe-core/playwright into an existing project.

Focus-order testing is the capability that justifies Playwright over a crawler even when the axe results are identical. No axe rule can evaluate WCAG 2.2 SC 2.4.3 (Focus Order), because the correct order is a property of the page’s meaning rather than of its markup; what a test can do is press Tab repeatedly, record the accessible name and role of each stop, and assert that the sequence matches a committed expectation. That turns focus order into a diffable artifact — a reviewer sees “the skip link moved after the search box” instead of “some tab order changed” — and it catches the specific regression class where a modal fails to trap focus or a hidden element remains reachable. Testing keyboard focus order with Playwright covers the walk, the escape conditions and the snapshot format.

The cost of this runner is discipline about timing. An analyze() call that fires before a route has settled produces a violation list describing a DOM that no user ever saw, and the resulting flakiness is invariably attributed to accessibility rather than to the wait. Anchor every scan on something the application asserts about itself — a visible heading, a resolved loading state, an aria-busy that has been removed — and never on a fixed timeout. Applications with heavy client-side routing need more than that, and the wait strategies for them belong to custom rule development and context-aware testing.

# Cypress a11y Testing Workflows

cypress-axe occupies a different niche from the Playwright plugin despite wrapping the same engine: it is at its best inside Cypress component testing, where a single component is mounted in a real browser with no application shell around it. That isolation is exactly what a design-system team wants — the scan result describes the component and nothing else, so a color-contrast failure is unambiguously the component’s token choice rather than a page background it happens to sit on. Cypress a11y testing workflows covers the support-file setup, the custom command shape and the reporting path.

The API is two commands and one gotcha. cy.injectAxe() must run after the page or component has mounted, because it injects the engine into the current document and a Cypress page load discards it; cy.checkA11y(context, options, callback, skipFailures) runs the scan and fails the test unless skipFailures is true. The fourth argument is worth understanding rather than avoiding — combined with the callback it gives a run that records every violation into the terminal and an artifact while failing only on the impacts the gate cares about, which is the mechanism for a soak period on a newly introduced check. Component-level configuration, including scanning each Storybook-equivalent state of a component, is covered in configuring cypress-axe for component testing.

Authenticated pages are the other place Cypress earns its keep, because the interesting accessibility surface of most products sits behind a login. Scanning a signed-in area needs session state that is established once and restored per spec rather than a login form driven at the top of every test, and it needs the scan to happen after the authenticated shell has rendered its user menu and notification region. Doing this without leaking credentials into artifacts or screenshots takes a little care, and scanning authenticated pages in Cypress a11y runs sets out the session, fixture and redaction pattern.

Where a team already has a substantial Cypress suite, adding accessibility assertions to existing specs is cheaper than standing up a second runner, and the marginal cost is close to zero: two lines per spec that already navigates to the right state. Where there is no existing suite, the tooling comparison matters more, and the trade-offs — parallelism model, browser coverage, iframe handling, retry semantics — are laid out in comparing Playwright and Cypress for WCAG compliance testing. The one thing not worth doing is running both against the same routes; two runners reporting the same rule IDs doubles the maintenance and halves the trust.

# Lighthouse CI Baseline Configuration

Lighthouse CI answers a question the other runners do not: has this deploy regressed overall, across performance, best practices and accessibility, compared with a stored baseline. Its accessibility category runs a curated subset of axe rules and weights them into a 0–100 score, and lhci autorun collects, asserts and uploads in one command, which makes it the cheapest thing on this list to add to a pipeline. Lighthouse CI baseline configuration covers the config file, the assertion syntax and the server deployment.

Assertions are where a Lighthouse gate becomes useful rather than decorative. A minScore on the accessibility category is a coarse guardrail; asserting individual audits by ID — color-contrast, image-alt, label, aria-allowed-attr — is a real gate, because those assertions fail with a name a developer can act on. Configure both: a category floor that catches broad collapse and an off-to-error list of specific audits that must never regress. The threshold arithmetic, including which audits are safe to assert at error on a legacy codebase, is in setting up Lighthouse CI thresholds for WCAG 2.2 AA.

{
  "ci": {
    "collect": {
      "url": ["http://127.0.0.1:4173/", "http://127.0.0.1:4173/checkout"],
      "numberOfRuns": 3,
      "settings": { "onlyCategories": ["accessibility"], "preset": "desktop" }
    },
    "assert": {
      "assertions": {
        "categories:accessibility": ["error", { "minScore": 0.95 }],
        "color-contrast": "error",
        "image-alt": "error",
        "label": "error",
        "aria-allowed-attr": "error",
        "heading-order": ["warn", { "minScore": 1 }]
      }
    },
    "upload": { "target": "lhci", "serverBaseUrl": "http://lhci.internal:9001" }
  }
}

The score-versus-violation-count distinction trips up every team that runs Lighthouse and axe side by side. Lighthouse weights audits by an internal importance model and reports a single number, so fixing three instances of one low-weight audit can leave the score unchanged while axe’s violation count drops by three; conversely one high-weight audit failing on a single node can cost several points. Neither tool is wrong — they are reporting different things — but only one of them can be the gate for a given rule, and the reconciliation procedure is set out in Lighthouse accessibility score vs axe violation counts.

A shared baseline server is what turns Lighthouse CI from a per-pull-request check into a trend record. The LHCI server stores every run against a commit, so a reviewer can see whether the score dropped in this branch or has been drifting for six weeks, and lhci assert --preset comparisons stop being sensitive to the ambient noise of a busy runner. Run three collections per URL and let the tool take the median; a single Lighthouse run on a shared CI machine has enough variance to produce a two-point swing with no code change at all, and a gate that fails on ambient variance teaches everyone to re-run the job until it passes.

# Pa11y CI Integration

pa11y-ci is the right tool for breadth. It takes a list of URLs — hand-written or expanded from a sitemap — loads each one in a headless browser, runs the configured engines, and returns a single exit code against a per-URL error threshold. For a content site with two hundred pages built from a dozen templates, that sweep finds template-level failures far more efficiently than an end-to-end suite would, and it needs no test code at all. Pa11y CI integration covers the .pa11yci config, sitemap expansion, thresholds and per-URL overrides.

Its distinguishing feature is the HTML CodeSniffer runner, which evaluates WCAG techniques directly rather than axe’s curated rule set. htmlcs reports notices and warnings alongside errors, and those advisory categories are the source of pa11y’s reputation for noise — a default configuration will report things like “check that this heading is not being used purely for presentation”, which is real advice and terrible gate material. Run htmlcs with errors only, or run both engines and gate exclusively on the axe results while treating the htmlcs output as a manual-review queue. Getting this wrong is the most common reason a pa11y gate gets removed.

Because pa11y loads a URL and scans, it cannot reach interaction-dependent state, and that is the practical trigger for migration. Once a site becomes an application — once the important accessibility surface lives behind a login, inside a dialog or after a client-side route change — the URL-list model stops covering it and a browser-driving runner has to take over. Doing this without losing the URL coverage that already exists is a mapping exercise from pa11y config to axe run options, covered in migrating from pa11y to axe-core in CI, and it is usually worth running both for a sprint so the two violation lists can be reconciled before the old job is deleted.

Choosing which engine holds the blocking role is the decision that ties this section together. axe-core has stable rule IDs, documented remediation, per-node output and an impact field, which makes it a good gate; Lighthouse produces a score that is a good trend line and a poor gate; pa11y produces broad URL coverage that is a good sweep and an ambiguous gate. The concrete comparison — exit-code control, speed per URL, rule overlap, false-positive profile — is in axe-core vs Lighthouse CI for PR gating, and the matrix below summarises what each runner can actually catch.

# Screen Reader Automation Testing

The right-hand columns of that matrix are where screen reader automation testing comes in. Rule-based scanning asks whether the markup satisfies a rule; screen-reader-oriented automation asks what the assistive technology would actually receive, which is a different question with different failure modes. Three techniques cover most of the ground: snapshotting the computed accessibility tree, capturing live-region announcements as they happen, and driving a real screen reader on a runner and diffing its speech output.

Accessibility-tree snapshots are the cheapest of the three and the one to adopt first. The browser exposes the tree it hands to assistive technology, so a test can capture the roles, names, levels and states of a subtree and commit that capture as an expectation. The resulting diff is remarkably readable — a heading that silently became a <div>, a button whose accessible name changed from “Delete invoice” to “Delete”, a checkbox that lost its checked state — and it catches regressions that no rule would flag because the markup remains technically valid. Asserting accessibility tree names with Playwright snapshots covers the snapshot format, the normalisation needed to stop copy edits from churning it, and where the expectation files belong.

Live-region behaviour needs a different mechanism, because a live region is an event rather than a state. WCAG 2.2 SC 4.1.3 (Status Messages) is satisfied when a change is announced without moving focus, and a DOM scan taken after the fact cannot distinguish a region that announced correctly from one that was inserted into the document together with its text and therefore said nothing at all. The workable approach is to observe the region with a MutationObserver installed before the triggering action, record every text change with a timestamp, and assert on that sequence — which also catches the double-announcement bug where a framework re-renders the region and the text is read twice. Verifying live-region announcements in automated tests covers the observer, the timing window and the assertion shape.

Driving a real screen reader is the heaviest option and the only one that tests what a user hears. NVDA can be scripted on a Windows runner with its speech output redirected to a log, so a test can assert that tabbing to the payment field announces the label, the required state and the format hint in that order. The cost is real: a Windows runner, a screen-reader installation step, a speech-synthesiser stub and output that is sensitive to verbosity settings, so this belongs in a nightly job over a handful of critical journeys rather than in a pull-request gate. Automating NVDA output capture in a Windows CI runner covers the runner image, the capture plumbing and the normalisation that keeps the assertions stable.

None of this raises the 30–40% ceiling by much, and it is worth being clear about why: these techniques move specific criteria from “unverifiable” to “verifiable”, they do not confer judgment. A tree snapshot proves the accessible name did not change; it cannot decide whether “Delete invoice” is a better name than “Remove”. That decision stays with a human, and the pipeline’s contribution is to guarantee that once the human has made it, nobody breaks it silently.

# Reference Pipeline

The four files below are a complete minimal stack: install the pinned dependencies, declare the scan configuration once, scan a route list from one Playwright spec, and wire the whole thing to two npm scripts and a single workflow job. Nothing here needs a custom rule, a baseline file or a reporting service; those are the next layer, and they belong to CI/CD integration and automated quality gating.

# Pin every moving part: engine, plugin, runner and browser build.
npm install --save-dev \
  axe-core@4.10.2 \
  @axe-core/playwright@4.10.1 \
  @playwright/test@1.49.1
# Installs the matching Chromium plus the system libraries a CI image lacks.
npx playwright install --with-deps chromium

The configuration module is the only place the conformance target, the global exclusions and the blocking impacts are written down. Every runner imports it, which is what makes a local failure and a gate failure describe the same thing.

// a11y/axe-config.js — single source of truth for every accessibility scan.

// The conformance claim. Adding a tag here widens the gate everywhere at once.
export const CONFORMANCE_TAGS = [
  'wcag2a',
  'wcag2aa',
  'wcag21a',
  'wcag21aa',
  'wcag22aa',
];

// Impacts that fail a build. moderate and minor are reported, never blocking.
export const BLOCKING_IMPACTS = new Set(['critical', 'serious']);

// Subtrees nobody on this team can fix. Each entry needs an owner and a date.
export const GLOBAL_EXCLUDES = [
  '#vendor-chat',      // third-party launcher, vendor ticket SUP-1194
  '.map-embed',        // map iframe, cross-origin, review 2026-11-01
];

// Applied to an AxeBuilder so the tag list and excludes cannot drift per spec.
export function withProjectConfig(builder) {
  const configured = builder.withTags(CONFORMANCE_TAGS);
  for (const selector of GLOBAL_EXCLUDES) {
    configured.exclude(selector);
  }
  return configured;
}

// Returns one compact line per blocking violation, for the assertion message.
export function blockingViolations(results) {
  return results.violations
    .filter((violation) => BLOCKING_IMPACTS.has(violation.impact))
    .map((violation) => `${violation.id} (${violation.impact}) x${violation.nodes.length}`)
    .sort();
}

One spec covers the route list. It iterates a plain array, so adding a route is a one-line change, and it attaches the full JSON result to the test so a failure is debuggable from the run artifacts without reproducing anything locally.

// tests/a11y/routes.spec.ts
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
import { withProjectConfig, blockingViolations } from '../../a11y/axe-config.js';

// Add a route here and it is gated on the next pull request.
const ROUTES = ['/', '/search?q=chair', '/product/desk-lamp', '/checkout', '/account'];

test.describe('WCAG 2.2 AA route sweep', () => {
  for (const route of ROUTES) {
    test(`no blocking violations on ${route}`, async ({ page }, testInfo) => {
      await page.goto(route, { waitUntil: 'domcontentloaded' });
      // An application-owned signal, not a timeout: the main landmark is
      // rendered only after hydration has attached its event handlers.
      await page.getByRole('main').waitFor({ state: 'visible' });

      const results = await withProjectConfig(new AxeBuilder({ page })).analyze();

      await testInfo.attach(`axe${route.replace(/\W+/g, '-')}.json`, {
        body: JSON.stringify(results, null, 2),
        contentType: 'application/json',
      });

      // Rule IDs, not a count: the failure message names what to fix.
      expect(blockingViolations(results)).toEqual([]);
    });
  }
});

Playwright owns the build-and-serve lifecycle so one command works identically on a laptop and on a runner, and the scripts give the two entry points a developer and the pipeline each need.

{
  "scripts": {
    "a11y": "playwright test tests/a11y --reporter=list",
    "a11y:ci": "playwright test tests/a11y --reporter=list,json",
    "a11y:one": "playwright test tests/a11y --grep"
  },
  "devDependencies": {
    "@axe-core/playwright": "4.10.1",
    "@playwright/test": "1.49.1",
    "axe-core": "4.10.2"
  }
}

The workflow job is deliberately short. It builds, scans, always uploads the report, and lets the test step alone decide the exit code.

name: a11y-route-sweep
on:
  pull_request:
    paths:
      - 'src/**'
      - 'a11y/**'
      - 'tests/a11y/**'
      - '.github/workflows/a11y-route-sweep.yml'
concurrency:
  group: a11y-route-sweep-${{ github.head_ref }}
  cancel-in-progress: true
jobs:
  route-sweep:
    runs-on: ubuntu-24.04
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - name: Install the pinned browser build
        run: npx playwright install --with-deps chromium
      - name: Scan the route list
        run: npm run a11y:ci # non-zero exit on any critical or serious violation
      - uses: actions/upload-artifact@v4
        if: always() # the failing run is the one whose report is worth keeping
        with:
          name: a11y-route-sweep-report
          path: |
            test-results/
            playwright-report/
          retention-days: 14

# WCAG 2.2 Coverage Mapping

The table below is the honest version of the coverage claim: which success criteria this toolchain decides, which rule IDs make the decision, and what remains for a human. Cite it verbatim in a conformance statement rather than paraphrasing “we test WCAG 2.2 AA automatically”, because the fourth column is the part auditors ask about.

Two entries in that table deserve emphasis. color-contrast returns incomplete rather than a violation whenever it cannot determine a background colour — an element over an image, a partially transparent overlay, a canvas — and a pipeline that only reads violations silently discards those, which is why the run options above request incomplete as well. And SC 2.4.3, SC 2.4.7 and SC 4.1.3 have no rule at all: they are covered in this section by test assertions, not by the scanner, which is the concrete reason a pipeline built only on a URL crawler cannot claim them.

# Common Pitfalls

• Gating on best-practice-tagged rules such as region or landmark-one-main, which fail on perfectly conformant pages and burn the team’s willingness to treat the gate as authoritative.
• Running color-contrast under jsdom, where there is no layout or paint, so the rule reports nothing and a whole success criterion silently drops out of coverage.
• Reading only results.violations and discarding results.incomplete, which throws away every case the engine flagged as needing a human look — overlays, transparent backgrounds, closed shadow roots.
• Using include when exclude was meant, so the scan quietly covers one subtree and the header, navigation and footer are never evaluated by any rule again.
• Disabling a rule globally to silence one node, then discovering at audit time that the rule had been suppressing forty real failures across the estate.
• Letting two runners gate the same routes, so a pull request can be blocked by Lighthouse and passed by axe on the same commit and nobody knows which verdict to act on.
• Treating a Lighthouse accessibility score as a violation count, which makes trend charts unreadable and makes “the score went up” compatible with “three new rules are failing”.
• Scanning before the application has settled and then blaming accessibility for the flakiness, instead of anchoring the scan on a signal the application itself asserts.
• Upgrading axe-core inside a batch dependency bump, so a newly added rule such as target-size turns the gate red in a pull request that has nothing to do with the failure.
• Publishing “automated accessibility testing passes” as a conformance claim without publishing the criteria the automation cannot decide.

# FAQ

Which single runner should a team pick if it can only afford one? @axe-core/playwright, because it is the only option that covers both the static rule set and interaction-dependent state, and because its output is per-rule and per-node rather than a score. A URL crawler is cheaper to set up but structurally cannot reach anything behind a login, a dialog or a client-side route change, and Lighthouse cannot be extended with a custom rule when the design system needs one. The cost is that a Playwright suite is test code that has to be maintained, so budget for the waits as well as for the scans.

Why does Lighthouse report a 96 while axe reports eleven violations on the same page? They are measuring different things on purpose. Lighthouse runs a curated subset of axe rules and weights each audit into a single score, so eleven violations concentrated in low-weight audits can cost four points, while one failing high-weight audit on a single node can cost more. Compare rule IDs across the two reports rather than comparing a score with a count, and let only one of the two tools hold the blocking role for any given rule.

Is the 30–40% figure a criticism of the tools? No — it is a property of the criteria. Roughly a third of WCAG success criteria can be decided from the DOM, the CSSOM and the accessibility tree, and the rest depend on whether something is meaningful, sequenced sensibly or understandable, which no static analysis can determine. The useful response is to automate the machine-decidable third completely, so that human review time is spent entirely on the questions that actually need judgment rather than on re-finding missing alt attributes.

How should incomplete results be handled in a gate? Report them, never block on them. An incomplete result means the engine could not decide — a background it could not compute, a control it could not reach, a shadow root it could not enter — and failing a build on uncertainty trains developers to add exclusions rather than to investigate. Post the incomplete list as a pull-request annotation, review it during triage, and convert each recurring entry into either a scoped exclusion with an owner or a manual-test step.

Can this pipeline replace a manual accessibility audit? It replaces the part of an audit that consists of finding mechanical failures, which is usually most of the raw finding count and almost none of the value. What remains for a specialist is the judgment work: whether the focus order matches the task, whether error messages explain the error, whether a custom widget behaves the way its role promises, and whether the whole journey is completable with a screen reader. A good pipeline makes an audit shorter and more expensive per finding, which is the correct direction.

# Related

• axe-core Configuration & Setup — tags, include/exclude, suppression policy and the framework wiring every other runner inherits.
• Playwright Accessibility Plugin Integration — scoped scans, focus-order walks and the fixture that keeps specs short.
• Screen Reader Automation Testing — accessibility-tree snapshots, live-region capture and NVDA output on a Windows runner.
• CI/CD Integration & Automated Quality Gating — where these exit codes become branch protection, thresholds and dashboards.
• Custom Rule Development & Context-Aware Testing — what to do when a design-system contract needs a rule the engine does not ship.