# Pa11y CI Integration as a Breadth-First Accessibility Gate

This guide is part of Web Accessibility Testing Fundamentals & Tool Selection, and it treats pa11y-ci as the one thing it is uniquely good at: sweeping a whole site’s URL list in a single job and returning one exit code. Every other scanner on this site is depth-first — it drives a browser through a scenario and asserts on one page at a time. pa11y-ci is breadth-first: hand it a sitemap and it will load two hundred documents, run one or two rule engines against each, aggregate the issue counts, and fail the build if any URL is over its budget. That shape maps almost exactly onto a marketing site, a documentation set, a government service catalogue, or any product whose accessibility risk is spread thinly across many templates rather than concentrated in one application shell.

# Problem Statement

The failure mode pa11y-ci solves is coverage drift. A team stands up an accessibility gate with three or four hand-picked URLs in it, ships forty new pages over the next two quarters, and the gate keeps passing because it never learned about the new pages. Nobody notices, because the job is green. Six months later an audit finds that the newest templates — the ones written after the gate existed — are the worst offenders, and the pipeline reported success on every one of the commits that introduced them. A URL list that a human maintains is a URL list that goes stale.

The second problem is engine blind spots. Different rule engines encode WCAG differently: axe-core is strongest on computed properties it can only see in a real browser — colour contrast against the resolved background for WCAG 2.2 SC 1.4.3, ARIA role validity for SC 4.1.2, focus-order side effects — while HTML CodeSniffer works from documented WCAG techniques and failures, which makes it noisier but gives it checks axe simply does not ship, such as flagging a paragraph styled to look like a heading. Running one engine and calling the result “our accessibility coverage” quietly narrows what the gate can ever catch. pa11y-ci is the cheapest way to run both in one pass and get one number out.

The third problem is that a breadth-first sweep is easy to make slow and easy to make flaky. Two hundred URLs at four seconds each is thirteen minutes of serial browser work, and the naive fix — raise concurrency until the wall clock looks acceptable — starves a two-vCPU runner and turns timeouts into phantom failures that appear on unrelated pull requests. Getting pa11y-ci onto a pull-request path means budgeting concurrency against the runner you actually have, not against the one you wish you had.

Key implementation targets:

• A .pa11yci file whose defaults block sets the conformance standard, both runners, the per-URL timeout and a container-safe Chrome launch configuration.
• URL discovery from sitemap.xml, with host rewriting so the production sitemap can drive a scan of a preview deployment.
• A threshold policy that tolerates known debt per URL without silencing a rule across the whole site, plus hideElements for third-party subtrees nobody on the team can fix.
• Per-URL overrides and an actions sequence that logs in, dismisses a consent dialog, or opens a disclosure before the check runs.
• A CI invocation that writes machine-readable JSON, uploads it as an artifact, and turns a non-zero exit into a merge block.
• A concurrency figure derived from the runner’s CPU count rather than guessed.

# Prerequisites

• Node.js 20.x or newer, with pa11y-ci 3.x installed as a dev dependency (npm i -D pa11y-ci)
• A served build reachable over HTTP — pa11y-ci navigates a real browser, so file:// paths and unserved dist/ folders are not testable
• A sitemap.xml published by the build, or a committed list of routes if the site has no sitemap
• Chromium available to Puppeteer in the runner image, or PUPPETEER_EXECUTABLE_PATH pointing at a system Chrome
• A decision on the conformance target — WCAG2AA is the default for almost every compliance obligation
• A CI job that can run a Node step, keep a background server alive, and archive a JSON file
• Credentials for one low-privilege test account in CI secrets, if any scanned route sits behind a login

# 1. Install pa11y-ci and Author the Config File

pa11y-ci reads a JSON file named .pa11yci from the working directory unless --config says otherwise. The file has exactly two meaningful top-level keys: defaults, which is merged into every URL’s options, and urls, which is an array of either plain strings or objects that override the defaults. Everything else about the tool is a consequence of those two keys, so it is worth setting the defaults deliberately once rather than passing flags on the command line where nobody will find them later.

The block below is a complete production defaults section. standard selects the HTML CodeSniffer ruleset — it has no effect on the axe runner, which is scoped by its own rule tags — and runners turns on both engines. timeout is the total budget for loading and testing one URL; the 30-second default is generous for a static page and too tight for a heavy dashboard. chromeLaunchConfig is passed straight through to Puppeteer, which is where the --no-sandbox argument belongs: container runners commonly execute as a user without the kernel capabilities Chrome’s sandbox needs, and --disable-dev-shm-usage avoids the 64 MB /dev/shm that Docker gives a container by default and that Chrome will happily exhaust on a large page.

{
  "defaults": {
    "standard": "WCAG2AA",
    "runners": ["axe", "htmlcs"],
    "timeout": 45000,
    "wait": 500,
    "threshold": 0,
    "includeWarnings": false,
    "includeNotices": false,
    "viewport": { "width": 1280, "height": 900 },
    "hideElements": "#onetrust-consent-sdk, iframe[title='Intercom live chat'], .adsbygoogle",
    "chromeLaunchConfig": {
      "args": [
        "--no-sandbox",
        "--disable-dev-shm-usage",
        "--disable-gpu"
      ]
    }
  },
  "urls": [
    "http://127.0.0.1:4173/",
    "http://127.0.0.1:4173/pricing",
    "http://127.0.0.1:4173/docs/getting-started"
  ]
}

Three of those keys carry more weight than they look. includeWarnings and includeNotices are false on purpose: HTML CodeSniffer emits a large volume of warning and notice results that are advisory rather than failures, and enabling them on a gating job converts a useful signal into a wall of text that nobody reads twice. hideElements takes a comma-separated CSS selector list and applies visibility: hidden to every match before the check runs, which removes the element and its descendants from both engines’ view — the correct tool for a consent banner, a chat widget or an ad slot injected by a vendor whose markup the team cannot change. threshold is the per-URL tolerance for issue count; start it at 0 for a new site so every finding blocks, and only raise it where a legacy template carries debt you have consciously agreed to carry.

Keep the URL list in the config file rather than on the command line even when the list is eventually generated, because the same file then documents the scan for the next person. If the list is long, generate .pa11yci from the sitemap at build time and commit the generator, not the output.

# 2. Discover URLs From a Sitemap Instead of a Hand-Written List

The --sitemap flag replaces the urls array at run time: pa11y-ci fetches the XML, extracts every <loc>, and tests the resulting list under the config’s defaults. That single change is what stops coverage drift, because a new page that does not appear in the sitemap is a page that search engines cannot see either — which makes the sitemap a self-correcting source of truth rather than another list to maintain.

The complication is host mismatch. A production sitemap contains production URLs, and the whole point of a pull-request gate is to test the build under review, which lives on a preview host or 127.0.0.1. --sitemap-find and --sitemap-replace do a literal string substitution on every extracted URL, and --sitemap-exclude takes a regular expression that drops matching URLs before the scan — the right place to remove routes that are meaningless to scan unauthenticated, such as an admin area that only ever renders a redirect.

The invocation below is what runs on a pull request. --sitemap-exclude is a JavaScript regular expression evaluated against the whole rewritten URL, so anchoring it matters; an unanchored admin would also drop /docs/administration.

#!/usr/bin/env bash
set -euo pipefail

npx pa11y-ci \
  --config .pa11yci \
  --sitemap http://127.0.0.1:4173/sitemap.xml \
  --sitemap-find "https://www.acme.io" \
  --sitemap-replace "http://127.0.0.1:4173" \
  --sitemap-exclude "/admin($|/)" \
  --concurrency 4 \
  --json > reports/pa11y-ci.json

One caveat worth knowing before it costs an afternoon: when --sitemap is used, the urls array in .pa11yci is ignored entirely rather than appended to. If a route needs actions — a login, a dialog dismissal — it cannot come from the sitemap, because a sitemap entry carries no options. The pattern that resolves this is two jobs sharing one config: a sitemap sweep for the public surface, and a second pa11y-ci run whose urls array holds the handful of objects that need per-URL setup. Section 4 covers the second job.

# 3. Choose Runners and a Conformance Standard

runners accepts htmlcs, axe, or both. The two engines are not redundant and they are not interchangeable, which is the whole argument for paying the extra second or two per URL to run both.

The axe runner executes the axe-core engine in the page and reports its violations. Its rules are conservative by design and tuned against real production sites, so a finding is almost always actionable, and it is the only one of the two that can reason about computed style — resolved colour contrast, whether an element is actually visible, whether a focusable node is inside an aria-hidden subtree. The standard key does not scope it; the axe runner reports the violations axe considers applicable, and narrowing that set means moving to a dedicated axe run, which is what the axe-core configuration and setup guide exists for.

The htmlcs runner walks the DOM against HTML CodeSniffer’s transcription of the WCAG techniques, and its issue codes name the technique directly — WCAG2AA.Principle1.Guideline1_1.1_1_1.H37 is technique H37, the alt attribute on img. That naming is genuinely useful in an audit conversation, because a code maps to a published technique rather than to a vendor rule name. The cost is precision: htmlcs cannot see computed style, so it hedges, and a meaningful share of its output is warning and notice rather than error. Leave includeWarnings off and the noise problem largely disappears.

The table below is the decision most teams need to make once, at the point they choose what the sweep is for.

Set standard to WCAG2AA unless a specific obligation says otherwise. WCAG2AAA adds criteria most products have not committed to and will bury the AA failures that actually block release; WCAG2A leaves out contrast and several form-labelling techniques that are almost always in scope.

# 4. Per-URL Overrides and Actions for Pages That Need Setup

Any entry in urls can be an object instead of a string, and every key that is legal in defaults is legal in that object, where it wins. This is the mechanism for the two situations a flat sweep cannot express: a page that carries agreed debt and therefore needs a higher threshold, and a page that is not in a testable state until something has been clicked.

actions is a small imperative language interpreted by Pa11y before the check runs. The verbs that matter are navigate to <url>, set field <selector> to <value>, check field <selector>, click element <selector>, wait for element <selector> to be added|removed|visible|hidden, and wait for url|path|fragment to be <value>. Each action runs in order in the same page context, and the accessibility check runs against whatever DOM exists after the last one — which makes the final action the single most important line in the sequence, because it is the assertion that the page is ready.

The second config file below is the companion job referenced in section 2: a short urls array of objects for routes that need setup, sharing the same defaults philosophy but with the per-URL detail a sitemap cannot carry. Environment variables are interpolated by the shell, so this file is generated at run time by envsubst or a two-line Node script rather than committed with a password in it.

{
  "defaults": {
    "standard": "WCAG2AA",
    "runners": ["axe", "htmlcs"],
    "timeout": 60000,
    "threshold": 0,
    "chromeLaunchConfig": { "args": ["--no-sandbox", "--disable-dev-shm-usage"] }
  },
  "urls": [
    {
      "url": "http://127.0.0.1:4173/dashboard",
      "actions": [
        "navigate to http://127.0.0.1:4173/login",
        "set field #email to a11y-bot@acme.io",
        "set field #password to ${A11Y_BOT_PASSWORD}",
        "click element form#signin button[type='submit']",
        "wait for path to be /dashboard",
        "wait for element [data-dashboard-ready] to be added"
      ]
    },
    {
      "url": "http://127.0.0.1:4173/legacy/report-builder",
      "threshold": 6,
      "timeout": 90000,
      "hideElements": ".legacy-grid-vendor-widget",
      "actions": [
        "click element #open-advanced-filters",
        "wait for element #advanced-filters to be visible"
      ]
    }
  ]
}

The threshold: 6 on the second entry is a deliberate, documented allowance, not a shrug. It says: this template currently emits six issues, we have decided not to fix them this quarter, and a seventh will fail the build. That is a much narrower concession than adding the offending code to ignore, which switches a rule off across every URL in the sweep and hides the same defect on every future page that repeats it. Ratchet those per-URL numbers down on a schedule rather than leaving them; the mechanics of doing that safely are in progressive threshold management.

# 5. Invoke the Gate in CI With a JSON Reporter

--json makes pa11y-ci print a machine-readable report to stdout instead of the human log. The shape is stable and small: a total, passes and errors count, plus a results object keyed by URL whose values are arrays of issue objects carrying code, type, message, selector, context and runner. Everything a pull-request annotation needs is in there, including which engine produced each finding — which is exactly what makes deduplication possible.

Two operational rules apply to the exit code. First, treat any non-zero value as failure and never branch on the specific number; the meaningful distinction is over-threshold versus not, and a script that tests for a particular integer will break on a version bump. Second, capture the JSON even when the command fails, which means the redirect has to happen before the shell can abort — hence set +e around the run rather than set -e over it.

#!/usr/bin/env bash
# scripts/pa11y-gate.sh — run the sweep, keep the report, propagate the verdict.
set -uo pipefail
mkdir -p reports

# --concurrency is sized from the runner's CPU count, capped at 4 (see below).
CORES="$(node -p 'require("node:os").availableParallelism()')"
CONCURRENCY="$(( CORES < 4 ? CORES : 4 ))"

set +e   # a threshold breach must not abort the script before the upload step
npx pa11y-ci \
  --config .pa11yci \
  --sitemap http://127.0.0.1:4173/sitemap.xml \
  --sitemap-find "https://www.acme.io" \
  --sitemap-replace "http://127.0.0.1:4173" \
  --sitemap-exclude "/admin($|/)" \
  --concurrency "$CONCURRENCY" \
  --json > reports/pa11y-ci.json
PA11Y_STATUS=$?
set -e

node scripts/pa11y-summary.mjs reports/pa11y-ci.json
exit "$PA11Y_STATUS"   # non-zero here is what blocks the merge

Concurrency is the single setting most likely to make this job untrustworthy. Each concurrent URL is a Chromium page — often a whole browser process — and a GitHub-hosted Linux runner has two vCPUs. Pushing past one page per core does not add throughput; it adds contention, and contention shows up as timeout errors on whichever URLs happened to be in flight, which look identical to real failures in the report. The measurements below come from a 96-URL documentation sweep on a two-vCPU runner with both runners enabled.

The summary script turns the JSON into something a reviewer reads without downloading an artifact. It also does the deduplication the dual-runner setup requires: two findings on the same selector for the same success criterion are one defect, so the script groups by selector and by the criterion parsed out of the issue code.

// scripts/pa11y-summary.mjs — usage: node scripts/pa11y-summary.mjs report.json
import { readFileSync } from 'node:fs';

const report = JSON.parse(readFileSync(process.argv[2], 'utf8'));

// htmlcs codes carry the criterion (…Guideline1_1.1_1_1.H37); axe rule ids do not,
// so axe findings are keyed by rule id and only ever merge with themselves.
const criterion = (code) => code.match(/\.(\d+_\d+_\d+)\./)?.[1] ?? code;

const rows = [];
for (const [url, issues] of Object.entries(report.results)) {
  const errors = issues.filter((i) => i.type === 'error');
  const distinct = new Set(errors.map((i) => `${i.selector}::${criterion(i.code)}`));
  rows.push({ url, raw: errors.length, distinct: distinct.size });
}
rows.sort((a, b) => b.distinct - a.distinct);

console.log(`### pa11y-ci: ${report.passes}/${report.total} URLs under threshold\n`);
console.log('| URL | raw errors | distinct defects |');
console.log('|---|---|---|');
for (const row of rows.filter((r) => r.raw > 0).slice(0, 20)) {
  console.log(`| ${row.url} | ${row.raw} | ${row.distinct} |`);
}

# Pipeline Integration

The job below is the whole gate. It builds the site, serves the build on a fixed port, waits for the port to answer before scanning — a scan that starts before the server is listening produces a wall of connection errors that read like accessibility failures — runs the sweep, writes the summary into the run summary, and uploads the JSON whether the sweep passed or not. The pa11y-authenticated job is the companion from section 4; it is a separate job so that a broken test account cannot take the public sweep down with it.

name: pa11y-sweep
on:
  pull_request:
    paths:
      - 'src/**'
      - 'content/**'
      - '.pa11yci'
      - '.github/workflows/pa11y-sweep.yml'
concurrency:
  group: pa11y-sweep-${{ github.head_ref }}
  cancel-in-progress: true
jobs:
  pa11y-public:
    runs-on: ubuntu-24.04
    timeout-minutes: 25
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - run: npm run build
      - name: Serve the build under test
        run: npx --yes http-server dist -p 4173 --silent &
      - name: Wait for the port to answer
        run: npx --yes wait-on http://127.0.0.1:4173/sitemap.xml -t 60000
      - name: Sweep every sitemap URL
        run: bash scripts/pa11y-gate.sh
      - name: Publish the per-URL summary
        if: always()
        run: node scripts/pa11y-summary.mjs reports/pa11y-ci.json >> "$GITHUB_STEP_SUMMARY"
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: pa11y-ci-report
          path: reports/pa11y-ci.json
          retention-days: 30
  pa11y-authenticated:
    runs-on: ubuntu-24.04
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - run: npm run build
      - run: npx --yes http-server dist -p 4173 --silent &
      - run: npx --yes wait-on http://127.0.0.1:4173/ -t 60000
      - name: Render the authenticated config from the template
        env:
          A11Y_BOT_PASSWORD: ${{ secrets.A11Y_BOT_PASSWORD }}
        run: npx --yes envsub .pa11yci.auth.tmpl .pa11yci.auth
      - name: Scan routes that need a session
        run: npx pa11y-ci --config .pa11yci.auth --json > reports/pa11y-auth.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: pa11y-auth-report
          path: reports/pa11y-auth.json

Two wiring decisions follow from this. First, only pa11y-public should be a required status check while the gate is new; make the sweep advisory for a sprint, watch which URLs fail, and only then promote it — the promotion path is the subject of auto-fail versus warning workflows. Second, the uploaded JSON is the input to everything downstream: PR annotations, a compliance export, or a violation trend, all of which are handled in reporting, dashboards and violation tracking. If Chromium’s behaviour differs between a developer’s machine and the runner, pin the browser and the Node version inside a container image so both environments resolve the same binary.

Two companion pages continue from here. If the question is whether this sweep should be the blocking check at all, axe-core versus Lighthouse CI for pull-request gating works through the choice with the merge gate specifically in view. If the sweep already exists and the team wants to move onto axe-core directly, migrating from pa11y to axe-core in CI covers doing it without losing coverage in the handover.

# Troubleshooting and Flaky-Test Mitigation

Error: Truffler timed out or Chrome failed to launch. Almost always the sandbox. An unprivileged container user cannot create the namespaces Chrome’s sandbox requires, and the --no-sandbox argument in chromeLaunchConfig is the standard workaround. It is a real reduction in isolation, so prefer granting the runner SYS_ADMIN or using a Chrome-provisioned base image where that is possible, and keep --no-sandbox for hosted runners you do not control.

ECONNREFUSED on every URL. The scan started before the server bound its port. Backgrounding a server with & returns immediately; it does not wait for the listener. Always follow it with an explicit readiness poll, and poll a path the build actually produces — polling / on a single-page app can succeed while sitemap.xml is still being written.

Intermittent timeout results on a rotating subset of URLs. This is concurrency contention, not a page problem, and the tell is that the failing URLs change between runs while the count stays roughly constant. Reduce --concurrency, and only then raise timeout. Raising the timeout first masks the contention and doubles the job’s worst case.

A page passes locally and fails in CI on a rule about a rendered element. The viewport differs. Pa11y’s default window is smaller than most laptops, so a responsive layout can collapse into a mobile variant in CI where a navigation menu is hidden behind a toggle and never scanned. Pin viewport in defaults so the scanned layout is deterministic, and add a second URL entry with a mobile viewport if the mobile layout needs its own coverage.

Hydration races on a JavaScript-rendered page. wait is a flat delay applied after load, and a flat delay is a bet on runner speed that eventually loses. Replace it with an actions entry — wait for element [data-hydrated] to be added — so the readiness condition is a fact the application asserts rather than a number somebody guessed. Where the application exposes no such marker, adding one is a smaller change than debugging a flaky gate every fortnight.

Duplicate findings inflating a threshold. With both runners on, one missing alt attribute produces an axe image-alt violation and an htmlcs 1_1_1.H37 error, so a threshold: 3 is really about one and a half defects. Either set thresholds from a dual-runner baseline you actually measured, or deduplicate before comparing — the summary script above does the latter, and its distinct column is the number to negotiate with.

A third-party widget failing a rule nobody can fix. Use hideElements with a selector for the widget’s outermost container, not ignore with the rule code. Hiding the subtree keeps the rule live everywhere else on the site; disabling the rule turns off that check for every page in the sweep, forever, including pages written next year.

# Common Pitfalls

• Leaving urls in the config while passing --sitemap. The flag replaces the array rather than extending it, so a carefully curated entry with actions on it silently stops running.
• Using ignore where threshold or hideElements belongs. An ignored code is invisible site-wide and there is nothing in the report to remind anyone it was ignored.
• An unanchored --sitemap-exclude pattern. admin also matches /docs/administration; anchor with ($|/) and check the resulting URL count against the sitemap’s entry count.
• Turning on includeWarnings for a blocking gate. htmlcs warnings are advisory by construction, and a gate that fails on advice gets bypassed within a week.
• Setting concurrency from the URL count. It should come from the runner’s core count. Ninety-six URLs on two vCPUs still means about four concurrent pages.
• Scanning only the default viewport. A collapsed mobile navigation is unscanned markup, and it is frequently the least accessible markup on the page.
• Treating a raw dual-runner issue count as a defect count. It overstates by twenty to thirty percent, which makes every threshold negotiation start from a wrong number.
• Reading the exit code as a severity signal. pa11y-ci reports over-threshold or not; severity lives in the JSON, and any policy finer than pass/fail has to be computed from it.

# FAQ

Is pa11y-ci still worth running if the team already has axe-core in Playwright? Yes, but for a different job. The Playwright suite is depth-first: it drives scenarios, carries session state, and asserts on states that only exist after interaction. pa11y-ci is breadth-first: it answers “does every published URL still pass” for a hundred documents in one job, which a scenario suite is a clumsy and slow way to express. Teams that run both usually gate on the scenario suite for application routes and on the sweep for content pages.

Why do the two runners report different totals for what looks like the same page? They implement different rule sets against different evidence. The axe runner evaluates computed style and the accessibility tree, so it alone can judge contrast against a resolved background or spot a focusable node inside an aria-hidden subtree. HTML CodeSniffer works from documented WCAG techniques and flags things axe does not model at all, such as a styled paragraph substituting for a heading under WCAG 2.2 SC 1.3.1. Neither total is the true count; the union, deduplicated by selector and criterion, is the closest thing to it.

How should the threshold be set on a legacy site with thousands of existing issues? Measure first, then freeze. Run the sweep once with threshold high enough that nothing fails, take the per-URL error count from the JSON, and write those counts back into the config as per-URL thresholds. The gate then blocks any regression from day one without demanding that anyone fix the backlog, and the numbers become a visible, per-template debt register that can be lowered on a schedule.

Can pa11y-ci scan a preview deployment on a real domain instead of localhost? It can, and the --sitemap-find and --sitemap-replace pair exists precisely for that: point the find string at the production origin and the replace string at the preview origin the deployment platform issued for the branch. Pass any required access token through headers in defaults, keep the token in CI secrets, and be aware that a scan over the public internet is slower and more variable than one over loopback, so give timeout more room.

What happens to actions when a selector never appears? The action fails, Pa11y reports an error for that URL, and no accessibility results are produced for it — the URL counts as a failure rather than silently passing, which is the correct behaviour. It also means a renamed selector in the login form shows up as an accessibility failure on a route that has no accessibility problem, so keep the selectors in actions on attributes the application treats as a contract, such as data-testid, rather than on classes a redesign will change.

# Related

• Web Accessibility Testing Fundamentals & Tool Selection — the parent section on picking and configuring scanners.
• axe-core vs Lighthouse CI for PR Gating — which engine belongs on the blocking path once the sweep exists.
• Migrating from pa11y to axe-core in CI — retiring this sweep onto axe-core without a coverage gap.
• axe-core Configuration & Setup — the engine behind the axe runner, configured directly.
• Progressive Threshold Management — ratcheting the per-URL numbers down sprint by sprint.