GitHub Actions a11y Pipeline Setup
An accessibility scan is about twenty lines of JavaScript. Everything expensive about running one on every pull request is the workflow wrapped around it: which events start it, what build it runs against, how many copies run at the same time, where the evidence lands, and which single job is allowed to hold the merge. This guide is part of CI/CD Integration & Automated Quality Gating, and it treats the workflow as the unit of design — a four-job graph whose stages carry deliberately different permissions, different failure semantics and different retry behaviour.
Problem Statement
The workflow that most teams end up with is one job with eleven steps. It checks out, installs, builds, starts a server, scans, and calls a script that both prints results and exits non-zero. That shape works on the day it is written and then fails in five predictable ways. The scan runs against a dev server whose hot-reload overlay changes computed styles, so color-contrast results differ from production. The build is repeated inside every matrix leg, so a two-viewport matrix pays for the build twice. The job needs pull-requests: write in order to comment, which means the whole job — including anything that executes repository JavaScript — runs with a token that can write to the pull request. A force-push leaves two runs in flight and the older one reports its status last. And because the reporting and the failing happen in the same step, a token permission error on a fork pull request takes the gate down with it.
Splitting the work into build, scan, report and gate fixes all five, and it costs one artifact upload plus roughly forty seconds of extra runner time. The build happens once and every shard downloads the same immutable dist/. The scan jobs need no token at all. Only report gets write access to the pull request, and it is allowed to fail without changing the merge decision. Only gate is registered in branch protection, and it is a single job with a stable name — which matters, because a required check whose name contains matrix values changes every time the matrix does.
Key implementation targets:
- A trigger set that covers pull requests, trunk pushes and manual reruns, with
pull_request_targetdeliberately excluded from any workflow that executes contributor code. - A
permissionsblock that iscontents: readat workflow level, raised topull-requests: writeon exactly one job. - A
buildjob that produces one artifact consumed by every scan shard, so the bytes under test are identical across the matrix. - A
scanmatrix over viewport widths and route groups, withfail-fast: falseso one broken group does not hide the others. - Two caches with different keys: the npm cache keyed on the lockfile, and the Playwright browser cache keyed on the resolved browser version.
- Artifact upload of the per-shard JSON report and the Playwright trace, retained long enough to debug a Friday failure on Monday.
- A
concurrencygroup that cancels superseded pull-request runs but never cancels a trunk run. - One job —
gate— whose exit code is the required status check.
Prerequisites
1. The Trigger and Permissions Block
Three events cover every legitimate reason to scan. pull_request is the feedback loop; it gives the workflow a merge-preview commit, which is the right thing to test because it is what will land. push on the trunk is the baseline; it is the only run whose numbers should be written to a trend store, because a pull-request run measures a commit that may never exist again. workflow_dispatch is the escape hatch for reruns after an infrastructure failure and for scanning a single route group while debugging a rule, which is why it takes an input rather than always scanning everything.
pull_request_target is the one to leave out. It exists so that workflows can access secrets and a writable token on pull requests from forks, and it achieves that by running the workflow definition from the base branch with full repository context. The moment such a workflow checks out the pull request head — or runs npm ci, which executes lifecycle scripts from the contributor’s package.json — arbitrary code from an untrusted branch is running next to a token that can push commits and edit pull requests. An accessibility scan needs to execute the contributor’s build, so it is exactly the wrong workload for that trigger. The correct fork story is the opposite: run the scan under plain pull_request with a read-only token, publish results through the run summary, and accept that the sticky comment will be missing on fork pull requests.
paths filtering keeps the workflow off documentation-only changes, and the workflow file itself must be in the list or a change to the pipeline will not be validated by the pipeline. The concurrency expression deserves care: keying on the pull-request number groups all runs for that pull request, while keying on github.ref for pushes keeps trunk runs in their own group. Making cancel-in-progress conditional is the detail most workflows miss — cancelling a superseded pull-request run saves minutes, but cancelling a trunk run loses a baseline data point permanently.
# .github/workflows/a11y.yml
name: a11y
on:
pull_request:
branches: [main]
paths:
- 'apps/web/**'
- 'packages/ui/**'
- 'scripts/a11y/**'
- '.github/workflows/a11y.yml' # the pipeline gates changes to itself
push:
branches: [main] # trunk runs feed the trend store
workflow_dispatch:
inputs:
group:
description: 'Route group: marketing, app-shell, checkout, or all'
default: all
type: string
permissions:
contents: read # every job inherits read-only by default
concurrency:
# One group per pull request; trunk pushes get their own group per ref.
group: a11y-${{ github.event.pull_request.number || github.ref }}
# Never cancel a trunk run: its numbers are the baseline.
cancel-in-progress: ${{ github.event_name == 'pull_request' }}The concurrency group earns its keep during the review loop, when an author force-pushes a fixup within a minute of the previous push. Without a group, both runs continue to completion and both report a status for their own head commit; the pull request shows whichever finished last, and on a busy runner pool that is frequently the older one, so a fixed violation keeps failing and a reviewer starts re-running jobs by hand. Worse, two scans on the same repository can contend for the same artifact names when the second run reaches upload while the first is still writing. Cancelling the superseded run makes the head commit the only thing with a status and halves the runner minutes an average pull request consumes.
2. Build and Serve the App Under Test
Scan the production build. A dev server is a different application: it ships an error overlay, injects unminified CSS with different cascade order, adds framework devtools hooks, and in several frameworks renders an extra root element that owns a role. Contrast results in particular diverge, because minified builds inline critical CSS that the dev server serves as a separate stylesheet applied a frame later. Building once and serving static files also removes an entire class of flakiness: there is no compilation happening while the scanner walks the DOM.
Building once means the build is its own job, and its output crosses the job boundary as an artifact named after the commit SHA. The alternative — rebuilding inside each matrix leg — is not merely slower; it lets two shards test two different bundles if a cache behaves differently on one runner, which produces a violation that reproduces on the 375-pixel shard and not the 1440-pixel one for reasons that have nothing to do with viewport width.
Two caches matter here and they have different keys. The npm cache is handled by setup-node and keys on the lockfile, so it invalidates when dependencies change. The browser binary cache must key on the resolved Playwright version rather than the lockfile, otherwise every unrelated dependency bump throws away a 180 MB download; the pattern and its trade-offs are worked through in caching axe-core browser binaries in CI containers. Note the asymmetry in the restore path: a cache hit restores the browser bundle but not the apt packages Chromium links against, so a hit still needs install-deps.
jobs:
build:
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20.18.1
cache: npm
cache-dependency-path: package-lock.json
- run: npm ci
- run: npm run build --workspace apps/web
- name: Publish the bytes every shard will scan
uses: actions/upload-artifact@v4
with:
name: web-dist-${{ github.sha }}
path: apps/web/dist
retention-days: 3 # only needed until the scan shards finish
if-no-files-found: error # a silent empty artifact scans as a blank pageThe serving half lives inside the scan job, and it has one non-obvious failure mode. A background process started with & survives between steps, but if its stdout is still attached to the step’s pipe the step can hang waiting for the stream to close. Redirect both streams to a file, then poll the URL rather than sleeping — a fixed sleep 5 is either wasted time or an intermittent failure depending on how loaded the runner is.
# scripts/a11y/serve.sh — start the static server and block until it answers.
set -euo pipefail
npx --yes http-server apps/web/dist -p 4173 --silent > server.log 2>&1 &
echo "$!" > server.pid # kept so a later step can stop it deliberately
for attempt in $(seq 1 60); do
# --fail turns a 4xx/5xx into a non-zero exit; -s keeps the log quiet.
if curl -sf -o /dev/null "http://127.0.0.1:4173/"; then
echo "server answered on attempt ${attempt}"
exit 0
fi
sleep 1
done
echo "server never answered; last 40 lines of server.log:" >&2
tail -n 40 server.log >&2
exit 13. The Scan Step
The matrix has two axes and they answer different questions. Viewport width changes which rules can fail at all: reflow, target size, focus-visible behaviour on hover-only controls, and any contrast failure that only exists in the mobile navigation drawer. Route groups change what gets scanned and who owns the failure — grouping routes by owning team means a failing shard names its owner in the job title before anyone opens the report. Six shards over 3 groups and 2 widths is the shape that fits comfortably inside a 20-minute job for a mid-sized application; beyond about a dozen route groups the axis wants to become a dynamic matrix generated from a manifest, which is covered in sharding axe-core scans across parallel CI jobs.
fail-fast: false is mandatory rather than a preference. The default cancels sibling legs on the first failure, so a critical violation in the checkout group hides whatever the marketing group would have reported, and the author fixes one thing per push instead of everything at once. Artifact names must be unique per shard as well: upload-artifact v4 rejects a second upload to the same name instead of merging into it, so a11y-${{ matrix.group }}-${{ matrix.width }} is a correctness requirement, not a nicety.
scan:
needs: build
runs-on: ubuntu-24.04
timeout-minutes: 20
strategy:
fail-fast: false # one broken group must not hide the others
matrix:
width: [375, 1440]
group: [marketing, app-shell, checkout]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20.18.1
cache: npm
- run: npm ci
- uses: actions/download-artifact@v4
with:
name: web-dist-${{ github.sha }}
path: apps/web/dist
- name: Resolve the Playwright version for the cache key
id: pw
run: |
version=$(node -p "require('@playwright/test/package.json').version")
echo "version=${version}" >> "$GITHUB_OUTPUT"
- uses: actions/cache@v4
id: browsers
with:
path: ~/.cache/ms-playwright
key: pw-${{ runner.os }}-${{ steps.pw.outputs.version }}
- if: steps.browsers.outputs.cache-hit != 'true'
run: npx playwright install --with-deps chromium
- if: steps.browsers.outputs.cache-hit == 'true'
run: npx playwright install-deps chromium # apt libs are never cached
- run: bash scripts/a11y/serve.sh
- name: Scan this shard
run: node scripts/a11y/scan.mjs
env:
A11Y_WIDTH: ${{ matrix.width }}
A11Y_GROUP: ${{ inputs.group == 'all' && matrix.group || inputs.group }}
- uses: actions/upload-artifact@v4
if: always() # a crashed scan still has a trace worth keeping
with:
name: a11y-${{ matrix.group }}-${{ matrix.width }}
path: |
a11y-out/*.json
a11y-out/*.zip
retention-days: 14The scan script itself stays deliberately dumb: it visits each route in the group, waits for a signal the application controls, runs axe, and writes one JSON file plus one trace per route. It makes no decisions about failure — that belongs to the gate job — so the scan step exits zero unless the browser itself broke. Wiring axe into an existing Playwright suite instead of a standalone script is covered in integrating axe-core Playwright into an existing project.
// scripts/a11y/scan.mjs — one shard: one width, one route group.
import { mkdirSync, writeFileSync } from 'node:fs';
import { chromium } from 'playwright';
import AxeBuilder from '@axe-core/playwright';
import { ROUTE_GROUPS } from './routes.mjs';
const width = Number(process.env.A11Y_WIDTH ?? 1440);
const group = process.env.A11Y_GROUP ?? 'marketing';
const base = 'http://127.0.0.1:4173';
const routes = ROUTE_GROUPS[group];
if (!routes) throw new Error(`unknown route group: ${group}`);
mkdirSync('a11y-out', { recursive: true });
const browser = await chromium.launch();
const context = await browser.newContext({
viewport: { width, height: 900 },
// Kills CSS transitions so a mid-animation opacity never fails contrast.
reducedMotion: 'reduce',
});
await context.tracing.start({ screenshots: true, snapshots: true });
const page = await context.newPage();
const results = [];
for (const route of routes) {
await page.goto(`${base}${route}`, { waitUntil: 'load' });
// The app sets this attribute in its own post-hydration effect.
await page.waitForSelector('body[data-hydrated="true"]', { timeout: 15_000 });
const run = await new AxeBuilder({ page })
.withTags(['wcag2a', 'wcag2aa', 'wcag21aa', 'wcag22aa'])
.analyze();
results.push({ route, width, group, violations: run.violations });
}
await context.tracing.stop({ path: `a11y-out/trace-${group}-${width}.zip` });
await browser.close();
const file = `a11y-out/axe-${group}-${width}.json`;
writeFileSync(file, JSON.stringify({ group, width, results }, null, 2));
console.log(`wrote ${file} for ${routes.length} route(s)`);4. The Report and Annotate Step
This is the only job that gets a write scope, and it gets exactly one: pull-requests: write. It downloads every shard artifact with a glob pattern, merges them, writes a table to the run summary, and updates a single sticky comment. It also carries continue-on-error: true, because a reporting failure is an inconvenience while a false green gate is a bug — and the two must not share a fate. On a fork pull request the comment API returns 403 and this job goes yellow; the run summary still renders because writing to $GITHUB_STEP_SUMMARY needs no token at all. The mechanics of the sticky comment, the annotation limits and the fork fallback are the subject of annotating pull requests with axe-core violation comments.
report:
needs: scan
if: always() && github.event_name == 'pull_request'
runs-on: ubuntu-24.04
timeout-minutes: 10
continue-on-error: true # reporting never decides the merge
permissions:
contents: read
pull-requests: write # the only elevated scope in the workflow
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20.18.1
- uses: actions/download-artifact@v4
with:
pattern: a11y-* # collects all six shard artifacts
merge-multiple: true
path: a11y-out
- name: Summarise and update the sticky comment
run: node scripts/a11y/report.mjs a11y-out
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PR_NUMBER: ${{ github.event.pull_request.number }}5. The Gate Step
The gate is a separate job with a read-only token, no network calls beyond the artifact download, and one responsibility: decide. It reads the merged shard reports, filters to the blocking impacts, counts failing nodes, prints a line per finding naming the rule ID and the selector, and exits accordingly. It runs if: always() so a cancelled or errored shard cannot produce a green gate by omission — a missing shard file is itself a failure condition, which is the check that catches a scan job that died before writing anything.
Keeping the gate out of the reporting job also keeps the required check honest across permission boundaries. A required check must be able to pass or fail identically for a maintainer’s branch and a contributor’s fork; the moment the deciding job needs a write token, fork pull requests get a different outcome for a reason unrelated to accessibility. The filtering logic, node counting and the expiring allowlist live in blocking pull requests on critical accessibility violations, and the first end-to-end version of this workflow for a team with nothing in place yet is in configuring GitHub Actions for automated WCAG checks.
gate:
needs: scan
if: always() # a cancelled shard must not read as a pass
runs-on: ubuntu-24.04
timeout-minutes: 5
permissions:
contents: read
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20.18.1
- uses: actions/download-artifact@v4
with:
pattern: a11y-*
merge-multiple: true
path: a11y-out
- name: Enforce the accessibility budget
run: node scripts/a11y/gate.mjs a11y-out --expect-shards 6Pipeline Integration
The workflow exposes exactly one contract to the rest of the engineering process: the a11y / gate check. Register that name in branch protection and nothing else — matrix legs appear as a11y / scan (375, marketing) and their names change whenever the matrix does, so requiring them turns a routine matrix edit into a set of permanently pending checks on every open pull request. The mechanics of registering it, including what happens to pull requests opened before the check existed, are in requiring accessibility status checks in branch protection.
Artifacts are the audit trail. The per-shard JSON is the machine-readable record and the trace is the human one: opening a trace shows the DOM snapshot, the computed styles and the network log at the moment the scan ran, which settles the “it does not reproduce locally” conversation in about ninety seconds. Fourteen days of retention covers a two-week sprint; the build artifact needs three, because nothing reads it after the scan shards finish. Trend storage is a different concern with a different lifetime and belongs in the destination described by reporting, dashboards and violation tracking, fed only by trunk runs.
Rollout order matters more than configuration. Land the workflow with the gate script in warning mode — printing findings and exiting zero — and leave it there long enough to see what a normal week produces, which is the soak period argued for in auto-fail versus warning workflows. Then flip the exit code, then add the required check. Doing those three things in one pull request means the first infrastructure hiccup gets read as “the accessibility gate is broken”, and the team’s response to that is to remove the required check rather than fix the flake.
Troubleshooting and Flaky-Test Mitigation
The scan step reports connection refused. The server process either never started or died. Check server.log, which the serve script writes and the artifact upload can collect: the usual causes are a port already bound by a previous step, a missing dist/ because the download-artifact step silently produced an empty directory, and a static server that exits immediately when given a path that does not exist. The if-no-files-found: error on the build upload catches the second case at its source.
A step hangs forever with no output. A background process holding the step’s stdout open is the classic cause. Redirecting to a file, as the serve script does, removes it. Set timeout-minutes on every job regardless; without it a hung job burns the full six-hour default before the workflow gives up.
Chromium fails to launch after a cache hit. The browser bundle restored from the cache but the shared libraries it links against did not, because those are apt packages outside ~/.cache/ms-playwright. Running install-deps on the cache-hit path fixes it. The error text mentions missing libnss3 or libatk and looks nothing like a cache problem.
Contrast violations appear only in CI. The runner lacks the webfont, so text renders in a fallback face at a different weight and the measured ratio changes. Self-host the font files in the build rather than loading them from a third-party origin, and confirm the font actually loaded by asserting document.fonts.status === 'loaded' before the scan. Pinning the whole environment in a container removes this class entirely, which is the argument made in Docker-based pipeline execution.
A shard passes and its sibling fails on the same route. Usually a genuine viewport-dependent failure — a drawer that only exists below 768 pixels, a target-size failure on a compact toolbar. Before assuming flakiness, open both traces and compare the DOM snapshots; if the markup is identical, the difference is timing rather than width, and the hydration wait needs a stronger signal.
Violations appear and disappear between reruns on an unchanged commit. Something is animating or loading late. reducedMotion: 'reduce' in the browser context removes transition-driven contrast noise; a lazily rendered banner needs an explicit wait for its own marker rather than a longer global timeout. Waiting on route-level signals rather than network idle is covered in waiting for route transitions before an axe scan.
Artifact upload fails with a conflict. Two shards used the same artifact name, which upload-artifact v4 refuses rather than merging. Include every matrix dimension in the name. The symmetric mistake is downloading with pattern: but forgetting merge-multiple: true, which nests each artifact in its own subdirectory and leaves the merge script finding nothing.
Common Pitfalls
- Reaching for
pull_request_targetto make the sticky comment work on fork pull requests, which trades a cosmetic gap for a repository-write vulnerability. - Granting
pull-requests: writeat workflow level, so the job that runs the contributor’s build and install scripts holds a writable token. - Rebuilding inside every matrix leg, which doubles cost and allows two shards to test two different bundles.
- Leaving
fail-fastat its default, so the first failing shard cancels its siblings and the author sees one problem per push. - Keying the browser cache on the lockfile hash, which discards a 180 MB download on every unrelated dependency bump.
- Requiring matrix job names in branch protection, so editing the matrix leaves every open pull request with a permanently pending check.
- Setting
cancel-in-progress: trueunconditionally, which cancels trunk runs and leaves gaps in the baseline series. - Uploading the JSON report but not the trace, which leaves nothing to inspect when a violation does not reproduce on a laptop.
- Letting the scan step own the exit code, so a reporting step that runs after it never executes and the pull request shows a failure with no explanation.
FAQ
Should the scan run against a preview deployment instead of a locally served build? A preview deployment is closer to production and tests the real CDN, redirects and headers, which makes it valuable — but it introduces a dependency on a third-party deploy finishing before the scan starts, and that dependency is the most common source of pipeline flakiness in practice. Serving the build inside the job keeps the workflow self-contained and reproducible; if the environment differs enough to matter, run the local scan as the gate on pull requests and a preview scan on a schedule.
How long does a four-job workflow take compared with one job? On a mid-sized application the split adds one artifact round-trip per shard, which is roughly 15 to 40 seconds, and removes a duplicated build per shard, which is usually 60 to 180 seconds. With six shards the split is faster in wall-clock terms and cheaper in runner minutes, because the build runs once instead of six times. The gain grows with the matrix size and disappears entirely at a single shard, where one job is the right answer.
Can the gate job run before all the shards finish?
No, and it should not try. It declares needs: scan, which in a matrix means every leg, and it asserts the expected shard count so a missing file is a failure rather than a silent pass. Partial gating sounds attractive for fast feedback but produces a check that flips from failing to passing as later shards arrive, which is worse than waiting ninety seconds.
What happens on a pull request from a fork with this setup?
The build, scan and gate jobs all run normally, because they need nothing but a read-only token. The report job attempts its comment update, receives a 403, and goes yellow without changing the merge decision because of continue-on-error. Reviewers get the full findings from the run summary, which is written with a shell redirect and needs no permissions, so the only thing missing is the convenience of the comment appearing in the conversation.
Is workflow_dispatch worth the extra input plumbing?
Yes, for two reasons that show up within the first month. Infrastructure failures — a registry timeout, a cancelled runner — need a rerun that does not require pushing an empty commit, and debugging a noisy rule needs a way to scan one route group repeatedly without waiting on the full matrix. The input costs six lines and one expression in the scan job’s environment block.
Related
- Configuring GitHub Actions for Automated WCAG Checks — the single-file starting point before the workflow is split into jobs.
- Annotating Pull Requests with axe-core Violation Comments — what the report job does with the merged shard output.
- Blocking Pull Requests on Critical Accessibility Violations — the filtering, node counting and allowlist rules inside the gate job.
- Pull Request Gating & Branch Policies — where the gate’s status check becomes a merge requirement.
- Monorepo Parallel Test Sharding — growing the matrix past a handful of hand-listed route groups.