# Choosing the Engine That Blocks the Merge

This guide is part of Pa11y CI Integration, and it settles one decision only: which accessibility engine is allowed to turn a pull request red. That is a narrower question than “which tool is better”, and it has a different answer, because a merge gate has requirements that a reporting tool does not — it must return the same verdict twice on the same commit, and its failure message must be enough for the author to act on without opening a second tool. The short version is that axe-core belongs on the blocking path and Lighthouse CI belongs beside it as a trend signal, and the rest of this page is the evidence and the configuration that makes running both cost almost nothing extra.

# Root Cause

A gate that blocks a merge is a claim about the code. When the claim is wrong the cost is not a wasted minute; it is a permanent loss of authority, because the first time a gate blocks a pull request that contains no accessibility change at all, somebody will re-run the job, watch it pass, and quietly conclude the check is decoration. Lighthouse CI’s blocking behaviour fails exactly there, and it fails for a structural reason rather than a bug. The accessibility category is a weighted average of audit scores, most of which are binary, so the aggregate lands on a small number of discrete values — and the audits that feed it depend on rendering that a loaded CI runner does not reproduce identically twice. A contrast audit needs the element painted, an audit on a lazily mounted control needs the control mounted, and under CPU contention some of those conditions are simply not met on run three when they were met on run one. The score moves; the code did not.

axe-core has the opposite profile on the same runner. It evaluates the DOM and the computed accessibility tree at the moment it is called, and it reports a set of violations, each with a rule id, an impact level, a CSS selector for the failing node and the node’s outer HTML. Two runs on the same commit against the same served build return the same set — not the same approximate number, the same set — provided the scan waits for a real readiness signal rather than a timer. That reproducibility is the entire reason it can be a required check. The same property makes its failures cheap to act on: a report that says label failed at form#signup > div:nth-child(2) > input is a one-line fix, whereas an accessibility category that dropped from 0.94 to 0.91 is a research task.

Per-rule control is the third structural difference and the one that decides long-term survivability. axe-core lets a gate name the rules it enforces and the impact levels it blocks on — for instance, block on serious and critical, annotate moderate, ignore nothing silently. Lighthouse CI can assert individual audits at error, which is genuinely useful, but the assertion is on the audit’s pass/fail state and its weighting inside a score rather than on a node-level result, so the granularity available to a policy is coarser. The numeric relationship between the two outputs — why a score of 96 tells you nothing about how many axe violations exist — is a separate question worked through in Lighthouse accessibility score vs axe violation counts; this page assumes the two numbers differ and asks only which one gets veto power.

The five decision axes, measured on the same 24-route preview deployment, come out like this.

The last row deserves a sentence, because preview deployments are where most gates actually run. A freshly deployed preview has an empty CDN cache, so the first request pays origin latency that the second does not. axe-core does not care — it scans whatever DOM exists once the readiness condition is met. Lighthouse’s audits that depend on paint timing do care, which is why Lighthouse CI defaults to multiple runs and a median, and why three runs is the floor rather than a nicety.

# Configuration

The recommendation is axe-core as the required check and Lighthouse CI as an advisory trend signal in the same workflow. The trap is implementing that as two independent jobs, because each job then checks out, installs, builds and serves the application again — roughly ninety-five seconds of duplicated work that has nothing to do with accessibility. The fix is one job that does the setup once and runs the two engines against the same live server, with the Lighthouse step scoped to three representative URLs rather than all twenty-four and marked so its exit status cannot fail the job.

name: a11y-merge-gate
on:
  pull_request:
    paths:
      - 'app/**'
      - 'packages/ui/**'
      - '.github/workflows/a11y-merge-gate.yml'
concurrency:
  group: a11y-merge-gate-${{ github.head_ref }}
  cancel-in-progress: true
jobs:
  gate:
    runs-on: ubuntu-24.04
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - run: npm run build
      - name: Serve the build once for both engines
        run: npx --yes http-server build -p 5188 --silent &
      - run: npx --yes wait-on http://127.0.0.1:5188/ -t 60000
      # Blocking: axe-core across every route. Non-zero exit fails the job.
      - name: axe-core gate
        run: node scripts/axe-gate.mjs
      # Advisory: Lighthouse on three representative URLs, same running server.
      # continue-on-error keeps a score wobble from ever blocking the merge.
      - name: Lighthouse trend
        continue-on-error: true
        run: npx --yes @lhci/cli autorun --config=lighthouserc.trend.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: a11y-gate-artifacts
          path: |
            reports/axe-gate.json
            .lighthouseci/
          retention-days: 30

The Lighthouse configuration that goes with it asserts nothing that can fail. Its job is to produce a number worth plotting, so assertions is empty and numberOfRuns is three because a single run on a shared runner is not a measurement. Uploading to temporary-public-storage is the zero-infrastructure option; a self-hosted server is better once the trend matters enough to keep.

{
  "ci": {
    "collect": {
      "url": [
        "http://127.0.0.1:5188/",
        "http://127.0.0.1:5188/checkout",
        "http://127.0.0.1:5188/account/settings"
      ],
      "numberOfRuns": 3,
      "settings": { "onlyCategories": ["accessibility"] }
    },
    "assert": { "assertions": {} },
    "upload": { "target": "temporary-public-storage" }
  }
}

onlyCategories matters more than it looks: restricting the collection to the accessibility category drops the per-URL cost from about thirty-one seconds to roughly nineteen, because performance tracing and the network-throttling passes are skipped. Three URLs at three runs each is therefore under two minutes, and it overlaps the axe step’s thirty-seven seconds rather than following a second build.

The axe side is a small script rather than the CLI, because a policy expressed in JavaScript is auditable and a policy expressed in shell flags is not. It blocks on serious and critical, prints moderate and minor for information, and writes the full report for the artifact.

// scripts/axe-gate.mjs — blocking axe-core sweep over the served build.
import { writeFileSync, mkdirSync } from 'node:fs';
import { chromium } from 'playwright';
import { AxeBuilder } from '@axe-core/playwright';

const ORIGIN = 'http://127.0.0.1:5188';
const ROUTES = ['/', '/checkout', '/account/settings', '/search?q=chair'];
const BLOCKING = new Set(['serious', 'critical']);

const browser = await chromium.launch();
const context = await browser.newContext({ reducedMotion: 'reduce' });
const report = [];
let blocking = 0;

for (const route of ROUTES) {
  const page = await context.newPage();
  await page.goto(ORIGIN + route);
  // A readiness attribute, not a timer: this is what makes the run repeatable.
  await page.locator('[data-app-ready]').waitFor({ timeout: 15000 });

  const { violations } = await new AxeBuilder({ page })
    .withTags(['wcag2a', 'wcag2aa', 'wcag21aa', 'wcag22aa'])
    .analyze();

  const gating = violations.filter((v) => BLOCKING.has(v.impact));
  blocking += gating.length;
  report.push({ route, violations });

  for (const v of gating) {
    console.log(`::error::${route} ${v.id} (${v.impact}) ${v.nodes[0].target}`);
  }
  for (const v of violations.filter((v) => !BLOCKING.has(v.impact))) {
    console.log(`::notice::${route} ${v.id} (${v.impact}) advisory only`);
  }
  await page.close();
}

await browser.close();
mkdirSync('reports', { recursive: true });
writeFileSync('reports/axe-gate.json', JSON.stringify(report, null, 2));
process.exit(blocking > 0 ? 1 : 0); // the only signal branch protection reads

# Validation

The claim that justifies this split is determinism, so validate it rather than trusting it. Run the axe gate five times against one unchanged build and assert that the sorted list of rule ids and node counts is byte-identical every time. If it is not, the problem is the readiness wait, not the engine, and fixing it is a prerequisite to making the check required.

#!/usr/bin/env bash
# Prove the gate is deterministic before making it a required status check.
set -uo pipefail
npx --yes http-server build -p 5188 --silent &
npx --yes wait-on http://127.0.0.1:5188/ -t 60000

for i in 1 2 3 4 5; do
  node scripts/axe-gate.mjs > /dev/null
  # Fingerprint = every route's rule ids and node counts, order-independent.
  node -e '
    const r = require("./reports/axe-gate.json");
    const fp = r.flatMap(x => x.violations.map(v => `${x.route}|${v.id}|${v.nodes.length}`));
    console.log(fp.sort().join("\n"));
  ' > "run-$i.txt"
done

# Identical output across all five runs means zero unique fingerprints beyond one.
sort -u run-1.txt run-2.txt run-3.txt run-4.txt run-5.txt > union.txt
diff <(sort run-1.txt) union.txt && echo "deterministic: safe to require"

The expected output is a single line, and anything else is a finding worth chasing before the gate becomes required.

deterministic: safe to require

Run the same exercise against lhci autorun with numberOfRuns: 1 and the category score will typically move by one to four points across five attempts on a shared runner. That spread is the number to quote when someone proposes a minScore gate: an assertion whose threshold sits inside the noise band will fail roughly as often as the noise crosses it.

# Edge Cases and Conditional Guards

• A route only reachable with a session. axe-core inherits whatever the browser context carries, so a Playwright storage-state file makes an authenticated route scannable in the same sweep. Lighthouse needs the credential injected into its own Chrome launch, which is enough extra machinery that authenticated routes are usually better left out of the trend set entirely rather than half-configured.
• A single-page application where the first paint is a skeleton. Both engines will happily scan the skeleton. axe-core’s guard is the [data-app-ready] locator in the script above; Lighthouse’s is that its median smooths some of it, which is smoothing rather than correctness — treat a trend number collected on a skeleton as meaningless rather than merely noisy.
• A monorepo where only one package changed. Scoping the axe route list to the affected package keeps the blocking step under a minute, but the Lighthouse trend must keep collecting the same three URLs every run or the series becomes uncomparable. Vary the blocking scope; never vary the trend scope.

# Pipeline Impact

Only one of these two steps should ever appear in branch protection. Make the job that runs scripts/axe-gate.mjs the required check, and leave the Lighthouse step inside it under continue-on-error so a wobbling score produces a yellow annotation and no more — the mechanics of choosing which checks are required and who may bypass them belong to pull request gating and branch policies. The artifacts diverge accordingly: reports/axe-gate.json is the input to a PR annotation, and .lighthouseci/ is the input to a dashboard.

The severity policy is the second lever. Starting at “block on serious and critical” is right for a gate that has to be adopted, and tightening later to include moderate is a policy change rather than a tooling change — worth pairing with the phased approach in progressive threshold management so the tightening lands on a schedule the team agreed to. If the same repository already sweeps its content pages with the breadth-first setup described in the parent guide, keep the two gates separate: the sweep answers “did any published URL regress” and this gate answers “does this diff introduce a serious violation”, and merging them into one required check makes both harder to reason about.

# Common Pitfalls

• Gating on categories:accessibility with a minScore inside the noise band. Measure the spread across five runs first; if the band is three points wide, a threshold three points below the current score is a coin flip.
• Appending || true to the axe step. It converts the only deterministic signal in the pipeline into decoration, and it is invisible in the workflow summary because the job still shows green.
• Building twice. Two jobs that each check out and build turn a three-minute gate into a five-minute one for no additional coverage.
• Leaving numberOfRuns at 1 for the trend. A single Lighthouse run on a shared runner is a sample, not a measurement, and it will make the trend line look like the code is oscillating.
• Letting an unpinned browser drift. A runner image that upgrades Chrome shifts both the audit set and the score, so a trend that spans an image bump has a step change in it that no commit caused.

# FAQ

Can Lighthouse CI ever be the blocking check? It can, in one specific configuration: pin individual audits to error rather than asserting on the category score, pin the browser version, and accept a slower gate. That gives per-audit determinism close to axe’s, because a binary audit either passed or did not. What it does not give is a node-level failure message, so the author still has to open the report to find the element — which is why even teams that gate on Lighthouse audits usually end up adding axe alongside for the message quality rather than the coverage.

Does running both engines mean the same violation blocks twice? No, because only one of them can block. The axe step’s exit code is the sole input to the merge decision, and the Lighthouse step is fenced off with continue-on-error. Both will report the same missing alt attribute, and that is fine — the Lighthouse record exists to move a number on a chart over sprints, not to cast a second vote on this pull request.

What is the argument for keeping Lighthouse at all if it never blocks? A trend line is a different artifact from a gate, and it answers a question the gate cannot: is the product getting more accessible over time, or is the gate merely holding the line at the level it was introduced at? A category score aggregated across three stable URLs, collected on every merge to the default branch, is a serviceable answer to that, and it comes almost free once the build and the server are already running for the blocking step.

# Related

• Pa11y CI Integration — the parent guide on breadth-first sweeps that sit beside this diff-level gate.
• Migrating from pa11y to axe-core in CI — moving an existing sweep onto the engine recommended here.
• Lighthouse CI Baseline Configuration — configuring the trend collection and its assertion states in full.