# Migrating from pa11y to axe-core Without a Coverage Gap

This guide is part of Pa11y CI Integration, and it addresses the one risk that makes this migration worth planning: the new gate can be green on day one while quietly checking less than the old one did. The mechanics of installing axe-core are trivial. The hard part is proving that every defect class the old job would have caught is still caught by something, and the only honest way to prove it is to run both jobs against the same commits for several weeks and diff their reports until the difference is empty and explained.

# Root Cause

A pa11y-ci job configured with the HTML CodeSniffer runner and an axe-core job do not describe the world in the same vocabulary, and the mismatch is a set difference rather than a rename. HTML CodeSniffer emits identifiers built from the WCAG document itself — WCAG2AA.Principle1.Guideline1_1.1_1_1.H37 names the standard, the principle, the guideline, the success criterion and the specific technique. axe-core emits short rule names like image-alt that were designed around what a browser can prove, not around the structure of the specification. Some codes have an exact counterpart, some map onto several axe rules depending on the element, and some have no axe rule at all because axe deliberately does not attempt checks it cannot decide reliably.

The second mismatch is the gate’s shape. Pa11y counts issues per URL and compares the count to a threshold; axe-core has no count threshold at all, and building one is usually the wrong move because a count treats a minor finding and a critical one as equal. The equivalent construct is a per-route budget expressed in terms of impact — and a threshold: 4 that was chosen two years ago to make a legacy template pass does not translate into any principled impact policy. That number has to be re-derived from the actual violations axe reports on that route, which is one more reason the soak period exists.

The third mismatch is exclusion semantics, and it is the one that silently changes results. Pa11y’s hideElements applies visibility: hidden to matching elements before the check runs, which means the element is still in the DOM but is treated as invisible — so rules about visible content skip it while rules about the tree may not, and descendants inherit the hiding. axe-core’s exclude() removes a selector’s subtree from the scan scope entirely, and the two are not the same thing: a rule such as aria-hidden-focus reasons about focusable content inside hidden subtrees, so hiding an element can create a finding while excluding it cannot. Port each hideElements selector to exclude() and then check whether the violation count on that route moved in a direction you can explain.

# Configuration

Start with a mapping file, checked into the repository as data rather than encoded in a script. Every code the old job has ever reported gets an entry, and the value is either an array of axe rule ids or null with a required note explaining where the check went. null without a note is how coverage gets lost, so the loader rejects it.

// a11y/migration/htmlcs-to-axe.mjs
// Key: the stable prefix of an htmlcs code (the trailing .Fail/.1/.2 varies).
// Value: axe rule ids that cover the same defect, or null plus a note.
export const MAP = {
  'WCAG2AA.Principle1.Guideline1_1.1_1_1.H37': ['image-alt'],
  'WCAG2AA.Principle1.Guideline1_1.1_1_1.H67': ['image-alt', 'role-img-alt'],
  'WCAG2AA.Principle1.Guideline1_3.1_3_1.F68': ['label', 'select-name'],
  'WCAG2AA.Principle1.Guideline1_3.1_3_1.H39': ['empty-table-header'],
  'WCAG2AA.Principle1.Guideline1_3.1_3_1.H43': ['td-headers-attr', 'th-has-data-cells'],
  'WCAG2AA.Principle1.Guideline1_4.1_4_3.G18': ['color-contrast'],
  'WCAG2AA.Principle2.Guideline2_4.2_4_1.H64': ['frame-title'],
  'WCAG2AA.Principle2.Guideline2_4.2_4_2.H25': ['document-title'],
  'WCAG2AA.Principle3.Guideline3_1.3_1_1.H57': ['html-has-lang', 'html-lang-valid'],
  'WCAG2AA.Principle4.Guideline4_1.4_1_2.H91': ['button-name', 'link-name', 'input-button-name'],
  'WCAG2AA.Principle1.Guideline1_3.1_3_1.H42': null,
  'WCAG2AA.Principle1.Guideline1_3.1_3_1.H48': null,
  'WCAG2AA.Principle1.Guideline1_3.1_3_1.H49': null,
};

export const NOTES = {
  'WCAG2AA.Principle1.Guideline1_3.1_3_1.H42': 'styled fake heading — custom axe rule',
  'WCAG2AA.Principle1.Guideline1_3.1_3_1.H48': 'list markup — custom axe rule',
  'WCAG2AA.Principle1.Guideline1_3.1_3_1.H49': 'b/i for stress — manual review checklist',
};

export function lookup(code) {
  // htmlcs appends variant suffixes; match on the longest known prefix.
  const key = Object.keys(MAP).find((k) => code.startsWith(k));
  if (!key) return { key: null, rules: undefined, note: undefined };
  const rules = MAP[key];
  if (rules === null && !NOTES[key]) {
    throw new Error(`unmapped code ${key} has no disposition note`);
  }
  return { key, rules, note: NOTES[key] };
}

The axe side that replaces the pa11y sweep reads its route list and its per-route exclusions from one config object, so the port from .pa11yci is reviewable as a diff rather than as an archaeology exercise. Note the two translations happening here: hideElements becomes exclude(), and the old numeric threshold becomes an explicit allowance keyed by impact.

// a11y/migration/axe-sweep.mjs — the replacement gate, still advisory in the soak.
import { writeFileSync, mkdirSync } from 'node:fs';
import { chromium } from 'playwright';
import { AxeBuilder } from '@axe-core/playwright';

const ORIGIN = process.env.A11Y_ORIGIN ?? 'http://127.0.0.1:4173';
const ROUTES = [
  { path: '/', exclude: ['#onetrust-consent-sdk'], allow: {} },
  { path: '/pricing', exclude: ['#onetrust-consent-sdk'], allow: {} },
  // Was threshold: 4 in .pa11yci — re-derived as a named, per-impact allowance.
  { path: '/legacy/report-builder',
    exclude: ['#onetrust-consent-sdk', '.legacy-grid-vendor-widget'],
    allow: { moderate: 2, serious: 1 } },
];

const browser = await chromium.launch();
const out = [];

for (const route of ROUTES) {
  const page = await browser.newPage();
  await page.goto(ORIGIN + route.path);
  await page.locator('main').waitFor({ timeout: 15000 });

  let builder = new AxeBuilder({ page })
    .withTags(['wcag2a', 'wcag2aa', 'wcag21aa', 'wcag22aa']); // == standard WCAG2AA
  for (const sel of route.exclude) builder = builder.exclude(sel);
  const { violations } = await builder.analyze();

  const over = violations.filter((v) => {
    const budget = route.allow[v.impact] ?? 0;
    return v.nodes.length > budget;
  });
  out.push({ path: route.path, violations, overBudget: over.map((v) => v.id) });
  await page.close();
}

await browser.close();
mkdirSync('out', { recursive: true });
writeFileSync('out/axe-sweep.json', JSON.stringify(out, null, 2));
console.log(`axe: ${out.reduce((n, r) => n + r.overBudget.length, 0)} over budget`);

The reconciler is the piece that makes the migration safe. It reads both reports, projects every pa11y error through the mapping table, and asks a single question per finding: did axe report at least one of the mapped rule ids on the same route? Anything that answers no is either a genuine coverage gap or a missing map entry, and both need a decision before cutover.

// a11y/migration/reconcile.mjs — usage: node reconcile.mjs pa11y.json axe-sweep.json
import { readFileSync } from 'node:fs';
import { lookup } from './htmlcs-to-axe.mjs';

const pa11y = JSON.parse(readFileSync(process.argv[2], 'utf8'));
const axe = JSON.parse(readFileSync(process.argv[3], 'utf8'));

// axe rule ids reported per route path, so comparison is per URL not site-wide.
const axeByPath = new Map(
  axe.map((r) => [r.path, new Set(r.violations.map((v) => v.id))]),
);

const gaps = [];
for (const [url, issues] of Object.entries(pa11y.results)) {
  const path = new URL(url).pathname;
  const axeRules = axeByPath.get(path) ?? new Set();
  for (const issue of issues) {
    if (issue.type !== 'error' || issue.runner !== 'htmlcs') continue;
    const { key, rules, note } = lookup(issue.code);
    if (!key) { gaps.push({ path, code: issue.code, reason: 'no map entry' }); continue; }
    if (rules === null) continue;                    // deliberately routed elsewhere
    if (!rules.some((id) => axeRules.has(id))) {
      gaps.push({ path, code: key, reason: `axe silent on ${rules.join('/')}` });
    }
  }
}

const unique = [...new Map(gaps.map((g) => [`${g.path}|${g.code}`, g])).values()];
for (const g of unique) console.log(`GAP ${g.path} ${g.code} — ${g.reason}`);
console.log(`\n${unique.length} unreconciled findings this run.`);
process.exitCode = 0; // reporting only during the soak; never blocks

The workflow runs the old gate as the required check and the new one alongside it. Both write JSON, the reconciler runs on both, and its output goes into the run summary where reviewers will see it without being blocked by it.

name: a11y-migration-soak
on:
  pull_request:
  schedule:
    - cron: '0 3 * * 1-5'   # a nightly sample widens the code sample fast
jobs:
  soak:
    runs-on: ubuntu-24.04
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - run: npm run build
      - run: npx --yes http-server dist -p 4173 --silent &
      - run: npx --yes wait-on http://127.0.0.1:4173/ -t 60000
      # Still the required check: the old gate keeps the branch protected.
      - name: pa11y-ci (blocking, unchanged)
        run: npx pa11y-ci --config .pa11yci --json > out/pa11y.json
      # The replacement, advisory only until the diff is empty.
      - name: axe-core sweep (advisory)
        if: always()
        continue-on-error: true
        run: node a11y/migration/axe-sweep.mjs
      - name: Reconcile the two reports
        if: always()
        run: node a11y/migration/reconcile.mjs out/pa11y.json out/axe-sweep.json
             >> "$GITHUB_STEP_SUMMARY"
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: migration-soak-${{ github.run_number }}
          path: out/
          retention-days: 90

# Validation

The soak has one exit condition: the reconciler prints zero unreconciled findings, and every null in the mapping table has a disposition that has actually shipped. In practice the first run produces a long list, most of it missing map entries rather than real gaps, and the list shortens quickly as the table fills in. The measured shape of that decline on a 96-URL documentation site was six weeks from forty-one distinct codes to zero.

Reading the reconciler’s output is a triage exercise with four possible answers, and the counts below are what the same site ended with.

The output that authorises the cutover looks like this, and it has to hold for several consecutive runs rather than once, because a single pull request only exercises the pages it touched.

0 unreconciled findings this run.

Once that line is stable, flip the required status check to the axe job, delete the pa11y step, and keep .pa11yci in version control for one release so a revert is a one-line change rather than a reconstruction. The mapping file stays permanently — it is the record of why three checks are no longer automated.

# Edge Cases and Conditional Guards

• The old config already used the axe runner. Then the engine is not changing and only the wrapper is, so the reconciler will report near-zero gaps on day one for those findings. Filter the reconciler to issue.runner === 'htmlcs' as the script above does, because comparing axe-through-pa11y against axe-directly measures nothing except a version difference.
• Per-URL ignore entries in the old config. An ignored code produced no output, so the soak will never observe it and the reconciler cannot tell you whether axe covers it. Read every ignore entry manually, look it up in the mapping table, and confirm the axe rule is either enabled or explicitly disabled with a note — this is the single most common place a coverage gap survives the migration.
• Routes that only existed in the sitemap. A sitemap-driven sweep tests URLs that no hand-written route list contains, so the replacement job’s ROUTES array will be shorter than what pa11y actually scanned unless it is generated from the same sitemap. Compare the URL count in the two reports before trusting any per-route comparison; a gap of zero across ten routes says nothing about the other eighty-six.

# Pipeline Impact

During the soak the pipeline carries two accessibility jobs and one of them is deliberately incapable of failing, which is worth stating in the workflow file rather than leaving as folklore — the continue-on-error and the comment above it are the documentation. Wall-clock cost is roughly the axe sweep’s duration on top of the existing job, because the build and the server are shared; on the reference site that was thirty-four seconds added to a four-minute job. The soak also produces a ninety-day artifact history, which is what makes it possible to answer “did this code ever appear” after the old job is gone.

Cutover is a branch-protection change, not a code change: the axe job becomes required and the pa11y job is removed from the required list first, then deleted from the workflow a release later. Sequencing it that way means a mistake is reversible without a revert commit, and the mechanics of switching a required check belong to pull request gating and branch policies. If the new gate needs to run in warning mode for a while after cutover — which is sensible if its impact policy is stricter than the old thresholds were — the ramp is the subject of auto-fail versus warning workflows. The two checks that became a custom rule are authored against the axe-core configuration the new sweep already loads.

# Common Pitfalls

• Mapping by success criterion instead of by technique. Three codes under SC 1.3.1 can have three different fates; collapsing them to “1.3.1 is covered by axe” is how a real gap gets signed off.
• Treating hideElements and exclude() as equivalent. Hiding leaves the subtree in the DOM and changes which rules apply; excluding removes it from scope. Check the per-route count after each port.
• Converting threshold: 4 into “allow four violations”. The old number counted issues from two runners with overlap. Re-derive the allowance from what axe actually reports on that route, per impact level.
• Ending the soak because the diff was empty once. One pull request touches a handful of pages. Require several consecutive clean runs plus at least one full nightly sweep before removing the old check.
• Deleting null map entries to make the loader quiet. The note requirement exists so that a check leaving the automated set has a named destination; a missing entry is indistinguishable from a forgotten one six months later.

# FAQ

How long should the parallel soak actually run? Long enough for the pull-request traffic to have touched every template, which on most sites is three to six weeks rather than three to six days. The nightly scheduled run in the workflow above shortens it considerably because it sweeps the whole URL list rather than only the pages a pull request happened to modify, so the code sample grows even in a quiet week. Stop when the reconciler has printed zero across at least five consecutive nightly runs.

What if a code has no axe counterpart and writing a custom rule is not worth it? Then it moves to a documented manual check with a named owner and a cadence, and that decision is recorded as the note in the mapping file. This is a real reduction in automated coverage and should be visible as such — the honest framing is that the migration traded three automated checks for better output quality on the other thirty-eight. Pretending the check still happens because the pipeline is green is the outcome the mapping file exists to prevent.

Can the reconciler compare node-level results rather than rule presence? Not reliably, and it is not worth attempting. The two engines produce different selectors for the same element — one reports the failing img, the other may report the enclosing figure — so a node-level join produces false gaps that cost more triage time than they save. Presence of the mapped rule id on the same route is a coarse comparison that answers the actual question, which is whether the defect class is still detected at all.

# Related

• Pa11y CI Integration — the parent guide describing the gate being retired here.
• axe-core vs Lighthouse CI for PR Gating — why axe-core is the engine worth migrating onto.
• CI/CD Integration & Automated Quality Gating — the section covering required checks, thresholds and warning-mode ramps.