Adding a Regression Test for Every Fixed Violation

A closed accessibility finding leaves behind a small, precise record: a rule id, a target selector, a failureSummary, and — once a sweep has resolved it — a source location and a fingerprint. That record contains everything needed to write the assertion that keeps the fix in place, and the conversion from one to the other is mechanical enough to be done the same way every time. This guide is part of Regression Prevention After Fixes, and it covers that conversion end to end: turning the report’s brittle selector into a locator that survives refactors, writing an assertion that genuinely fails on the pre-fix code, proving it does by running it against the parent commit, and wiring the check that flags a pull request which closes a rule id without adding a test for it.

The worked example is a critical label finding on the cart quantity stepper at src/cart/QuantityStepper.tsx:28, route /cart, fingerprint 5d0c91a4. The input had no programmatic label; the fix associated the visible product name with it. Everything below is the process that turns that closed finding into a spec nobody has to remember to write.

Root Cause

The report’s target is not a locator, and using it as one is the single most common way these tests rot. axe generates that string with a selector serialiser whose only goal is to uniquely identify a node in the document it just scanned, so it reaches for whatever is available: hashed class names from CSS modules, nth-child positions, and ancestor chains. The finding above arrived as .cart-grid > div:nth-child(3) > div > input. Every segment of that is a hostage. Moving the quantity column left of the price column changes nth-child(3). A CSS-modules upgrade rewrites .cart-grid to .cart-grid__a7f31. Wrapping the input in a tooltip provider adds a div. None of those changes has anything to do with accessibility, and all of them break the selector.

A broken selector would be tolerable if it failed loudly. It does not. AxeBuilder.include() with a selector that matches nothing returns zero violations, and a scoped scan over zero nodes is a green test. So a regression test built on the report’s target does not start failing when the DOM moves — it starts passing unconditionally, and it keeps reporting success for as long as the repository exists. The test looks like coverage in every dashboard and protects nothing.

The failureSummary is the field that tells you what to assert, and it is worth reading rather than skipping. For this finding it reads “Fix any of the following: Form element does not have an implicit (wrapped) <label>; Form element does not have an explicit <label>; aria-label attribute does not exist or is empty”. That is a list of the mechanisms that could satisfy the rule, which means the assertion must not test any one of them. Asserting htmlFor is present would fail the day someone legitimately switches to aria-labelledby. Assert the outcome — the control has a non-empty accessible name — and let the implementation move.

Configuration

Resolve the node once, on the fixed code, and read what it actually exposes. This is the step that produces a durable locator, and it has to happen after the fix because the durable locator is written against the accessible name the fix created. Injecting axe-core and using its own commons gives the same role and name resolution the scanner used, so the values match the report rather than approximating it.

// a11y/locks/resolve-locator.mjs
// Usage: node a11y/locks/resolve-locator.mjs /cart '.cart-grid > div:nth-child(3) > div > input'
// Run against the FIXED code: the durable locator uses the name the fix created.
import { chromium } from 'playwright';
import axe from 'axe-core';

const [route, target] = process.argv.slice(2);
const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto(new URL(route, 'http://localhost:3000').href);
// axe.source is the packaged browser bundle; injecting it gives us the same
// role and accessible-name resolution the scan used to report the finding.
await page.addScriptTag({ content: axe.source });

const node = await page.evaluate((selector) => {
  const el = document.querySelector(selector);
  if (!el) return null;
  return {
    tag: el.tagName.toLowerCase(),
    role: window.axe.commons.aria.getRole(el),
    name: window.axe.commons.text.accessibleText(el).trim(),
    testId: el.getAttribute('data-testid'),
  };
}, target);

await browser.close();

if (!node) {
  console.error(`target did not resolve on ${route}: ${target}`);
  process.exit(1);
}
console.log(node);

// Preference order: role + accessible name, then a test id the fix added.
if (node.role && node.name) {
  console.log(`page.getByRole('${node.role}', { name: '${node.name}' })`);
} else if (node.testId) {
  console.log(`page.getByTestId('${node.testId}')`);
} else {
  // No durable handle exists yet, so creating one is part of the fix.
  console.error('add a data-testid to the repaired element in the fix commit');
  process.exit(2);
}

For this finding the script prints role: 'spinbutton', name: 'Quantity for Trailhead Jacket', and the locator page.getByRole('spinbutton', { name: 'Quantity for Trailhead Jacket' }). Role plus accessible name is the first choice for a reason beyond durability: it is the same pair a screen-reader user navigates by, so a locator that stops resolving is itself evidence that something a user relies on has changed. When the repaired element has no name and no role worth addressing — a decorative wrapper, a container fixed for aria-allowed-attr — add a data-testid in the same commit as the fix, never in the test commit, so the handle the lock depends on is part of the change being reviewed.

From report target to durable locator to lock spec The first panel holds the violation record: rule id label, critical impact, the positional target selector and the failure summary. The second panel holds the values resolved once on the fixed page: role spinbutton, accessible name Quantity for Trailhead Jacket, and a test id. The third panel holds the durable locator written from those values. A band beneath describes the single spec that results, named with the rule id and fingerprint. Three panels, one mechanical conversion id: label impact: critical target: .cart-grid > div:nth- child(3) > div > input failureSummary: no label element resolved once on the fixed page role: spinbutton name: Quantity for Trailhead Jacket testId: cart-qty-1 durable locator getByRole('spinbutton', { name: /^Quantity for / }) survives a grid re-order and a class rename one spec: reach the state, guard the count, assert the name, re-run rule label named "label stays fixed [5d0c91a4]" so a failure in 2027 traces back to this fix The rose selector is the only field in the record that must never reach the spec file.
Only the middle panel requires running anything; the conversion either produces a role-and-name pair or tells you a test id is missing from the fix.

The spec itself has four moving parts and no cleverness. It seeds the state the finding was reported in, guards that the locator resolves to exactly one node, asserts the outcome named by failureSummary, and re-runs the one rule that was violated over the repaired subtree. The name assertion uses a pattern anchored at the start so a product rename does not break the lock while a lost label still does.

// tests/a11y/locks/cart-quantity.lock.spec.ts
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

const FINGERPRINT = '5d0c91a4';

test.describe('@a11y-lock cart quantity stepper', () => {
  test.beforeEach(async ({ page, request }) => {
    // Seeded basket: one line item, so the accessible name is deterministic.
    await request.post('/test/basket', { data: { sku: 'TRL-JKT-01', qty: 2 } });
    await page.goto('/cart');
    await page.getByTestId('cart-line-1').waitFor();
  });

  test(`label stays fixed [${FINGERPRINT}]`, async ({ page }) => {
    // Pattern, not a literal: the product name is data, "Quantity for" is not.
    const qty = page.getByRole('spinbutton', { name: /^Quantity for / });

    // Without this, a drifted locator makes the scan below pass on zero nodes.
    await expect(qty).toHaveCount(1);

    // Assert the outcome from failureSummary, not the mechanism: htmlFor,
    // aria-label and aria-labelledby must all be allowed to satisfy it.
    await expect(qty).toHaveAccessibleName(/^Quantity for \S/);

    const results = await new AxeBuilder({ page })
      .include('[data-testid="cart-line-1"]') // only the repaired subtree
      .withRules(['label'])                   // only the rule that was closed
      .analyze();

    expect(results.violations).toEqual([]);
  });
});

Validation

A regression test that has never failed is a hypothesis. Prove it by running the new spec against the commit immediately before the fix, where it must fail — and must fail for the right reason, which is the locator resolving to zero nodes because the accessible name does not exist yet. A worktree does this without disturbing the working copy.

# From the branch that contains both the fix and the new lock spec.
FIX=$(git rev-parse HEAD)
PARENT=$(git rev-parse HEAD~1)         # the commit before the fix landed

git worktree add ../pre-fix "$PARENT"  # a second checkout, no stashing needed
cp tests/a11y/locks/cart-quantity.lock.spec.ts ../pre-fix/tests/a11y/locks/

cd ../pre-fix
npm ci --silent
npx playwright test tests/a11y/locks/cart-quantity.lock.spec.ts --reporter=list
#   ✘ label stays fixed [5d0c91a4]
#     Expected: 1
#     Received: 0
#     Locator: getByRole('spinbutton', { name: /^Quantity for / })
#   1 failed, exit code 1

cd - && git worktree remove ../pre-fix  # leaves no trace in the main checkout

Two failure signatures at this step mean different things and both are useful. Received: 0 is the expected outcome: the control had no accessible name before the fix, so a role-and-name locator cannot find it, which proves the assertion depends on the fix. Received: 1 followed by a passing axe scan means the spec is not testing what you think — usually because the pattern is loose enough to match something else on the page, or because the subtree include resolved to a different node. Tighten the pattern and re-run before trusting anything.

Then run it on the fix commit, which should be green in under a second, and record the pair of results in the pull-request body. That two-line note is what makes the box on the fix checklist meaningful rather than decorative: “red at a4f1e2 (Received: 0), green at e21d55 (1 passed, 0.9s)”.

Proving the assertion fails on the pre-fix commit A commit line shows a4f1e2 as the parent without the fix, b7c390 as the fix, and e21d55 as the lock spec. Running the new spec in a worktree at the parent commit fails with expected one and received zero from the count guard, which proves the assertion is falsifiable. Running the same spec at the lock commit passes in under a second. The same spec, two commits, two outcomes parent — no fix a4f1e2 the fix b7c390 the lock spec e21d55 run in a worktree at a4f1e2 Expected: 1 Received: 0 the count guard fires first the assertion is falsifiable run the same spec at e21d55 1 passed in 0.9s the fix is why it passes both results go in the PR body A spec that passes at the parent commit is not a regression test; it is a spec about something else.
The worktree keeps the proof cheap enough to do every time: a second checkout, one copied file, one run, then removed.

Edge Cases and Conditional Guards

  • The repaired node needs an interaction to exist. When the finding was reported inside a menu, a step of a wizard or a collapsed panel, the spec must replay that interaction in beforeEach and the locator must be asserted after it. A sweep that records a reachedBy field makes this mechanical; where it does not, read the route’s own test helpers rather than guessing, because a lock that scans the wrong state passes for the wrong reason.
  • The accessible name is entirely data. A row action named “Remove Trailhead Jacket” has no stable prefix if the product changes, and a locator anchored on the data will break on every fixture reset. Add a data-testid in the fix commit and locate by it, then assert the name with a pattern that pins the verb: /^Remove \S/. That keeps the assertion about the label’s existence and shape rather than its content.
  • Several findings on one node. A single input can carry label, aria-allowed-attr and aria-valid-attr-value findings at once. Write one test per rule id inside a shared describe, each with its own withRules call and its own fingerprint in the name, so a later regression names exactly which rule came back rather than reporting a generic failure on a busy element.
  • Shadow DOM boundaries. AxeBuilder.include() accepts an array of frame or shadow selectors, and a report target for a node inside a shadow root arrives as a nested array for that reason. Locate the host by test id, pass the include as the array form, and never flatten it into a single string — the flattened version silently matches nothing.

Pipeline Impact

The lock spec runs inside the tagged Playwright project described in the parent guide, so it adds one route load, one interaction and one single-rule scan — roughly 0.9 seconds — to the pull-request job. That cost only buys anything if the test exists, which is what the second half of this page is for: a check that compares the rule ids a pull request claims to have fixed against the specs it adds.

The claim comes from a commit trailer, A11y-Fixes: label@5d0c91a4, one line per closed finding. The guard reads the trailers between the merge base and HEAD, reads the lock spec files the branch touched, and requires that some changed spec mentions both the rule id and the fingerprint. Matching on the fingerprint as well as the rule id is what stops a rename or an unrelated edit to an existing spec from satisfying a fresh claim.

// a11y/locks/fix-has-test.mjs
// Every rule id claimed fixed in a trailer needs a spec naming its fingerprint.
import { execFileSync } from 'node:child_process';

const base = process.argv[process.argv.indexOf('--base') + 1];
const git = (...args) => execFileSync('git', args, { encoding: 'utf8' });

// 1. What this branch claims to have closed.
const claimed = new Map(); // rule id -> fingerprint
for (const line of git('log', `${base}...HEAD`, '--pretty=%B').split('\n')) {
  const trailer = line.match(/^A11y-Fixes:\s*([a-z0-9-]+)@([0-9a-f]{6,})\s*$/);
  if (trailer) claimed.set(trailer[1], trailer[2]);
}
if (claimed.size === 0) {
  console.log('no A11y-Fixes trailers; nothing to enforce');
  process.exit(0);
}

// 2. The lock specs this branch added or changed, read at HEAD.
const specs = git('diff', '--name-only', `${base}...HEAD`)
  .split('\n')
  .filter((file) => /\.lock\.spec\.[tj]s$/.test(file));
const specText = specs.map((file) => git('show', `HEAD:${file}`)).join('\n');

// 3. A claim is satisfied only when both identifiers appear in a changed spec.
const missing = [...claimed].filter(
  ([rule, fingerprint]) =>
    !(specText.includes(rule) && specText.includes(fingerprint)),
);

for (const [rule, fingerprint] of missing) {
  // GitHub renders this as an annotation on the run, next to the diff.
  console.log(`::error::${rule}@${fingerprint} was closed with no lock spec`);
}
console.log(`${claimed.size} claimed, ${claimed.size - missing.length} locked`);
process.exit(missing.length === 0 ? 0 : 1);

Register the guard as its own required status check rather than folding it into the test job, as covered in requiring accessibility status checks in branch protection, because its failure has a different remedy from a failing test: the author adds a spec rather than fixing code. When the annotations are also being forwarded to a review channel, emit them in the shape described in structuring JSON violation output for Slack and GitHub annotations so a missing lock is distinguishable from a new violation. The structural half of the same fix — the subtree snapshot that catches drift no rule reports — is covered in snapshot testing accessibility trees to prevent regressions, and the shared Playwright harness these specs run in is set up in integrating axe-core Playwright into an existing project.

Matching claimed fixes against added lock specs Four trailers claim closed findings: label at 5d0c91a4, aria-input-field-name at 7f21ac0b, button-name at 1c88d0f2 and duplicate-id-aria at 93ba4f10. Three changed lock specs match the first three claims. The fourth claim matches nothing, so the guard posts a review annotation and exits non-zero. A11y-Fixes trailers (4) lock specs changed (3) label@5d0c91a4 aria-input-field-name@7f21ac0b button-name@1c88d0f2 duplicate-id-aria@93ba4f10 cart-quantity.lock.spec.ts saved-search.lock.spec.ts toolbar-name.lock.spec.ts no spec names 93ba4f10 annotation posted, job exits 1 A claim is satisfied only when a changed spec contains both the rule id and the fingerprint. Editing an existing spec therefore cannot launder a new claim through the guard.
The guard does not judge whether the test is good — that is review's job — it only makes the absence of one impossible to miss.

Common Pitfalls

  • Pasting the report’s target into the spec, which produces a locator that stops resolving and starts passing rather than failing.
  • Omitting the count guard, so a scoped scan over zero nodes reports success for years.
  • Asserting the mechanism (htmlFor is present) instead of the outcome (the control has a name), which breaks the first time someone switches to aria-labelledby for a good reason.
  • Adding the data-testid in the test commit rather than the fix commit, so the lock depends on a change nobody reviewed as part of the fix.
  • Skipping the pre-fix run. Without it there is no evidence the assertion depends on the fix, and a tautological test is indistinguishable from a real one in every report.
  • Matching the guard on rule id alone, which lets an unrelated edit to an existing spec satisfy a brand-new claim.
  • Writing the trailer by hand from memory. Copy the rule id and fingerprint from the sweep’s plan so the strings match exactly; a typo reads as a missing lock.

FAQ

Why not generate these specs automatically from the report? Generation is fine for the boilerplate and useless for the parts that matter. A script can emit the route, the rule id, the fingerprint and the file name, and doing so saves real time on a sweep that healed thirty findings. What it cannot do is choose the durable locator, decide which part of an accessible name is data, or replay the interaction that makes the node exist — and those three decisions are exactly what separates a lock from a test that passes forever. Generate the skeleton, then resolve and edit each one by hand.

How many of these accumulate before the suite becomes a burden? Slower than teams expect, because each spec is one route load and one single-rule scan. A suite of 37 locks across 14 components runs in under a minute on two workers, and the marginal cost of the thirty-eighth is under a second. The real limit is not runtime but attention: locks grouped into one file per component, each with a shared beforeEach, stay readable, while 37 separate files with 37 copies of the same setup do not. When a component is deleted, delete its lock file in the same pull request.

What if the violation was fixed long ago and there is no fingerprint or parent commit? Write the lock against the current code and prove it differently: break the fix locally — delete the htmlFor, remove the aria-labelledby — confirm the spec goes red, then revert. It is the same falsifiability check with a hand-made counterexample instead of a historical one. Generate a fingerprint at that point from the rule id and the durable locator so the spec name and any future trailer have something stable to refer to, and note in the commit that the lock was added retroactively.

Back to Regression Prevention After Fixes