Snapshot Testing Accessibility Trees to Prevent Regressions

An aria-dialog-name finding on the refund dialog in src/orders/RefundDialog.tsx was closed in April by pointing aria-labelledby at the dialog’s <h2>. The rule-level lock on that fix asserts the dialog still has a name. It says nothing about the heading dropping to level 4, about the reason field leaving the group that gave it context, or about the confirmation status region disappearing from the tree entirely — three changes that alter what a screen-reader user is told and leave every axe rule green. This guide is part of Regression Prevention After Fixes, and it covers the second lock in that set: a committed snapshot of the repaired subtree whose only job is to fail when the shape of a landed fix drifts.

Keep this separate from the neighbouring technique. Asserting accessibility tree names with Playwright snapshots is about authoring name and role expectations in the first place — deciding what a component’s tree should contain and writing that down deliberately. This page starts later, with a violation that has already been closed, and treats the snapshot as an artifact whose value is measured entirely by whether it is still trustworthy in eight months.

Root Cause

The default instinct is to snapshot the page the fix was found on, and it is the reason most of these files end up worthless. The refund route’s tree is 268 nodes: a banner, a primary navigation, an order table, a support launcher, a footer, and — somewhere inside it — the fourteen nodes of the repaired dialog. Five teams can change that file. Over the six weeks after the fix landed, the page-level snapshot was regenerated twenty-three times; nineteen of those regenerations changed nothing inside the dialog.

That ratio is what destroys the lock, and it does so through the reviewer rather than through the tooling. A diff of two hundred lines from a copy change in the promotional banner cannot be read, so it gets regenerated with an update flag and approved. Once that has happened four or five times, the update flag becomes the standard response to a red snapshot, and the twentieth failure — the one where the dialog’s heading actually moved — is laundered into the expected value by the same reflex. A blanket-updated snapshot does not merely stop catching regressions; it records them as intended behaviour, which is worse than having no file at all.

Volatile content produces the same outcome by a different route. Accessible names absorb subtree text, so the dialog’s own tree contains an order number, a currency total and a relative timestamp — “Refund order #48219”, “Refund 48.20 GBP”, “requested 2 minutes ago”. A snapshot that pins those exact strings fails on the next fixture reset, on a slower runner, and on any machine with a different locale, and a lock that fails for reasons unrelated to accessibility is a lock that gets deleted within a sprint. The two failure modes are opposites — too much scope and too much precision — and both are fixed before the file is committed rather than after it starts complaining.

Snapshot churn at page scope versus subtree scope Above a six-week axis, rose chips record five, four, six, three, three and two updates to the whole-page snapshot, totalling twenty-three, nineteen of which were blanket regenerations. Below the axis, two teal chips mark the only updates to the subtree-scoped snapshot: the week the fix landed and the week a close button was added. Six weeks after the refund-dialog fix landed whole-page snapshot: 23 updates, 19 of them blanket regenerations 5 4 6 3 3 2 week 1 week 2 week 3 week 4 week 5 week 6 scoped to the 14-node dialog subtree: 2 updates, both explained in the body fix landed close button added Nineteen of the twenty-three page-level updates changed nothing inside the repaired dialog.
The churn count is the number that predicts whether a snapshot will still be read in six months, and it is a direct consequence of the scope chosen on day one.

Configuration

Scope with the component’s own root, and pick that root by something structural rather than by the property under protection. A data-testid on the dialog wrapper is the right handle here; getByRole('dialog', { name: 'Refund order #48219' }) is not, because when the fix regresses and the name disappears the locator stops resolving and the failure reads “locator not found” instead of showing which node lost its name. Scope by identity, assert about semantics.

Determinism comes before patterns. Two of the three volatile values in this dialog can be removed at the source: seed the fixture so the order number is fixed, and freeze the clock so “requested 2 minutes ago” is computed from a known instant. Only the values that genuinely vary per run — here, nothing, once both of those are pinned — need a pattern in the expected file. Patterns are the fallback, not the first move, because every pattern is a small amount of coverage traded away.

// tests/a11y/locks/refund-dialog.lock.spec.ts
import { test, expect } from '@playwright/test';

// Fingerprint of the closed aria-dialog-name finding, from the sweep manifest.
const FINGERPRINT = 'c40b8e17';

test.describe('@a11y-lock refund dialog', () => {
  test.beforeEach(async ({ page }) => {
    // Freeze time before navigation so the "requested … ago" label is a
    // constant rather than something the snapshot has to tolerate.
    await page.clock.setFixedTime(new Date('2026-04-14T09:30:00Z'));
    await page.goto('/orders/48219');            // seeded fixture order
    await page.getByRole('button', { name: 'Refund' }).click();
    // The dialog sets data-ready once focus has moved and its labelling
    // relationship is wired; a tree captured before that contains skeletons.
    await page.locator('[data-testid="refund-dialog"][data-ready="true"]').waitFor();
  });

  test(`refund dialog subtree is unchanged [${FINGERPRINT}]`, async ({ page }) => {
    const dialog = page.getByTestId('refund-dialog');
    // Scoped by test id, not by accessible name: if the repaired name is
    // lost, the diff must show the missing name, not fail to find the node.
    await expect(dialog).toMatchAriaSnapshot({ name: 'refund-dialog.aria.yml' });
  });
});

The committed file is the artifact that matters, and it is worth reading line by line before the first commit — a snapshot generated from currently-broken markup promotes the defect to the expected value. Fourteen lines is short enough that this review takes a minute and long enough to encode the whole contract the fix established: the dialog has a name, the name comes from the heading, the heading is level 2, the reason field is inside a named group, and the confirmation region exists with a live role.

# tests/a11y/locks/__snapshots__/refund-dialog.aria.yml
# Locks the subtree repaired by fix c40b8e17 (aria-dialog-name, SC 4.1.2).
- dialog "Refund order #48219":
  # Level matters: this heading is also the dialog's accessible name source.
  - heading "Refund order #48219" [level=2]
  - group "Refund details":
    - textbox "Refund amount"
    # The amount is basket-dependent even with a seeded order, so the shape
    # is pinned and the digits are allowed to vary.
    - text /^Maximum refundable: \d+\.\d{2} GBP$/
    - combobox "Reason for refund"
    - textbox "Notes for the customer"
  - text "requested 14 April 2026 at 09:30"
  - button "Confirm refund"
  - button "Cancel"
  # Empty until the request resolves; its presence is the contract, not its text.
  - status
What to freeze and what to pattern Four rows compare the raw captured node, the reason it churns, and the committed line. The order number is frozen by a seeded fixture, the relative timestamp is frozen by a fixed clock, the refundable amount keeps a pattern because it is basket-dependent, and the reason combobox is pinned exactly because nothing about it varies. raw captured node why it churns committed line heading "Refund order #48219" id varies per fixture frozen: seed order 48219 text "requested 2 minutes ago" drifts on a slow runner frozen: fixed clock text "Maximum refundable: 48.20 GBP" basket-dependent pattern: \d+\.\d{2} GBP combobox "Reason for refund" nothing varies pinned exactly Freezing removes the variance; a pattern only hides it, and hides some coverage with it. A pattern loose enough to survive every fixture — /Maximum.*/ — has stopped testing anything.
Two of the three volatile values in this dialog disappear once the fixture and the clock are pinned, which leaves exactly one pattern in the committed file.

Validation

Prove the lock bites before trusting it, and break it in the way the code will actually break. The realistic regression here is not someone deleting aria-labelledby; it is a visual refactor that turns the <h2> into a styled <div> so the dialog title can share typography with the page title. The dialog looks identical, aria-labelledby still points at a real element, and the accessible name survives — but the heading node is gone from the tree and the level information with it.

# Baseline: the lock is green against the committed contract.
npx playwright test tests/a11y/locks/refund-dialog.lock.spec.ts --reporter=list
#   ✓ refund dialog subtree is unchanged [c40b8e17] (1.1s)

# After the h2 becomes a styled div:
npx playwright test tests/a11y/locks/refund-dialog.lock.spec.ts --reporter=list
#   ✘ refund dialog subtree is unchanged [c40b8e17]
#     - - heading "Refund order #48219" [level=2]
#     + - text "Refund order #48219"
#   1 failed, exit code 1

Commit the snapshot in the same pull request as the fix, never in a follow-up. The file is the record of what was reviewed, and its value depends on it being generated from the reviewed state — a snapshot added a week later is generated from whatever the code does then, including any drift that arrived in between. In practice this means the fix commit touches three things: the component, the lock spec, and the __snapshots__ file, which is exactly what the pull-request guard in the parent guide looks for.

When a change to the file is genuine — the dialog gains a close button, a field is renamed by product — the update is deliberate and it is explained. The mechanism is a one-line claim per touched file in the pull-request body, checked by a small script against the diff. This is the check that keeps every other snapshot honest, because it is the only one an update flag cannot satisfy.

// a11y/locks/snapshot-justified.mjs
// Fail when a committed .aria.yml changed and the PR body does not say why.
import { execFileSync } from 'node:child_process';

const base = process.argv[process.argv.indexOf('--base') + 1];
const body = process.env.PR_BODY ?? '';

const changed = execFileSync('git', ['diff', '--name-only', `${base}...HEAD`], {
  encoding: 'utf8',
})
  .split('\n')
  .filter((file) => file.endsWith('.aria.yml'));

if (changed.length === 0) {
  console.log('no snapshot files changed in this pull request');
  process.exit(0);
}

// One line per touched file: "Snapshot-Change: <path> - <reason>". The reason
// must clear 20 characters, so "updated" and "regenerated" do not pass.
const explained = new Map();
for (const line of body.split('\n')) {
  const claim = line.match(/^Snapshot-Change:\s*(\S+)\s*[-—]\s*(.+)$/);
  if (claim && claim[2].trim().length >= 20) {
    explained.set(claim[1], claim[2].trim());
  }
}

const unexplained = changed.filter((file) => !explained.has(file));
for (const file of unexplained) {
  console.error(`unexplained snapshot change: ${file}`);
}
console.log(`${changed.length} changed, ${changed.length - unexplained.length} explained`);
process.exit(unexplained.length === 0 ? 0 : 1);

The regeneration itself is two commands and one read. Run the update flag locally, read the diff as prose — “the status region moved above the buttons”, “a button appeared” — and only then write the claim line that describes it. A reviewer who sees a two-line diff and a sentence explaining it can approve in seconds; a reviewer who sees a regenerated file and no sentence has been handed a decision they cannot make.

# Intentional change: the dialog gained an explicit close button.
npx playwright test tests/a11y/locks/refund-dialog.lock.spec.ts --update-snapshots
git diff tests/a11y/locks/__snapshots__/refund-dialog.aria.yml
#   + - button "Close refund dialog"

# Then the body of the pull request carries the matching claim:
#   Snapshot-Change: tests/a11y/locks/__snapshots__/refund-dialog.aria.yml
#     - added an explicit close button alongside Cancel per design review 214
Lifecycle of a committed snapshot file State one is the snapshot committed with the fix and read line by line in review. State two is CI reporting a diff because a refactor moved the subtree. From there the pull request either reaches state three, an accepted update carrying a Snapshot-Change claim and a reviewer sign-off, which becomes the new contract, or state three-b, blocked because the diff has no stated reason. A snapshot only stays a contract if every change is claimed 1 committed with the fix 14 lines, all read 2 CI reports a diff a refactor moved a node 3 update accepted claim + sign-off 3b merge blocked no reason in the body no claim line found the reviewed file becomes the new contract
State 3b is not an obstacle to be routed around: it is the state that stops a regression being renamed as an update.

Edge Cases and Conditional Guards

  • Skeleton and aria-busy states. While the dialog is fetching refund limits it may set aria-busy="true" and render placeholder nodes with no names; a capture taken then bakes them into the contract and every later run diffs against them. Gate the capture on the component’s own readiness attribute, and if there is no such attribute, assert aria-busy is false before capturing. Framework-specific timing for these attributes is covered in handling dynamic ARIA states in modern JavaScript frameworks.
  • One file per state, never a merged one. The refund dialog has an idle tree, a submitting tree with a busy status, and an error tree with a validation summary. Expressing all three in one expectation produces a file that matches none of them precisely. Capture each state in its own snapshot with its own name, and let the spec drive the component into that state first.
  • Portals and stacked overlays. A dialog rendered through a portal sits outside its logical parent in the DOM, so a locator anchored to the surrounding page section resolves to nothing. Anchor on the portal root’s own test id. If a second overlay can stack above it, dismiss it in beforeEach rather than tolerating an extra branch in the file.
  • Browser upgrades that re-indent the tree. The engine’s “interesting node” filtering changes between revisions, so an upgrade can introduce a generic node and shift a whole branch. Pin the browser in the lockfile, do the regeneration in a commit that touches nothing else, and give it a claim line naming the upgrade.

Pipeline Impact

The lock is an ordinary Playwright expectation, so a failure exits non-zero and needs no extra reporter plumbing. Run it inside the tagged lock project rather than alongside the rule scan, because a subtree drift and a rule violation need different reviewers and different remedies — the scan configuration itself lives with axe-core configuration and setup and has no opinion about snapshots. On a suite of fourteen locked components, the snapshot half of the job costs about eighteen seconds, which is cheap enough to be a required check as described in requiring accessibility status checks in branch protection.

Two pipeline rules are absolute. --update-snapshots never runs in CI, under any condition, including the “first run writes the file” convenience — a missing snapshot must fail the job and be committed by a human. And snapshot-justified.mjs runs on the same job, so a pull request cannot be green with an unexplained snapshot diff. Upload the received YAML as a failure artifact so the drift can be read from the run page without a local reproduction, and add the __snapshots__ directory to CODEOWNERS under the accessibility group when the components belong to a shared design system.

Common Pitfalls

  • Snapshotting the page the fix was found on rather than the subtree the fix repaired, which guarantees churn from teams that have never seen the component.
  • Scoping the locator by the accessible name the fix restored, so a regression surfaces as “locator not found” instead of as a readable diff.
  • Committing a file generated before the fix was reviewed, which promotes whatever the code currently does — including the defect — to the expected value.
  • Reaching for a pattern when the value could have been frozen at the source with a seeded fixture or a fixed clock.
  • Writing a pattern loose enough to survive anything (/Maximum.*/), which passes for an empty string and stops testing.
  • Landing the snapshot in a follow-up pull request, so the contract records a state nobody reviewed.
  • Treating a red snapshot as a tooling problem. It is a claim that something in the accessibility tree changed, and the only valid responses are to fix the code or to justify the change.

FAQ

Does this replace the targeted assertion on the same fix? No, and the two fail on deliberately different changes. The assertion answers “does this node still satisfy the rule that was violated” and produces a one-line message naming the rule id, which is what a triage engineer wants at three in the afternoon. The snapshot answers “is the shape of this subtree still what was reviewed” and catches the changes no rule has an opinion about — a heading level, a lost grouping, a status region that vanished. A fix protected by only the snapshot is hard to triage; a fix protected by only the assertion is blind to structural drift.

How large is too large for a scoped snapshot? Past roughly thirty nodes the diff stops being readable as prose, and readability is the entire mechanism — a file nobody reads is a file that gets regenerated. If a component’s tree is bigger than that, it is usually two components: snapshot the parts separately, each anchored on its own test id, and keep the container’s own assertion to its role and name. The refund dialog at fourteen nodes sits comfortably inside that limit even with its error state captured in a second file.

What happens when the whole component is deleted? The lock fails with a locator error, and that is the correct trigger for a decision rather than a cleanup. Delete the spec and the snapshot in the same pull request that deletes the component, and put the fingerprint in the commit message so the original finding is traceable to a removal rather than to a lost lock. If the component was replaced rather than removed, the replacement inherits the contract: generate a fresh snapshot for it, read it against the old file, and confirm every relationship the fix established still exists.

Back to Regression Prevention After Fixes