# Screen Reader Automation Testing in a CI Pipeline

A rule scanner reads markup and decides whether it conforms; a screen reader reads the platform accessibility tree and decides what to say. Those are different questions, and the gap between them is where the expensive bugs live: an icon button whose label is technically present but computes to an empty name, a filter panel that updates twelve results without ever telling anyone, a dialog that opens with focus still parked on the page behind it. This guide is part of Web Accessibility Testing Fundamentals & Tool Selection, and it covers how much of that assistive-technology layer can genuinely be automated, which parts belong on a merge gate, and which parts belong on a nightly job that is allowed to be slow and occasionally wrong.

# Problem Statement

The instinct when a team decides to “automate screen reader testing” is to reach for the screen reader. That produces a test suite that boots NVDA on a Windows runner, sends keystrokes, and compares spoken phrases against expected strings — and it works, right up until NVDA ships a new version that changes the phrasing of a table cell announcement, or the runner’s session loses focus, or a synthesizer takes 300 ms longer than the last run and the transcript arrives in a different order. That suite ends up either disabled or permanently marked as allowed-to-fail, at which point it protects nothing while still costing twelve minutes of runner time per push.

The opposite instinct is equally wrong. Asserting only what axe-core reports means the pipeline never checks announcement behaviour at all, because a static scan sees one instant of the DOM and an announcement is a change between two instants. axe can confirm that a role="status" container exists and is well-formed; nothing in a snapshot-based scan can confirm that text was written into that container while it was already attached, which is the actual precondition for a screen reader to speak it.

The workable answer is to stop treating “screen reader testing” as one activity. Three distinct things can be verified, they cost wildly different amounts, and they have different failure characteristics. The platform accessibility tree — the roles, names, states and structure the operating system exposes to any assistive technology — is computed by the browser and can be read out deterministically in milliseconds. Announcement semantics — whether a status message, a focus move or a state change would be conveyed — are provable from the DOM contract that drives them, observed over time rather than sampled once. Actual speech output is the only tier that proves a specific reader on a specific platform says a specific sentence, and it is the only tier that needs a driven operating system. Tiers one and two belong on the pull request. Tier three belongs on a schedule.

Key implementation targets:

• A per-component accessibility-tree snapshot committed to the repository, scoped to the component root so it does not churn on unrelated page changes.
• Explicit computed accessible-name and role assertions on the controls whose labels are most likely to be refactored away.
• An announcement recorder installed before application code runs, producing an ordered transcript of every mutation to every live region.
• Focus-move assertions after route changes, dialog opens and form submissions, so the reader’s reading cursor lands somewhere deliberate.
• A nightly windows-latest job that drives a real NVDA build, writes speech to a transcript file, and asserts phrase order.
• Two separate result channels, so a flaky nightly transcript raises an issue while the axe gate keeps its own clean pass/fail signal on every merge.

The table below is the trade-off that decides where a new assistive-technology assertion goes. Read it as a routing table, not a ranking: a naming regression should never be chased with a speech transcript when a 40-millisecond tree assertion catches it, and no amount of tree assertion proves that NVDA in browse mode reads a summary table’s caption before its first row.

# Prerequisites

• Node.js 20.x or newer, with Playwright 1.49+ (locator.ariaSnapshot() and toMatchAriaSnapshot are required by section 1).
• A pinned browser revision in package-lock.json — an unpinned Chromium changes tree output between runs and invalidates every snapshot.
• A component harness or route that renders one component in isolation, so tier-1 snapshots are scoped to a subtree rather than a whole page.
• A windows-latest runner available to the repository, plus permission to install software on it, for the nightly tier-3 job.
• NVDA 2024.4 or newer pinned by exact installer version, plus its testing add-on, cached between nightly runs.
• actions/upload-artifact@v4 retention of at least 14 days, so a nightly transcript is still readable when someone triages the issue it opened.
• Branch protection configured with the axe status check required and the nightly reader check explicitly not required.
• A fixed TZ and locale in the test configuration, because both the tree and the transcript contain formatted dates.

# 1. Capture an Accessibility-Tree Snapshot Per Component

Tier one starts with a manifest, not with a page. Snapshotting whole pages produces enormous trees whose diffs nobody reads, and any change anywhere on the page rewrites the file. Snapshot the component instead: mount it in a harness route, take the tree for that subtree only, and commit the result next to the component. The diff then reads as a sentence — “the remove button lost its name” — instead of as four hundred lines of YAML.

Playwright’s locator.ariaSnapshot() returns the subtree as indented YAML of roles and accessible names, which is far more reviewable than the JSON shape returned by page.accessibility.snapshot(). Pair it with expect(locator).toMatchAriaSnapshot({ name }) so the expected value lives in a file under __snapshots__ that a reviewer can read in the pull request diff. Keep one file per component and per meaningful state — collapsed and expanded, empty and populated, idle and error — because a component’s tree in its error state is the one nobody checks by hand.

// tests/reader/tree-contract.spec.ts
import { test, expect } from '@playwright/test';

// One entry per component state. The harness route renders exactly one
// component into #harness-root, so the snapshot has no page chrome in it.
const CONTRACT = [
  { id: 'filter-panel-idle', route: '/harness/filter-panel?state=idle' },
  { id: 'filter-panel-error', route: '/harness/filter-panel?state=error' },
  { id: 'cart-row-populated', route: '/harness/cart-row?qty=2' },
  { id: 'date-picker-open', route: '/harness/date-picker?open=1' },
];

for (const entry of CONTRACT) {
  test(`accessibility tree contract: ${entry.id}`, async ({ page }) => {
    await page.goto(entry.route);
    const root = page.locator('#harness-root');
    // Wait for the component's own readiness flag, not for networkidle:
    // the harness sets it after effects have run and refs are attached.
    await root.locator('[data-harness-ready="true"]').waitFor();
    await expect(root).toMatchAriaSnapshot({ name: `${entry.id}.aria.yml` });
  });
}

The committed file is the contract. It is worth spending review attention on the first version of each one, because a snapshot written from broken markup locks the breakage in. A useful discipline: when the file is created, read it out loud. If a control appears as - button with no name, or a heading level jumps from heading "Filters" [level=2] to heading "Sort" [level=4], the snapshot has just documented a real defect and the fix belongs in the same pull request. The mechanics of pruning and of tolerating volatile text are worked through in asserting accessibility tree names with Playwright snapshots.

# 2. Assert Accessible Names and Roles Explicitly

A snapshot proves the whole subtree is unchanged. It does not say which parts of it matter, and it will happily be regenerated by a developer running the update flag on a red build. For the controls that carry real risk — the icon-only buttons, the inputs whose labels live in a sibling component, the tabs whose names come from a translation catalogue — write the assertion out by hand as well. A named assertion cannot be silently updated, and its failure message says exactly what was expected.

Playwright exposes the computed values directly: toHaveRole reads the resolved ARIA role, toHaveAccessibleName runs the full accessible-name computation, and toHaveAccessibleDescription covers aria-describedby text. These are the assertions that map to WCAG 2.2 SC 4.1.2 (Name, Role, Value) one-to-one, so a failing test names the criterion it breaks without any interpretation step.

// tests/reader/name-role.spec.ts
import { test, expect } from '@playwright/test';

test('toolbar controls keep their computed names (SC 4.1.2)', async ({ page }) => {
  await page.goto('/reports/quarterly');
  const toolbar = page.getByRole('toolbar', { name: 'Report actions' });

  // Icon-only controls: the label is invisible, so nothing on screen
  // regresses when it disappears from the tree.
  await expect(toolbar.getByTestId('export')).toHaveRole('button');
  await expect(toolbar.getByTestId('export')).toHaveAccessibleName('Export as CSV');
  await expect(toolbar.getByTestId('share')).toHaveAccessibleName('Share report');

  // A description carries the destructive-action warning; losing it is
  // invisible on screen and total for a reader.
  await expect(toolbar.getByTestId('purge')).toHaveAccessibleDescription(
    'Deletes the saved report for everyone in the workspace',
  );

  // State, not identity: aria-pressed must resolve on the node with the role.
  await expect(toolbar.getByTestId('compare')).toHaveAttribute('aria-pressed', 'false');
});

Choose the locators for these assertions by risk, not by coverage. Twelve hand-written name assertions on the controls that a design-system upgrade is most likely to break are worth more than four hundred generated ones, because the generated set will be regenerated the first time it goes red under deadline pressure. A practical selection rule: every control whose visible content is an icon, every control whose name is interpolated from data, and every control that appears in more than one product.

# 3. Observe Live-Region Mutations as an Ordered Transcript

Tier two is where the pipeline starts testing behaviour instead of structure. The unit of behaviour is the announcement, and an announcement is not a DOM state — it is a change to a node that was already in the accessibility tree. That makes it observable with a MutationObserver installed before the application boots, recording every write into every live region with a timestamp and the region’s resolved politeness.

Install the recorder with page.addInitScript rather than page.addScriptTag. An init script runs in every new document before any application script, so it catches the first announcement of the page load; a script tag added after goto starts observing after the app has already had its first chance to speak. The recorder below is the minimal version — it records what was written and where — and the full politeness-resolution and ordering treatment is in verifying live region announcements in automated tests.

// tests/reader/announcements.spec.ts
import { test, expect } from '@playwright/test';

const RECORDER = `
window.__announced = [];
new MutationObserver((records) => {
  for (const record of records) {
    const region = record.target.closest?.(
      '[aria-live], [role="status"], [role="alert"], [role="log"]');
    if (!region) continue;
    const text = region.textContent.trim();
    if (!text) continue;                       // clearing a region is not speech
    window.__announced.push({ at: Math.round(performance.now()), text });
  }
}).observe(document.documentElement, {
  childList: true, subtree: true, characterData: true,
});
`;

test('applying a filter announces the new result count', async ({ page }) => {
  await page.addInitScript({ content: RECORDER });  // before app code, not after
  await page.goto('/reports/quarterly');

  await page.getByRole('combobox', { name: 'Region' }).selectOption('EMEA');
  await page.getByRole('button', { name: 'Apply filters' }).click();

  // Poll the recorder instead of sleeping: the assertion retries until the
  // announcement lands or the expect timeout fires.
  await expect
    .poll(() => page.evaluate(() => window.__announced.map((a) => a.text)))
    .toContain('12 results, filtered by region EMEA');
});

Focus moves belong in the same tier and are recorded the same way. After a client-side route change, a dialog open or a form submission, the reading cursor has to land on something with a name — otherwise the reader either keeps reading the old view or restarts from the top of the document. Assert it with expect(locator).toBeFocused() on the intended landing target, and pair it with a name assertion so a heading that receives focus but has lost its text still fails. The routing-specific version of this problem, including the second-navigation failures that a fresh page load hides, is covered in handling single-page application routing.

# 4. Wire a Nightly OS-Level Screen-Reader Job

Tier three exists because tiers one and two model a reader rather than being one. They cannot tell you that NVDA reads a <caption> before the first row, that a role="grid" announces “row 4 of 200 column 2 Price”, or that a custom slider is silent in browse mode but fine in focus mode. Those are reader behaviours, and only a reader proves them.

Run it on windows-latest, on a schedule, in its own workflow file. Keep the workflow separate from the pull-request workflow: a separate file means a separate required-check name, and a required-check name that never appears on a pull request cannot accidentally become a blocker. Cache the pinned NVDA installer, launch the reader before the browser, capture speech to a transcript file, and treat the transcript as an artifact that a human reads when the job fails.

name: nightly-screen-reader
on:
  schedule:
    - cron: '20 3 * * 1-5'   # weeknights only; nobody triages a Sunday failure
  workflow_dispatch: {}
permissions:
  contents: read
  issues: write              # the only write scope this job needs
jobs:
  nvda-transcript:
    runs-on: windows-latest
    timeout-minutes: 35      # a hung reader must not burn the runner budget
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - name: Restore the pinned NVDA installer
        uses: actions/cache@v4
        with:
          path: .cache/nvda
          key: nvda-2024.4.1-${{ runner.os }}
      - name: Install NVDA and its testing add-on
        run: npm run reader:setup
      - name: Walk the reader route list
        run: npm run reader:transcript -- --pages reader/pages.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: nvda-transcripts
          path: reader-out/
          retention-days: 21
      - name: File an issue instead of failing a branch
        if: failure()
        uses: actions/github-script@v7
        with:
          script: |
            const run = `${context.serverUrl}/${context.repo.owner}/` +
              `${context.repo.repo}/actions/runs/${context.runId}`;
            await github.rest.issues.create({
              ...context.repo,
              title: `Nightly screen reader transcript drift (${context.runId})`,
              labels: ['accessibility', 'nightly'],
              body: `The NVDA transcript job failed. Transcripts are attached to ${run}.`,
            });

The step that does the work is deliberately hidden behind two npm scripts, because the interesting details — silent installation, the speech-capture channel, and the launch order that makes the reader attach to the browser rather than to an empty desktop — are specific enough to need their own treatment in automating NVDA output capture in a Windows CI runner. Two rules survive whatever tooling is chosen: pin the reader version by installer filename, and never assert an exact full transcript. Assert that expected phrases appear, in the expected relative order, allowing unknown phrases between them.

# 5. Report Reader Results Separately From the axe Gate

The single decision that keeps this whole arrangement alive is refusing to merge the two report streams. If the nightly transcript and the pull-request scan write into the same summary, the same dashboard row and the same status check, then one flaky reader run makes the accessibility signal untrustworthy, and an untrustworthy signal gets ignored within a sprint.

Keep two artifacts with two schemas and two audiences. The pull-request artifact is a pass/fail gate consumed by branch protection: axe violations plus tier-1 and tier-2 assertions, one status check, blocking. The nightly artifact is a trend document consumed by whoever owns accessibility: transcripts, diffs against yesterday’s phrases, and an issue when something drifts. The script below writes both from the same test runner output so the split is enforced by code rather than by convention.

// reader/scripts/split-report.mjs
// usage: node reader/scripts/split-report.mjs playwright-report.json
import { readFileSync, writeFileSync, mkdirSync } from 'node:fs';

const report = JSON.parse(readFileSync(process.argv[2], 'utf8'));
const specs = report.suites.flatMap((suite) => suite.specs ?? []);

// The project name, set in playwright.config.ts, is the routing key.
const isBlocking = (spec) => spec.tags?.includes('@gate') ?? true;

const gate = specs.filter(isBlocking);
const advisory = specs.filter((spec) => !isBlocking(spec));
const failed = (list) => list.filter((spec) => !spec.ok);

mkdirSync('reader-out', { recursive: true });
writeFileSync(
  'reader-out/gate.json',
  JSON.stringify(
    { channel: 'pull-request', total: gate.length, failed: failed(gate).length },
    null,
    2,
  ),
);
writeFileSync(
  'reader-out/advisory.json',
  JSON.stringify(
    { channel: 'nightly', total: advisory.length, failed: failed(advisory).length },
    null,
    2,
  ),
);

// Only the gate channel is allowed to influence the process exit code.
process.exitCode = failed(gate).length > 0 ? 1 : 0;
console.log(
  `gate: ${failed(gate).length}/${gate.length} failed, ` +
    `advisory: ${failed(advisory).length}/${advisory.length} failed`,
);

# Pipeline Integration

Tiers one and two join the existing accessibility job rather than getting one of their own. Concretely: the same Playwright invocation that runs the axe scan runs the tree contract and the announcement specs, in a separate project so the reporter groups them, and the job exits non-zero if any of them fail. That gives one required status check for the whole automated accessibility surface, which is the only arrangement that survives contact with branch protection — reviewers ignore a second, third and fourth accessibility check very quickly.

Artifacts matter more here than for a rule scan, because a tree-snapshot failure is unreadable without the diff. Upload the __snapshots__ directory alongside the actual received values, and upload the recorded announcement transcript as JSON for every failed announcement spec. A reviewer who can read “expected 12 results at position 2, received Loading… at position 2” fixes the bug without reproducing it locally.

The nightly job integrates through issues rather than through checks. Its failure signal is an issue with the run URL and the transcripts attached, labelled so it lands in the accessibility triage queue. Feed the pass rate into the same trend store the rest of the programme uses, described in reporting dashboards and violation tracking, and review it on a weekly cadence rather than a per-commit one. If the nightly job is failing more than roughly one run in ten for reasons unrelated to the product, the assertions are too strict — loosen the phrase matching before anyone learns to ignore the label.

# Troubleshooting and Flaky-Test Mitigation

A tree snapshot changes after a browser upgrade, with no product change. Chromium’s name computation and its notion of an “interesting” node both evolve. Two mitigations: pin the browser revision in the lockfile so upgrades are deliberate, and scope every snapshot to a component root so an upgrade touches a handful of files instead of all of them. When an upgrade does move things, review the diff as a batch, in its own commit, with no product changes mixed in.

An announcement assertion passes locally and fails in CI. Almost always a recorder that was installed too late. addScriptTag after page.goto misses everything the app announced during boot, and a slower runner shifts more of the app’s work into that window. Move to addInitScript, which is re-injected for every document including after a full reload.

The transcript records an announcement twice. A MutationObserver observing childList and characterData on the same subtree fires twice when a framework replaces a text node instead of mutating it. De-duplicate on the pair of region identity and text rather than on text alone, and keep the deduplication window short — a genuinely repeated status (“still saving…”) must survive it.

A focus assertion races hydration. toBeFocused() immediately after a click can run while the framework is still moving focus, and it can also run after a focus trap has already moved it again. Assert on the settled state by asserting the name of the focused element instead of its identity, and give the assertion a longer timeout rather than adding a fixed wait.

NVDA produces an empty transcript. The reader was running but nothing had focus, or the browser was launched headless, or the browser started before the reader finished initialising. Launch order is load-bearing: reader first, wait until it reports ready, then launch a windowed browser and focus the window before sending a single keystroke.

The nightly job hangs instead of failing. A modal dialog from an installer, a Windows update prompt, or a reader waiting on a synthesizer with no audio device will all hang. Set an explicit timeout-minutes on the job, configure the reader with a silent synthesizer so no audio device is needed, and always upload the transcript directory with if: always() so a hung run still yields evidence.

# Common Pitfalls

• Snapshotting the whole page instead of a component subtree, which produces diffs so large that reviewers approve them without reading.
• Regenerating tree snapshots as the first response to a red build, which converts a caught regression into a committed baseline.
• Treating page.accessibility.snapshot() output as what a screen reader says. It is what the browser exposes; the reader adds its own verbosity settings, punctuation rules and mode switches on top.
• Asserting an exact NVDA transcript. Readers change phrasing between versions, so assert phrase presence and relative order instead.
• Making the nightly reader job a required status check because it looked green for a week, which guarantees a blocked release the first time NVDA ships an update.
• Recording live-region mutations without checking whether the region existed beforehand, so a silent-by-construction status message is recorded as a successful announcement.
• Using fixed waits before reading the announcement transcript, which trades a two-second test for a five-second one and still races on a loaded runner.
• Running tier-1 assertions under an unpinned locale or timezone, so formatted dates and numbers inside accessible names churn between machines.

# FAQ

Does automating screen readers raise the 30–40% of WCAG issues that scanners catch? Somewhat, and in a specific direction. Tier-1 and tier-2 assertions add coverage for name and role regressions and for status messages, which are criteria a static scan can only partially judge — a scan can see that a live region exists, not that it announced. What none of these tiers add is judgement: whether the announced sentence is useful, whether the reading order makes sense in a novel widget, whether an error message explains the error. Those stay manual, so the honest claim is a few points of extra coverage on criteria that regress often, not a change in the ceiling.

Why not just run the real screen reader on every pull request and skip the first two tiers? Cost and determinism. A driven reader takes 30–60 seconds per page against 40 milliseconds per component, needs a Windows or macOS runner with a windowed session, and fails for reasons unrelated to the product — a reader upgrade, a lost focus, a synthesizer timing shift. A gate that fails for unrelated reasons stops being a gate. The first two tiers catch the large majority of real regressions at a hundredth of the cost, which leaves the reader to prove the handful of things only it can.

Which reader should the nightly job drive? Start with one, and pick it by user data rather than by preference. NVDA with Chrome on Windows is the most common combination in most product analytics and is the easiest to automate on hosted runners, which makes it the pragmatic default. Adding VoiceOver on macOS doubles the runner cost and roughly doubles the triage load, so it earns its place only once the NVDA transcripts have been stable and useful for a quarter.

How should the accessibility tree snapshots be reviewed in a pull request? As specifications, not as generated files. A tree snapshot diff is three lines long when scoped properly, and every line has a meaning a reviewer can read: a role changed, a name changed, a node disappeared. Require that any snapshot change be explained in the pull request description, and treat an unexplained snapshot update the same way as an unexplained change to an assertion — as something to ask about before approving.

Can these tiers run against a component library instead of a deployed app? Yes, and the tier-1 assertions are better there. A component harness renders one component per route with no page chrome, which makes the tree small and the snapshot stable, and it lets a design system prove its own contract before any product consumes it. Tier-2 announcement tests need a bit more scaffolding, because a live region usually lives in the application shell rather than in the component, so the harness has to provide one. Tier three stays at the application level — a reader walking a component in isolation tells you very little about a real page.

# Related

• Asserting Accessibility Tree Names with Playwright Snapshots — pruning a snapshot to one subtree and locking computed names against refactors.
• Automating NVDA Output Capture in a Windows CI Runner — silent install, speech transcripts and launch order for the nightly job.
• Verifying Live Region Announcements in Automated Tests — politeness resolution and the ordered transcript that proves a status message.
• Playwright Accessibility Plugin Integration — the scan harness the tier-1 and tier-2 specs share a job with.
• Handling Single-Page Application Routing — focus and live-region failures that only appear on the second navigation.