Automating NVDA Output Capture in a Windows CI Runner

Everything else in automated accessibility testing inspects what the browser exposes. This is the one job that listens to what a real screen reader says: NVDA running on a Windows runner, driven by keystrokes, with its speech redirected from an audio device into a text transcript that a test can assert against. This guide is part of Screen Reader Automation Testing, and it is deliberately scoped to a nightly job. The setup below takes three minutes to install, tens of seconds per page to run, and breaks when NVDA changes its phrasing — none of which belongs anywhere near a pull-request gate.

Root Cause

A screen reader is a desktop application that reads the focused window and speaks through a synthesizer. Neither half of that survives a normal CI setup. There is no audio device on a hosted runner, so the default synthesizer either fails to initialise or silently discards output; and a headless browser has no window to focus, so even a perfectly configured reader has nothing to read. Any attempt to bolt a reader onto an existing headless pipeline produces an empty transcript and a green test, which is the worst of all outcomes: a job that proves nothing while appearing to prove everything.

Getting speech as text needs a deliberate channel. NVDA offers three, with different properties. Its Speech Viewer displays every spoken phrase in a window, which is excellent for a human watching a session and useless to a script because there is no file behind it. Its log, at debug level, contains a Speaking entry for each utterance, which is scriptable but noisy and tied to internal log formatting. The channel built for automation is the community testing add-on, which exposes speech over a local socket as it is produced; the @guidepup/guidepup package wraps that socket in a Node API with a spoken-phrase log. Use the add-on as the primary channel and keep the debug log as corroborating evidence when a run produces something surprising.

The last obstacle is ordering. NVDA builds a virtual buffer for a document when that document gets focus, and it enters sleep mode for applications it does not recognise. Launch the browser first and the reader may attach to a window it has already classified, or miss the document load entirely; launch the reader and immediately start typing and the keystrokes land before it is ready to interpret them. The sequence that works is: start NVDA, wait until it reports readiness, launch a windowed browser, bring that window to the front, and only then send the first navigation key.

Launch order for reader and browser The working sequence starts NVDA, waits for readiness, launches a headed browser, brings the window to the front, then sends keystrokes and collects the phrase log. Two failing variants are shown: launching a headless browser, and sending keystrokes before the reader reports ready. Working order 1 start NVDA silent synth 2 await ready socket answers 3 headed browser real window 4 bring to front buffer built 5 walk phrases logged Failing variant A — headless browser NVDA running headless launch no window exists transcript: 0 phrases Failing variant B — keys sent too early NVDA starting keys sent at once no virtual buffer partial, unordered log Both variants pass if the test forgets to require phrases.
An empty transcript is the default failure of this job, so the test must assert that the log is non-empty before it asserts anything about its contents.

Configuration

Set up the reader in a script rather than inline in the workflow, so the same steps run on a maintainer’s Windows machine. Download a pinned installer into a cached directory, create a portable copy so nothing touches the system-wide screen-reader flag, seed a configuration that uses the silent synthesizer, and drop the testing add-on into the portable copy’s add-on directory. The @guidepup/setup package performs an equivalent installation in one command; the explicit version below exists because pinning and caching the installer is what keeps a nightly job reproducible.

# reader/setup-nvda.ps1 — run on windows-latest with: pwsh reader/setup-nvda.ps1
$ErrorActionPreference = 'Stop'
$version = '2024.4.1'                  # pinned: a reader upgrade is a code change
$cache   = Join-Path $PSScriptRoot '..\.cache\nvda'
$root    = 'C:\nvda-portable'
New-Item -ItemType Directory -Force -Path $cache, $root | Out-Null

$installer = Join-Path $cache "nvda_$version.exe"
if (-not (Test-Path $installer)) {
  Invoke-WebRequest -Uri $env:NVDA_INSTALLER_URL -OutFile $installer
}

# A portable copy never registers itself as the system screen reader and never
# survives the runner, which is exactly what a disposable CI machine wants.
Start-Process -FilePath $installer -Wait -ArgumentList `
  '--create-portable-silent', "--portable-path=$root", '--no-sr-flag'

# Speech goes to the add-on socket; the synthesizer is 'silence' so no audio
# device is needed and nothing blocks waiting for one.
$config = Join-Path $root 'userConfig'
New-Item -ItemType Directory -Force -Path $config | Out-Null
@'
schemaVersion = 12
[general]
	loggingLevel = DEBUG
	showWelcomeDialogAtStartup = False
	playStartAndExitSounds = False
[update]
	autoCheck = False
[speech]
	synth = silence
[speechViewer]
	showSpeechViewer = True
'@ | Set-Content -Path (Join-Path $config 'nvda.ini') -Encoding UTF8

# An add-on package is a zip; extracting it into userConfig\addons registers it.
$addonDir = Join-Path $config 'addons\nvda-testing-driver'
New-Item -ItemType Directory -Force -Path $addonDir | Out-Null
Expand-Archive -Path (Join-Path $cache 'nvda-testing-driver.nvda-addon') `
  -DestinationPath $addonDir -Force
Write-Host "NVDA $version prepared at $root"

The driver script owns the launch order. It starts the reader, waits for it to answer, launches a headed Chromium, brings the window forward, walks the document with the reader’s own navigation keys, and writes the phrase log to disk as JSON. Every step has a timeout, because a hung reader is a far more common failure than a wrong phrase.

// reader/transcript.ts — run with: npx tsx reader/transcript.ts /reports/quarterly
import { writeFileSync, mkdirSync } from 'node:fs';
import { nvda } from '@guidepup/guidepup';
import { chromium } from 'playwright';

const route = process.argv[2] ?? '/';
const BASE = process.env.READER_BASE_URL ?? 'http://127.0.0.1:4173';
const STEPS = 40;                       // bounded walk: never loop the document

async function main() {
  await nvda.start();                   // step 1: reader first, always
  await nvda.clearSpokenPhraseLog();    // step 2: it speaks on start; discard that

  const browser = await chromium.launch({ headless: false });  // step 3: a window
  const page = await browser.newPage({ viewport: null });
  await page.goto(`${BASE}${route}`, { waitUntil: 'load' });
  await page.bringToFront();            // step 4: NVDA reads the focused window
  await page.locator('main').waitFor();

  // Step 5: walk with the reader's browse-mode navigation, not the app's UI.
  for (let i = 0; i < STEPS; i += 1) {
    await nvda.next();
  }

  const phrases = await nvda.spokenPhraseLog();
  mkdirSync('reader-out', { recursive: true });
  const file = `reader-out/${route.replace(/\W+/g, '_')}.json`;
  writeFileSync(file, JSON.stringify({ route, phrases }, null, 2));
  console.log(`${phrases.length} phrases captured to ${file}`);

  await browser.close();
  await nvda.stop();                    // leaving it running hangs the runner
  if (phrases.length === 0) {
    throw new Error('empty transcript: reader attached to no document');
  }
}

main().catch((error) => {
  console.error(error);
  process.exitCode = 1;
});

Validation

A captured transcript is a list of phrases in the order NVDA produced them. Assert against it as an ordered subsequence: every expected phrase must appear, in the expected relative order, with any number of unknown phrases in between. Exact-transcript equality fails on the next reader upgrade for reasons that have nothing to do with the product.

// reader/assert-phrases.mjs
// usage: node reader/assert-phrases.mjs reader-out/_reports_quarterly.json
import { readFileSync } from 'node:fs';

const { route, phrases } = JSON.parse(readFileSync(process.argv[2], 'utf8'));

// Patterns, not literals: NVDA's wording around roles changes between versions.
const EXPECTED = [
  /main landmark/i,
  /Quarterly revenue.*heading.*level 1/i,
  /Region.*combo ?box/i,
  /Apply filters.*button/i,
  /table.*with 5 rows and 4 columns/i,
];

let cursor = 0;
const missing = [];
for (const pattern of EXPECTED) {
  const hit = phrases.findIndex((p, i) => i >= cursor && pattern.test(p));
  if (hit === -1) missing.push(String(pattern));
  else cursor = hit + 1;               // enforces order, tolerates extra speech
}

if (missing.length > 0) {
  console.error(`${route}: ${missing.length} expected phrase(s) missing or out of order`);
  for (const pattern of missing) console.error(`  missing: ${pattern}`);
  console.error(`captured ${phrases.length} phrases; see the artifact for the full log`);
  process.exit(1);                     // nightly job fails, no branch is blocked
}
console.log(`${route}: all ${EXPECTED.length} expected phrases present and in order`);

Running the pair end to end shows what a real failure looks like. The transcript below is the correct run; removing the <caption> from the revenue table drops one phrase and the ordered check reports it by pattern rather than by line number.

npx tsx reader/transcript.ts /reports/quarterly
# 63 phrases captured to reader-out/_reports_quarterly.json
node reader/assert-phrases.mjs reader-out/_reports_quarterly.json
# /reports/quarterly: all 5 expected phrases present and in order

# After deleting the table <caption>:
node reader/assert-phrases.mjs reader-out/_reports_quarterly.json
# /reports/quarterly: 1 expected phrase(s) missing or out of order
#   missing: /table.*with 5 rows and 4 columns/i
# captured 61 phrases; see the artifact for the full log
# exit status 1
Where NVDA speech goes on a runner NVDA produces speech that fans out to three destinations: a silence synthesizer that discards audio, the testing add-on socket that the Node driver reads into a JSON transcript, and a debug log file kept as an artifact for triage. NVDA speech one utterance stream synth = silence audio discarded, no device testing add-on socket phrases as they happen nvda.log at DEBUG Speaking entries, noisy transcript JSON asserted, uploaded triage artifact read only on failure Only the teal path is asserted on; the others are for humans.
Splitting the channels keeps the assertion narrow and still leaves a full debug log for the run that fails for an unclear reason.

Edge Cases and Conditional Guards

  • Browse mode versus focus mode. NVDA switches modes automatically when focus enters an editable field or an application-role widget, and the same keystroke means different things in each mode. Walk the document in browse mode only, and when a widget must be exercised, enter focus mode explicitly, capture, then leave — otherwise arrow keys type into an input and the transcript fills with character echoes.
  • A dialog that steals focus mid-walk. A cookie banner, a session-expiry modal or a Windows notification moves focus and the remaining keystrokes address the wrong window. Dismiss known interstitials before the walk starts using ordinary Playwright actions, and treat any phrase mentioning a notification as a signal to discard the run rather than to fail the assertion.
  • Reader upgrades. Pin the installer by exact version and treat a version bump as its own pull request with the expected phrase patterns updated alongside it. NVDA changes how it announces tables, landmarks and state between releases; discovering that on the same night as a product change makes both harder to diagnose.

Pipeline Impact

This job is scheduled, not gated. It runs on windows-latest in its own workflow file so it can never contribute a required status check, and its failure signal is a labelled issue containing the run URL and the transcript artifact. A nightly reader job that has been quietly promoted to a required check will eventually block a release on a phrasing change in a third-party screen reader, which is an outage caused entirely by test infrastructure — the warning-versus-blocking distinction covered in auto-fail versus warning workflows applies here in its strongest form.

The outcome states are worth writing down before the first run, because three of the four have nothing to do with the product. A reader that never reports ready is an infrastructure fault; an empty transcript is a launch-order or headless fault; a transcript that is present but missing an expected phrase is the only state that indicates an accessibility regression. Distinguishing them in the job’s own output saves the triage step of deciding whether to open a product ticket at all.

Outcome states of the nightly transcript job Three sequential checks: did the reader report ready, is the transcript non-empty after at most one retry, and are the expected phrases present in order. The three failing exits are labelled infrastructure fault, launch fault and accessibility regression, and none of them produces a status check. reader reports ready? transcript non-empty? phrases present, in order? infrastructure fault upload nvda.log, label ci-infra launch fault, retry once still empty: label ci-infra accessibility regression issue with the missing pattern transcript archived tomorrow's comparison baseline no no no yes yes yes Every terminal state writes an issue or an artifact. None of them writes a status check.
Separating infrastructure faults from real regressions in the job's own exit path is what keeps the nightly label meaningful after a few weeks.

Budget the runtime honestly. Installation is roughly three minutes on a cold cache and forty seconds warm; reader startup is ten to fifteen seconds; a forty-step walk of one page is thirty to sixty seconds depending on how much the page has to say. Twenty pages therefore costs a comfortable twenty-minute job, which is fine at 03:00 and unacceptable on a pull request. Set timeout-minutes on the job, upload reader-out/ with if: always(), and keep the per-page transcripts as separate files so a diff against the previous night’s run points at one page. Feed the pass rate into the trend store described in reporting dashboards and violation tracking and review it weekly; a job that fails more than about one night in ten for infrastructure reasons needs its assertions loosened before anyone starts ignoring the label.

Common Pitfalls

  • Running the browser headless, which produces an empty transcript and — without an explicit non-empty assertion — a green job that tests nothing.
  • Leaving the default synthesizer configured, so the reader blocks or errors on a runner with no audio device and the job times out instead of failing clearly.
  • Installing NVDA system-wide rather than as a portable copy, which sets the global screen-reader flag and changes the behaviour of other applications on the runner.
  • Asserting the transcript with exact string equality, guaranteeing a red job on the next reader release for a phrasing change nobody in the product caused.
  • Forgetting to clear the spoken-phrase log after startup, so the reader’s own boot announcements appear at the front of every transcript and shift the ordered comparison.
  • Never calling nvda.stop(), leaving a reader process holding the session so the next step of the workflow hangs until the job timeout.

FAQ

Can this run on a self-hosted runner instead of a hosted one? Yes, and it is often more stable, because the machine keeps its NVDA installation and its cached installer between runs, cutting three minutes from every job. The requirement is an interactive desktop session: a runner configured as a Windows service with no session cannot give a browser window focus, so the reader has nothing to read. Run the agent in a logged-in user session, disable screen lock and sleep, and keep the machine off the general update channel so a reader or Windows upgrade never lands mid-week.

Why not use the debug log as the only speech channel? It works, and it is a reasonable fallback when the add-on cannot be installed, but it is a worse contract. The log interleaves speech with internal diagnostics, its formatting is an implementation detail that changes between releases, and it is written asynchronously so the last utterance may not be flushed when the script reads the file. The add-on socket delivers phrases as discrete strings in order, which is what an ordered-subsequence assertion needs.

How many pages should the nightly walk cover? Start with three to five pages that represent distinct reading experiences — a data table, a multi-step form, a dashboard with live updates — rather than a broad crawl. The value of this job is depth on complex reading behaviour, not coverage; breadth is what the tier-one and tier-two assertions on every pull request are for, including the announcement checks in verifying live region announcements in automated tests. Grow the page list only when the existing transcripts have been stable for several weeks.

Back to Screen Reader Automation Testing