# The CI Gate a Machine-Authored ARIA Patch Has to Clear

A patch written by a machine arrives with no author to ask questions of, so the pull request has to answer them by itself. This guide is part of AI-assisted accessibility remediation and covers only the verification half: the workflow that applies the patch to a branch, collects six pieces of evidence, and closes the pull request automatically when any one of them is missing.

# Root Cause

The instinctive gate for a machine-authored accessibility patch is “run the scanner and see if it is green”, and that gate is close to worthless. A scanner reports the absence of the rules it knows about, and a patch that changes an accessible name has a specific relationship to that fact: the rule which could have detected a bad name is the same rule the patch switched off. Green after such a patch means “the rule no longer applies”, which is the expected outcome of both a correct fix and a wrong one. A total count that stays flat is equally uninformative on its own, because a patch can remove one button-name node and add one aria-hidden-focus node and leave the total unchanged.

The second gap is that a normal test run is not scoped to the patch. A machine-authored diff has three properties a human diff usually lacks: it was produced by something with no stake in the outcome, it can be regenerated instantly if rejected, and it has no reason not to take the shortest path to a green pipeline. Anything in the repository that can turn a red run green is therefore part of the attack surface — an assertion in a spec file, an entry in a scanner suppression list, a threshold in a budget file, a committed snapshot. A gate that inspects only the scan result and the test exit code will happily approve a patch whose actual content was “delete the assertion that failed”.

The third gap is the baseline. Comparing a patched scan against a JSON file committed last week measures the patch plus every unrelated change since that file was written, and on any active repository the noise swamps the signal within days. The baseline has to come from the merge base, scanned on the same runner, in the same browser build, in the same job — which is why the workflow below checks out two revisions rather than trusting a stored artifact.

# Configuration

The workflow triggers only on branches the drafting pipeline creates, so a human’s pull request never pays for this machinery. It orders the steps by cost: the diff-scope check runs before any install, the test suite before the browser scans, and the scans last. permissions grants exactly what the automatic close needs and nothing more.

name: a11y-verify-machine-patch
on:
  pull_request:
    branches: [main]
    types: [opened, synchronize, reopened]
    paths-ignore: ['docs/**', '**/*.md']
permissions:
  contents: read
  pull-requests: write # required only for the automatic close and comment
concurrency:
  group: a11y-verify-${{ github.head_ref }}
  cancel-in-progress: true
jobs:
  verify:
    # Only branches the drafting pipeline owns are gated this way.
    if: startsWith(github.head_ref, 'a11y/draft/')
    runs-on: ubuntu-24.04
    timeout-minutes: 25
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0 # merge-base resolution needs full history

      - name: Resolve the merge base
        id: base
        run: |
          BASE=$(git merge-base origin/${{ github.base_ref }} HEAD)
          echo "sha=$BASE" >> "$GITHUB_OUTPUT"

      # Cheapest and most decisive: no install, no build, no browser.
      - name: Condition 5 and 6 — diff scope
        run: node a11y/ci/check-diff-scope.mjs "${{ steps.base.outputs.sha }}"

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium

      - name: Condition 3 — the existing suite must still pass
        run: npm test -- --run # no --update-snapshots, ever, on this branch

      - name: Build the patched revision and serve it on 4311
        run: |
          npm run build
          npx serve dist --listen 4311 --no-clipboard &
          npx wait-on -t 60000 http://127.0.0.1:4311

      # A second worktree at the merge base gives a same-runner baseline.
      - name: Build the merge base and serve it on 4312
        run: |
          git worktree add ../base "${{ steps.base.outputs.sha }}"
          npm --prefix ../base ci
          npm --prefix ../base run build
          npx serve ../base/dist --listen 4312 --no-clipboard &
          npx wait-on -t 60000 http://127.0.0.1:4312

      - name: Conditions 1 and 2 — fingerprint and total count
        run: |
          node a11y/ci/scan.mjs http://127.0.0.1:4312 --out base-scan.json
          node a11y/ci/scan.mjs http://127.0.0.1:4311 --out head-scan.json
          node a11y/ci/compare-scans.mjs base-scan.json head-scan.json \
            --suggestion suggestion.json

      - name: Condition 4 — tree snapshot scoped to the target node
        run: npx playwright test tests/a11y/tree-snapshot.spec.ts

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: a11y-verify-evidence
          path: |
            base-scan.json
            head-scan.json
            suggestion.json
            test-results/
          retention-days: 30

      # Any failed step lands here. A machine-authored patch is not amended.
      - name: Close the pull request on any failed condition
        if: failure()
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          gh pr comment "${{ github.event.pull_request.number }}" \
            --body "A verification condition failed; see the run log and the
          a11y-verify-evidence artifact. Regenerate rather than amend."
          gh pr close "${{ github.event.pull_request.number }}" --delete-branch

The comparison script implements conditions one and two together, because they answer different questions from the same pair of scans. Condition one is fingerprint-based: the finding the patch claimed to repair must be gone, matched on a stable identity rather than on a count, so a patch that fixes a different node of the same rule cannot pass by arithmetic. Condition two is a per-rule and total comparison across every scanned route, which is what catches an attribute whose effect leaked into a parent’s accessible name.

// a11y/ci/compare-scans.mjs
// usage: node compare-scans.mjs base.json head.json --suggestion s.json
import { readFileSync } from 'node:fs';
import { createHash } from 'node:crypto';

const [basePath, headPath] = process.argv.slice(2, 4);
const sIdx = process.argv.indexOf('--suggestion');
const suggestion = JSON.parse(readFileSync(process.argv[sIdx + 1], 'utf8'));

// A fingerprint identifies a finding across runs without relying on a CSS
// selector, which changes whenever the markup around the node changes.
const fingerprint = (ruleId, route, node) =>
  createHash('sha1')
    .update(`${ruleId}|${route}|${node.html.replace(/\s+/g, ' ').trim()}`)
    .digest('hex')
    .slice(0, 12);

function index(scan) {
  const prints = new Set();
  const perRule = new Map();
  let total = 0;
  for (const route of scan.routes) {
    for (const v of route.violations) {
      for (const node of v.nodes) {
        prints.add(fingerprint(v.id, route.url, node));
        total += 1;
      }
      perRule.set(v.id, (perRule.get(v.id) ?? 0) + v.nodes.length);
    }
  }
  return { prints, perRule, total };
}

const base = index(JSON.parse(readFileSync(basePath, 'utf8')));
const head = index(JSON.parse(readFileSync(headPath, 'utf8')));
const problems = [];

// Condition 1: the exact finding the patch claimed must be absent.
if (head.prints.has(suggestion.finding.fingerprint)) {
  problems.push(`target finding ${suggestion.finding.fingerprint} still present`);
}
// A vanished target proves nothing if the node itself vanished.
if (!head.nodeSeen && suggestion.requireNodePresent !== false) {
  // scan.mjs records whether the patched selector resolved during the run.
  const seen = JSON.parse(readFileSync(headPath, 'utf8')).targetsResolved ?? [];
  if (!seen.includes(suggestion.finding.target)) {
    problems.push(`target node ${suggestion.finding.target} was not in the DOM`);
  }
}
// Condition 2: nothing rose, anywhere.
if (head.total > base.total) {
  problems.push(`total nodes rose ${base.total} -> ${head.total}`);
}
for (const [ruleId, count] of head.perRule) {
  const before = base.perRule.get(ruleId) ?? 0;
  if (count > before) problems.push(`${ruleId} rose ${before} -> ${count}`);
}
// New fingerprints are collateral damage even when a total happens to match.
for (const print of head.prints) {
  if (!base.prints.has(print)) problems.push(`new finding ${print} appeared`);
}

for (const p of problems) console.error(`FAIL ${p}`);
console.log(`base ${base.total} nodes, head ${head.total} nodes`);
if (problems.length > 0) process.exit(1);
console.log('conditions 1 and 2 passed');

The targetsResolved check deserves a note. A finding disappears from a scan for two reasons: it was fixed, or the node stopped existing. The second is far more common than teams expect — a stale selector, a route that failed to render, a virtualised row that scrolled out — and it produces a clean green comparison that means nothing at all. Recording which of the patch’s targets actually resolved during the run turns that silent pass into an explicit failure.

# Validation

The diff-scope check enforces conditions five and six and is the one place where “the model has no authority over the judge” becomes executable. It denies a path list, requires the diff to be a single file matching the suggestion’s declared source path, and then reduces the one replaced line to a skeleton with the declared attribute removed — if the skeletons of the removed and added lines are not identical, the patch changed something it did not declare.

// a11y/ci/check-diff-scope.mjs — usage: node check-diff-scope.mjs <base-sha>
import { execFileSync } from 'node:child_process';
import { readFileSync } from 'node:fs';

const baseSha = process.argv[2];
if (!baseSha) throw new Error('a base sha is required');
const suggestion = JSON.parse(readFileSync('suggestion.json', 'utf8'));
const git = (...args) => execFileSync('git', args, { encoding: 'utf8' });

// Anything that can turn a red run green is out of bounds for a patch.
const DENY = [
  [/\.(test|spec)\.[cm]?[jt]sx?$/, 'a test file'],
  [/(^|\/)__tests__\//, 'a test directory'],
  [/(^|\/)tests?\//, 'a test directory'],
  [/(^|\/)__snapshots__\//, 'a committed snapshot'],
  [/\.snap$/, 'a committed snapshot'],
  [/(^|\/)a11y\/suppressions\.json$/, 'a suppression list'],
  [/(^|\/)a11y\/budget\.json$/, 'a violation budget'],
  [/(^|\/)\.axe-?(rc|ignore)$/, 'a scanner configuration'],
  [/(^|\/)(playwright|vitest|jest)\.config\.[cm]?[jt]s$/, 'a runner config'],
  [/(^|\/)\.github\/workflows\//, 'a workflow definition'],
  [/(^|\/)(package-lock\.json|pnpm-lock\.yaml|yarn\.lock)$/, 'a lockfile'],
];

const files = git('diff', '--name-only', `${baseSha}...HEAD`)
  .split('\n').filter(Boolean);
const problems = [];

if (files.length === 0) problems.push('empty diff: nothing to verify');
if (files.length > 1) problems.push(`${files.length} files changed, expected 1`);
for (const file of files) {
  const hit = DENY.find(([rx]) => rx.test(file));
  if (hit) problems.push(`${file} is ${hit[1]} and must never be patched`);
}
if (files[0] && files[0] !== suggestion.finding.sourcePath) {
  problems.push(
    `changed ${files[0]}, suggestion declared ${suggestion.finding.sourcePath}`);
}

// --unified=0 gives only the changed lines, with no surrounding context.
const lines = git('diff', '--unified=0', `${baseSha}...HEAD`).split('\n');
const removed = lines.filter((l) => l.startsWith('-') && !l.startsWith('---'));
const added = lines.filter((l) => l.startsWith('+') && !l.startsWith('+++'));

if (removed.length !== 1 || added.length !== 1) {
  problems.push(`expected one replaced line, got -${removed.length} +${added.length}`);
} else {
  const { attribute, value } = suggestion.envelope;
  const inserted = `${attribute}="${value}"`;
  // Strip any form of the declared attribute from both sides, then compare:
  // identical skeletons prove nothing else on the line moved.
  const attrRe = new RegExp(`\\s*${attribute}="[^"]*"`, 'g');
  const skeleton = (l) => l.slice(1).replace(attrRe, '').replace(/\s+/g, ' ').trim();
  if (skeleton(added[0]) !== skeleton(removed[0])) {
    problems.push('the added line changes more than the declared attribute');
  }
  if (!added[0].includes(inserted)) {
    problems.push(`the added line does not contain the approved value ${inserted}`);
  }
}

for (const p of problems) console.error(`FAIL ${p}`);
if (problems.length > 0) {
  console.error(`${problems.length} scope violation(s); the patch is rejected.`);
  process.exit(1);
}
console.log(`conditions 5 and 6 passed on ${files[0]}`);

Prove the check rejects before trusting it to accept. Point it at a branch that also edited the spec covering the patched component, and it fails in under a second with no browser and no build:

node a11y/ci/check-diff-scope.mjs "$(git merge-base origin/main HEAD)"
# FAIL 2 files changed, expected 1
# FAIL src/components/OrderRow.test.tsx is a test file and must never be patched
# FAIL expected one replaced line, got -3 +2
# 3 scope violation(s); the patch is rejected.
echo $?
# 1

A clean single-attribute patch produces one line and exit 0:

node a11y/ci/check-diff-scope.mjs "$(git merge-base origin/main HEAD)"
# conditions 5 and 6 passed on src/components/OrderRow.tsx
echo $?
# 0

Condition four is the accessibility-tree snapshot, and its assertion is about the shape of the diff rather than its content. Capture the tree for the patched route, compare it to the committed baseline node by node, and require that exactly one node differs and that it is the node the suggestion targeted. The capture and serialisation mechanics are covered in snapshot testing accessibility trees to prevent regressions, and the naming assertions that make the snapshot readable are in asserting accessibility tree names with Playwright snapshots.

# Edge Cases and Conditional Guards

• aria-busy and unsettled routes. If the patched node sits inside a container that is still aria-busy="true" when the scan runs, the comparison reads a transitional tree and the fingerprint diff is noise. Wait for the container to flip to false and assert the patched selector resolves before calling the scanner; a target that never resolved must fail condition one rather than pass it silently.
• Virtualised and conditionally rendered targets. A patch aimed at a table row that is unmounted at scan time produces a clean green run that proves nothing. Record targetsResolved during the scan, and treat an unresolved target the same way as a still-present violation — the evidence is missing, so the condition is not met.
• A legitimate net improvement that adds a violation elsewhere. Occasionally a correct fix genuinely raises another rule’s count, for instance when naming a previously ignored control brings it into the scope of a name-uniqueness rule. The gate must still reject it: split the work into two patches, or make the change by hand where a human author can explain it in the pull request. Widening the gate for the exception removes it for everything, and the alternative discipline of a slowly tightened budget belongs in a human-authored workflow, not this one.

# Pipeline Impact

Make this job a required status check on the default branch so the automatic close is a formality rather than the enforcement — closing the pull request is a tidiness measure, and branch protection is what actually prevents the merge. The same reasoning applies to critical findings on human pull requests, where the mechanics of a blocking check are set out in blocking pull requests on critical accessibility violations.

Two operational numbers matter. The first is the job’s wall-clock time, which is dominated by building two revisions; caching node_modules for the worktree and reusing a warm browser image keeps a typical run near six minutes, and a gate that takes twenty minutes gets bypassed. The second is the distribution of failures by condition, published to the same store as the rest of the trend data. A gate that only ever fails on condition one is a gate whose other five checks have never been exercised, which usually means the drafting pipeline is producing suggestions so narrow that the interesting failures never arrive — or that a check has silently broken.

Retain the evidence artifact for thirty days rather than the default fourteen. When a name accepted by this gate turns out to be wrong, the two scans and the suggestion envelope are the record that shows whether the gate was fooled or the reviewer was, and those are very different problems with very different fixes. The constraints that should have caught a wrong name earlier — before it ever became a patch — are the subject of using LLMs to suggest ARIA labels safely.

# Common Pitfalls

• Comparing against a committed baseline JSON file instead of a scan of the merge base on the same runner, which attributes a week of unrelated drift to one attribute change.
• Diffing totals only. A patch that removes one button-name node and adds one aria-hidden-focus node leaves the total flat, which is why the comparison also diffs per-rule counts and individual fingerprints.
• Treating a missing target node as a pass. The most common cause of a green verification run on a broken patch is a selector that resolved to nothing at scan time.
• Allowing --update-snapshots anywhere in the job. A patch branch that can rewrite the baseline it is being measured against has no gate at all, and the flag is easy to inherit from a shared npm script.
• Running the diff-scope check after the build. It needs no dependencies and it catches the most serious class of failure, so putting it last wastes several minutes on patches that were never admissible.
• Amending a rejected patch instead of regenerating it. An amended machine patch accumulates a history no reviewer can follow; close the pull request, fix the drafting input, and produce a fresh one.
• Granting the workflow contents: write so it can “just fix” a failing condition. Verification is read-only by design, and the only write it needs is the comment and close on the pull request.

# FAQ

Why close the pull request automatically instead of leaving it red for someone to look at? Because a queue of red machine-authored pull requests trains reviewers to ignore the label, and within a few weeks a genuinely interesting failure is invisible among twenty stale ones. A rejected patch also has nothing worth salvaging: the input that produced it is still in the drafting job’s artifact, regenerating is cheap, and the failure reason is in the run log and the evidence artifact. Closing keeps the review surface at exactly the size a person will actually read.

Does the test suite really need to run, given the scans already ran? Yes, and it is often the condition that catches the most subtle problems. A getByRole('button', { name: 'Ship' }) query is a written specification of an accessible name, so a patch that changes a name and breaks that query has changed a contract somebody wrote down deliberately. The scanners have no opinion about that; they only know whether a rule fires. A broken query on a name change is signal, and the correct response is to close the pull request and decide which of the two — the name or the specification — is wrong.

How does this gate behave on a pull request from a fork? It should not run at all, and the if: startsWith(github.head_ref, 'a11y/draft/') condition is what ensures that: the drafting identity pushes branches into the repository, so a fork branch never matches the prefix and the job is skipped. Do not reach for pull_request_target to make it work on forks — that runs the base branch’s workflow with write-capable secrets against untrusted code, which is a far larger problem than the one it solves.

# Related

• AI-Assisted Accessibility Remediation — the operating model that produces the patches this gate verifies, and the review step that follows it.
• Using LLMs to Suggest ARIA Labels Safely — the constraints that reject an unsafe accessible name before it ever becomes a commit.
• CI/CD Integration & Automated Quality Gating — required status checks, branch policies and the wider gating machinery this job plugs into.