AI-Assisted Accessibility Remediation with a Deterministic Approval Gate

A language model is a fast, cheap, plausible-text generator, and a large share of accessibility findings are missing text: an image with no alt, an icon control with no accessible name, an untitled iframe. That match is real, and it is also the trap, because plausible text is exactly the kind of wrong answer no scanner can detect. This guide is part of Automated Remediation & Accessibility Fixing Patterns, and it describes an operating model in which the model drafts and something deterministic decides — never the other way round.

Problem Statement

The failure mode that makes this dangerous is not a model that returns nonsense. Nonsense is caught: it fails a JSON parse, it fails a length check, it fails a regular expression. The dangerous output is a fluent, confident accessible name that reads correctly to a reviewer skimming a diff, satisfies every automated rule, and makes the page worse for the people the fix was for.

Take an image submit button in a legacy admin table. The graphic reads Delete, the markup is <input type="image" src="/i/remove-item.png" alt="">, and axe reports input-image-alt at serious impact. A model handed the element and its surroundings sees the filename remove-item.png and the column header “Remove”, and proposes alt="Remove item". The scan turns green. The accessible name no longer matches the visible label, so WCAG 2.2 SC 2.5.3 (Label in Name) now fails: a speech-input user who says “click Delete” — the only word they can see — gets no match, and a screen-reader user who is being coached over the phone hears a different word from the one their colleague is reading aloud. The original defect was an unlabelled control, which is at least discoverable by touch and by trial. The replacement defect is a control that lies about its own name, and no rule in any scanner’s catalogue will ever fire on it again.

A suggested alt text that turns the gate green and breaks Label in Name The left column shows an image button with an empty alt attribute, one serious axe violation, and a speech-input user with no name to speak. The right column shows the same button with the suggested alt Remove item, zero axe violations, and a speech-input user whose spoken word Delete no longer matches. A band across the bottom names the new failure as SC 2.5.3. Before: the reported finding After: the suggested attribute <input type="image" src="/i/remove-item.png" alt=""> <input type="image" src="/i/remove-item.png" alt="Remove item"> axe: 1 violation, serious rule id input-image-alt axe: 0 violations, gate green no rule id fires again Speech input: nothing to say control has no name at all Speech input: "click Delete" no match, command is dropped The graphic reads "Delete", so the new name fails SC 2.5.3 Label in Name. A green scan now proves nothing: the only check left is a human looking at the pixels.
The suggestion removed the rule that could detect the problem and replaced it with a problem no rule detects, which is why a green re-scan is a necessary condition and never a sufficient one.

There is a second shape of the same failure: the model puts a correct-sounding value on the wrong node. Asked to fix an image-alt finding on a decorative icon inside a button that already reads “Delete”, a model will frequently propose aria-label="Remove item" on the button rather than alt="" on the icon, because labelling the interactive ancestor is the pattern it has seen most often. That single misplacement overrides an accessible name that was already correct, produces the same SC 2.5.3 failure, and again clears every rule. The lesson is not “models are bad at ARIA”. The lesson is that the choice of node, the choice of attribute and the decision to accept are three separate authorities, and a drafting model can hold none of them.

So the division of labour has to be written down and enforced by the pipeline, not by a convention in a runbook. The model proposes a value for a pre-selected attribute on a pre-selected node. The scanner and the test suite decide whether the proposal is admissible. A named human decides whether it is true. Everything the model must never be able to reach — an assertion, a spec file, a suppression list, a violation budget, the merge button — is enforced by a check on the diff itself, because a suggestion loop that is allowed to edit the thing that judges it will converge on deleting the judge.

The authority boundary around a drafting model The left enclosure lists four permitted proposals: one ARIA attribute value, an alt value on one image, a visible label string, and an explicit abstention. The right enclosure lists four things outside the model's authority: editing an assertion, widening a suppression list, raising a violation budget, and approving or merging a branch. Both point down into a bar naming the gate and a reviewer as the only approvers. The model may propose one value for one named attribute alt text for one already-chosen node a visible label string to render an abstention with a stated reason The model has no authority over editing or deleting an assertion widening a suppression list raising a violation budget approving or merging a branch Only the deterministic gate and a named reviewer can approve a change. The right-hand list is enforced by a check on the diff, not by a prompt instruction.
Everything on the right is enforced against the produced diff, because an instruction in a prompt is a request and a check on the diff is a guarantee.

Key implementation targets:

  • An eligibility list that names the exact rule ids a drafted fix is allowed to address, and a blocked list for everything whose repair is structural rather than textual.
  • A context payload assembled from the DOM and the component source, containing the visible text the fix must agree with and nothing about the test suite.
  • An output contract that admits exactly one attribute value on exactly one node, plus a first-class way for the model to abstain.
  • A verification gate whose six conditions are all necessary and which closes the pull request automatically when any of them fails.
  • A review surface that puts the visible text and the proposed name side by side, so the one judgement a machine cannot make is the only thing left for the human to make.
  • Provenance on every accepted change — model identifier, payload hash, gate verdicts — so a bad name found six months later is traceable to the batch that produced it.

Prerequisites

1. Scope Which Rule IDs Are Eligible for a Suggested Fix

Eligibility is the cheapest safety control on this list and the one most often skipped. A rule id belongs on the eligible list only when the entire repair is a short human-readable string, when the node that must carry that string is unambiguous, and when no deterministic transform can already produce it. That is a small set: image-alt, input-image-alt, area-alt, frame-title, and the two accessible-name rules button-name and link-name when the control genuinely has no text node to promote.

Everything else is blocked, and it is worth being blunt about why. color-contrast is a design-token decision with a brand owner. heading-order is a document-outline change whose correct answer depends on content the model cannot see below the fold. aria-required-children needs a structural edit that moves nodes. A model will produce confident output for all three, and every one of those outputs is a guess dressed as a fix. Route them to a scored ticket instead.

The second half of the routing decision is where the finding lives. A missing label inside a shared component should never be patched at the call site: fixing it once in the component removes it from every consumer, which is the job of design-system accessibility defaults. And a finding with dozens of identical nodes is a transform, not a suggestion — a repeated mechanical edit belongs in codemod-driven accessibility fixes, where the diff is generated by an AST and is reviewable as one pattern rather than as forty independent strings.

// a11y/ai/eligibility.mjs
// A finding may be drafted by a model only when the whole repair is one short
// string on one unambiguous node. Everything else routes elsewhere.
export const ELIGIBLE = {
  'image-alt':       { attribute: 'alt',        onNode: 'self' },
  'input-image-alt': { attribute: 'alt',        onNode: 'self' },
  'area-alt':        { attribute: 'alt',        onNode: 'self' },
  'frame-title':     { attribute: 'title',      onNode: 'self' },
  'button-name':     { attribute: 'aria-label', onNode: 'self' },
  'link-name':       { attribute: 'aria-label', onNode: 'self' },
};

// Never drafted: the repair is structural, visual, or depends on content the
// payload cannot contain. These go to a scored ticket with an owner.
export const BLOCKED = new Set([
  'color-contrast', 'color-contrast-enhanced', 'heading-order',
  'aria-required-children', 'aria-required-parent', 'listitem',
  'aria-hidden-focus', 'scrollable-region-focusable', 'region',
  'duplicate-id-aria', 'nested-interactive', 'tabindex',
]);

const SHARED_PREFIXES = ['packages/ui/', 'packages/icons/'];

export function route(finding) {
  if (BLOCKED.has(finding.ruleId)) return { strategy: 'ticket' };
  const spec = ELIGIBLE[finding.ruleId];
  if (!spec) return { strategy: 'ticket' }; // default deny, not default allow
  if (SHARED_PREFIXES.some((p) => finding.sourcePath.startsWith(p))) {
    return { strategy: 'component-default' }; // fix once, not per call site
  }
  // A repeated mechanical edit is a transform; a one-off string is a draft.
  if (finding.nodeCount > 12) return { strategy: 'codemod' };
  return { strategy: 'model-draft', ...spec };
}

Default deny is the important line in that file. A new axe minor release adds rule ids, and an allow-list that falls through to “draft it” will silently start generating patches for rules nobody evaluated. Pin the list, and treat adding an entry to it as a reviewed change with its own justification.

How many findings actually reach the drafting model One report node holding 181 findings across 26 rule ids fans out to four strategy boxes: 94 findings to a deterministic codemod, 41 to a component-level default, 31 to a model-drafted string, and 15 to a scored ticket. Only the model-draft box is marked as requiring the verification gate. 181 findings 26 rule ids one nightly scan 94 to a deterministic codemod repeated attribute surgery, one commit per rule id 41 to a component-level default fixed once in the shared package, gone everywhere 31 to a model-drafted string every one must clear all six gate conditions 15 to a scored ticket with an owner contrast, heading order, required children 17% of one report reaches the model, and none of it reaches the default branch unreviewed
Routing is what keeps the risky path small: on this report seventeen percent of findings were drafted, and the other eighty-three percent were repaired by mechanisms that need no judgement at all.

2. Assemble the Context Payload

A model’s answer is a function of what it was shown. Almost every bad accessible name in practice traces back to a payload that contained the filename, the URL and the class names, but not the pixels — so the model inferred meaning from developer-facing strings rather than from user-facing ones. The payload has one job: put the visible truth in front of the model and leave the developer folklore out.

Four things belong in it. First, the element itself, serialised and truncated, so the model can see the tag, the role and any attributes already present. Second, the visible text on and around the element as the browser computes it after layout — the element’s own rendered text, the rendered text of up to three visible siblings, and the text of any <label> that points at it. Third, up to three ancestor tags with their roles, which is how the model learns that this control lives inside a table row rather than a toolbar. Fourth, a small slice of the component source around the reported line, because a prop named confirmDestructive carries real information that the DOM alone does not.

Two things are deliberately excluded. The payload contains no test file, no spec name and no snapshot content, so the model has no material with which to propose an edit to its own judge. And it contains no scanner suppression file or budget file, for the same reason. Excluding them from the context is not the enforcement mechanism — the diff-scope check in the gate is — but a model that has never seen a suppression list is markedly less likely to invent one.

// a11y/ai/context.mjs — assembles the only context the model receives.
import { readFileSync } from 'node:fs';
import { createHash } from 'node:crypto';

export async function buildPayload(page, finding, { sourceLines = 20 } = {}) {
  const dom = await page.evaluate((selector) => {
    const el = document.querySelector(selector);
    if (!el) return null;
    // "Visible" means laid out and not hidden: this is the text a sighted
    // user can read, which is the only text SC 2.5.3 cares about.
    const shown = (n) => {
      const r = n.getBoundingClientRect();
      return r.width > 0 && r.height > 0 &&
        getComputedStyle(n).visibility !== 'hidden';
    };
    const ancestors = [];
    for (let n = el.parentElement; n && ancestors.length < 3; n = n.parentElement) {
      const role = n.getAttribute('role');
      ancestors.push(n.tagName.toLowerCase() + (role ? `[role=${role}]` : ''));
    }
    const siblings = Array.from(el.parentElement?.children ?? [])
      .filter((n) => n !== el && shown(n))
      .map((n) => n.innerText.trim())
      .filter(Boolean)
      .slice(0, 3);
    const forId = el.id
      ? document.querySelector(`label[for="${el.id}"]`)
      : null;
    return {
      outerHTML: el.outerHTML.slice(0, 600), // truncated: no giant subtrees
      visibleText: shown(el) ? el.innerText.trim() : '',
      visibleLabel: forId && shown(forId) ? forId.innerText.trim() : '',
      siblingText: siblings,
      ancestorPath: ancestors.reverse().join(' > '),
      // Names already used by sibling controls, for the duplicate check later.
      siblingNames: Array.from(document.querySelectorAll('button, a[href]'))
        .filter((n) => n !== el && n.parentElement === el.parentElement)
        .map((n) => (n.getAttribute('aria-label') || n.innerText).trim())
        .filter(Boolean),
    };
  }, finding.target);

  if (!dom) throw new Error(`target vanished before capture: ${finding.target}`);

  const lines = readFileSync(finding.sourcePath, 'utf8').split('\n');
  const from = Math.max(0, finding.sourceLine - Math.floor(sourceLines / 2));
  const payload = {
    ruleId: finding.ruleId,
    attribute: finding.attribute,
    target: finding.target,
    documentLang: finding.documentLang,
    sourceSlice: lines.slice(from, from + sourceLines).join('\n'),
    ...dom,
  };
  // Hash the payload so an accepted suggestion is reproducible later.
  payload.payloadHash = createHash('sha256')
    .update(JSON.stringify(payload))
    .digest('hex')
    .slice(0, 12);
  return payload;
}

Capture the payload once per finding and store it next to the suggestion. When a reviewer six weeks later asks why the model wrote “Remove item”, the answer is in the payload, and if the answer is “because the payload never contained the word Delete”, the fix is to the capture step rather than to the prompt.

3. Define the Constrained Output Contract

The output contract is where “one attribute on one node” stops being a policy and becomes a parse error. The model returns a single JSON object with a closed set of fields, and the validator rejects anything else before the value reaches a DOM, a file or a person. Unknown fields are a rejection rather than a warning, because an extra field is the first sign that the model has decided to explain itself in prose or to bundle a second edit.

Three parts of the contract earn their keep. The attribute field must equal the attribute the eligibility routing already chose — the model is confirming, not selecting, and a mismatch is a rejection. The value field is plain text: no angle brackets, no ampersand entities, no newlines, and a length ceiling, so a suggestion can never smuggle markup into an attribute. And abstain is a first-class success, not an error path: a model that answers “the payload does not contain the visible text of this control” has produced the single most useful output in the whole system, and a contract with no abstention forces a guess.

// a11y/ai/contract.mjs — the suggestion envelope and its validator.
const ALLOWED_FIELDS = new Set([
  'fingerprint', 'ruleId', 'target', 'attribute', 'value', 'rationale',
  'abstain', 'abstainReason',
]);
const MAX_VALUE = 90;      // characters; a name, not a description
const MAX_RATIONALE = 240; // characters; for the reviewer, not for the DOM

export function validateEnvelope(raw, finding, spec) {
  const problems = [];
  let env;
  try {
    env = JSON.parse(raw);
  } catch {
    return { ok: false, problems: ['not valid JSON'] };
  }
  if (env === null || typeof env !== 'object' || Array.isArray(env)) {
    return { ok: false, problems: ['envelope is not a JSON object'] };
  }
  for (const key of Object.keys(env)) {
    if (!ALLOWED_FIELDS.has(key)) problems.push(`unknown field "${key}"`);
  }
  if (env.fingerprint !== finding.fingerprint) problems.push('fingerprint mismatch');
  if (env.ruleId !== finding.ruleId) problems.push('ruleId mismatch');
  if (env.target !== finding.target) problems.push('target reassigned by model');
  if (env.abstain === true) {
    if (!env.abstainReason) problems.push('abstain without a reason');
    return { ok: problems.length === 0, abstain: true, problems, envelope: env };
  }
  // The attribute was chosen by routing; the model may only confirm it.
  if (env.attribute !== spec.attribute) {
    problems.push(`attribute "${env.attribute}" is not "${spec.attribute}"`);
  }
  if (typeof env.value !== 'string') problems.push('value is not a string');
  else {
    if (env.value !== env.value.trim()) problems.push('value has edge whitespace');
    if (env.value.length === 0) problems.push('value is empty');
    if (env.value.length > MAX_VALUE) problems.push(`value over ${MAX_VALUE} chars`);
    if (/[<>&\n\r]/.test(env.value)) problems.push('value contains markup or newline');
  }
  if (typeof env.rationale !== 'string' || env.rationale.length > MAX_RATIONALE) {
    problems.push('rationale missing or too long');
  }
  return { ok: problems.length === 0, abstain: false, problems, envelope: env };
}

The name-specific constraints — that the value must contain the visible text verbatim, that it must not be generic, that it must not collide with a sibling control’s name — are the subject of using LLMs to suggest ARIA labels safely, which sets out both the hard rules in the prompt and the post-filter that applies them again after the fact. Keep those checks in their own module: the envelope validator answers “is this a well-formed proposal”, and the name filter answers “is this a defensible name”. Conflating them produces error messages nobody can act on.

4. Make the Verification Gate the Only Approver

A suggestion becomes a candidate change by being committed to a throwaway branch, and it becomes a merge candidate only by clearing six conditions. All six are necessary; none is sufficient; and the gate reports each one separately, because “the AI fix failed” is not an actionable message while “the total violation count rose from 12 to 14 on /orders” is.

The first condition is that the specific finding the patch claimed to fix is absent from the re-scan, matched by fingerprint rather than by count. The second is that the total violation count has not risen anywhere in the scanned routes, which catches the collateral damage of an attribute that changes a parent’s accessible name. The third is that the existing test suite is still green — a suggested aria-label frequently breaks a getByRole('button', { name: … }) query, and that broken query is a signal, not an obstacle. The fourth is that the accessibility-tree snapshot diff touches only the node the patch targeted, the technique described in regression prevention after fixes. The fifth is that the diff touches no test file and no suppression list. The sixth is that the diff is genuinely one attribute on one element, which is what stops a patch from quietly reformatting a file.

// a11y/ai/verify.mjs — the only thing allowed to approve a suggestion.
// Usage: node a11y/ai/verify.mjs suggestion.json
import { readFileSync } from 'node:fs';
import { targetFindingAbsent, totalDidNotRise } from './checks/scan.mjs';
import { suiteIsGreen } from './checks/suite.mjs';
import { snapshotScopedToNode } from './checks/snapshot.mjs';
import { diffTouchesNoJudge, diffIsSingleAttribute } from './checks/diff.mjs';

const suggestion = JSON.parse(readFileSync(process.argv[2], 'utf8'));

const CONDITIONS = [
  ['target finding absent',       targetFindingAbsent],
  ['total count did not rise',    totalDidNotRise],
  ['test suite green',            suiteIsGreen],
  ['snapshot scoped to one node', snapshotScopedToNode],
  ['no test or suppression edit', diffTouchesNoJudge],
  ['one attribute on one node',   diffIsSingleAttribute],
];

const verdicts = [];
for (const [name, check] of CONDITIONS) {
  // Every condition runs even after a failure, so one report explains
  // everything that is wrong rather than only the first thing.
  const result = await check(suggestion);
  verdicts.push({ name, pass: result.pass, detail: result.detail });
}

for (const v of verdicts) {
  console.log(`${v.pass ? 'PASS' : 'FAIL'}  ${v.name}  ${v.detail}`);
}
const failed = verdicts.filter((v) => !v.pass);
if (failed.length > 0) {
  console.error(`${failed.length} of 6 conditions failed; closing the branch.`);
  process.exit(1); // the workflow closes the pull request on this exit code
}
console.log('6 of 6 conditions passed; requesting human review.');
The six conditions a machine-authored patch must clear Six stacked conditions run in order: the target finding is absent, the total count did not rise, the test suite is green, the tree snapshot diff is limited to one node, no test file or suppression list was touched, and the diff is one attribute on one element. Each has an arrow into a tall panel on the right stating that any failure closes the pull request automatically. Clearing all six leads to a green box requesting human review. All six are necessary 1 target finding absent from the re-scan 2 total violation count did not rise 3 existing test suite still green 4 tree snapshot diff limited to one node 5 no test file and no suppression edited 6 diff is one attribute on one element Any one fails the pull request closes and the branch is deleted no retry loop, no override review requested from a named person
Clearing all six conditions earns a review request and nothing more, because the gate can only prove a change is admissible and never that it is true.

The workflow that implements these conditions, including the check that inspects the changed file paths and the automatic close, is written out in validating AI-generated ARIA fixes in CI. Two design decisions there are worth stating here because they are easy to get wrong. Compare against a baseline captured from the merge base on the same runner, not against a stored file from last week, or normal drift in the application will be attributed to the patch. And run every condition even after the first failure, so one report tells a reviewer everything that is wrong.

5. Require Human Review for Anything Semantic

Every accessible name is semantic, so in practice every drafted fix reaches a person. The review step’s design goal is therefore to make the one judgement that matters cheap to make in ten seconds, and to make it impossible to make by accident.

Cheap means the reviewer never opens the app. Put the visible text and the proposed name adjacent in the pull-request body, in that order, with a rendered screenshot of the element’s bounding box attached as an artifact. A reviewer comparing the word in the picture against the word in the attribute is doing exactly the check no machine can do, and it takes seconds. Bury the same information in a unified diff of a JSX file and the check silently degrades into “the CI is green, approve”.

Impossible-by-accident means the bot cannot approve, cannot dismiss a stale review, and cannot enable auto-merge. Grant the identity that pushes the branch contents: write and nothing else, require one approving review from a named team, and require that team to be a human team rather than a group containing the bot. Configure branch protection so the verification job is a required status check, as set out in pull request gating and branch policies; a gate that a service account can bypass is a suggestion.

// a11y/ai/review-comment.mjs — renders the surface a human decides on.
// Usage: node a11y/ai/review-comment.mjs suggestion.json verdicts.json
import { readFileSync } from 'node:fs';

const s = JSON.parse(readFileSync(process.argv[2], 'utf8'));
const verdicts = JSON.parse(readFileSync(process.argv[3], 'utf8'));

const rows = [
  ['Visible text on the control', s.payload.visibleText || '(none rendered)'],
  ['Proposed accessible name', s.envelope.value],
  ['Attribute and node', `${s.envelope.attribute} on ${s.envelope.target}`],
  ['Rule id repaired', s.finding.ruleId],
  ['Source', `${s.finding.sourcePath}:${s.finding.sourceLine}`],
  ['Model and payload', `${s.provenance.model} / ${s.payload.payloadHash}`],
];

console.log('### One question for the reviewer\n');
console.log('Does the proposed name match the words a sighted user can read');
console.log('on this control? If not, close this pull request.\n');
console.log('| Field | Value |');
console.log('|---|---|');
for (const [k, v] of rows) console.log(`| ${k} | ${v} |`);

console.log('\n### Gate verdicts\n');
for (const v of verdicts) {
  console.log(`- ${v.pass ? 'pass' : 'FAIL'} — ${v.name}: ${v.detail}`);
}
// A visible-text mismatch is legal only when there is no visible text at all.
if (s.payload.visibleText &&
    !s.envelope.value.toLowerCase().includes(s.payload.visibleText.toLowerCase())) {
  console.log('\n> The name does not contain the visible text (SC 2.5.3).');
  console.log('> This pull request should be closed, not amended.');
}

One organisational rule makes the difference between this working and this rotting: the reviewer who approves a drafted name owns that name. Not the platform team that built the pipeline, not the model. If a name turns out to be wrong in a user report three months later, the trail leads to a person who looked at a picture and a string and said yes. That accountability is what keeps the ten-second check honest, and it is the only reason the whole arrangement is safer than typing the labels by hand.

Pipeline Integration

Generation and verification belong in separate workflows with separate triggers, separate permissions and separate failure semantics. Generation runs on a schedule against the default branch, reads a report, produces suggestions, and opens one pull request per rule id with at most eight nodes in it. Verification runs on pull_request and is the required status check. Splitting them means a model outage never blocks a developer’s pull request, and a verification bug never allows a generation run to merge anything.

name: a11y-draft-suggestions
on:
  schedule:
    - cron: '20 3 * * 2' # Tuesday 03:20 UTC, so review lands mid-week
  workflow_dispatch:
permissions:
  contents: write       # push the branch
  pull-requests: write  # open the pull request
  # deliberately absent: no checks:write, no admin, no approval rights
concurrency:
  group: a11y-draft-suggestions
  cancel-in-progress: false
jobs:
  draft:
    runs-on: ubuntu-24.04
    timeout-minutes: 25
    env:
      A11Y_MODEL_ENDPOINT: ${{ secrets.A11Y_MODEL_ENDPOINT }}
      A11Y_MODEL_KEY: ${{ secrets.A11Y_MODEL_KEY }}
      A11Y_MAX_SUGGESTIONS: '8' # cap the blast radius of one bad batch
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Scan and route findings
        run: node a11y/ai/scan-and-route.mjs --out routed.json
      - name: Draft suggestions for eligible findings only
        run: node a11y/ai/draft.mjs routed.json --out suggestions.json
      - name: Open one pull request per rule id
        run: node a11y/ai/open-prs.mjs suggestions.json
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: a11y-draft-provenance
          path: |
            routed.json
            suggestions.json
            a11y/ai/payloads/
          retention-days: 90

The provenance artifact retention is set to ninety days rather than the usual fourteen for a specific reason: the questions people ask about a machine-authored label arrive long after the pull request is merged, and a payload that has expired makes the answer unknowable. Route the batch-level counts — suggestions drafted, abstentions, gate failures by condition — into the same store as the rest of the trend data described in the reporting and violation-tracking guide, because the abstention rate is the single best health metric this pipeline has.

Troubleshooting and Flaky-Test Mitigation

The model returns prose around the JSON. Do not strip fences with a regular expression and retry silently; that is how a partially-parsed object with a truncated value reaches a DOM. Count the parse failure, retry once with the same payload, and on a second failure record an abstention with reason unparseable. A model that cannot emit an object for a given payload is telling you the payload is confusing.

The re-scan says the finding is gone, but only sometimes. This is a hydration race, not a fix. The scan is running before the framework has attached the attribute the patch added, so the node under test does not exist yet and no rule matches it. Wait for an application-asserted signal — a settled-route attribute, a resolved promise on window — and assert the target node is present before calling analyze(). Treating “target node not found” as a pass is the most common way this gate silently stops working.

A suggestion breaks a getByRole query. Expected, and load-bearing. A test that asks for a button by accessible name is a specification of that name, and a patch that changes the name without changing the test has changed the contract. Close the pull request, then decide deliberately whether the test or the name is wrong. Never let the drafting pipeline update the query, which is exactly what condition five exists to prevent.

The accessibility-tree snapshot diff is larger than one node. Usually an aria-label on an ancestor has suppressed the names of descendants, or a role change has moved children in the tree. This is real collateral damage and the correct outcome is a failure. If the extra nodes are genuinely unrelated churn — a timestamp, a generated id — normalise them in the snapshot serialiser rather than widening the scope of the check.

Two suggestions in one batch produce identical names. Sibling controls with the same accessible name are a usability failure even though no rule fires. Compare names within a batch before opening pull requests, and abstain on the collision rather than shipping both and hoping a reviewer notices.

The gate passes and the reviewer approves a wrong name anyway. The counter-measure is not more automation, it is the screenshot. If reviewers are approving without looking at the element, the review surface is failing, and the fix is to attach the cropped image of the control’s bounding box to the pull request body instead of a link to a build.

Common Pitfalls

  • Letting the eligibility list fall through to “draft it” for unknown rule ids, so an axe upgrade quietly widens the model’s scope with no review.
  • Giving the model the finding’s CSS selector and letting it choose the node, which produces attributes on interactive ancestors that override names that were already correct.
  • Building a payload from filenames, class names and URLs, so the model infers meaning from developer strings the user cannot see.
  • Comparing total violation counts against a stored baseline instead of one captured from the merge base on the same runner, which attributes ordinary drift to the patch.
  • Treating a green re-scan as approval, when a green re-scan on a name change proves only that the rule which could detect the mistake has been switched off by the mistake.
  • Allowing the drafting identity to enable auto-merge, dismiss reviews or push to a branch with no required checks — three separate misconfigurations with the same result.
  • Retrying an abstention until the model produces a value, which converts the most valuable output in the system into noise.
  • Omitting a provenance record, so a bad name discovered later cannot be traced to a batch, a payload or a decision.

FAQ

Is a model worth using at all if a human reviews every suggestion anyway? It depends entirely on what the human’s job becomes. Typing two hundred alt strings means opening two hundred files, finding two hundred elements and inventing two hundred phrasings; reviewing two hundred suggestions means comparing a picture and a string two hundred times, which is perhaps a tenth of the effort and a task humans do well. The gain is real but it is a review-throughput gain, not an automation gain, and any pipeline sold on the promise of removing the human has removed the only component that can detect the failure described at the top of this page.

What abstention rate should be expected? On a well-built payload, somewhere between fifteen and thirty percent of eligible findings — controls whose meaning genuinely is not present in the rendered page, images whose content the payload cannot describe, icons whose function depends on application state. A pipeline reporting a two percent abstention rate is not doing better; it is either guessing or its abstention path is broken. Track the rate per rule id and investigate a sudden drop the way you would investigate a sudden drop in test failures.

Can the same approach draft anything other than text? Cautiously, and only where the repair is still a single value with a small legal domain: a lang attribute on a foreign-language phrase, a scope value on a table header, a type on a button. All of them need their own gate conditions and their own review surface, and none of them has the property that makes text suggestions tractable, which is that a human can verify the answer by looking at one element for one second. Structural repairs — moving nodes, changing heading levels, adding required children — should not go through this pipeline at all.

How does this interact with a progressive violation budget? It must never touch it. The budget is part of the judge, so the diff-scope check treats the budget file exactly like a test file, and a patch that lowers a threshold to pass is a closed pull request. Ratcheting a budget down is a human decision made on trend data, and drafted fixes should show up as budget headroom appearing on its own rather than as an edit to the number.

What happens when the model endpoint is unavailable during a scheduled run? Nothing that matters, which is the point of separating the workflows. The drafting run fails, no branch is pushed, no pull request is opened, and the verification workflow — the one wired into branch protection — is untouched because it never calls a model. Developers’ pull requests keep passing the accessibility gate on their own merits, and the next scheduled drafting run picks the report up again.

In This Section

Back to Automated Remediation Fixing Patterns