# Auto-Fail vs Warning Workflows: Deciding What a Finding Does

A scanner answers one question — is this rule satisfied on this node — and a pipeline has to answer a different one: what should that answer do. This guide is part of CI/CD Integration & Automated Quality Gating, and it covers the decision that sits between the two: the policy that turns a result into a blocked merge, a message the author reads, or a row in a table nobody looks at until the quarter ends. Getting that policy wrong is the most common reason an accessibility programme stalls, and it is almost never a tooling problem.

# Problem Statement

Two failure modes account for nearly every abandoned accessibility gate. The first is enforcing everything on day one: a team enables the full WCAG AA rule set with a non-zero exit on any finding, the next fourteen pull requests all go red for color-contrast on a legacy footer, and by the end of the sprint someone has added the job to a skip list or set the whole thing to advisory “temporarily”. The second is enforcing nothing: the job runs, prints a violation count into a log that requires four clicks to reach, and eighteen months later the count is four times larger and nobody can name a single finding the job prevented.

Both come from treating severity and policy as the same thing. axe assigns each result an impact — a statement about how badly a user relying on assistive technology is affected — and teams read that as an instruction about pipeline behaviour. It is not. Impact tells you how much a failure costs a user; policy tells you how much a false failure costs the team, how confident you are that the rule is right on this codebase, and whether the person who can fix it is the person whose pull request is being blocked. Those are separate inputs, and the policy needs all of them.

The workable model is a three-tier response — block, warn, record — applied per rule rather than per severity band, with a written promotion path from the weakest tier to the strongest. Every rule enters at record. It earns warn by being observed, and it earns block by proving, on real pull requests, that it does not fire when it should not. Alongside those three tiers sits a fourth bucket the model has to handle explicitly: axe’s incomplete results, which are neither pass nor fail and are the one output that must reach a human rather than a threshold.

Key implementation targets:

• A committed policy file that maps every enabled rule id to block, warn, or record, with an owner and an expiry date on every warn entry.
• A classification step that reads raw scanner JSON and emits four labelled buckets, so no other step in the pipeline has to know the impact model.
• An advisory job that cannot fail the build, writes a readable run summary, and is never a required status check.
• A blocking job that is a separate check run with a stable name, evaluating only the block tier.
• An incomplete review artifact plus one tracked issue per rule, so undecidable results are triaged instead of averaged away.
• A promotion procedure with numeric entry criteria and a named rollback trigger, so tightening the gate is a routine change rather than an argument.

# Prerequisites

• Node.js 20.x or newer on the runner — the classifier below uses node:crypto and top-level await.
• axe-core 4.10 or newer, so every result carries both a stable impact value and a populated tags array.
• A scan step that writes one JSON file per URL into scan/ and sets no exit code of its own.
• .a11y/policy.json committed at the repository root with a review group listed as its owner.
• Workflow token permissions contents: read, pull-requests: write, and checks: write.
• Administrator access to branch protection, from someone who can add and remove a required check the same day.
• Somewhere to keep recorded findings for at least two quarters: a metrics store, a SQL table, or an object bucket.
• At least 30 pull requests per month against the protected branch, or the promotion numbers never reach a usable sample size.

# The Severity Contract

impact is per-rule metadata shipped with axe-core, occasionally refined at run time by the check itself (a contrast ratio of 1.2:1 and one of 4.4:1 are both failures, and axe reports the more severe one accordingly). A violation’s impact is the most serious impact among its failing nodes, which is why a single color-contrast violation covering forty nodes reports one impact value rather than forty. The four values mean the following, in terms of what a user actually experiences.

The critical property of that table is that the right-hand column is a default, not a derivation. Impact describes harm, and harm is uncorrelated with the cost of the fix. image-alt is critical and is usually a one-line change. aria-required-children is also severe and can require restructuring a component that six teams consume. landmark-one-main is moderate and is typically fixed by adding one element to a layout template. If a team lets fix cost leak into the tier decision, the policy quietly becomes “block the cheap things”, which is the opposite of a user-centred gate.

Two more properties of the contract matter in practice. First, results tagged best-practice carry an impact value but map to no WCAG success criterion; region and landmark-unique are the usual examples. They are useful advice and they must never be able to block a merge, because there is no external standard to appeal to when an author disagrees. Second, impact is a judgement made by the axe maintainers against the general web, not against your product. Re-mapping a rule is legitimate — a data-visualisation product may reasonably treat svg-img-alt as blocking even though its default impact is serious and its soak has not finished — but the re-map belongs in a reviewed file, with a comment, rather than in a shell condition inside a workflow.

# Three Responses: Block, Warn, Record

Each tier is defined by three things: where the result appears, who is expected to act, and how reversible the consequence is.

block is the only tier that spends someone else’s time, so it carries the highest evidence bar. A blocking rule must be deterministic on the same commit, must produce a message an author can act on without asking anyone, and must fire on something the author’s diff could plausibly have caused. A rule that fails a pull request touching only a build script, because a page it never rendered has a contrast problem from 2021, is technically correct and organisationally fatal — that is what diff-scoped scanning and pull request gating and branch policies exist to prevent.

warn is the tier that gets abused. A warning is not a weaker block; it is a message with a recipient. If no person’s name is attached to a warning tier, and no date by which the rule will either be promoted or deleted, the tier is a landfill. record is the honest version of that landfill: findings are stored, counted and trended, with zero output on the pull request, which is exactly right for minor and best-practice results and for any rule whose behaviour on this codebase is still unknown.

# A warning nobody reads is worse than no check at all

That claim sounds rhetorical and is not. An unread warning has four measurable costs that an absent check does not. It burns runner minutes on every pull request. It trains everyone on the team to scroll past the accessibility section of a run, which also devalues the blocking findings that appear in the same place. It produces a coverage claim — “we scan for that” — that survives into audit questionnaires and vendor accessibility statements even though nothing changed. And it consumes the political capital you will need later, because the second attempt to introduce a gate has to argue past “we already had that and it did nothing”.

The numbers below are from one product team’s quarter, measured as the share of first-appearance findings that were fixed within thirty days. Findings that were merely recorded almost never got fixed, which is fine — recording is a measurement tier, not an intervention. Findings warned about with no owner did barely better than recorded ones. The difference between a warning that works and a warning that does not is entirely the owner and the digest.

# The incomplete bucket

axe returns four arrays, and only two of them are verdicts. passes and violations are decided; inapplicable means no node matched; incomplete means a check ran, matched a node, and could not determine an answer. The common producers are color-contrast against a background image or a gradient, frame-tested when a cross-origin frame could not be instrumented, and any custom check whose evaluate returns undefined because the DOM does not contain enough information to decide.

An incomplete result must never fail a build, because the scanner has explicitly declined to make a claim, and failing on it manufactures a verdict axe refused to give. It must equally never be dropped, because the undecidable cases are disproportionately where real failures hide — text over a hero image is exactly the situation where contrast is worst and where automation is blindest. The only correct destination is a human, which in pipeline terms means a durable artifact plus a tracked item, configured in section 4 below. Automated scanning reliably resolves roughly 30–40% of WCAG failures; incomplete is the part of the remainder that the scanner is honest enough to flag, and treating it as a manual-testing queue is the highest-yield use of that honesty.

# 1. Classifying Results by Impact in the Report Step

Exactly one component in the pipeline should know the impact model. Everything downstream reads a labelled bucket. That keeps the policy auditable — one file, one review group — and means promoting a rule never involves editing a workflow.

The policy file lists only deviations from the impact defaults, plus the owner and expiry metadata that make the warn tier real:

{
  "version": 4,
  "defaultTier": "record",
  "impactDefaults": {
    "critical": "block",
    "serious": "warn",
    "moderate": "warn",
    "minor": "record"
  },
  "rules": {
    "color-contrast": "block",
    "link-name": "block",
    "target-size": "warn",
    "heading-order": "warn",
    "svg-img-alt": "record"
  },
  "warn": {
    "target-size": { "owner": "@acme/web-platform", "expires": "2026-09-15" },
    "heading-order": { "owner": "@acme/content-eng", "expires": "2026-10-01" }
  }
}

svg-img-alt sits at record deliberately: its default impact is serious, but on a charting-heavy product it fires on hundreds of generated glyphs, so it is being measured before anyone argues about enforcement. Note that serious defaults to warn here rather than block — the two serious rules that do block are listed explicitly, which is the whole point.

The classifier reads every scan file, applies the policy, and writes one JSON document with four buckets. It also fingerprints each finding so the same problem is recognisable across runs, which is what makes trend counting and de-duplication possible later:

// a11y/policy/classify.mjs
// Usage: node a11y/policy/classify.mjs scan/*.json > classified.json
import { readFileSync } from 'node:fs';
import { createHash } from 'node:crypto';

const policy = JSON.parse(readFileSync('.a11y/policy.json', 'utf8'));
const files = process.argv.slice(2);
if (files.length === 0) throw new Error('no scan files passed to the classifier');

// A fingerprint that survives re-runs: rule + the first CSS selector of the node.
const fingerprint = (ruleId, target) =>
  createHash('sha1').update(`${ruleId}|${target}`).digest('hex').slice(0, 12);

function tierFor(result) {
  // An explicit rule entry always wins, so every deviation is visible in review.
  if (policy.rules[result.id]) return policy.rules[result.id];
  // best-practice results map to no success criterion and can never block.
  if (result.tags.includes('best-practice')) return 'record';
  return policy.impactDefaults[result.impact] ?? policy.defaultTier;
}

const buckets = { block: [], warn: [], record: [], review: [] };

for (const file of files) {
  const page = JSON.parse(readFileSync(file, 'utf8'));
  const url = page.url ?? file;

  for (const result of page.violations) {
    const tier = tierFor(result);
    for (const node of result.nodes) {
      const target = Array.isArray(node.target) ? node.target[0] : String(node.target);
      buckets[tier].push({
        id: fingerprint(result.id, target),
        rule: result.id,
        // node.impact is the per-node severity; result.impact is the worst of them.
        impact: node.impact ?? result.impact,
        tier, url, target,
        help: result.help,
      });
    }
  }

  // incomplete is a third state: the check ran and declined to decide.
  for (const result of page.incomplete) {
    for (const node of result.nodes) {
      const target = Array.isArray(node.target) ? node.target[0] : String(node.target);
      buckets.review.push({
        id: fingerprint(result.id, target),
        rule: result.id,
        reason: node.any?.[0]?.message ?? 'check returned undefined',
        url, target,
      });
    }
  }
}

process.stdout.write(JSON.stringify({ policyVersion: policy.version, buckets }, null, 2));
process.exitCode = 0; // classification never sets pipeline status

That last line is not decoration. The classifier is a reporting step and must always succeed; if it can also fail, then a policy parsing bug becomes an accessibility failure and the two are indistinguishable in the checks list.

# 2. An Advisory Job That Always Succeeds

The advisory job’s contract is that it cannot go red for accessibility reasons. It scans, classifies, writes a summary humans will actually see, and exits zero. It may still fail for infrastructure reasons, and it should — a job that is green when the browser never launched is a lie, and distinguishing those two cases is the subject of choosing exit codes for warning and blocking a11y jobs.

name: a11y-advisory
on:
  pull_request:
    branches: [main]
permissions:
  contents: read
concurrency:
  group: a11y-advisory-${{ github.head_ref }}
  cancel-in-progress: true
jobs:
  advise:
    runs-on: ubuntu-24.04
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Scan the route list into scan/
        run: node a11y/scan.mjs --out scan  # writes JSON, never sets an exit code
      - name: Classify against the policy
        run: node a11y/policy/classify.mjs scan/*.json > classified.json
      - name: Write the run summary
        run: node a11y/policy/summary.mjs classified.json >> "$GITHUB_STEP_SUMMARY"
      - name: Keep the evidence
        if: always()  # a crashed scan still uploads whatever it produced
        uses: actions/upload-artifact@v4
        with:
          name: a11y-advisory-${{ github.run_id }}
          path: |
            scan/
            classified.json
          retention-days: 30

The summary writer groups by rule rather than by page, because an author fixing target-size fixes it once in a component and not eleven times across eleven URLs. It also prints the owner and expiry straight from the policy, which is what converts a warning into an assignment:

// a11y/policy/summary.mjs — usage: node a11y/policy/summary.mjs classified.json
import { readFileSync } from 'node:fs';

const { buckets } = JSON.parse(readFileSync(process.argv[2], 'utf8'));
const policy = JSON.parse(readFileSync('.a11y/policy.json', 'utf8'));

const byRule = (rows) =>
  rows.reduce((acc, r) => ((acc[r.rule] ??= []).push(r), acc), {});

console.log('## Accessibility advisory\n');
console.log(`Blocking findings: **${buckets.block.length}** · `
  + `warnings: **${buckets.warn.length}** · `
  + `needs a human: **${buckets.review.length}**\n`);

console.log('| Rule | Nodes | Owner | Decision due |');
console.log('|---|---|---|---|');
for (const [rule, rows] of Object.entries(byRule(buckets.warn))) {
  const meta = policy.warn[rule] ?? {};
  console.log(`| \`${rule}\` | ${rows.length} | ${meta.owner ?? 'UNOWNED'} `
    + `| ${meta.expires ?? 'UNSCHEDULED'} |`);
}
// UNOWNED and UNSCHEDULED are deliberately loud: they are policy bugs.
process.exitCode = 0;

# 3. The Blocking Job and Its Separate Status Check

The blocking decision belongs in its own job, not a later step of the advisory job, for one structural reason: a job maps one-to-one to a check run, and branch protection selects required checks by name. One job means one name, and one name means you can require a11y-block while leaving a11y-advisory optional — and you can remove that requirement in thirty seconds if the gate misbehaves at 4 p.m. on a Friday.

name: a11y-block
on:
  pull_request:
    branches: [main]
permissions:
  contents: read
jobs:
  block:
    runs-on: ubuntu-24.04
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Scan and classify
        run: |
          set -o pipefail  # a scanner crash must not be hidden by the next command
          node a11y/scan.mjs --out scan
          node a11y/policy/classify.mjs scan/*.json > classified.json
      - name: Enforce the block tier only
        run: node a11y/policy/enforce.mjs classified.json

enforce.mjs reads one bucket and nothing else. It never re-reads the raw scan, never re-applies the impact model, and prints each blocking finding in a form an author can paste into a search:

// a11y/policy/enforce.mjs — usage: node a11y/policy/enforce.mjs classified.json
import { readFileSync } from 'node:fs';

const { buckets, policyVersion } = JSON.parse(readFileSync(process.argv[2], 'utf8'));

if (buckets.block.length === 0) {
  console.log(`No blocking accessibility findings (policy v${policyVersion}).`);
  process.exit(0);
}

console.error(`${buckets.block.length} blocking finding(s), policy v${policyVersion}:`);
for (const f of buckets.block) {
  console.error(`  ${f.rule} [${f.impact}] ${f.url}`);
  console.error(`    selector: ${f.target}`);
  console.error(`    fix: ${f.help}`);
}
process.exit(1); // the only deliberate non-zero exit in the whole policy layer

Register that job name in branch protection once it has earned it, and keep the legacy allowance separate: a repository mid-remediation should carry a budget that shrinks on a schedule rather than a flat zero, which is what progressive threshold management is for. The tier decides which findings can block; the budget decides how many are tolerated while the backlog burns down.

# 4. Routing incomplete Results to a Review Artifact

The review bucket needs three properties: it must be readable without a JSON viewer, it must be bounded so nobody has to triage nine hundred rows, and it must not re-notify anyone about a case they already dismissed. A committed dismissal file supplies the third, keyed by the same fingerprint the classifier emits.

// a11y/policy/review.mjs
// Usage: node a11y/policy/review.mjs classified.json > a11y-review.md
import { readFileSync, existsSync } from 'node:fs';

const { buckets } = JSON.parse(readFileSync(process.argv[2], 'utf8'));
// .a11y/reviewed.json: { "<fingerprint>": "checked 2026-06-02, text sits on a solid panel" }
const reviewed = existsSync('.a11y/reviewed.json')
  ? JSON.parse(readFileSync('.a11y/reviewed.json', 'utf8'))
  : {};

const open = buckets.review.filter((r) => !reviewed[r.id]);
const MAX_ROWS = 40; // a triage queue longer than this gets abandoned, not worked

console.log('# Accessibility results needing a human decision\n');
console.log(`${open.length} open, ${buckets.review.length - open.length} `
  + 'previously reviewed and dismissed.\n');
console.log('These are axe `incomplete` results: the check ran and could not decide.');
console.log('Confirm each one manually, then either fix it or record the decision in');
console.log('`.a11y/reviewed.json` with the reason.\n');

for (const r of open.slice(0, MAX_ROWS)) {
  console.log(`- \`${r.rule}\` — ${r.url}`);
  console.log(`  - selector: \`${r.target}\``);
  console.log(`  - why undecided: ${r.reason}`);
  console.log(`  - fingerprint: \`${r.id}\``);
}
if (open.length > MAX_ROWS) {
  console.log(`\n_${open.length - MAX_ROWS} further items truncated; `
    + 'triage the listed ones first._');
}
process.exitCode = 0; // review output is never a build verdict

Wire it into the advisory job as an artifact rather than a pull-request comment. incomplete results are usually stable across a branch and re-posting them on every push is precisely how a review queue becomes noise:

      - name: Build the human-review queue
        run: node a11y/policy/review.mjs classified.json > a11y-review.md
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: a11y-needs-review-${{ github.run_id }}
          path: a11y-review.md
          retention-days: 90  # long enough to survive a triage sprint

Run the same script on a weekly schedule against the main branch and open a single issue per rule with the counts, so the queue has an owner outside the pull-request flow. Aggregate counts belong in the trend store described in reporting, dashboards and violation tracking, tracked as its own series — a rising incomplete count usually means a new background-image pattern shipped, not that accessibility got worse.

# 5. Promoting a Rule From Warn to Block

Promotion is a change to one line of policy.json, and everything difficult about it happens before that line changes. The procedure below is deliberately mechanical, because the failure mode it prevents is a well-meaning engineer promoting a rule on a Tuesday afternoon based on a feeling.

1. Enter at record. The rule is enabled in the scan and appears only in the metrics store. No pull-request output. Minimum 14 days.
2. Promote to warn once the record data shows a plausible finding volume — under roughly 30 new nodes per week for that rule, or the warn tier is unreadable from day one. Assign an owner and an expiry date in the same commit.
3. Soak in warn mode for a fixed window while collecting per-run results keyed by rule id, and compute the two numbers that decide the outcome: the flake rate (same commit, different verdict) and the false-positive rate (findings a human triaged as not real). The measurement itself is a job of its own, set out in soak-testing a new accessibility gate in warning mode.
4. Check the entry criteria. All three must hold: at least 150 scored runs for the rule, flake rate at or below 2%, false-positive rate at or below 5%. Two out of three is a hold, not a promotion.
5. Announce, then promote. Post the numbers and the switch-on date to the team channel at least three working days ahead, then open the one-line policy pull request. The announcement is what turns the first red check from an ambush into an expected event.
6. Watch the first 20 runs. One confirmed false block — a pull request stopped by a finding a human agrees is not real — reverts the entry to warn the same day, with the fingerprint recorded so the next soak can be aimed at the specific case that broke.

Demotion deserves the same formality as promotion. Record the reason in the policy commit message, keep the fingerprint that caused it, and set a new expiry date. A rule that has been demoted twice for the same reason is telling you the rule is wrong for this codebase, and the honest response is to disable it and cover the underlying criterion in manual testing rather than to keep re-litigating it.

# Pipeline Integration

Two workflows, one classifier, one artifact set. a11y-advisory runs on every pull request and can only be red for infrastructure reasons; a11y-block runs on the same event and is the only name registered in branch protection. Both call the same scan.mjs and the same classify.mjs, so a finding cannot be blocking in one job and advisory in the other — a class of bug that is very hard to see when two workflows carry two copies of the impact logic.

Artifacts follow a naming convention that makes retrieval scriptable: a11y-advisory-<run_id> for the raw scan plus classified.json, and a11y-needs-review-<run_id> for the triage queue. Retention differs on purpose — 30 days for evidence, 90 for the review queue, because triage happens on a slower cadence than merging. Every upload step carries if: always() so a failed run still yields its evidence; without it the one run you most want to inspect is the one with no artifact.

Annotations come from the advisory job, never the blocking one. The blocking job’s job is to be a red or green dot with a clear log; duplicating its findings as inline comments doubles the notification volume for the author who is already staring at a failed check. Warn-tier findings, on the other hand, need to appear where the author is looking, and the per-run summary plus the owner digest is the delivery mechanism.

For the exit-code plumbing that sits underneath all of this — which non-zero codes mean “accessibility” versus “the browser died”, how continue-on-error shows up in the checks list, and why set -o pipefail belongs in every scan step that pipes into a formatter — work through the exit-code guide rather than reinventing it per workflow.

# Troubleshooting and Flaky-Test Mitigation

The advisory job is green but the summary is empty. The scan crashed, wrote nothing to scan/, and the glob scan/*.json expanded to a literal that the classifier treated as a missing file. The classifier above throws when it receives zero files, which converts a silent pass into a loud infrastructure failure. Add a floor assertion too: fail the step if the number of scan files is lower than the number of URLs in the route list.

The blocking job fails on something in the review bucket. Almost always a filter written as impact === 'serious' against the raw scan instead of reading buckets.block. Every consumer downstream of the classifier must select by tier; the moment a second component re-derives severity, the two drift and the policy file stops being the source of truth.

A rule fires on one run and not the next, same commit. This is the flake case, and it is a timing problem in the scan, not a policy problem. The usual causes are scanning before hydration settles, a lazily loaded region that renders after the scan on a fast runner and before it on a slow one, and animations that leave an element mid-transition when contrast is sampled. Fix the wait before touching the tier — promoting a flaky rule to block produces red checks that turn green on re-run, and that pattern teaches everyone to re-run rather than to read.

Node-level and violation-level impact disagree. result.impact is the worst impact among the failing nodes, so a violation reported as critical can contain nodes that are individually serious. Classify per node, as the script above does, or a single severe node drags forty mild ones into the blocking bucket and the author cannot tell which of the forty actually matters.

A best-practice rule reaches the block tier. This happens when the scan is configured by tag (withTags(['wcag2aa', 'best-practice'])) and the policy relies purely on impact defaults. The explicit best-practice guard in tierFor is the backstop; keep it even after you think the tag list is clean, because the tag list changes when someone widens the scan to catch one extra rule.

The two jobs disagree about the same pull request. They scanned different URL sets. This happens when one workflow’s route list is generated from a sitemap fetched at run time and the other reads a committed file. Generate once, commit the generated list, and have both jobs read it — a gate whose scope is nondeterministic cannot be a required check.

# Common Pitfalls

• Deriving the tier from impact alone, so the day axe-core changes a rule’s default impact in a patch release, the gate’s behaviour changes with it and nobody reviewed the change.
• Putting the blocking decision in a later step of the advisory job, which makes the two inseparable in branch protection and forces an all-or-nothing choice under pressure.
• Letting warn entries accumulate without an owner or expiry, producing a summary section that is long, unread, and cited as evidence of coverage.
• Failing the build on incomplete, which fabricates a verdict the scanner explicitly declined to give and trains authors to add suppressions to silence uncertainty.
• Discarding incomplete instead, which hides exactly the cases automation is worst at: text over imagery, cross-origin frames, and canvas-rendered content.
• Re-posting the whole review queue as a pull-request comment on every push, so a genuinely useful list of forty items becomes forty notifications per branch.
• Promoting a rule with a clean flake rate but no false-positive triage, which passes the easy test and skips the one that predicts developer trust.
• Treating a demotion as a defeat and arguing instead of reverting, which converts a five-minute policy commit into a two-week debate while the branch stays blocked.

# FAQ

Should serious block by default, or only critical? Start with critical blocking and serious warning, then promote individual serious rules once each has its own soak numbers. serious is a wide band: link-name is unambiguous and cheap to enforce, while color-contrast interacts with brand palettes, background images and third-party embeds and generates most of the incomplete bucket. Promoting the band wholesale is how teams end up disabling the whole gate to get one legacy page merged.

How is a false positive different from a finding a team does not want to fix? A false positive is a finding where the accessibility claim is wrong — the element does have an accessible name from a mechanism the rule did not check, or the contrast is fine because the sampled background is not the rendered one. “We do not want to fix it” is a real violation with a deferral decision attached, and it belongs in the budget and backlog, not in the false-positive rate. Mixing the two inflates the rate, blocks promotion of a rule that works perfectly, and hides the rules that genuinely misfire.

Can different repositories in one organisation use different tiers for the same rule? Yes, and they should, because the soak evidence is per codebase. Distribute a shared policy.base.json with the impact defaults and the best-practice guard, and let each repository ship an overrides file that is merged on top at classification time. What must not vary is the shape of the four buckets or the fingerprint format, since the trend store and the review queue aggregate across repositories.

What blocks a merge when the accessibility job cannot run at all? The blocking job should fail, and it should fail with a code that says “harness”, not “accessibility”. A required check that goes green when the browser failed to launch is worse than having no check, because it reports coverage that did not happen. Keep the distinction in the wrapper’s exit code and in the log’s first line, so the author reading the failure knows whether to fix their markup or re-run the job.

How often should the policy file change? Expect one or two entries to move per month in an active programme, and treat a month with no changes as a signal that the promotion path has stalled rather than that the policy is finished. Every warn entry has an expiry date precisely so the file forces a decision on a schedule: promote it, demote it, or extend the date with a stated reason.

# Related

• Progressive Threshold Management — how many findings the block tier tolerates while a legacy backlog burns down.
• Pull Request Gating & Branch Policies — registering a11y-block as a required check and scoping it to changed pages.
• Soak-Testing an A11y Gate in Warning Mode — the measurement period that produces the two promotion numbers.
• Exit Codes for Warning & Blocking A11y Jobs — the shell and workflow mechanics under every tier in this guide.
• Reporting, Dashboards & Violation Tracking — where the record tier and the incomplete count become trends.