Soak-Testing a New Accessibility Gate in Warning Mode

A soak is a fixed measurement period during which a candidate rule runs against real pull requests, produces no enforcement, and is instrumented well enough to answer two questions with numbers. This guide is part of Auto-Fail vs Warning A11y Pipeline Gates, and it covers only the measurement: the collection mechanism, the two rates it produces, the thresholds those rates have to clear, and the trigger that ends a soak early.

Root Cause

A single green run proves nothing about a rule. CI conditions are hostile to determinism in ways that a local run never reveals: a shared runner under load renders a lazily hydrated component 300 ms later than a quiet one, a font that arrives from cache locally arrives over the network in CI and shifts the text a scanner samples for contrast, and a CSS transition that has finished by the time a developer looks at the page is still mid-flight when the scanner reads computed styles. None of that changes the rule’s correctness — it changes whether the rule produces the same verdict twice on identical source, which is the only property a required status check actually needs.

The second unknown is orthogonal and cannot be measured by any amount of re-running: whether the findings are real. axe rules are tuned against the general web, and a rule that is precise on a content site can be systematically wrong on one product — the mechanics of that mismatch are covered in reducing false positives in automated accessibility scanners, and a soak is how you find out whether your codebase has the problem. target-size fires on icon buttons that a design system deliberately pads with a larger hit area implemented through a pseudo-element the rule cannot see. color-contrast returns findings against a sampled background that is not the rendered one whenever a decorative layer sits between text and its parent. Those are not flakes — they reproduce perfectly every run — and enforcing them costs exactly as much developer trust as a genuinely random failure.

So a soak measures two independent things and refuses to promote unless both are clean: flake rate, the share of scored runs where the same commit produced different results for the rule, and false-positive rate, the share of triaged findings that a human judged not to be real accessibility problems. A rule with a 0% flake rate and a 30% false-positive rate is worse than useless as a gate; a rule with a 0% false-positive rate and a 6% flake rate will teach the team to press re-run. Fourteen days is the minimum useful window because it spans two weekly release rhythms and catches the Monday-morning dependency bumps that a three-day trial misses.

Configuration

Getting a flake rate normally means comparing two runs of the same commit, and waiting for developers to happen to re-run jobs produces a sample of about four data points a month. The trick that makes the measurement practical is a twin scan: inside a single job, scan the candidate URL set twice into two directories, and treat the pair as two independent observations of one commit. Every pull-request run then contributes a flake data point, and the extra cost is one scan pass rather than a whole job.

name: a11y-soak
on:
  pull_request:
    branches: [main]
permissions:
  contents: read
jobs:
  soak:
    runs-on: ubuntu-24.04
    timeout-minutes: 25
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Twin scan of the candidate rules only
        run: |
          set -o pipefail
          # Two passes, same commit, same URL list, fresh browser context each time.
          node a11y/scan.mjs --rules-file .a11y/soak-candidates.json --out scan-a
          node a11y/scan.mjs --rules-file .a11y/soak-candidates.json --out scan-b
      - name: Append this run to the soak store
        run: |
          node a11y/soak/record.mjs \
            --a scan-a --b scan-b \
            --sha "${{ github.event.pull_request.head.sha }}" \
            --pr "${{ github.event.pull_request.number }}" \
            --out soak-records.ndjson
      - uses: actions/upload-artifact@v4
        if: always()   # a crashed pass must still leave a record of the attempt
        with:
          name: a11y-soak-${{ github.run_id }}-${{ github.run_attempt }}
          path: |
            soak-records.ndjson
            scan-a/
            scan-b/
          retention-days: 45   # longer than the longest soak window

The candidate list is a plain file so that adding a rule to the soak is a one-line reviewable change and never a workflow edit:

{
  "soakStarted": "2026-07-14",
  "rules": ["target-size", "color-contrast", "aria-required-children",
            "link-in-text-block", "heading-order"]
}

record.mjs emits one NDJSON line per rule per run. Keying by rule id rather than by page is what makes the arithmetic possible later: a rule is promoted or held as a unit, so every number has to be attributable to a rule id without re-parsing the raw scans.

// a11y/soak/record.mjs
// Usage: node a11y/soak/record.mjs --a scan-a --b scan-b --sha <sha> --pr <n> --out f.ndjson
import { readdirSync, readFileSync, appendFileSync } from 'node:fs';
import { createHash } from 'node:crypto';
import { join } from 'node:path';

const arg = (name) => {
  const i = process.argv.indexOf(`--${name}`);
  if (i === -1) throw new Error(`missing --${name}`);
  return process.argv[i + 1];
};

const fingerprint = (rule, url, target) =>
  createHash('sha1').update(`${rule}|${url}|${target}`).digest('hex').slice(0, 12);

// Collect { rule -> Set(fingerprint) } for one scan directory.
function observe(dir) {
  const byRule = new Map();
  for (const file of readdirSync(dir).filter((f) => f.endsWith('.json'))) {
    const page = JSON.parse(readFileSync(join(dir, file), 'utf8'));
    for (const violation of page.violations) {
      const set = byRule.get(violation.id) ?? new Set();
      for (const node of violation.nodes) {
        set.add(fingerprint(violation.id, page.url, node.target[0]));
      }
      byRule.set(violation.id, set);
    }
  }
  return byRule;
}

const a = observe(arg('a'));
const b = observe(arg('b'));
const rules = new Set([...a.keys(), ...b.keys()]);
const same = (x, y) =>
  x.size === y.size && [...x].every((v) => y.has(v)); // set equality, order-free

for (const rule of rules) {
  const setA = a.get(rule) ?? new Set();
  const setB = b.get(rule) ?? new Set();
  appendFileSync(arg('out'), JSON.stringify({
    ts: new Date().toISOString(),
    pr: Number(arg('pr')),
    sha: arg('sha'),
    rule,
    nodesA: setA.size,
    nodesB: setB.size,
    twinMatch: same(setA, setB),
    // The union is what triage works from: every distinct problem seen this run.
    fingerprints: [...new Set([...setA, ...setB])].sort(),
  }) + '\n');
}
process.exitCode = 0; // the soak job is never allowed to fail for accessibility
Soak collection: one run, two observations, two derived rates Three pull-request runs each perform scan A and scan B of the same commit. A recording script compares the two fingerprint sets and appends one line per rule to an NDJSON artifact. A weekly report job derives the flake rate from the A versus B disagreements, while human triage of the recorded fingerprints derives the false-positive rate. One pull-request run, two observations, two derived rates PR 4821 sha a1b2c3d scan A + scan B PR 4822 sha c9d4e5f scan A + scan B PR 4823 sha 7f8a0b1 scan A + scan B one line per rule, per run record.mjs compares A with B records.ndjson one artifact per run report.mjs weekly job human triage real / not-real false-positive rate not-real over triaged flake rate A differs from B The A/B pair inside one run yields the flake rate; the triage verdicts on the same fingerprints yield the false-positive rate.
Because both observations come from one job, every pull request contributes a flake data point instead of only the rare re-run.

Triage verdicts live in a committed file keyed by the same fingerprint the recorder emits, which is what lets a decision made in week one still count in week two:

{
  "3f9a1c74be02": { "verdict": "real", "note": "icon button genuinely 22x22 CSS px" },
  "8d21ee0b47af": { "verdict": "not-real",
    "note": "hit area comes from a ::after overlay; target-size cannot see it" },
  "b05c7f19aa63": { "verdict": "not-real",
    "note": "contrast sampled against the parent, real backdrop is the panel" },
  "c74e0b39d182": { "verdict": "real", "note": "3.9:1 body copy on the marketing banner" }
}

Validation

The aggregator downloads every soak artifact produced in the window, concatenates the NDJSON, and prints one row per rule. Run it weekly and once more on the day the soak ends.

name: a11y-soak-report
on:
  schedule:
    - cron: '0 7 * * 1'   # Monday 07:00 UTC, before the week's first merges
  workflow_dispatch:
permissions:
  contents: read
  actions: read           # required to list and download other runs' artifacts
jobs:
  aggregate:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - name: Pull every soak artifact from the window
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          set -o pipefail
          mkdir -p soak-in
          gh run list --workflow a11y-soak.yml --limit 400 \
            --json databaseId --jq '.[].databaseId' > run-ids.txt
          while read -r id; do
            # A run whose artifact expired is skipped rather than failing the job.
            gh run download "$id" --dir "soak-in/$id" --pattern 'a11y-soak-*' || true
          done < run-ids.txt
          find soak-in -name 'soak-records.ndjson' -exec cat {} + > all-records.ndjson
      - name: Compute the two rates per rule
        run: node a11y/soak/report.mjs all-records.ndjson > soak-report.md
      - uses: actions/upload-artifact@v4
        with:
          name: a11y-soak-report
          path: soak-report.md
          retention-days: 90

The report script holds the promotion thresholds as named constants, so the criteria are code rather than folklore:

// a11y/soak/report.mjs — usage: node a11y/soak/report.mjs all-records.ndjson
import { readFileSync, existsSync } from 'node:fs';

const THRESHOLDS = {
  minRuns: 150,          // below this the rates are noise, whatever they say
  minDays: 14,           // two weekly release rhythms
  maxFlakeRate: 0.02,    // 2% of scored runs may disagree with themselves
  maxFalsePositive: 0.05,// 5% of triaged findings may be judged not real
  minTriageCoverage: 0.6,// 60% of distinct fingerprints must have a verdict
};

const lines = readFileSync(process.argv[2], 'utf8').trim().split('\n');
const records = lines.filter(Boolean).map((l) => JSON.parse(l));
const triage = existsSync('.a11y/soak-triage.json')
  ? JSON.parse(readFileSync('.a11y/soak-triage.json', 'utf8'))
  : {};
const candidates = JSON.parse(readFileSync('.a11y/soak-candidates.json', 'utf8'));
const dayMs = 86_400_000;
const elapsedDays = (Date.now() - Date.parse(candidates.soakStarted)) / dayMs;

console.log('# Soak report\n');
console.log(`Window: ${elapsedDays.toFixed(1)} days since ${candidates.soakStarted}\n`);
console.log('| Rule | Runs | Flake | Triaged | FP rate | Verdict |');
console.log('|---|---|---|---|---|---|');

for (const rule of candidates.rules) {
  const rows = records.filter((r) => r.rule === rule);
  // Deduplicate by sha+run so a re-run of the same commit is not double counted.
  const scored = new Map(rows.map((r) => [`${r.sha}|${r.ts}`, r]));
  const runs = scored.size;
  const mismatches = [...scored.values()].filter((r) => !r.twinMatch).length;
  const flake = runs === 0 ? 0 : mismatches / runs;

  const seen = new Set(rows.flatMap((r) => r.fingerprints));
  const verdicts = [...seen].map((f) => triage[f]?.verdict).filter(Boolean);
  const coverage = seen.size === 0 ? 1 : verdicts.length / seen.size;
  const notReal = verdicts.filter((v) => v === 'not-real').length;
  const fp = verdicts.length === 0 ? 0 : notReal / verdicts.length;

  const reasons = [];
  if (runs < THRESHOLDS.minRuns) reasons.push('sample');
  if (elapsedDays < THRESHOLDS.minDays) reasons.push('window');
  if (flake > THRESHOLDS.maxFlakeRate) reasons.push('flake');
  if (coverage < THRESHOLDS.minTriageCoverage) reasons.push('triage coverage');
  if (fp > THRESHOLDS.maxFalsePositive) reasons.push('false positives');

  const verdict = reasons.length === 0 ? 'PROMOTE' : `hold — ${reasons.join(', ')}`;
  console.log(`| \`${rule}\` | ${runs} | ${(flake * 100).toFixed(1)}% `
    + `| ${verdicts.length}/${seen.size} | ${(fp * 100).toFixed(1)}% | ${verdict} |`);
}
process.exitCode = 0; // a report is never a build verdict

The flake calculation is worth reading twice, because it is the part people get wrong. A run is scored once; a run where the two scans produced different fingerprint sets is a mismatch; the rate is mismatches over scored runs. Node counts are not compared directly — two scans can both find three nodes and still disagree about which three, and that is a flake even though the count is stable.

Twin scans expose a non-deterministic rule Four commits are each scanned twice in the same job. Three commits produce identical fingerprint sets in scan A and scan B and are counted as stable. One commit produces one node in scan A and zero in scan B and is counted as a flake, giving one flake in four scored runs. Twin scans of target-size across four scored runs commit scan A scan B sets differ counted as a1b2c3d 3 nodes 3 nodes no stable c9d4e5f 1 node 0 nodes yes flake 7f8a0b1 5 nodes 5 nodes no stable b2c3d4e 2 nodes 2 nodes no stable flake rate = mismatched runs divided by scored runs = 1 over 4 = 25.0% The ceiling is 2.0%, so this rule stays in warning mode and the wait gets fixed first.
A lazily rendered row is present in one pass and absent in the next, which the twin scan turns into a number instead of an argument.

Reading the report is mechanical: a rule promotes only when the runs column clears 150, the window clears 14 days, flake is at or under 2.0%, triage coverage is at or above 60%, and the false-positive rate is at or under 5.0%. Anything else is a hold with a named reason, and the reason tells you what to do next — a flake hold means fix the scan’s wait condition, a false positives hold means the rule needs an exclude selector or narrowed run options in the axe-core configuration, and a sample hold means keep waiting.

Five candidate rules against three thresholds color-contrast with 214 runs, 0.0 percent flake and 1.4 percent false positives promotes. link-in-text-block with 196 runs, 0.9 percent flake and 3.1 percent false positives promotes. target-size holds on an 11.2 percent false-positive rate. aria-required-children holds on a 4.1 percent flake rate. heading-order holds on only 37 runs. End-of-soak verdicts for five candidate rules rule runs flake FP rate verdict color-contrast 214 0.0% 1.4% promote link-in-text-block 196 0.9% 3.1% promote target-size 188 0.5% 11.2% hold — triage aria-required-children 203 4.1% 2.0% hold — flake heading-order 37 0.0% 0.0% hold — sample Thresholds: 150 or more scored runs, flake at most 2.0%, false positives at most 5.0% — all three, or it stays a warning.
Three holds for three different reasons, each pointing at a different next action: narrow the rule, fix the wait, or keep collecting.

The rollback trigger belongs in the same document as the promotion criteria, decided before anyone has an emotional stake in the outcome. After promotion, revert the rule to warning mode on the same working day if any of three things happens: one pull request is blocked by a finding a reviewer and an accessibility specialist agree is not real; the trailing-20-run flake rate on the default branch exceeds 2%; or the rule blocks more than three pull requests in one day whose diffs do not touch the failing component, which means the scan’s scope is wrong rather than the rule.

Edge Cases and Conditional Guards

  • Two scans in one job share a warm cache, so the twin pair detects render-timing and layout flake but not cold-start flake. Add a third observation from the nightly run on the default branch, where the browser and CDN caches are cold, and treat a disagreement between the nightly and the pull-request scans as a flake against the same 2% ceiling.
  • A rebased or force-pushed branch changes the head SHA while the content is unchanged, so run counts inflate and the same fingerprints are re-counted. Deduplicate by sha when counting runs, as the report does, and rely on fingerprints rather than SHAs for triage identity so a rebase never invalidates a verdict.
  • Fork pull requests get a read-only token, so the artifact upload works but any push-based store does not. Keep the soak store artifact-based for exactly this reason, and if the majority of contributions come from forks, run the soak on push to the default branch instead and accept a slower accumulation of runs.

Pipeline Impact

The soak job is advisory by construction: record.mjs sets process.exitCode = 0, nothing reads its output during the run, and the job name is never added to branch protection. It must still be able to fail when the harness breaks, which is a distinction the wrapper described in choosing exit codes for warning and blocking a11y jobs handles with a dedicated exit code. It should not post pull-request comments either — a candidate rule that comments is already enforcing socially, and authors will start fixing findings to make the comment go away, which biases the very false-positive rate you are trying to measure.

Budget the cost honestly. A twin scan roughly doubles the scan portion of the job, which on a 24-URL route list is about 90 extra seconds, and it runs in parallel with the existing advisory and blocking jobs rather than in series with them. Artifact retention of 45 days at a few kilobytes of NDJSON per run is negligible; the scan-a/ and scan-b/ directories are the expensive part, so drop them from the upload once the collection script is trusted.

When a rule finally promotes, the only change is one line in the policy file, and the soak job stops scanning that rule because it leaves soak-candidates.json. Then the numbers that justified the change should be pasted into the promotion commit message, because the next person to ask “why does this block?” deserves an answer with digits in it rather than a name and a date.

Common Pitfalls

  • Comparing node counts instead of fingerprint sets, which scores a run as stable whenever two different findings happen to total the same number.
  • Counting a re-run of the same commit as a fresh run, which inflates the denominator and makes a flaky rule look stable in exactly the situation that produced the extra data.
  • Letting the soak run without triage, so the false-positive rate is computed from three verdicts and the coverage check is the only thing standing between you and a confidently wrong promotion.
  • Ending the soak early because the flake rate looks perfect on day four — a 14-day window exists to catch the weekly dependency bump and the Friday release freeze, not to satisfy a calendar.
  • Soaking ten rules at once, which makes triage unaffordable; three to five candidates per window is the practical ceiling for one team.
  • Treating “the team does not want to fix this” as a false positive, which pollutes the rate with backlog decisions and permanently blocks promotion of a rule that works correctly.

FAQ

Why 150 runs rather than a confidence interval? Because 150 is the point at which a 2% ceiling has enough resolution to be meaningful — three mismatches take a rule over the line, and fewer than that is within the noise of one bad afternoon on shared runners. A formal interval is defensible and harder to explain in a promotion review; if a rule sits right at the boundary, the correct move is to extend the window rather than to argue about statistics.

Can the soak run on the default branch instead of pull requests? It can, and for a repository with mostly external contributions it should, but the sample changes character. Default-branch runs measure a merged, settled state, while pull-request runs measure the messy intermediate states where flake actually lives — half-finished components, feature flags in odd combinations, unbuilt assets. If both are available, collect both and compute the flake rate separately for each, because a rule that is stable on the default branch and flaky on pull requests will fail as a pull-request gate.

What if a candidate rule finds nothing during the whole soak? That is a pass on both rates and a fail on usefulness, and it usually means the scan never reached the pages where the rule applies. Before promoting a silent rule, verify it can fire at all by scanning a fixture page that deliberately breaks it; a rule that reports zero because it never matched a node is indistinguishable from a rule that reports zero because the code is clean, and only one of those is good news.

Back to Auto Fail Vs Warning Workflows