Burning Down an Accessibility Backlog Across Quarters

A backlog of 4,812 findings is a multi-quarter programme whether or not anyone plans it as one. This guide is part of Accessibility Debt Triage & Prioritization, and it covers the measurement that makes such a programme survivable: two curves computed separately from stored run history, a quarter target derived from throughput the team has actually demonstrated, an exit criterion specific enough to be met, and a recurring report short enough to be read.

Root Cause

The default chart every dashboard produces is total open violations over time, and it is the wrong chart because it sums two populations that behave differently and mean different things. The first is the pre-existing backlog: a fixed cohort of findings that existed when the gate went live, which should shrink monotonically and never grow. The second is newly introduced findings: defects merged after the gate went live, which should be zero at all times, because preventing them is the entire purpose of the gate.

Adding those two together destroys the only signal that matters. Take a real week from the middle of a programme: the total moved from 2,266 to 2,254. Twelve fewer findings, a small green arrow on the dashboard, an uneventful weekly review. What actually happened was 74 findings fixed and 62 findings newly introduced — the gate had been silently skipped on a release branch for eleven days, and the team was running to stand still. On the combined curve that is indistinguishable from a slow week of remediation. On two curves it is unmissable: the remaining line drops normally and the introduced line lifts off zero, which is an incident, not a trend.

The second failure of the single-total view is planning. A total that includes new arrivals cannot support a forecast, because the forecast needs a rate: findings closed per week, measured over enough weeks to include a holiday, an incident and a release freeze. Teams that skip the measurement set a target by working backwards from a date somebody wants — “clear it by year end” — and the target is missed in the first quarter, at which point the programme’s credibility is spent. Throughput is measurable from the same run history that produces the curves, and the honest number it produces is usually about half of what was originally promised.

Configuration

Everything here derives from one append-only artifact: a run-history file that records which finding fingerprints were open at each scan. One line per run keeps the file small — 4,812 twelve-character ids is roughly 60 KB per run, and a weekly cadence is under 4 MB a year, which is comfortable to commit.

// a11y/history/record.mjs
// Usage: node a11y/history/record.mjs triage/findings.ndjson >> a11y/history/runs.ndjson
import { readFileSync } from 'node:fs';

const rows = readFileSync(process.argv[2], 'utf8').trim().split('\n').map(JSON.parse);
const ids = rows.map((r) => r.id);
process.stdout.write(
  JSON.stringify({
    runId: process.env.GITHUB_RUN_ID ?? `local-${Date.now()}`,
    scannedAt: new Date().toISOString().slice(0, 10),
    routeCount: new Set(rows.map((r) => r.route)).size, // coverage, tracked separately
    ruleSetVersion: process.env.AXE_VERSION ?? 'unknown',
    openIds: ids,
    // Blocking subset is reported on its own: it is the number leadership acts on.
    blockingIds: rows.filter((r) => r.factors?.harm === 'blocksTask').map((r) => r.id),
  }) + '\n',
);

The two curves come from cohort membership, not from counting. Fix the gate-live date once, treat every id present in the first run on or after that date as the baseline cohort, and classify everything else by when it first appeared. A baseline id that disappears and later returns is a third category — a regression of already-fixed work — and it belongs on the introduced curve, because it is a failure of prevention rather than a remnant of history.

// a11y/history/curves.mjs
// Usage: node a11y/history/curves.mjs a11y/history/runs.ndjson 2026-01-05
import { readFileSync } from 'node:fs';

const runs = readFileSync(process.argv[2], 'utf8').trim().split('\n').map(JSON.parse);
const gateLive = process.argv[3];
runs.sort((a, b) => a.scannedAt.localeCompare(b.scannedAt));

const firstAfterGate = runs.find((r) => r.scannedAt >= gateLive);
if (!firstAfterGate) throw new Error(`no run on or after the gate-live date ${gateLive}`);
const baseline = new Set(firstAfterGate.openIds);

// Ids seen in any earlier run of this cohort, used to detect reappearance.
const everClosed = new Set();
let previousOpen = new Set(firstAfterGate.openIds);

const series = [];
for (const run of runs.filter((r) => r.scannedAt >= gateLive)) {
  const open = new Set(run.openIds);
  for (const id of previousOpen) if (!open.has(id)) everClosed.add(id);

  let remaining = 0;   // baseline cohort still open: must fall
  let introduced = 0;  // never in the baseline: must stay at zero
  let regressed = 0;   // baseline, closed once, open again: also a prevention failure
  for (const id of open) {
    if (!baseline.has(id)) introduced += 1;
    else if (everClosed.has(id)) regressed += 1;
    else remaining += 1;
  }
  series.push({
    date: run.scannedAt,
    remaining,
    introduced: introduced + regressed,
    blocking: run.blockingIds.length,
    routeCount: run.routeCount, // coverage growth is never debt growth
  });
  previousOpen = open;
}

console.log(JSON.stringify(series, null, 2));

routeCount travels with every point for one reason: adding routes to the scan manifest raises every count without anything getting worse. When the remaining curve jumps, the first question is whether coverage changed, and having the number in the series answers it without a git archaeology session. Report coverage as its own line on the chart rather than normalising it away — a backlog that grew because the scan started looking harder is good news reported badly.

Two curves tracked apart across six quarters The remaining baseline cohort falls from four thousand eight hundred and twelve to zero over six quarters. The introduced curve sits on zero except for a spike of one hundred and forty-eight findings in the fourth period, when the gate was skipped on a release branch. The dashed combined total falls smoothly through that period, concealing the spike. Remaining backlog and newly introduced findings, tracked apart remaining (baseline cohort) introduced combined total 4800 3600 2400 1200 0 Q4: the gate was skipped on a release branch for eleven days and 148 findings arrived — yet the combined total still fell by 726 that quarter. gate Q2 Q3 Q4 Q5 Q6 Q7 quarter boundary (weekly scans aggregated to the last run in each quarter) The dashed line is what a single-total dashboard shows: smooth, reassuring, and blind to the Q4 leak.
The remaining curve answers "are we making progress" and the introduced curve answers "is the gate working"; a chart that adds them can answer neither question.

Cohort membership is worth spelling out because it is the part teams implement loosely and then cannot explain. A fingerprint’s first appearance decides its cohort permanently. Nothing moves from introduced to baseline, ever — otherwise a quarter of bad prevention quietly becomes historical debt, which is the most comfortable and least honest accounting available.

How a fingerprint is assigned to a cohort A horizontal timeline marks the gate-live date. Fingerprints first seen in the run on or after that date form the baseline cohort. Fingerprints first seen later are introduced. A baseline fingerprint that closes and reappears is counted as regressed and reported with the introduced curve, never back on the baseline curve. First appearance decides the cohort, permanently gate live 2026-01-05 scan runs, weekly 4,812 ids in the first run baseline cohort — the remaining curve closes once and is gone; target falls every quarter introduced — id first seen after the gate went live target is zero; any non-zero point is an incident to explain regressed — a baseline id that closed and came back reported with introduced: prevention failed, not history Nothing ever moves from the introduced cohort into the baseline cohort: that would turn a prevention failure into historical debt.
The regressed branch is why fingerprints have to be stable: without a durable id, a returning finding enters as new work and both curves are wrong at once.

Validation

The target has to be derived, not chosen. Measure closures per week from the same history file, take the median rather than the mean so one heroic sprint does not set the plan, and discount for the weeks that reliably disappear — holidays, incident response, release freezes. Eight weeks is the minimum useful window; a quarter is better.

// a11y/history/target.mjs
// Usage: node a11y/history/target.mjs a11y/history/runs.ndjson 2026-01-05
import { readFileSync } from 'node:fs';
import { execFileSync } from 'node:child_process';

const series = JSON.parse(
  execFileSync('node', ['a11y/history/curves.mjs', process.argv[2], process.argv[3]]),
);

// Weekly closures = fall in the remaining curve between consecutive runs.
const closures = [];
for (let i = 1; i < series.length; i += 1) {
  closures.push(Math.max(series[i - 1].remaining - series[i].remaining, 0));
}
const window = closures.slice(-8);
const sorted = [...window].sort((a, b) => a - b);
const mid = Math.floor(sorted.length / 2);
// Median, not mean: one heroic codemod week must not set the plan.
const median = sorted.length % 2 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2;

const WEEKS_PER_QUARTER = 13;
const AVAILABILITY = 0.8; // holidays, incidents and freezes measured over a year
const perQuarter = Math.floor(median * WEEKS_PER_QUARTER * AVAILABILITY);
const remaining = series.at(-1).remaining;

console.log(`measured weekly closures: ${window.join(', ')}`);
console.log(`median: ${median}/week  →  quarter target: ${perQuarter}`);
console.log(`remaining: ${remaining}  →  quarters to exit: ${Math.ceil(remaining / perQuarter)}`);
if (series.at(-1).introduced > 0) {
  console.log(`WARNING: ${series.at(-1).introduced} introduced findings open — fix the gate first`);
}

Run against a real eight-week window, the output is deflating and useful in equal measure:

measured weekly closures: 96, 74, 121, 0, 58, 143, 88, 67
median: 81/week  →  quarter target: 842
remaining: 4812  →  quarters to exit: 6
Quarter target derived from measured weekly throughput Eight columns show findings closed per week: ninety-six, seventy-four, one hundred and twenty-one, zero during an incident week, fifty-eight, one hundred and forty-three, eighty-eight and sixty-seven. A dashed median line sits at eighty-one per week. A panel below shows the derivation of the quarter target of eight hundred and forty-two and the six-quarter exit estimate. Findings closed per week — measured, not promised week 4 was an incident week: no accessibility work happened, and the median absorbs it 150 100 50 0 96 74 121 0 58 143 88 67 median 81 W1 W2 W3 W4 W5 W6 W7 W8 81 per week x 13 weeks x 0.8 availability = 842 per quarter 4,812 remaining / 842 = 6 quarters to exit, not "by year end"
The median rather than the mean is deliberate: the mean of this window is 81 too, but a single 300-finding codemod week would push the mean to 108 and set a target the team cannot repeat.

An exit criterion has to be a set of conditions a run can evaluate, not a sentiment. Three conditions, all of which must hold on two consecutive scheduled runs before the backlog programme closes: zero remaining baseline findings that block task completion on primary-journey routes; the introduced curve at zero for two full quarters; and every remaining P3 finding either closed or carrying a documented, dated deferral. Note what is absent — “zero findings”. Total zero is not an exit criterion for any real application, because the scan surface keeps growing and incomplete results keep arriving; a programme whose success condition is unreachable never gets to declare success and quietly loses its funding instead.

// a11y/history/exit-criterion.mjs — exits 0 when the backlog programme can close.
import { readFileSync } from 'node:fs';

const series = JSON.parse(readFileSync('a11y/history/series.json', 'utf8'));
const queue = readFileSync('triage/queue.ndjson', 'utf8').trim().split('\n').map(JSON.parse);
const recent = series.slice(-2);        // two consecutive scheduled runs
const twoQuarters = series.slice(-26);  // twenty-six weekly runs

const blockingOnPrimary = queue.filter(
  (r) => r.factors.harm === 'blocksTask' && r.factors.tier === 'primaryJourney',
);

const checks = {
  noBlockingOnPrimaryJourneys:
    blockingOnPrimary.length === 0 && recent.every((p) => p.blocking === 0),
  preventionHeldTwoQuarters: twoQuarters.every((p) => p.introduced === 0),
  deferralsDocumented: queue
    .filter((r) => r.band === 'P3')
    .every((r) => r.deferredUntil && r.deferredReason),
};

for (const [name, ok] of Object.entries(checks)) console.log(`${ok ? 'PASS' : 'FAIL'} ${name}`);
process.exitCode = Object.values(checks).every(Boolean) ? 0 : 1;

Edge Cases and Conditional Guards

  • A missed scheduled run. A cancelled runner or an expired token leaves a gap in the history, and the closure calculation for the following week then covers fourteen days and doubles that week’s apparent throughput. Compute closures per elapsed day and multiply by seven rather than trusting the run index, and drop any interval longer than twenty-one days from the throughput window entirely.
  • A rule-set upgrade mid-programme. Upgrading axe-core adds rules, retires others and occasionally re-scopes an existing rule, which mints new fingerprints for defects that were always there. Those must not land on the introduced curve. Record ruleSetVersion per run, and on any change re-baseline only the rules whose ids are new: their findings join the baseline cohort with a note, and the report shows the re-baseline as an annotation on the chart.
  • A route removed from the product. Deleting a page closes its findings, which is legitimate progress and also the easiest way to fake a burndown. Track routeCount alongside the curves and require the weekly report to state closures split by cause — fixed, deleted, or re-scoped — because a quarter target met by deleting an admin surface teaches the programme nothing about its own capacity.

Pipeline Impact

The history, curve and target scripts run in the scheduled triage job, after the queue is regenerated and before the report is posted. None of them fails the build on a number: a rising remaining curve is a planning problem, not a broken pipeline. Two conditions do warrant a non-zero exit — a gap in the history file that makes the curves unreliable, and a non-zero introduced count, because that means the pull-request gate let something through and the leak is worth more attention than the backlog.

The recurring report is four lines, posted to the same channel every week and to the quarterly review verbatim: remaining, introduced, blocking-on-primary-journeys, and quarters-to-exit at current measured throughput. Publish it as a job summary and a chat message, and keep the series JSON as a build artifact so the violation-trend view renders from the same numbers the report quotes — two systems computing the backlog differently is how a review turns into an argument about tooling.

      - name: Compute curves, target and the weekly report
        run: |
          node a11y/history/record.mjs triage/findings.ndjson >> a11y/history/runs.ndjson
          node a11y/history/curves.mjs a11y/history/runs.ndjson 2026-01-05 \
            > a11y/history/series.json
          node a11y/history/target.mjs a11y/history/runs.ndjson 2026-01-05 \
            | tee -a "$GITHUB_STEP_SUMMARY"
      - name: Fail if the gate leaked new findings this week
        run: |
          introduced=$(jq '.[-1].introduced' a11y/history/series.json)
          test "$introduced" -eq 0 || {
            echo "::error::$introduced new findings introduced since the gate went live"
            exit 1  # a prevention failure outranks any backlog number
          }

Per-team throughput is worth computing separately once the buckets from ownership routing exist, because a single organisation-wide median hides the fact that one team is clearing 60 findings a week and four are clearing none. A quarter target allocated proportionally to measured per-team throughput is defensible in a planning meeting; one allocated by bucket size is a demand that the team with the largest bucket work fastest, which is exactly backwards.

Common Pitfalls

  • Charting one combined total, which makes a week of 74 fixes and 62 regressions look like ordinary slow progress.
  • Setting the quarter target from a desired end date rather than from measured throughput, which spends the programme’s credibility in its first quarter.
  • Using the mean of weekly closures, so one large codemod week sets a target no ordinary week can meet.
  • Letting introduced findings age into the baseline cohort, which converts a prevention failure into comfortable historical debt.
  • Counting a route deletion as a fix without labelling it, which lets a quarter target be met without any accessibility work happening.
  • Defining the exit criterion as zero findings, which is unreachable on a growing product and guarantees the programme never declares success.
  • Ignoring ruleSetVersion changes, so an axe-core upgrade appears on the introduced curve as a gate failure and triggers an incident review of a working gate.

FAQ

Why weekly scans rather than daily? Weekly matches the cadence at which the numbers change and at which anyone acts on them, and it keeps the history file small enough to commit and diff. Daily full-coverage scans of 400 routes cost real runner minutes and produce six points of noise for every point of signal, since fixes land in batches at sprint boundaries. Where a faster signal is wanted, get it from the pull-request gate — that is a per-commit check already — and leave the full scan on a weekly rhythm.

What if the remaining curve flattens for a whole quarter? Read the throughput numbers before the priority model. Flat usually means capacity went elsewhere — an incident, a migration, a reorganisation — and the honest response is to restate the exit estimate with the new measured median rather than to keep quoting the old plan. If capacity was genuinely spent on accessibility and the curve is still flat, the queue is stuck on high-cost groups, and the fix is to schedule the design decisions those groups need rather than to ask for more effort.

Should the two curves be reported to leadership separately? Yes, and in that order: prevention first, then remediation. The introduced count is a yes-or-no question about whether the gate works, it needs no context to interpret, and a non-zero value is the only number in the report that requires action this week. Remaining is a trend that only means something against the quarter target, so report it as “1,452 remaining against a target of 842 closed this quarter, five quarters to exit” — a number, its target and a forecast, which is the shape an engineering leader can actually plan against.

Back to Accessibility Debt Triage Prioritization