# Making an Accessibility Violation Count Comparable Across Sprints

Sprint 41 closed with 148 violations and sprint 42 closed with 171, and that +23 is worth nothing on its own. Six routes joined the scan manifest between those two dates, the runner picked up an axe-core minor that ships two rules the previous version did not have, and neither total can say whether a single defect was actually repaired. This guide is part of Reporting, Dashboards & Violation Tracking, and it covers what has to be recorded and computed before two counts can be subtracted at all: a scope denominator, a size denominator, a hard series break whenever the scanner changes, and a sprint report built from fingerprint sets rather than from totals.

# Root Cause

A violation total is the product of three variables, and only one of them is the quality of the application. The first is scope: how many routes the scan visited. A manifest is edited constantly — someone adds the new checkout step, someone excludes an admin page that takes ninety seconds to log into, someone splits one route into three. Every one of those edits moves the total, and none of them is a regression or a fix. A count with no route denominator is a measurement whose units changed halfway through the experiment.

The second variable is size. Rules that scale with the amount of markup on a page — color-contrast over every text node, td-has-header over every data cell — produce counts proportional to page weight. A route that grows from a twelve-row table to a two-hundred-row table gains 188 new chances to fail one rule without a single line of component code changing. The denominator that corrects for this is already in the report and almost always thrown away: the number of nodes the rule set actually inspected, which is the sum of node counts across violations, passes and incomplete. Store it per run and a count becomes a density.

The third variable is the rule set. Every count is conditional on which rules ran, and a scanner upgrade changes that set silently. A minor version can add rules, retire a check, reclassify an impact from moderate to serious, or tighten an existing evaluator so that markup which passed last week now fails. None of that is visible in a total. The scanner version and the effective rule set therefore belong in the series key, not in a footnote: two runs with different keys are two different measurements, and the series must break between them rather than draw a line across. The same discipline applies to a house rule bundle, which is why versioning custom rules without breaking existing pipelines treats widening a selector as a breaking change.

Finally, subtraction cannot see identity. A sprint in which twelve defects were fixed and twelve appeared produces exactly the same delta as a sprint in which nothing happened, and the second reading is the one every stakeholder will assume. The unit of comparison has to be the fingerprint of a finding — the stable identity defined in the reporting guide — so that the sprint report can distinguish fixed, new and carried over. Those three numbers, not the total, are what a team can act on.

Four quantities therefore have to be recorded per run, and each answers a different question.

# Configuration

The append step adds one row per run to a series_point table. It deliberately does not re-store findings: the normalised findings and runs tables built in the reporting guide already hold them, and the series table exists only to carry the denominators and the series identity alongside a pre-counted total. Sprint boundaries live in their own table, because real sprints slip by a day or two and a modulo over the timestamp will silently reassign commits to the wrong window the moment the cadence changes.

-- a11y/trend/series.sql — applied before every append, idempotent.
CREATE TABLE IF NOT EXISTS series_point (
  run_id               TEXT PRIMARY KEY REFERENCES runs(run_id),
  observed_at          TEXT NOT NULL,
  commit_sha           TEXT NOT NULL,
  series_id            TEXT NOT NULL,    -- axe version + effective rule set
  routes_scanned       INTEGER NOT NULL, -- scope denominator
  nodes_evaluated      INTEGER NOT NULL, -- size denominator
  defects              INTEGER NOT NULL, -- distinct fingerprints, violations only
  defects_serious_plus INTEGER NOT NULL,
  is_bridge            INTEGER NOT NULL DEFAULT 0  -- 1 = re-scan of an old commit
);

-- The sprint calendar is data the team already owns, not arithmetic on a date.
CREATE TABLE IF NOT EXISTS sprint (
  sprint_id TEXT PRIMARY KEY,            -- '2026-S42'
  starts_on TEXT NOT NULL,               -- inclusive, YYYY-MM-DD
  ends_on   TEXT NOT NULL                -- inclusive
);

CREATE VIEW IF NOT EXISTS series_density AS
SELECT run_id, observed_at, series_id,
       ROUND(defects * 1.0 / routes_scanned, 2)     AS defects_per_route,
       ROUND(defects * 1000.0 / nodes_evaluated, 2) AS defects_per_1k_nodes
FROM series_point
WHERE nodes_evaluated > 0;

CREATE INDEX IF NOT EXISTS series_time_idx ON series_point (observed_at);

The appender runs after the load step, reads the raw per-route reports once for the node denominator, and derives the series identity from the scanner version plus the sorted list of every rule that executed. Hashing the rule list rather than the version alone matters because a team can change the rule set without changing the version — disabling region, adding a house bundle, or switching the tag selection in the shared axe-core configuration all produce a new measurement regime under an unchanged 4.10.2.

// a11y/trend/append.mjs
// usage: node a11y/trend/append.mjs axe-raw/*.json
// Run after a11y/report/load.mjs, which inserts the runs and findings rows.
import { readFileSync } from 'node:fs';
import { createHash } from 'node:crypto';
import Database from 'better-sqlite3';

const files = process.argv.slice(2);
const db = new Database('a11y.db');
db.exec(readFileSync('a11y/trend/series.sql', 'utf8'));

const runId = `${process.env.GITHUB_RUN_ID ?? 'local'}.${process.env.GITHUB_RUN_ATTEMPT ?? '1'}`;
const run = db.prepare('SELECT * FROM runs WHERE run_id = ?').get(runId);
if (!run) throw new Error(`no runs row for ${runId}: load findings before appending`);

let nodesEvaluated = 0;
const ruleIds = new Set();
for (const file of files) {
  const r = JSON.parse(readFileSync(file, 'utf8'));
  // Every node a rule actually reached. inapplicable has no nodes but its rule
  // ids still belong in the series identity: the rule was enabled.
  for (const bucket of ['violations', 'passes', 'incomplete']) {
    for (const result of r[bucket]) {
      nodesEvaluated += result.nodes.length;
      ruleIds.add(result.id);
    }
  }
  for (const result of r.inapplicable) ruleIds.add(result.id);
}

const seriesId = createHash('sha1')
  .update([run.axe_version, [...ruleIds].sort().join(',')].join('|'))
  .digest('hex').slice(0, 10);

const count = (extra) => db.prepare(
  `SELECT COUNT(DISTINCT fingerprint) AS n FROM findings
   WHERE run_id = ? AND status = 'violation' ${extra}`).get(runId).n;

db.prepare(`INSERT OR REPLACE INTO series_point
  (run_id, observed_at, commit_sha, series_id, routes_scanned,
   nodes_evaluated, defects, defects_serious_plus)
  VALUES (?, ?, ?, ?, ?, ?, ?, ?)`).run(
  runId, run.started_at, run.commit_sha, seriesId, run.routes_scanned,
  nodesEvaluated, count(''), count('AND rank >= 3'));

console.log(`series ${seriesId}: ${count('')} defects, ${nodesEvaluated} nodes, ` +
  `${run.routes_scanned} routes`);

Only full-manifest runs on the default branch may write a point. A pull-request scan, and especially a diff-aware scan that visits three routes, is a gate rather than a measurement; letting it append produces a series whose denominator oscillates by an order of magnitude between adjacent rows.

- name: Append the series point
  if: always() && github.ref == 'refs/heads/main'
  env:
    A11Y_EXPECTED_ROUTES: '44'     # must match the manifest the scan job used
  run: node a11y/trend/append.mjs axe-raw/*.json

# Validation

The sprint report is one query. It picks the last complete run inside each sprint window as that sprint’s closing state, pairs each closing run with the previous sprint’s, labels the pair comparable or broken by comparing series ids, and then differences the fingerprint sets three ways.

-- a11y/trend/sprint-report.sql — new, fixed and carried-over per sprint.
WITH ranked AS (
  SELECT s.sprint_id, p.run_id, p.series_id, p.observed_at,
         p.routes_scanned, p.nodes_evaluated, p.defects,
         ROW_NUMBER() OVER (PARTITION BY s.sprint_id
                            ORDER BY p.observed_at DESC) AS rn
  FROM sprint s
  JOIN series_point p
    ON date(p.observed_at) BETWEEN s.starts_on AND s.ends_on
  JOIN runs r ON r.run_id = p.run_id
  WHERE r.branch = 'main'
    AND r.scan_status = 'complete'   -- a partial scan is not a closing state
    AND p.is_bridge = 0              -- bridges are comparisons, not observations
),
closing AS (SELECT * FROM ranked WHERE rn = 1),
paired AS (
  SELECT c.sprint_id, c.run_id AS head_run, c.series_id AS head_series,
         c.routes_scanned, c.nodes_evaluated, c.defects,
         LAG(c.run_id)    OVER (ORDER BY c.sprint_id) AS base_run,
         LAG(c.series_id) OVER (ORDER BY c.sprint_id) AS base_series
  FROM closing c
)
SELECT p.sprint_id,
       CASE WHEN p.base_run IS NULL          THEN 'no base'
            WHEN p.head_series <> p.base_series THEN 'series break'
            ELSE 'comparable' END                        AS comparability,
       p.defects,
       ROUND(p.defects * 1.0 / p.routes_scanned, 2)      AS per_route,
       ROUND(p.defects * 1000.0 / p.nodes_evaluated, 2)  AS per_1k_nodes,
       (SELECT COUNT(*) FROM findings h
         WHERE h.run_id = p.head_run AND h.status = 'violation'
           AND h.fingerprint NOT IN (SELECT b.fingerprint FROM findings b
                WHERE b.run_id = p.base_run AND b.status = 'violation')) AS new_defects,
       (SELECT COUNT(*) FROM findings b
         WHERE b.run_id = p.base_run AND b.status = 'violation'
           AND b.fingerprint NOT IN (SELECT h.fingerprint FROM findings h
                WHERE h.run_id = p.head_run AND h.status = 'violation')) AS fixed,
       (SELECT COUNT(*) FROM findings h
         WHERE h.run_id = p.head_run AND h.status = 'violation'
           AND h.fingerprint IN (SELECT b.fingerprint FROM findings b
                WHERE b.run_id = p.base_run AND b.status = 'violation')) AS carried
FROM paired p
ORDER BY p.sprint_id;

Run it with sqlite3 -column -header a11y.db < a11y/trend/sprint-report.sql and the four sprints from the example above come back like this. Two invariants make the output self-checking: carried + new_defects must equal this sprint’s defects, and carried + fixed must equal the previous sprint’s. If either identity fails, the fingerprints are churning and every number in the row is fiction.

sprint_id  comparability  defects  per_route  per_1k_nodes  new_defects  fixed  carried
2026-S40   no base        156      4.11       1.83          0            0      0
2026-S41   comparable     148      3.89       1.74          11           19     137
2026-S42   series break   171      3.89       1.71          29           6      142
2026-S43   comparable     165      3.75       1.66          8            14     157

Sprint 42 is the whole argument in one row. The raw count rose by 23, the per-route rate did not move at all to two decimal places, the per-node density fell, and the honest reading is 9 new defects against 6 repairs once the 14 findings on newly scanned routes and the 6 from the new rule are attributed to scope and coverage. Feed new_defects — never defects — into the ratchet described in ratcheting violation budgets down each sprint, and hand carried to the backlog, ordered by the user-impact scoring in scoring accessibility violations by user impact.

# Series Breaks and the Bridge Run

The awkward case is the sprint that contains the upgrade. axe-core 4.10.2 lands on a Wednesday, sprint 42 closes on the Friday, and its closing run was produced by a different rule set than sprint 41’s. Three responses exist. Ignoring it publishes a fabricated delta. Discarding the history throws away the only long-run evidence the team has. The third is to break the series and buy back exactly one comparable delta with a bridge run: check out the previous sprint’s closing commit, scan it with the new scanner against the current manifest, and store the result as a point flagged is_bridge = 1.

The bridge holds code constant and changes only the instrument, so the difference between the head run and the bridge is attributable to code alone. In the worked example the bridge came back at 168 — sprint 41’s 148 plus 6 from the new rule plus 14 from the six added routes — against a head of 171, which is the +3 that 9 new and 6 fixed findings imply.

name: a11y-bridge-run
on:
  workflow_dispatch:
    inputs:
      base_commit:
        description: Closing commit of the previous sprint
        required: true
jobs:
  bridge:
    runs-on: ubuntu-24.04
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ inputs.base_commit }}
          fetch-depth: 0
      # Old application code, current scanner and current manifest: the only
      # variable left between this point and the head run is the code.
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - run: npm install --no-save axe-core@4.10.2
      - run: cp "$GITHUB_WORKSPACE/../manifest/routes.json" tests/a11y/routes.json
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test tests/a11y --reporter=line
      - run: node a11y/report/normalise.mjs axe-raw/*.json > findings.ndjson
      - run: node a11y/report/load.mjs findings.ndjson a11y.db
      - name: Flag the point as a bridge
        run: |
          node a11y/trend/append.mjs axe-raw/*.json
          sqlite3 a11y.db "UPDATE series_point SET is_bridge = 1 \
            WHERE run_id = '${GITHUB_RUN_ID}.${GITHUB_RUN_ATTEMPT}';"

Reporting then prefers the bridge as the base whenever one exists for the pair, and falls back to labelling the row series break when it does not.

-- Bridge-aware base: the re-scanned old commit wins over the old closing run.
SELECT b.run_id AS base_run,
       CASE WHEN b.is_bridge = 1 THEN 'bridged' ELSE 'series break' END AS basis
FROM series_point b
WHERE b.commit_sha = :previous_closing_commit
  AND b.series_id  = :head_series_id     -- same instrument as the head run
ORDER BY b.is_bridge DESC, b.observed_at DESC
LIMIT 1;

# Edge Cases and Conditional Guards

• A sprint with no complete run. Release freezes, a broken preview deploy, or a week of partial scans leave a window with no closing state. LAG then pairs two non-adjacent sprints and attributes two sprints of work to one. Emit the sprint with a no base comparability label and a null delta rather than silently spanning the gap.
• Diff-aware and affected-package runs. A scan scoped to the routes a pull request touched has a valid but incomparable denominator. Gate on it, never append it: the github.ref == 'refs/heads/main' guard plus the scan_status = 'complete' filter keeps those runs out of the series entirely.
• Seed data that grows. nodes_evaluated is sensitive to fixture size, so a demo database that gains 5,000 orders raises the denominator and lowers density with no accessibility change at all. Pin the seed for the measurement run, or record the seed version alongside the point and treat a seed change as a series break like any other.

# Pipeline Impact

The append step runs inside the reporting job with if: always(), so a red gate still contributes a point — the sprints that break are the ones whose numbers matter. It never touches the exit code and never reads a threshold. The sprint report itself is a scheduled job that runs the morning after a sprint closes, writes the four-column table into the run summary, and exits 0 whatever the numbers say; the enforcement decision belongs to the gate. Because the store must survive ephemeral runners, restore it from cache before the load step and upload it as an artifact after; if the store is lost, insert a new series_id prefix rather than pretending the old and new histories are one line. The same points feed the panels described in visualizing WCAG compliance trends with Grafana, where the series break becomes a dashboard annotation.

# Common Pitfalls

• Reporting the raw total, so a sprint that added six routes to the manifest looks like a sprint that broke six pages.
• Deriving sprint windows by dividing a timestamp by fourteen, which reassigns commits to different sprints the moment the cadence slips or the divisor changes.
• Taking the newest run in a window instead of the newest complete run, so a crashed scan that reached eight of forty-four routes is published as a dramatic improvement.
• Subtracting totals rather than differencing fingerprint sets, which renders twelve fixes and twelve regressions indistinguishable from an idle sprint.
• Drawing one continuous line across a scanner upgrade, then spending a sprint retrospective explaining a step change that was two new rules.
• Building the node denominator from failing nodes only, which makes density rise as the application improves.

# FAQ

Per route or per scanned node — which denominator should the sprint report lead with? Lead with per route, because it is the one a non-engineer can interpret: it answers “how many defects does an average page of ours carry”. Keep per 1,000 evaluated nodes beside it as the control, since it is the only figure that survives a content change such as a table growing from twelve rows to two hundred. When the two disagree — per-route flat and per-node falling, as in sprint 42 — the disagreement is itself the finding, and it usually means the scanned pages got bigger rather than worse.

Does a scanner upgrade have to reset the whole history? No, and it should not. Keep every old point; just refuse to compute a delta across a series-id change. Charts render the old and new segments as separate lines, the sprint report labels the crossing row series break, and a single bridge run buys back the one delta anyone actually needs. Two or three breaks a year are normal, and a series with no breaks recorded almost always means the version was never stored rather than never changed.

Can defect density be compared between two applications or two teams? Not usefully. nodes_evaluated depends on how many rules are enabled and how much markup a framework emits, so a table-heavy internal tool will always look worse per node than a marketing site with the same real quality, and a team that disables three rules instantly looks better. Density is a within-application, within-series measure: compare an application to its own past, and compare teams on movement — new versus fixed — rather than on levels.

# Related

• Reporting, Dashboards & Violation Tracking — the parent guide that defines the run, finding and fingerprint records this series is built from.
• Visualizing WCAG Compliance Trends with Grafana — rendering these points as panels, with the series break as an annotation.
• Automated Accessibility Remediation Patterns — the section that turns a carried-over finding into scheduled remediation work.