# Scoring Accessibility Violations by User Impact

A priority score is a small piece of arithmetic that has to survive being disagreed with. This guide is part of Accessibility Debt Triage & Prioritization, and it builds one concrete function: five inputs per finding, four weighted terms, one effort multiplier, and a ranked table where every number that produced the score is printed in the same row. The output is a single ordering of every finding in a scan, and the test at the end proves that the worst finding in the report — a payment button with no accessible name — comes out at rank one.

# Root Cause

axe-core hands each result an impact of minor, moderate, serious or critical, and that field is a statement about the rule’s failure mode in the abstract. image-alt is critical because an image carrying meaning with no text alternative is categorically a hard failure. The field cannot express that this particular image is a decorative divider on a report builder that eleven internal users reach through a bookmark. Sorting on it produces an ordering in which the most severe rule outranks the most damaging instance, every time.

The naive alternative — sorting by node count — fails in the mirror image. Node count measures how loudly a defect shouts in the report, not how much it costs users. A single unmaintained admin table with 190 broken cells generates 190 rows and looks like the largest problem in the product, while the one unlabelled control on the highest-traffic form in the business generates one row and disappears. Both fields are real signals; neither is a priority, and no combination of just those two produces one, because the deciding facts — how many sessions touch the route, whether the defect blocks the task, how big the fix is — are not in the scan at all.

That gives the scoring function two jobs, and the second one is the harder one. It has to compute a defensible number, and it has to compute it in a way that nobody can dismiss. A score nobody trusts is worse than no score: the first time a team is handed a ranking it cannot reconstruct, it reorders locally, and the whole exercise degrades into a spreadsheet somebody maintains by hand. Two properties prevent that. The weights live in a checked-in file that changes only through review, so the ordering has an audit trail and an owner. And every row of the output prints its own inputs next to its score, so an engineer who thinks rank 3 is wrong can see immediately whether the disagreement is about the traffic weight, the blocking classification or the arithmetic — and two of those three are things they can fix with a pull request.

# Configuration

Five fields per finding drive the score. Three come straight from the axe report, one comes from analytics, and one is a human estimate:

The weights file holds every constant, including the mappings, so that the script contains arithmetic and nothing arguable. The four term weights sum to 1.00, which means the weighted base is always in 0…1 and the final score reads as a percentage of the worst possible finding.

{
  "termWeights": { "reach": 0.30, "harm": 0.28, "severity": 0.22, "spread": 0.20 },
  "severityScale": { "critical": 1.0, "serious": 0.75, "moderate": 0.45, "minor": 0.2 },
  "harmScale": { "blocking": 1.0, "degrading": 0.35 },
  "effortMultiplier": { "small": 1.15, "medium": 1.0, "large": 0.8 },
  "spreadSaturatesAt": 25,
  "blockingRules": [
    "button-name", "link-name", "label", "select-name", "input-button-name",
    "aria-required-attr", "aria-hidden-focus", "frame-title", "form-field-multiple-labels"
  ],
  "fixSizeByRule": {
    "image-alt": "small", "button-name": "small", "label": "small",
    "link-name": "medium", "aria-hidden-focus": "medium",
    "color-contrast": "large", "empty-table-header": "large"
  },
  "defaultFixSize": "medium",
  "reviewedBy": "a11y-guild",
  "reviewedOn": "2026-07-13"
}

The formula itself is one line of arithmetic. base is the weighted sum of four normalised terms; score scales it to 0–100 and applies the effort multiplier, which rewards cheap fixes without ever letting a trivial fix on an unvisited page outrank a blocking defect on the busiest route:

reach    = trafficWeight                       (0.00 … 1.00)
harm     = blocking ? 1.00 : 0.35
severity = severityScale[impact]
spread   = min(nodeCount / 25, 1.00)

base  = 0.30*reach + 0.28*harm + 0.22*severity + 0.20*spread
score = round(100 * base * effortMultiplier[fixSize])

That property is worth checking rather than asserting. Plotting score against traffic weight for the two extreme finding classes — a blocking, critical, small-fix defect and a non-blocking, critical, large-fix defect at full spread — shows two rising lines that never cross. Traffic separates findings within a harm class; it cannot promote a degrading defect above a blocking one, however popular the page.

The script below reads a directory of axe JSON reports plus the traffic-weight file, aggregates nodes per (rule, route) pair, scores each pair and prints a ranked table. It writes the table to stdout and the full scored rows to a JSON file, because the table is for people and the JSON is for the issue-routing step that runs next.

// a11y/score/rank.mjs
// Usage: node a11y/score/rank.mjs artifacts/scan a11y/score/traffic.json
import { readFileSync, writeFileSync, readdirSync } from 'node:fs';
import { join } from 'node:path';

const [scanDir, trafficFile] = process.argv.slice(2);
const W = JSON.parse(readFileSync('a11y/score/weights.json', 'utf8'));
const traffic = JSON.parse(readFileSync(trafficFile, 'utf8'));
const blocking = new Set(W.blockingRules);

// One entry per (rule, route): the unit a single fix usually closes.
const pairs = new Map();
for (const file of readdirSync(scanDir).filter((f) => f.endsWith('.json'))) {
  const report = JSON.parse(readFileSync(join(scanDir, file), 'utf8'));
  const route = new URL(report.url).pathname;
  for (const v of report.violations) {
    const key = `${v.id}|${route}`;
    const entry = pairs.get(key) ?? { rule: v.id, route, nodeCount: 0, impact: 'minor' };
    entry.nodeCount += v.nodes.length;
    // Keep the worst per-node impact seen on this route for this rule.
    for (const n of v.nodes) {
      const seen = n.impact ?? v.impact ?? 'minor';
      if (W.severityScale[seen] > W.severityScale[entry.impact]) entry.impact = seen;
    }
    pairs.set(key, entry);
  }
}

const scored = [...pairs.values()].map((e) => {
  // Unknown routes score 0 reach rather than throwing: a missing analytics row
  // must not silently promote a finding, and it shows up in the table as 0.00.
  const reach = traffic[e.route] ?? 0;
  const isBlocking = blocking.has(e.rule);
  const harm = isBlocking ? W.harmScale.blocking : W.harmScale.degrading;
  const severity = W.severityScale[e.impact];
  const spread = Math.min(e.nodeCount / W.spreadSaturatesAt, 1);
  const fixSize = W.fixSizeByRule[e.rule] ?? W.defaultFixSize;
  const t = W.termWeights;
  const base = t.reach * reach + t.harm * harm + t.severity * severity + t.spread * spread;
  return {
    ...e,
    reach, harm, severity, spread, fixSize,
    blocking: isBlocking,
    trafficMissing: !(e.route in traffic),
    score: Math.round(100 * base * W.effortMultiplier[fixSize]),
  };
});

scored.sort((a, b) => b.score - a.score || b.nodeCount - a.nodeCount);
writeFileSync('a11y/score/scored.json', JSON.stringify(scored, null, 2));

// The inputs print beside the score: a disputed rank is checkable in one glance.
console.log('| # | Score | Rule | Route | Reach | Blocks | Sev | Nodes | Fix |');
console.log('|---|---|---|---|---|---|---|---|---|');
scored.forEach((s, i) => {
  console.log(
    `| ${i + 1} | ${s.score} | ${s.rule} | ${s.route} | ${s.reach.toFixed(2)} ` +
      `| ${s.blocking ? 'yes' : 'no'} | ${s.impact} | ${s.nodeCount} | ${s.fixSize} |`,
  );
});

The traffic-weight file is the one input that comes from outside engineering, and it needs the least precision of the five. Normalise every route to the share of the busiest route’s sessions, round to two decimals, and accept that a route at 0.63 versus 0.61 changes nothing. Wildcard routes are collapsed to their template, because /product/8842 and /product/1170 are one route with one fix:

{
  "/checkout/payment": 1.0,
  "/checkout/cart": 0.94,
  "/signup": 0.71,
  "/product/:id": 0.64,
  "/search": 0.38,
  "/blog/:slug": 0.11,
  "/help": 0.09,
  "/admin/report-builder": 0.02
}

# Validation

The test that keeps the function honest is not a unit test on the arithmetic — that is one multiplication and rarely wrong. It is an ordering assertion against a fixture whose worst finding is known in advance by human judgment. Build a six-finding fixture, state which one must rank first and which must rank last, and assert the positions. If a weights change reorders the fixture, the change gets discussed before it ships.

// a11y/score/rank.test.mjs — run with: node --test a11y/score
import test from 'node:test';
import assert from 'node:assert/strict';
import { execFileSync } from 'node:child_process';
import { readFileSync } from 'node:fs';

// The fixture is six (rule, route) pairs with a hand-agreed worst and best case.
execFileSync('node', [
  'a11y/score/rank.mjs',
  'a11y/score/fixtures/scan',
  'a11y/score/fixtures/traffic.json',
]);
const scored = JSON.parse(readFileSync('a11y/score/scored.json', 'utf8'));

test('the blocking defect on the payment route ranks first', () => {
  assert.equal(scored[0].rule, 'button-name');
  assert.equal(scored[0].route, '/checkout/payment');
  assert.equal(scored[0].score, 95);
});

test('the critical-impact finding on the admin tool does not reach the top three', () => {
  const admin = scored.findIndex((s) => s.route === '/admin/report-builder');
  assert.ok(admin >= 3, `admin finding ranked ${admin + 1}, expected 4th or lower`);
  assert.equal(scored[admin].impact, 'critical'); // still critical, still not urgent
});

test('every scored row carries the inputs that produced it', () => {
  for (const row of scored) {
    for (const field of ['reach', 'harm', 'severity', 'spread', 'fixSize', 'nodeCount']) {
      assert.ok(row[field] !== undefined, `${row.rule}@${row.route} missing ${field}`);
    }
  }
});

test('no route is scored against missing traffic data without flagging it', () => {
  const silent = scored.filter((s) => s.trafficMissing && s.reach > 0);
  assert.deepEqual(silent, []);
});

Running the ranker on the fixture prints the table below. The interesting column is not the score; it is the pair of rows at 42 and 27, where a critical finding sits beneath a moderate one and the reason is visible in the same row.

| # | Score | Rule          | Route                 | Reach | Blocks | Sev      | Nodes | Fix    |
|---|-------|---------------|-----------------------|-------|--------|----------|-------|--------|
| 1 | 95    | button-name   | /checkout/payment     | 1.00  | yes    | critical | 3     | small  |
| 2 | 84    | label         | /signup               | 0.71  | yes    | critical | 2     | small  |
| 3 | 68    | link-name     | /blog/:slug           | 0.11  | yes    | serious  | 40    | medium |
| 4 | 55    | color-contrast| /product/:id          | 0.64  | no     | serious  | 12    | large  |
| 5 | 42    | image-alt     | /admin/report-builder | 0.02  | no     | critical | 190   | large  |
| 6 | 27    | region        | /help                 | 0.09  | no     | moderate | 1     | small  |

# Edge Cases and Conditional Guards

• A route with no traffic row. New routes, feature-flagged routes and locale variants regularly miss the analytics export. Scoring them as reach 0 is the safe default because it cannot promote a finding falsely, but it can bury a genuinely important new page. The trafficMissing flag exists so the weekly review can list those routes and assign them a provisional weight rather than leaving them silently at the bottom.
• A finding with no impact at all. axe omits node.impact for some results, and custom checks that forget metadata.impact produce undefined. Falling through to minor is deliberate — an unclassified rule should not outrank a classified one — but log the rule id, because an unclassified custom rule is a bug in the rule, not in the score.
• Spread saturating too early on virtualised or paginated content. A rule that fires once per rendered row reports 12 nodes on a virtualised list that has 10,000 rows, and 40 nodes on a table that renders in full. The saturation point at 25 keeps that discrepancy from dominating, but treat any pair whose nodeCount equals the rendered window size as a measurement rather than a count, and score the component rather than the route.

# Pipeline Impact

The ranker is a reporting step, not a gate. It runs after the full-coverage scan in the scheduled triage job, writes scored.json as a build artifact, and exits zero regardless of how bad the numbers are — a high score is information, and failing the job on it would only teach people to stop reading the job. The one condition worth failing on is a broken input: if the weights file does not parse, or a term weight has been edited so the four no longer sum to 1.00, exit non-zero, because every score in the artifact is then wrong in a way nobody will notice.

Downstream, three consumers read the same artifact. The issue-routing step turns the top of the ranking into per-team tickets. The pull-request annotation step looks up the score for any pre-existing finding on a changed route, so a reviewer touching /checkout/payment sees that a score-95 finding lives in the file they are editing — the cheapest possible nudge. And the trend store behind the reporting and violation-tracking dashboards keeps the score-weighted total over time, which is a far better progress signal than a raw count because fixing the payment button moves it visibly and fixing 190 admin cells barely does.

Keep the weights file’s review history visible to everyone who reads a score. A change to termWeights reorders the entire backlog, so treat it like a schema migration: require a reviewer from the accessibility guild, note the date in the file, and re-run the fixture test in the same pull request. When the ordering is questioned six months later, git log a11y/score/weights.json is the answer, and that is the difference between a score with authority and a number people work around.

# Common Pitfalls

• Hard-coding the weights inside rank.mjs, which makes every disagreement about the ranking into a code review of somebody’s script rather than a decision about priorities.
• Printing only the score in the ranked table, so a disputed rank cannot be checked without re-running the tool with a debugger attached.
• Letting spread grow without saturation, which puts a 190-node legacy table at the top of the queue and keeps it there for a quarter.
• Using raw pageviews rather than a normalised 0–1 share, which makes the score depend on absolute traffic and breaks the moment the business grows or a bot filter changes.
• Treating color-contrast as blocking because it is visually obvious, which floods the blocking class and destroys the harm term’s ability to discriminate.
• Rescoring with new weights without re-running the fixture assertions, so a well-intentioned tweak quietly demotes the worst finding in the product.

# FAQ

Why multiply by an effort term instead of subtracting a cost? A multiplier keeps the score in a bounded, readable range and expresses the right relationship: fix size modulates value rather than offsetting it. Subtracting a cost lets a large enough estimate drive a high-value finding negative, which produces the absurd result that a blocking defect on the payment route can rank below a decorative image because somebody estimated three weeks. Capping the multiplier at 1.15 and 0.80 also limits how much an estimate — the least reliable of the five inputs — can move a rank.

Should the score be recomputed on every pull request? No. Compute it in the scheduled full-coverage run, where every route is scanned and the traffic file is current, and have pull-request jobs read the committed scored.json for lookups. A pull request only scans changed pages, so recomputing there would produce scores whose spread term was measured against a partial report — different numbers for the same finding depending on which job ran, which is exactly the kind of inconsistency that makes people stop trusting the ranking.

How often should the weights change? Rarely, and always with a stated reason. In practice the term weights settle after two or three review cycles and then stay put for a year; what changes is the blockingRules list, as an accessibility specialist reclassifies a rule, and the fixSizeByRule map, as the team learns what a fix actually costs in this codebase. Those two are safe to adjust often because they move individual rows rather than reordering the whole backlog.

# Related

• Accessibility Debt Triage & Prioritization — the triage model this score plugs into, from normalised report to owned work queue.
• Burning Down an Accessibility Backlog Across Quarters — tracking the score-weighted total across quarters without hiding a regression.
• CI/CD Integration & Automated Quality Gating — where the scan that feeds this ranker runs, and how its exit code is decided.