# Lighthouse Accessibility Score vs axe Violation Counts

A pull request shows two accessibility checks. Lighthouse says the accessibility score fell from 0.97 to 0.89; axe says one violation, twelve nodes. Someone asks which number is wrong, and the answer is neither — the two tools counted different things about the same DOM. This guide is part of Lighthouse CI Baseline Configuration, and it is a reconciliation procedure: dump both result sets for one URL, join them on rule ID, classify each disagreement, and decide which number is allowed near a gate.

# Root Cause

Lighthouse runs axe-core. That fact is the source of the confusion, because it creates an expectation of agreement that the architecture does not support. Lighthouse bundles a pinned axe-core build, runs a selected subset of its rules, and then discards the shape of the results: each rule becomes one Lighthouse audit that is either scored 1 or scored 0, regardless of whether one node failed or two hundred did. Those binary audit scores are then multiplied by per-audit weights and averaged into the single category number. Three separate transformations sit between the raw axe output and the score on the pull request.

The first transformation is selection. Lighthouse’s accessibility category contains around sixty scored audits; axe-core 4.10 ships roughly a hundred rules across its WCAG and best-practice tags. Rules that exist only in axe — region, nested-interactive, autocomplete-valid, scrollable-region-focusable — have no Lighthouse audit, so they cannot affect the score by any amount. In the opposite direction, the category includes eleven manual placeholder audits that no engine evaluates at all.

The second is aggregation. axe reports one violation object per rule with an array of failing nodes, so its natural units are “violations” and “nodes”. Lighthouse reports one pass/fail per audit and weights it, so its natural unit is “score points”. A single broken component repeated twelve times is one axe violation with twelve nodes and one failed Lighthouse audit — and because weights differ per audit, that one failure might cost 3.7 points or 5.3 points depending on which rule it was. Three unrelated single-node failures are three axe violations with three nodes total and three failed audits, which can cost four times as much score as the twelve-node problem.

The third is rendering context. Lighthouse audits by default under an emulated mobile viewport of 412 by 823 CSS pixels at a device pixel ratio of 1.75, with a mobile user-agent string and simulated throttling. A standalone axe run driven from a desktop-sized Playwright page audits a different rendered document: a different breakpoint, different visible elements, a navigation that is expanded in one and collapsed behind a menu button in the other. Both engines are correct about the page they saw.

# Configuration

Reconciliation starts with two JSON dumps taken against the same URL in the same commit. Run them back to back against a locally served build so no deploy sits between them, and keep the Lighthouse settings identical to the ones in lighthouserc.js — a reconciliation done under different emulation than the gate uses answers a question nobody asked.

npm run build
npx serve dist --listen 4173 --single &   # one origin for both engines
URL=http://127.0.0.1:4173/checkout/cart/

# Lighthouse: accessibility only, one run, default mobile emulation.
npx lighthouse "$URL" \
  --only-categories=accessibility \
  --output=json --output-path=./lh.report.json \
  --chrome-flags="--headless=new --no-sandbox --disable-dev-shm-usage"

# axe-core CLI: the full WCAG 2.2 AA tag set, saved as JSON.
npx @axe-core/cli "$URL" \
  --tags wcag2a,wcag2aa,wcag21a,wcag21aa,wcag22aa \
  --save ./axe.report.json \
  --exit  # non-zero on any violation; keep it so the shell records the truth

The join script reads both files, keys them on rule ID, and prints one markdown row per rule. It also computes what each failed audit cost the score, because “points lost” is the unit that makes a Lighthouse number comparable to anything. Note where the weights come from: lhr.categories.accessibility.auditRefs carries a weight per audit, and the denominator is the sum of the weights of the audits that actually applied to this page — audits marked not applicable are excluded, so the total is page-specific and must be recomputed per URL.

// scripts/reconcile-a11y.mjs
// usage: node scripts/reconcile-a11y.mjs lh.report.json axe.report.json
import { readFileSync } from 'node:fs';

const lhr = JSON.parse(readFileSync(process.argv[2], 'utf8'));
const axeRaw = JSON.parse(readFileSync(process.argv[3], 'utf8'));
// @axe-core/cli writes an array with one result object per audited URL.
const axe = Array.isArray(axeRaw) ? axeRaw[0] : axeRaw;

// Only scored audits carry weight; manual and notApplicable ones do not.
const refs = lhr.categories.accessibility.auditRefs;
const scored = refs.filter((r) => {
  const mode = lhr.audits[r.id].scoreDisplayMode;
  return mode === 'binary' || mode === 'numeric';
});
const totalWeight = scored.reduce((sum, r) => sum + r.weight, 0);

const lhRows = new Map();
for (const ref of scored) {
  const audit = lhr.audits[ref.id];
  const failed = audit.score === 0;
  lhRows.set(ref.id, {
    failed,
    weight: ref.weight,
    // Lighthouse keeps the offending elements even though the audit is binary.
    nodes: audit.details?.items?.length ?? 0,
    lost: failed ? +((ref.weight / totalWeight) * 100).toFixed(1) : 0,
  });
}

const axeRows = new Map(axe.violations.map((v) => [v.id, v.nodes.length]));
const axeIncomplete = new Set(axe.incomplete.map((v) => v.id));

function classify(id, lh, nodes) {
  if (!lh) return 'not in the Lighthouse subset';
  if (nodes === undefined && lh.failed) return 'render context differs';
  if (nodes !== undefined && !lh.failed) {
    return axeIncomplete.has(id) ? 'axe incomplete' : 'render context differs';
  }
  if (nodes !== undefined && lh.failed && nodes !== lh.nodes) {
    return 'node count differs';
  }
  return lh.failed ? 'agrees: both fail' : 'agrees: both pass';
}

const ids = [...new Set([...lhRows.keys(), ...axeRows.keys()])].sort();
console.log('| rule id | axe nodes | LH audit | LH nodes | points lost | class |');
console.log('|---|---|---|---|---|---|');
for (const id of ids) {
  const lh = lhRows.get(id);
  const nodes = axeRows.get(id);
  // Skip the wall of mutual passes; only disagreements need a human.
  const cls = classify(id, lh, nodes);
  if (cls === 'agrees: both pass') continue;
  const audit = lh ? (lh.failed ? 'fail' : 'pass') : '—';
  console.log(
    `| \`${id}\` | ${nodes ?? '—'} | ${audit} `
    + `| ${lh?.nodes ?? '—'} | ${lh?.lost ?? '—'} | ${cls} |`,
  );
}
console.log(
  `\nApplicable weight: ${totalWeight}. `
  + `Category score: ${lhr.categories.accessibility.score}. `
  + `axe violations: ${axe.violations.length}, `
  + `nodes: ${axe.violations.reduce((n, v) => n + v.nodes.length, 0)}.`,
);

# Validation

Running the script against the cart page produces the table below. Read it as the answer to “why do the two checks disagree”, one row at a time.

The footer line reads: applicable weight 190, category score 0.81, axe violations 7, nodes 24. Both numbers are now explicable. axe’s seven violations include two rules Lighthouse never runs, so Lighthouse can only see five of them. Of the five, color-contrast accounts for exactly half the failing nodes and costs less score than image-alt, which failed on one node — because the weight of image-alt in this run is 10 against color-contrast’s 7. And target-size fails only in Lighthouse, because the desktop-width axe run had nothing under 24 by 24 CSS pixels while the 412-pixel emulated viewport did.

The third disagreement class deserves its own confirmation step, because it is the one people misdiagnose as a tool bug. Re-run axe under Lighthouse’s emulation and the target-size row moves to agreement.

// tests/a11y/match-lighthouse-emulation.spec.ts
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

test.use({
  // Lighthouse's default mobile emulation, reproduced exactly.
  viewport: { width: 412, height: 823 },
  deviceScaleFactor: 1.75,
  isMobile: true,
  hasTouch: true,
});

test('axe under Lighthouse emulation sees the same target-size failures', async ({ page }) => {
  await page.goto('http://127.0.0.1:4173/checkout/cart/');
  await page.waitForLoadState('networkidle');
  const results = await new AxeBuilder({ page })
    .withTags(['wcag22aa'])
    .analyze();
  const targetSize = results.violations.find((v) => v.id === 'target-size');
  // The desktop run found none; the emulated mobile run finds Lighthouse's nine.
  expect(targetSize?.nodes.length).toBe(9);
});

# Edge Cases and Conditional Guards

• axe incomplete results have no Lighthouse equivalent. axe returns a third array for nodes it could not decide — text over a background image, an element inside a closed shadow root. Lighthouse folds those into a pass, so a rule can be incomplete with four nodes in axe and score 1 in Lighthouse. Treat that row as “needs a human”, never as a Lighthouse false negative.
• A single component with twelve instances is one fix, not twelve. Node counts inflate with repetition, so a component rendered in a loop produces a scary axe number and a modest score drop. Deduplicate by nodes[].target selector prefix or by component name before anyone reports the count in a status meeting.
• Version skew silently changes the overlap. The axe build inside Lighthouse is pinned to the Lighthouse release, so a standalone axe-core at 4.10 and a Lighthouse bundling 4.9 differ in rule definitions and in which nodes a rule matches. Print both versions in the reconciliation output — lhr.lighthouseVersion and the testEngine.version field in the axe report — before debugging a one-node difference.

# Pipeline Impact

Only one of these numbers belongs on a blocking gate, and it is the axe node count. It has the three properties a gate needs: it is deterministic given a fixed viewport, it names the failing element, and it moves by exactly one when one problem is introduced or fixed. Gate on rule ID plus impact — block on serious and critical, annotate the rest — and the pull request comment can point at a selector rather than at an arithmetic delta.

The Lighthouse score belongs on a trend line. Its weighted average is genuinely useful for answering “is this route getting better over six months”, and it is the number a non-engineering stakeholder can read, which is a real feature. What it cannot do is answer “what changed in this pull request”, because a 0.08 drop is consistent with one new violation, with a previously not-applicable audit becoming applicable, or with a font that shifted a tap target under emulation. If Lighthouse must block, block on individual audit IDs at minScore: 1 as set out in setting up Lighthouse CI thresholds for WCAG 2.2 AA, never on the category score.

Wire the reconciliation itself into the pipeline as a diagnostic rather than a gate: run it nightly on the main routes, publish the joined table as a job summary, and let anyone who questions the two checks read the table instead of opening a tooling bug. The rule-level configuration that keeps the axe side stable lives in axe-core configuration and setup, and the trend side belongs with the dashboards in reporting, dashboards and violation tracking. If the open question is which engine should be required in branch protection in the first place rather than why the numbers differ, that argument is made in axe-core vs Lighthouse CI for PR gating.

# Common Pitfalls

• Reading a score delta as a violation count, so “we lost eight points” becomes “we introduced eight problems” in a status report.
• Comparing a Lighthouse category score against an axe violation total at all — the numbers have different units and different denominators, and no ratio between them is meaningful.
• Reconciling against two different renders: a Lighthouse run on a deployed preview and an axe run on a local build, with a commit or a feature flag between them.
• Forgetting that not-applicable audits leave the weighted average, so a page that gains its first data table can lose score without any regression.
• Assuming a rule missing from Lighthouse is a Lighthouse bug, when the rule was never in the audit list and never could have fired.
• Deduplicating nothing, then triaging twelve nodes of one component as twelve tickets.

# FAQ

If Lighthouse runs axe-core, why do the rule sets differ at all? Lighthouse curates its audit list, and each audit is a wrapper with its own title, weight and scoring behaviour rather than a passthrough of an axe rule. Rules the Lighthouse team judged too noisy or too advisory for a general-purpose report — region and nested-interactive among them — simply have no wrapper, so they cannot appear in the report at any severity. The overlap also shifts with every Lighthouse release, which is why the reconciliation script derives the list from auditRefs instead of hardcoding it.

Which number should go in a compliance report? The axe rule IDs with their node counts and their mapped success criteria, because an auditor asks which criteria failed and where, not what a weighted average was. The Lighthouse score is fine as supporting evidence of a trend over time, clearly labelled as a score over a subset of audits at one emulated viewport. Presenting the score as a conformance percentage is the fastest way to lose credibility in an audit conversation.

Can the two be made to agree exactly? Close, but not exactly, and chasing it is not worth the effort. Matching the viewport, device scale factor and user-agent gets the rendering-context class to zero; pinning the standalone axe-core to the version Lighthouse bundles removes rule-definition drift. What remains is structural — Lighthouse reports one binary result per audit and omits rules it never wrapped — so the goal is a table where every remaining difference has a named class, not a single shared number.

# Related

• Lighthouse CI Baseline Configuration — the parent guide on collect, assert and baseline storage.
• Lighthouse CI Thresholds for WCAG 2.2 AA — the per-audit assertion list that replaces a category-score gate.
• Web Accessibility Testing Fundamentals & Tool Selection — how the two engines sit alongside Playwright, Cypress and Pa11y.