Custom Rule Testing and Distribution

A custom axe-core rule that lives in one repository’s tests/ directory is a script. The same rule, once six product teams gate their merges on it, is infrastructure — and it needs the same treatment as any other shared dependency: a test suite, a version number, a registry, and a rollout plan. This guide is part of Custom Rule Development & Context-Aware Testing, and it covers the full lifecycle of a rule set as a shipped software product: how to lay out a rule package, how to unit-test rules against DOM fixtures in jsdom, how to build and publish it, and how to move consumers onto a new rule version without turning every downstream pipeline red on a Tuesday morning.

Problem Statement

Rules written with the axe.configure patterns in Component-Specific Rule Writing start life inline. Someone pastes an evaluate function into a Playwright spec, it catches a real defect, and within a month three other repositories have copy-pasted it. Now there are four divergent copies of the same check, each with slightly different selectors, and none of them has a test.

The failure mode is specific and it is not about code quality. It is that a rule’s behaviour is a contract with every pipeline that runs it. When a rule’s selector widens by one attribute, or its evaluate starts returning false where it used to return true, every consumer’s violation count changes. If those consumers gate merges on a violation threshold, a rule change is functionally a change to their merge policy — pushed to them without review, at install time. Teams discover it when a pull request that touches a README fails the accessibility job.

Copy-paste avoids that by accident: nobody’s rules ever change, so nobody’s gate ever moves. It also means a bug fixed in one repository stays broken in five others, and the accessible-name logic that took two days to get right in the design-system repo never reaches the checkout app. The way out is not more discipline about copying. It is to make the rule set a package with a tested public surface, a semantic version, and an explicit story for what a version bump is allowed to do to a consumer’s violation count.

Key implementation targets:

  • A rule package with a single configure() entry point, so consumers never reach into internals.
  • A jsdom harness that runs the real axe.run against HTML fixtures, asserting on all four result buckets.
  • A fixture corpus that pairs every rule with at least one passing and one failing DOM.
  • A build step that emits both a Node entry point and a browser bundle, smoke-tested before publish.
  • Consumer-side exact pinning, so a rule version reaches a pipeline only when someone merges it.

Prerequisites

1. Laying Out the Rule Package

Four directories and one file. checks/ holds the evaluate functions, one per file, each exporting a plain check object. rules/ holds the rule definitions that bind checks to selectors and tags. fixtures/ holds HTML files. src/configure.js is the only thing consumers import.

Keeping checks and rules in separate directories is not decoration. A check is reusable logic — “does this element have an accessible name” — and several rules may compose the same check with different selectors and impacts. If checks live inside rule files, the second rule that needs the same logic copies it, and you are back to divergent copies inside your own package.

Rule package layout and its two published entry points The checks, rules and fixtures directories are internal; only src/configure.js is public, and the build emits a CommonJS Node entry alongside a browser IIFE bundle. internal (not exported) checks/ evaluate functions rules/ selector + tags + impact fixtures/ good + bad DOM configure() the only public surface dist/index.cjs Node / jsdom tests dist/browser.js injected IIFE fixtures ship in the tarball so consumers can re-run the corpus
Only configure() crosses the package boundary; the build fans it into a Node entry and a browser bundle.

A check module carries no selector and no tags — it is pure logic plus an id:

// checks/accessible-name-present.js
module.exports = {
  id: 'a11y-name-present',
  evaluate: function (node) {
    const label = (node.getAttribute('aria-label') || '').trim();
    if (label) return true;
    const ref = node.getAttribute('aria-labelledby');
    // aria-labelledby only counts if the referenced node exists and has text.
    if (ref) {
      const target = node.ownerDocument.getElementById(ref.split(/\s+/)[0]);
      if (target && (target.textContent || '').trim()) return true;
    }
    if ((node.textContent || '').trim()) return true;
    this.data({ reason: 'no-name-source' });   // surfaces in result.nodes[].any[].data
    return false;
  },
};

The rule module binds it to a selector, a WCAG tag set and an impact. Note enabled: false on anything newly added — new rules arrive switched off, which is the mechanism the versioning guide leans on:

// rules/toolbar-controls-named.js
module.exports = {
  id: 'ds-toolbar-controls-named',
  selector: '[data-ds="toolbar"] button, [data-ds="toolbar"] [role="button"]',
  tags: ['wcag2a', 'wcag412', 'ds-rules', 'ds-optin'],  // ds-optin = not on by default
  impact: 'serious',
  enabled: false,          // consumers switch it on when they are ready
  all: ['a11y-name-present'],
  metadata: {
    description: 'Toolbar controls must expose an accessible name',
    help: 'Add aria-label or visible text to every toolbar control',
  },
};

And the entry point that assembles both, registering only the checks the selected rules actually reference:

// src/configure.js — the only public surface of the package.
const checks = require('../checks');   // index re-exporting every check module
const rules = require('../rules');     // index re-exporting every rule module

function configure(axe, opts = {}) {
  const selected = opts.only ? rules.filter((r) => opts.only.includes(r.id)) : rules;
  const needed = new Set(
    selected.flatMap((r) => [...(r.all || []), ...(r.any || []), ...(r.none || [])])
  );
  axe.configure({
    checks: checks.filter((c) => needed.has(c.id)),
    // enableAll forces the opt-in rules on; otherwise each rule's own flag wins.
    rules: selected.map((r) => ({
      ...r,
      enabled: opts.enableAll === true || r.enabled !== false,
    })),
  });
  return selected.map((r) => r.id);   // caller can assert on the registered id set
}

module.exports = { configure, ruleIds: rules.map((r) => r.id) };

Returning the registered id list is worth the two extra lines. It gives consumers something cheap to assert on, and it is the signal that catches a rule silently disappearing from a minor release.

2. A jsdom Harness That Runs axe.run Against Fixtures

The package’s own test suite does not need a browser. axe-core runs inside jsdom for any rule whose logic is attribute-and-tree based, which covers almost every custom rule worth writing. The harness below uses Node’s built-in test runner, so the package has no test-framework dependency at all — consumers who re-run the corpus in their own repositories are not forced onto a particular runner. Teams that already standardise on Jest can express the same harness there instead; the bucket-assertion details are worked through in unit-testing custom axe rules with Jest fixtures.

// test/harness.js
const { JSDOM } = require('jsdom');
const { readFileSync } = require('node:fs');
const { join } = require('node:path');

/**
 * Load a fixture into a fresh jsdom, register this package's rules, run axe
 * scoped to the fixture root, and hand back the raw result object.
 */
async function runFixture(name, opts = {}) {
  const html = readFileSync(join(__dirname, '..', 'fixtures', name), 'utf8');
  const dom = new JSDOM(`<!doctype html><html lang="en"><body>${html}</body></html>`, {
    pretendToBeVisual: true,   // gives axe requestAnimationFrame and a layout-ish window
    runScripts: 'outside-only',
  });
  // axe reads these off the global scope at require time, so set them first.
  global.window = dom.window;
  global.document = dom.window.document;
  global.Node = dom.window.Node;
  global.Element = dom.window.Element;
  global.NodeList = dom.window.NodeList;

  // Fresh module instance per fixture: axe.configure mutates global rule state.
  delete require.cache[require.resolve('axe-core')];
  const axe = require('axe-core');
  const { configure } = require('../src/configure.js');
  const registered = configure(axe, { enableAll: true, only: opts.only });

  const result = await axe.run(dom.window.document.body, {
    runOnly: { type: 'rule', values: opts.only || registered },
    // jsdom has no real layout: contrast and visibility checks would be noise.
    rules: { 'color-contrast': { enabled: false } },
    resultTypes: ['violations', 'passes', 'incomplete', 'inapplicable'],
  });
  dom.window.close();
  return result;
}

module.exports = { runFixture };

Three details in there decide whether the suite is trustworthy. pretendToBeVisual: true gives jsdom requestAnimationFrame, without which axe’s own scheduling never resolves and axe.run hangs until the test times out. Busting the axe-core entry out of require.cache between fixtures matters because axe.configure mutates a module-level registry: without it, the rule state from fixture one leaks into fixture two and a rule you disabled stays disabled for the rest of the run. And listing all four resultTypes explicitly is what lets a test assert that a rule was inapplicable rather than passing — axe trims buckets it thinks you do not want.

Fixture-to-buckets sequence in the jsdom harness A fixture file becomes a jsdom document, the package registers its rules, axe runs scoped to the fixture root, and every matched node lands in violations, passes, incomplete or inapplicable. fixture .html readFileSync new JSDOM pretendToBeVisual configure(axe) checks + rules axe.run(body) runOnly: rule ids violations bad fixture must be here passes good fixture must be here incomplete evaluate returned undefined inapplicable selector matched nothing an untested rule usually sits in inapplicable, which reads as green assert the bucket, not the absence of failures
Every fixture run sorts nodes into four buckets; a test that only checks violations.length === 0 cannot tell green from absent.

3. A Fixture Corpus With Passing and Failing DOM

One fixture per rule is not enough. Each rule needs at minimum a good fixture that must land in passes, a bad fixture that must land in violations, and a neutral fixture containing none of the rule’s target markup that must land in inapplicable. The third one is the guard against a broken selector: if a typo makes the selector match nothing, the good and bad fixtures both report zero violations and the suite stays green while the rule has stopped working entirely.

Drive the corpus from a manifest rather than writing one test per file. The manifest is also documentation — a reviewer can see at a glance which rules have thin coverage:

{
  "ds-toolbar-controls-named": {
    "pass": "toolbar-named.good.html",
    "fail": "toolbar-named.bad.html",
    "inapplicable": "no-toolbar.html",
    "expectedFailNodes": 2
  },
  "ds-grid-header-scope": {
    "pass": "grid-scope.good.html",
    "fail": "grid-scope.bad.html",
    "inapplicable": "no-toolbar.html",
    "expectedFailNodes": 1
  }
}

The bad fixture should contain more than one failing node. A single-node fixture cannot distinguish “the rule found the problem” from “the rule fires once per fixture regardless” — a real bug when an evaluate accidentally closes over the first match. Pinning expectedFailNodes catches it:

// test/corpus.test.js
const test = require('node:test');
const assert = require('node:assert/strict');
const { runFixture } = require('./harness.js');
const manifest = require('../fixtures/manifest.json');
const { ruleIds } = require('../src/configure.js');

for (const [ruleId, spec] of Object.entries(manifest)) {
  test(`${ruleId}: bad fixture produces a violation`, async () => {
    const res = await runFixture(spec.fail, { only: [ruleId] });
    const hit = res.violations.find((v) => v.id === ruleId);
    assert.ok(hit, `${ruleId} did not appear in violations for ${spec.fail}`);
    assert.equal(hit.nodes.length, spec.expectedFailNodes);
  });

  test(`${ruleId}: good fixture produces a pass`, async () => {
    const res = await runFixture(spec.pass, { only: [ruleId] });
    assert.equal(res.violations.length, 0);
    // Not just "no violations" — the rule must actually have run and passed.
    assert.ok(res.passes.some((p) => p.id === ruleId), `${ruleId} never applied`);
  });

  test(`${ruleId}: neutral fixture is inapplicable, not passing`, async () => {
    const res = await runFixture(spec.inapplicable, { only: [ruleId] });
    assert.ok(res.inapplicable.some((r) => r.id === ruleId));
  });
}

test('every rule in the package appears in the manifest', () => {
  const uncovered = ruleIds.filter((id) => !(id in manifest));
  assert.deepEqual(uncovered, [], `rules without fixtures: ${uncovered.join(', ')}`);
});

That last test is the one that keeps the corpus honest over time. Without it, a rule added in a hurry ships with no fixtures at all, and the suite reports full coverage of the rules it happens to know about.

Required result bucket for each fixture kind The good fixture must land in passes, the bad fixture in violations, and the neutral fixture in inapplicable; any other cell means the rule or its selector is wrong. fixture kind violations passes incomplete inapplicable good DOM required bad DOM x2 required neutral DOM required allowed allowed an empty cell is a test failure: the selector or the check is wrong
Three fixture kinds, one required bucket each — the neutral row is what catches a selector that matches nothing.

4. Building the Two Entry Points and Smoke-Testing the Tarball

Consumers need the rule set in two shapes. A Node process running jsdom or @axe-core/playwright in-process wants a CommonJS module it can require. A browser context — anything injected through page.addInitScript or a Cypress cy.window() call — wants a self-contained script that attaches to window with no module loader present. Emitting only one of the two forces every consumer to build the other, badly.

# scripts/build.sh — emits both entry points, then verifies the tarball.
set -euo pipefail

rm -rf dist && mkdir dist
cp src/configure.js dist/index.cjs        # Node entry: plain CommonJS, no bundling
cp -R checks rules dist/                  # required by index.cjs at runtime

npx esbuild src/browser-entry.js \
  --bundle \
  --format=iife \
  --global-name=dsA11yRules \             # window.dsA11yRules.configure(window.axe)
  --external:axe-core \                   # axe is provided by the host page
  --target=es2019 \                       # matches the oldest browser in the CI matrix
  --outfile=dist/browser.js

node --test test/                          # corpus must pass against the source
npm pack --dry-run                         # print the exact file list going to the registry
node scripts/verify-tarball.js             # fail if dist/ or fixtures/ are missing

--external:axe-core is the flag that matters most. Bundling axe-core into the browser build means a page ends up with two axe instances, and the one your rules registered against is not the one the test runner drives — the rules appear to vanish. Keeping axe external and taking it as a peerDependency guarantees the consumer’s axe version is the one configured.

The verification step is cheap insurance against the classic packaging mistake, which is publishing a package whose files array excludes the built output:

// scripts/verify-tarball.js
const { execFileSync } = require('node:child_process');
const assert = require('node:assert/strict');

const listing = JSON.parse(execFileSync('npm', ['pack', '--dry-run', '--json'], {
  encoding: 'utf8',
}));
const files = listing[0].files.map((f) => f.path);

for (const required of ['dist/index.cjs', 'dist/browser.js', 'fixtures/manifest.json']) {
  assert.ok(files.includes(required), `tarball is missing ${required}`);
}
// Fixtures are intentionally shipped so consumers can re-run the corpus locally.
assert.ok(files.some((f) => f.endsWith('.bad.html')), 'no failing fixtures in tarball');
console.log(`tarball verified: ${files.length} files, ${listing[0].size} bytes`);

The registry credentials, provenance and scope wiring are a topic of their own — see publishing a shared axe rule package to a private registry for the .npmrc and token handling that turn this build into a published version.

5. Consumer-Side Pinning

A consumer that installs the rule package with a caret range has handed control of its merge gate to whoever publishes next. Pin exactly, and let a bot open the upgrade pull request so the change is reviewed like any other:

{
  "devDependencies": {
    "@acme/axe-rules": "3.4.1",
    "axe-core": "4.10.2"
  },
  "scripts": {
    "test:a11y": "node --test tests/a11y/",
    "verify:rules": "node tests/a11y/rule-inventory.test.js"
  }
}

Then assert the rule inventory in the consumer repository, not just the violation count. This is the test that turns a surprise into a readable failure — the pipeline says “rule set changed” instead of “3 new violations in a PR that only touched CSS”:

// tests/a11y/rule-inventory.test.js
const test = require('node:test');
const assert = require('node:assert/strict');
const { ruleIds } = require('@acme/axe-rules');

// The exact set this repository has reviewed and agreed to gate on.
const AGREED = [
  'ds-grid-header-scope',
  'ds-toolbar-controls-named',
];

test('rule inventory matches the reviewed set', () => {
  assert.deepEqual([...ruleIds].sort(), AGREED);
});

Install with npm ci so the lockfile is authoritative; npm install in CI can quietly resolve a different transitive axe-core and change results between runs. The same discipline that governs a violation budget in progressive threshold management applies here: the number that gates a merge should only move when a human moves it.

Configure the update bot to group the rule package and axe-core into a single pull request. Split apart, an axe-core minor lands on Monday and the rule package on Thursday, and when the violation count moves nobody can say which bump caused it. Grouped, the reviewer sees one diff, one violation delta, and one decision. Label those pull requests distinctly — a11y-gate works — so the reviewer knows before opening it that merging changes what blocks their colleagues’ merges rather than just what the tests print.

Pipeline Integration

The rule package repository runs two workflows. The pull-request workflow is just the corpus: node --test test/ with a non-zero exit on any fixture mismatch, plus the tarball verification so a packaging regression is caught before it can be tagged. The release workflow runs on a version tag, rebuilds, republishes, and then does the thing most rule packages skip — it installs the freshly published version into a canary consumer and runs that consumer’s accessibility suite.

# .github/workflows/rule-package-release.yml
name: rule-package-release
on:
  push:
    tags: ['v*.*.*']

jobs:
  corpus:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20', cache: npm }
      - run: npm ci
      - run: bash scripts/build.sh
      - name: Upload the full axe result JSON for every fixture
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: fixture-results
          path: reports/fixtures/*.json
          retention-days: 30

  canary-consumer:
    needs: corpus
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
        with: { repository: acme/design-system-docs }
      - uses: actions/setup-node@v4
        with: { node-version: '20', cache: npm }
      - run: npm ci
      # Install the tag we just built, not a range, so the canary tests this exact build.
      - run: npm i -D "@acme/axe-rules@${GITHUB_REF_NAME#v}"
      - run: npm run test:a11y
      - name: Report the violation delta rather than a bare pass or fail
        if: always()
        run: node scripts/violation-delta.js --baseline .a11y-baseline.json

The canary-consumer job is the whole point of treating the rule set as a product. A rule change that is correct in isolation can still add forty violations to a real application, and the canary makes that visible before any other team’s pipeline sees it. Wire its output into the same annotation path you use for ordinary scans so the delta lands on the release pull request rather than in a log; the JSON shaping for that is covered in structuring JSON violation output for Slack and GitHub annotations.

Exit codes stay conventional: the corpus job exits non-zero on a fixture mismatch and blocks the release; the canary job exits non-zero only when the violation delta is positive, which is a signal to hold the version, not necessarily to revert the rule.

Release path from tag to pinned consumer upgrade A version tag triggers corpus tests and a build, publishes to the private registry, runs a canary consumer, and only then does a bot open pinned upgrade pull requests in each consumer repository. tag v3.5.0 rule repo corpus node --test publish registry canary delta check checkout app pin 3.5.0 admin portal pin 3.4.1 docs site pin 3.5.0 a red canary holds the version; consumers stay on their pin no consumer upgrades without a merged pull request
Publishing is not deployment: each consumer adopts the new version by merging its own pinned bump.

Troubleshooting and Flaky-Test Mitigation

axe.run never resolves in jsdom. axe schedules work through requestAnimationFrame, which jsdom only provides when constructed with pretendToBeVisual: true. Without it the promise hangs and the test dies on the runner’s timeout with no useful message. Set the flag, and give fixture tests a 10-second timeout so a genuine hang fails fast.

Rule state leaking between fixtures. axe.configure mutates a module-scoped registry inside axe-core, so a rule disabled in one test stays disabled in the next when the module is shared. Deleting require.cache[require.resolve('axe-core')] per fixture, as the harness does, isolates them. The symptom of getting this wrong is order-dependent flake: the suite passes with --test-concurrency=1 and fails in parallel.

Zero-size boxes breaking visibility logic. jsdom’s getBoundingClientRect returns all zeros because there is no layout engine. Any check that reasons about size, overlap or contrast will be wrong. Disable those rules in the harness options — as done above for color-contrast — and cover them in a real browser instead, alongside the other browser-only concerns such as shadow-root traversal and focus behaviour.

A duplicate rule id silently wins. If a consumer registers both an inline copy of a rule and the packaged version, the last axe.configure call replaces the earlier definition — no warning. The rule-inventory test catches the id set changing; add an assertion on axe.getRules() length in the consumer if inline rules are still in play.

Fixture drift after a framework upgrade. A static HTML fixture cannot track markup that a component library changes. Generate fixtures from the component library’s own rendered output where possible, and treat a fixture that no longer resembles production as a stale test rather than a passing one.

A check that passes in jsdom and fails in a real browser. The fixture harness runs against a DOM implementation with no layout engine, so anything derived from geometry or computed style is either absent or a default. A check that reads getBoundingClientRect() sees zeroes, one that reads getComputedStyle(node).display sees the initial value rather than the cascaded one, and one that relies on offsetParent sees null. Each of those returns a confident, wrong answer in the unit test and a different answer in the browser scan, which is the worst possible split: the rule looks proven and behaves differently where it matters.

Keep geometry out of evaluate wherever the contract can be expressed structurally — a combobox whose aria-controls target is missing is a structural fault and needs no layout. Where a rule genuinely depends on rendering, mark it in the rule metadata and give it a browser fixture instead of a jsdom one: a Playwright spec that loads the same HTML file from fixtures/, injects the bundle, and asserts the same buckets. The fixture corpus then has two tiers, and the manifest records which tier each fixture belongs to so the release job runs both and neither silently disappears.

Common Pitfalls

  • Bundling axe-core into the browser build, so the page has two axe instances and the registered rules appear to vanish.
  • Asserting only violations.length === 0 on the good fixture, which passes identically when the rule never applied.
  • Shipping without a neutral fixture, so a selector typo leaves the whole suite green.
  • Omitting dist/ from the package files array and publishing a tarball with no code in it.
  • Letting consumers install with a caret range, which turns every publish into an unreviewed merge-policy change.
  • Skipping the canary consumer, so the first real application to see a rule change is somebody else’s pull request.

FAQ

Why run rule tests in jsdom rather than a real browser? Most custom rules reason about attributes, roles and tree structure, all of which jsdom models faithfully, and a jsdom fixture run costs milliseconds against seconds for a browser launch. That speed is what makes a three-fixtures-per-rule corpus practical. Keep a small browser-based suite for anything involving layout, computed style or shadow DOM behaviour that jsdom does not implement.

Should the package depend on axe-core or peer-depend on it? Peer-depend, always. A hard dependency lets npm install a second copy of axe-core inside node_modules/@acme/axe-rules, and rules registered against that instance are invisible to the consumer’s scan. Declare the supported range as a peerDependency and let the consumer own the resolved version.

Can the same package serve Playwright, Cypress and a jsdom unit test? Yes, and that is why the build emits two entry points. Node-side runners require('@acme/axe-rules') and call configure(axe) in-process; browser-side runners inject dist/browser.js and call window.dsA11yRules.configure(window.axe) after axe has loaded. The rule definitions are identical in both paths, so results do not diverge by runner.

How many rules belong in one package? Group by ownership, not by count. One package per team that owns a set of components works because the version number then means something to the consumers who care about it. A single company-wide package forces every team to absorb every other team’s rule changes, which is exactly the coupling that pinning is meant to remove.

What happens to incomplete results from a packaged rule? They are the honest answer when a check cannot decide, and they must not be dropped. Have the harness assert the expected incomplete count for fixtures designed to be undecidable, and have consumers surface incompletes as a warning artifact rather than folding them into the pass or fail count.

In This Section

Back to Custom Rule Development Context Aware Testing