Versioning Custom Rules Without Breaking Existing Pipelines

Semantic versioning describes what happens to code that calls your API. A rule package has almost no API — one configure() function whose signature never changes — and yet a patch release can turn eleven pipelines red without touching a single call site. This guide is part of Custom Rule Testing & Distribution, and it sets out a version policy for rules that gate merges: why tightening a rule is a breaking change for every consumer, how to ship a new rule as low-impact behind an opt-in tag before it blocks anything, how to deprecate a rule ID without deleting it, and how to size a migration window against each consumer’s own threshold budget.

Root Cause

The contract a rule package offers is not its function signature. It is the number of violations it will report against a given DOM. Consumers gate merges on that number — directly, when the pipeline fails on any violation, or through a budget, when it fails above a count. Any change that makes the reported count larger for markup that already exists in a consumer’s codebase is a breaking change to them, regardless of which digit of the version moved.

That inverts the usual semver intuition in a way that catches teams out. Fixing a false negative — the rule should have flagged role="button" elements and now does — is unambiguously a bug fix, and a bug fix is a patch release. But the consumer whose admin console has forty unlabelled role="button" elements experiences that patch as a hard stop on every merge until someone fixes forty components. From their side there is no difference between a bug fix, a tightened selector, and a brand-new rule: the count went up and the gate closed.

The mirror case is quieter and worse. Loosening a rule, narrowing a selector or deleting a rule ID lowers the count, which never breaks a build — so nobody notices. A consumer that ratchets its budget down after each release, in the manner described in progressive threshold management, will lock in the lower number as its new ceiling. When the rule is later restored, the ratcheted budget makes the restoration look like a regression. Removals therefore need announcing precisely because they are silent.

Rule change kinds against consumer impact New rule and tightened rule both raise the reported violation count and are breaking for consumers, while loosened and removed rules lower the count and pass silently into a ratcheted budget. change to the rule set count moves release must be new rule, enabled by default up major selector widened / check tightened up major new rule behind an opt-in tag flat minor rule loosened or id removed down, silently major + announcement
Only the opt-in row is a safe minor; everything that moves a consumer's count in either direction is a major.

Configuration

Encode the policy in the rule definitions themselves so it survives staff turnover. A new rule arrives disabled, carrying an -optin tag and a deliberately low impact, plus explicit metadata about when it becomes default:

// rules/dialog-initial-focus.js — introduced in 4.1.0, default in 5.0.0
module.exports = {
  id: 'ds-dialog-initial-focus',
  selector: '[data-ds="dialog"][open]',
  // The -optin tag is how consumers select the new rules without naming each one.
  tags: ['wcag2a', 'wcag131', 'ds-rules', 'ds-optin'],
  // minor on purpose: even when enabled, it must not cross a critical/serious gate.
  impact: 'minor',
  enabled: false,
  all: ['ds-focus-lands-inside'],
  metadata: {
    description: 'An open dialog must move focus to an element inside itself',
    help: 'Focus the dialog container or its first control when it opens',
  },
};

Naming the promotion release in the file header — default in 5.0.0 — sounds like a comment and functions as a commitment. A rule introduced with no stated promotion version stays opt-in indefinitely, because nobody volunteers to own the release that closes a gate, and after four quarters the tag holds a dozen rules that nothing enforces and nobody has fixed.

impact: 'minor' is doing real work. Most gates fail on critical and serious only, so a consumer that opts a new rule in immediately gets a reported violation and a green build — a warning rather than a block. Promoting the rule to serious is then a separate, visible decision that belongs in a major release.

Deprecation works the same way in reverse. Never delete a rule ID: an ID that vanishes takes its violations with it, and it also breaks any consumer that names it in runOnly or in an ignore list, which fails with an unhelpful “unknown rule” error. Keep the ID, disable it, and mark it:

// rules/legacy-icon-button-title.js — superseded by ds-toolbar-controls-named in 4.2.0
module.exports = {
  id: 'ds-icon-button-title',
  selector: '[data-ds="icon-button"]',
  tags: ['ds-rules', 'ds-deprecated'],   // tag, not deletion: runOnly references survive
  impact: 'minor',
  enabled: false,
  all: ['ds-name-present'],
  metadata: {
    description: 'Deprecated: use ds-toolbar-controls-named instead',
    help: 'This rule is retained as a no-op and will be removed in 6.0.0',
  },
};

The ds-optin tag is what lets a consumer run the candidate rules without gating on them. Two scans, one set of options each, and only the second one influences the exit code:

// tests/a11y/axe-options.js — the consumer's split between gating and reporting.
const GATING = {
  runOnly: { type: 'tag', values: ['ds-rules'] },
  // Deprecated ids stay resolvable but must never contribute to the gate.
  rules: { 'ds-icon-button-title': { enabled: false } },
};

const REPORTING = {
  runOnly: { type: 'tag', values: ['ds-optin'] },  // candidates only
};

// The gate reads violations from GATING; REPORTING results are uploaded as an artifact.
module.exports = { GATING, REPORTING };

Because a rule tagged ds-optin also carries ds-rules, order matters: the gating scan must disable the opt-in ids explicitly, or filter its violations by impact, so an adopted candidate does not start blocking merges the moment a consumer enables it for measurement.

Then make the policy machine-checkable. This test reads the previous published version’s rule inventory and fails the release when a change and its version bump disagree:

// test/version-policy.test.js
const test = require('node:test');
const assert = require('node:assert/strict');
const current = require('../rules');
const previous = require('./fixtures/inventory-previous.json');  // committed on release
const { version } = require('../package.json');

const [major] = version.split('.').map(Number);
const byId = (list) => new Map(list.map((r) => [r.id, r]));
const now = byId(current);
const before = byId(previous);

test('no rule id disappears without a major bump', () => {
  const gone = [...before.keys()].filter((id) => !now.has(id));
  if (gone.length) assert.ok(major > previous.major, `removed ${gone} needs a major`);
});

test('a rule enabled by default for the first time needs a major', () => {
  const newlyDefault = [...now.values()].filter(
    (r) => r.enabled !== false && before.get(r.id)?.enabled === false
  );
  if (newlyDefault.length) {
    assert.ok(major > previous.major, `${newlyDefault.map((r) => r.id)} now default`);
  }
});

test('impact may only be raised in a major release', () => {
  const order = { minor: 1, moderate: 2, serious: 3, critical: 4 };
  const raised = [...now.values()].filter(
    (r) => before.has(r.id) && order[r.impact] > order[before.get(r.id).impact]
  );
  if (raised.length) assert.ok(major > previous.major, `impact raised: ${raised.length}`);
});

test('any new rule ships disabled and tagged ds-optin', () => {
  const added = [...now.values()].filter((r) => !before.has(r.id));
  for (const r of added) {
    assert.equal(r.enabled, false, `${r.id} must ship disabled`);
    assert.ok(r.tags.includes('ds-optin'), `${r.id} must carry ds-optin`);
  }
});
One rule across four releases A rule ships disabled and opt-in at 4.1.0, consumers enable it during the migration window at 4.2.0, it becomes enabled by default with a raised impact at 5.0.0, and the superseded rule id is retired as a no-op at 6.0.0. 4.1.0 4.2.0 5.0.0 6.0.0 added, disabled tag ds-optin impact minor consumers opt in counted, not gating budget absorbs it default + serious now gates merges major release old id retired no-op kept then removed the window between 4.1.0 and 5.0.0 is the migration period its length is set by the slowest consumer's remaining budget
A rule crosses from opt-in to gating exactly once, at a major boundary, with a measured window in between.

Validation

Size the migration window from data rather than from a calendar. Each consumer already knows its violation budget; run the candidate rule against each consumer in warning mode and compare the new violations it would add against the headroom that consumer has left:

# For each consumer, scan with the opt-in rules on and report what they would add.
for repo in checkout-web admin-portal docs-site; do
  npx axe-scan --repo "$repo" \
    --rules-tag ds-optin \          # only the candidate rules, nothing else
    --warn-only \                   # never fail: this run is measurement
    --json "reports/$repo.json"
done

node scripts/migration-window.js reports/*.json
# repo           budget  used  headroom  optin-adds  sprints-needed
# checkout-web      40     31         9           4             1
# admin-portal      60     58         2          37            19
# docs-site         25      6        19           2             1
#
# Slowest consumer: admin-portal (19 sprints at the current burn-down rate)
# Recommendation: split ds-dialog-initial-focus adoption per directory,
#   or hold 5.0.0 until admin-portal headroom exceeds 37.

Plotted side by side, the mismatch between headroom and cost is obvious in a way the table is not:

Headroom versus opt-in cost per consumer Checkout web has nine spare violations against four added and docs site nineteen against two, but admin portal has only two spare against thirty-seven added, so it sets the migration window. headroom left in the budget violations the opt-in rules add 9 4 checkout-web 1 sprint 2 37 admin-portal 19 sprints 19 2 docs-site 1 sprint one consumer sets the window; scope adoption per directory instead of waiting
Two consumers could adopt the rule next sprint; the third would need nineteen, which is the case the policy has to answer.

admin-portal is the whole reason for this exercise. Publishing the rule as default-on would add thirty-seven violations to a repository with two spare, so the gate closes on the next unrelated pull request. Nineteen sprints is not an acceptable window either, which is the signal to scope adoption per directory rather than per repository — the technique in per-directory accessibility budgets in legacy code — so new code adopts the rule immediately and the legacy directory carries an explicit exemption with an owner.

Verify the deprecation path too, and pin the expected behaviour in the corpus alongside the fixture assertions from unit-testing custom axe rules with Jest fixtures. A consumer still naming the retired ID must keep working:

node -e "
const axe = require('axe-core');
require('@acme/axe-rules').configure(axe, { enableAll: true });
const ids = axe.getRules().map((r) => r.ruleId);
console.log('deprecated id still resolvable:', ids.includes('ds-icon-button-title'));
"
# deprecated id still resolvable: true

Edge Cases and Conditional Guards

  • A rule that was wrong, not merely lenient. A check producing false positives has to be loosened immediately, and that lowers counts. Ship it as a patch, but announce it, because consumers that ratchet budgets after every release will bank the lower ceiling and later read the corrected rule as a regression.
  • axe-core minor upgrades underneath the package. A change in axe’s accessible-name computation can move counts with no change to your rules at all. Pin the peerDependency range narrowly, and treat widening it as a minor release with its own canary run rather than as housekeeping.
  • Consumers on different majors at once. Support at most two: the current major and the previous one. Backport genuine false-positive fixes to the older line, never new rules or tightened selectors, and publish an end-of-support version so a consumer cannot sit on a three-year-old rule set indefinitely.

Pipeline Impact

The version policy shows up in CI as three concrete jobs. The release workflow runs version-policy.test.js and refuses to publish when a removal, a new default or a raised impact appears without a major bump — a non-zero exit before npm publish runs, so a policy violation never reaches the registry. The measurement workflow runs the opt-in tag against every consumer on a schedule and writes the headroom table as an artifact, which is what makes the migration window a number rather than an argument. And each consumer’s own accessibility job splits the rule set by tag: ds-optin results are reported and uploaded but never gate, while the default rules gate as usual.

That split is the mechanism that lets a rule be visible for a full quarter before it can block a merge, and it is why the pattern in soak-testing a new accessibility gate in warning mode applies per rule rather than only per pipeline. Announce the promotion date in the release notes of the minor that introduces the rule, not in the major that enforces it — by then the notes arrive with the broken build.

Common Pitfalls

  • Shipping a false-negative fix as a patch, closing the gate for every consumer that already has that markup.
  • Deleting a deprecated rule ID, which breaks runOnly and ignore lists with an unknown-rule error.
  • Introducing a new rule as serious and enabled, so it gates on release day with no migration window.
  • Sizing the window by calendar rather than by the slowest consumer’s remaining budget.
  • Letting a loosened rule lower counts silently, which a ratcheting budget then locks in as the new ceiling.
  • Widening the axe-core peer range without a canary run, so counts move with no visible change to the rules.

FAQ

Is tightening a rule really a major version, even when the old behaviour was a bug? For a package that gates merges, yes. Semver is a promise about what the consumer has to do to adopt a release, and adopting a tightened rule requires fixing whatever it now reports. Calling that a patch means the consumer’s install command is the thing that broke their pipeline, which is precisely the situation versioning exists to prevent.

How long should the opt-in window be? Long enough for the slowest consumer to absorb the new violations at its actual burn-down rate, which the headroom table measures directly. In practice one or two sprints is typical, a quarter is common when a legacy application is involved, and anything beyond that is a signal to scope adoption per directory instead of holding the whole release.

What if a rule must gate immediately, for a legal or contractual deadline? Then it is not a rule-package decision. Publish it opt-in as usual, and have the consumers that are in scope enable it and set their own gate, so the urgency is applied where the obligation lives rather than to every repository that happens to install the package. Compliance deadlines rarely apply uniformly across an organisation’s products.

Back to Custom Rule Testing Distribution