Choosing Exit Codes for Warning and Blocking Accessibility Jobs

Making a job pass or fail on purpose sounds like a one-character decision and is actually four decisions stacked on top of each other: what the scanner returns, what the step result becomes, what the job conclusion becomes, and what the checks list shows a reviewer. This guide is part of Auto-Fail vs Warning A11y Pipeline Gates, and it covers only that plumbing — the shell and workflow mechanics that make a warning genuinely non-blocking and make a blocking check trustworthy.

Root Cause

Every accessibility scanner collapses two completely different outcomes into one non-zero exit code. The axe CLI exits 0 even when it found violations unless you pass --exit, and with --exit it returns non-zero for any violation at any impact — there is no flag that means “only serious and above”. A pa11y-ci run returns non-zero when a URL exceeds its configured error threshold and also returns non-zero when its configuration is invalid or a URL never loaded. lhci assert returns non-zero when an assertion fails, and lhci autorun returns non-zero when collection itself failed because Chrome could not start. In all three cases the shell sees the same thing: not zero.

Scanner Findings, no flag Flag that fails on findings Same code for a run failure
@axe-core/cli exits 0 --exit (impact-agnostic) yes
pa11y-ci non-zero over threshold threshold in config yes
lhci autorun non-zero on assertion failure assert block in config yes
Playwright + axe non-zero if the test asserts the assertion itself yes

The right-hand column is the whole problem: no scanner in that table lets the shell distinguish “the page has problems” from “I could not look at the page”. The trade-offs between the first three as pull-request gates are worked through in axe-core vs Lighthouse CI for PR gating; for the purposes of this guide they are interchangeable, because the wrapper makes their conventions irrelevant.

That ambiguity is the actual bug, and it is worse than a mis-tuned threshold. A required check that goes red because the app returned 502 for every URL tells the author to fix their markup, which they cannot do. A required check that goes green because someone silenced the scanner with || true reports coverage for a scan that never ran, which is the failure mode that survives into audit answers. Neither is a threshold problem; both are exit-code problems.

The second confusion is architectural. In GitHub Actions the scanner’s exit code sets the step result; the step result normally sets the job conclusion; the job conclusion is what becomes the check run that branch protection reads. continue-on-error: true cuts the link between the second and third of those — the step still fails, and steps.<id>.outcome still reports failure, but steps.<id>.conclusion becomes success and the job stays green. Understanding which layer you are editing is the difference between an advisory job that is genuinely advisory and one that quietly blocks merges.

From scanner exit code to check run, with and without continue-on-error In both columns the scanner exits one and the step result is failure. In the default column the job conclusion is failure and the check is red, blocking the merge. With continue-on-error set to true the job conclusion is success and the check is green, with the failed step surfaced only as a warning. The same scanner exit code, two different job conclusions default step continue-on-error: true scanner exit code step result job conclusion checks UI exit 1 exit 1 step result: failure step result: failure job: failure job: success check red merge blocked check green step shows a warning continue-on-error changes the job conclusion, never the step result — steps.id.outcome still reads failure.
Both columns run identical commands; only the third row differs, which is why an advisory job needs a decision about the layer it edits rather than about the scanner.

Configuration

The fix is a wrapper that owns the exit code, so no scanner’s convention ever reaches the step result directly. Three codes are enough, and three is also the maximum a team will reliably remember: 0 clean or within budget, 1 accessibility findings over budget, 2 harness or infrastructure error.

The three-code exit contract Exit code zero means the report is clean or within budget and the check stays green. Exit code one means accessibility findings exceeded the budget and the check goes red as an accessibility failure. Exit code two means the harness failed — a browser crash, a 502 from the application, or a timeout — and the check goes red as an infrastructure failure. A callout warns that appending logical-or true collapses both one and two into zero. One wrapper, three exit codes, three pipeline responses 0 report clean or within budget the scan ran and the gate agrees check green 1 findings over the budget a real accessibility signal for the author check red — accessibility 2 harness error browser crash, 502 from the app, timeout check red — infrastructure Appending a tolerant || true maps both 1 and 2 to 0 an infrastructure failure then looks exactly like a clean scan
The point of code 2 is that a reviewer can tell, from the check alone, whether to fix markup or re-run the job.

The shell wrapper runs the scanner, refuses to let the scanner’s own convention escape, and hands the report to a Node gate. It also publishes the code as a step output so a later step can branch on it:

#!/usr/bin/env bash
# scripts/a11y-gate.sh — one exit-code contract for every scanner in the repo.
# 0 = clean or within budget · 1 = findings over budget · 2 = harness error
set -Eeuo pipefail

TARGET_URL="${TARGET_URL:?set TARGET_URL}"
REPORT="${REPORT:-a11y-report.json}"
BUDGET="${BUDGET:-0}"

# Capture the scanner's status instead of letting it set the step result.
# --exit makes axe return non-zero on any violation; the gate reinterprets that.
set +e
npx axe "$TARGET_URL" \
  --tags wcag2a,wcag2aa \
  --exit \
  --save "$REPORT"
scanner_status=$?
set -e

# No parseable report means the scanner died, not that the page is accessible.
if [[ ! -s "$REPORT" ]]; then
  echo "::error::scanner exited ${scanner_status} without writing ${REPORT}"
  echo "code=2" >> "${GITHUB_OUTPUT:-/dev/null}"
  exit 2
fi

set +e
node a11y/gate.mjs "$REPORT" --budget "$BUDGET"
status=$?
set -e

# Publish before exiting so the value survives a non-zero code.
echo "code=${status}" >> "${GITHUB_OUTPUT:-/dev/null}"
exit "$status"

The Node gate is where “harness error” gets a real definition. A truncated JSON file, an empty page array, and a report with no testEngine block all mean the scan did not complete, and every one of them would otherwise read as zero violations:

// a11y/gate.mjs — usage: node a11y/gate.mjs a11y-report.json --budget 0
import { readFileSync } from 'node:fs';

const CLEAN = 0, FINDINGS = 1, HARNESS = 2;
const BLOCKING = new Set(['critical', 'serious']);
const budgetIndex = process.argv.indexOf('--budget');
const budget = budgetIndex === -1 ? 0 : Number(process.argv[budgetIndex + 1]);

function fail(message) {
  console.error(`::error::${message}`);
  // Setting exitCode rather than calling process.exit() lets stdout flush,
  // which matters when this command is on the left of a pipe.
  process.exitCode = HARNESS;
}

let pages;
try {
  pages = JSON.parse(readFileSync(process.argv[2], 'utf8'));
} catch (error) {
  fail(`report is not parseable JSON: ${error.message}`);
  process.exit(HARNESS);
}

// The axe CLI writes an array of per-URL results; anything else is a broken run.
if (!Array.isArray(pages) || pages.length === 0) {
  fail('report contains no page results — the scan did not complete');
  process.exit(HARNESS);
}
if (pages.some((page) => !page.testEngine?.version)) {
  fail('a page result has no testEngine block — axe never ran on it');
  process.exit(HARNESS);
}

const nodes = pages.flatMap((page) =>
  page.violations
    .filter((violation) => BLOCKING.has(violation.impact))
    .flatMap((violation) =>
      violation.nodes.map((node) => `${violation.id} ${page.url} ${node.target[0]}`)));

if (nodes.length > budget) {
  console.error(`${nodes.length} blocking node(s), budget ${budget}:`);
  for (const line of nodes) console.error(`  ${line}`);
  process.exit(FINDINGS);
}

console.log(`${nodes.length} blocking node(s) within a budget of ${budget}.`);
process.exit(CLEAN);

Wiring it up takes two steps in an advisory job and one in a blocking job. The advisory job marks the scan step continue-on-error: true, then re-raises only code 2:

      - name: Advisory accessibility scan
        id: advisory
        continue-on-error: true      # the step may fail; the job must not
        env:
          TARGET_URL: http://127.0.0.1:4173
          BUDGET: '0'
        run: scripts/a11y-gate.sh
      - name: Re-raise harness failures only
        # outcome is the pre-continue-on-error result; conclusion would be success.
        if: steps.advisory.outcome == 'failure' && steps.advisory.outputs.code == '2'
        run: |
          echo "::error::the accessibility harness failed; findings are unknown"
          exit 1
      - name: Publish the report whatever happened
        if: always()                 # runs even when an earlier step failed
        uses: actions/upload-artifact@v4
        with:
          name: a11y-report-${{ github.run_id }}
          path: a11y-report.json
          retention-days: 30

if: always() on the upload step is not optional decoration. The default condition on every step is success(), so a failing scan step skips every later step — including the one that would have uploaded the evidence you need to debug the failure. always() also runs the step after a workflow cancellation, which is usually harmless for an upload; when it is not, if: !cancelled() is the narrower form.

Validation

Prove the contract locally before any branch protection depends on it. Four fixtures and a five-line assertion helper are enough, and this script belongs in the same pull request as the wrapper:

#!/usr/bin/env bash
# scripts/test-a11y-gate.sh — proves the exit-code contract before CI trusts it.
set -uo pipefail   # not -e: a non-zero code is the thing under test

assert_code() {
  local expected="$1"; shift
  "$@" >/dev/null 2>&1
  local actual=$?
  if [[ "$actual" != "$expected" ]]; then
    echo "FAIL: expected ${expected}, got ${actual}: $*"
    return 1
  fi
  echo "ok exit ${actual}: $*"
}

failures=0
assert_code 0 node a11y/gate.mjs fixtures/clean.json --budget 0 || failures=1
assert_code 1 node a11y/gate.mjs fixtures/three-serious.json --budget 0 || failures=1
assert_code 0 node a11y/gate.mjs fixtures/three-serious.json --budget 5 || failures=1
assert_code 2 node a11y/gate.mjs fixtures/truncated.json --budget 0 || failures=1
assert_code 2 node a11y/gate.mjs fixtures/empty-array.json --budget 0 || failures=1
assert_code 2 node a11y/gate.mjs fixtures/no-such-file.json --budget 0 || failures=1
exit "$failures"

Expected output, which is also the documentation of the contract:

ok exit 0: node a11y/gate.mjs fixtures/clean.json --budget 0
ok exit 1: node a11y/gate.mjs fixtures/three-serious.json --budget 0
ok exit 0: node a11y/gate.mjs fixtures/three-serious.json --budget 5
ok exit 2: node a11y/gate.mjs fixtures/truncated.json --budget 0
ok exit 2: node a11y/gate.mjs fixtures/empty-array.json --budget 0
ok exit 2: node a11y/gate.mjs fixtures/no-such-file.json --budget 0

The last thing to validate is the pipe. The default shell for a run block on a Linux runner is bash -e, which does not enable pipefail; only an explicit shell: bash (or defaults.run.shell: bash) gives you bash --noprofile --norc -eo pipefail. Without it, the status of scanner | formatter is the formatter’s status, so a scanner that crashed mid-report is reported as a pass by a formatter that happily printed an empty table.

Why pipefail matters when a scanner feeds a formatter The axe command exits one because it found violations, while tee and the formatting script both exit zero. Under the default shell the pipeline status is the last command's zero, so the step passes and the findings disappear from the job status. With pipefail set, the pipeline status is the rightmost non-zero code, one, so the step fails and the wrapper can classify it. A scanner piped into a formatter, with and without pipefail npx axe --exit exits 1 — findings tee raw.json exits 0 node format.mjs exits 0 default bash -e: the status is the last command in the pipe = 0 the step passes and the findings never reach the job status set -o pipefail: the status is the rightmost non-zero code = 1 the step fails and the wrapper can classify the failure
The two rows run byte-identical commands; the only difference is one shell option, and it decides whether the finding is visible at all.

Edge Cases and Conditional Guards

  • Node truncates stdout when you call process.exit() on a pipe. process.exit() does not wait for asynchronous stdout flushes, so a long violation list piped into head or a formatter can be cut off mid-line. Set process.exitCode and return normally whenever the script writes more than a screen of output; reserve process.exit(code) for the early-abort paths where nothing has been written yet.
  • continue-on-error at job level is a different lever. jobs.<id>.continue-on-error stops a failing job from failing the workflow, but the job’s own check run still reports the failure, so a required check on that job name still blocks the merge. Step level is what makes an advisory step advisory; a genuinely advisory job needs its wrapper to exit 0.
  • A missing report and an empty report are both harness errors, but a stale report is worse. If the workflow re-uses a cached workspace, yesterday’s a11y-report.json can be sitting there when today’s scan crashes, and the gate happily grades an old scan. Delete the report file before the scan, or write it into a path that includes the commit SHA.

Pipeline Impact

Once every scanner goes through the same wrapper, the checks list becomes readable: a11y-block is red only for code 1, and code 2 shows up as a distinct error annotation that says the harness failed. That distinction is what makes a retry policy defensible — re-running a job on code 2 is diagnosis, whereas re-running on code 1 is hoping the accessibility problem goes away, and the two look identical if both exit 1.

The wrapper is also the right place to enforce a budget, because the budget is a number the report is compared against rather than a scanner flag. That keeps the tightening schedule in progressive threshold management independent of which scanner produced the report, and it means swapping the axe CLI for a Playwright-based scan changes one command inside a11y-gate.sh and nothing about the check names registered in pull request gating and branch policies.

One caution about --exit and its equivalents: keep them on, and reinterpret their result, rather than turning them off. A scanner that is allowed to signal its own opinion gives the wrapper a second, independent piece of evidence — if the scanner says “violations found” and the report contains none, the two disagree and that itself is a harness error worth surfacing.

Common Pitfalls

  • Ending a scan step with || true, which maps a crashed browser and a clean page to the same green check and is the single most common way an accessibility gate becomes decorative.
  • Using continue-on-error: true on the blocking job’s gate step, which leaves a required check permanently green while the log still prints violations nobody acts on.
  • Omitting shell: bash or set -o pipefail on a multi-command run block that pipes a scanner into a formatter, so the formatter’s success masks the scanner’s failure.
  • Leaving the report upload on its default if: success() condition, which deletes the evidence for exactly the runs that failed.
  • Branching on steps.<id>.conclusion instead of steps.<id>.outcome after setting continue-on-error, so the condition never matches and the harness-error branch is dead code.
  • Reusing exit code 1 for a missing configuration file, which reintroduces the ambiguity the wrapper exists to remove — configuration problems are harness problems and belong on code 2.

FAQ

Why not use more exit codes, one per severity? Because the consumer of an exit code is a shell and a checks list, neither of which can express nuance, and because severity belongs in the report rather than in a number between 3 and 125. The three codes answer the only three questions the pipeline has: proceed, stop for accessibility, stop for infrastructure. Anything finer belongs in the classified JSON that the summary and the annotations read.

Does this work the same on GitLab CI or Jenkins? The wrapper is portable, and the layer above it is not. GitLab has allow_failure: true on a job, which produces a warning state rather than the pass/fail pair Actions gives a step, and Jenkins needs catchError or a try block around the stage to keep the build result green while marking the stage unstable. Write the wrapper once, then translate only the advisory mechanism per platform.

How should a scheduled nightly job treat code 2? Fail loudly and page nobody. A nightly harness error is a real problem — tomorrow’s trend data has a hole in it — but it blocks no one, so the right response is a failed run plus an alert to the owning team rather than an on-call notification. What matters is that the trend store records “no data” rather than “zero violations”, which is only possible if the nightly job distinguishes the two codes.

Back to Auto Fail Vs Warning Workflows