Sharding axe-core Scans Across Parallel CI Jobs

Partitioning is the one decision that determines whether four parallel jobs finish in a quarter of the serial time or in half of it. This guide is part of Monorepo & Parallel Test Sharding, and it covers only the split: which routes each job scans, who computes that assignment, why the order routes are listed in changes the wall clock, and the two matrix settings that decide whether a failing shard hides the findings of every other shard.

Root Cause

The default tool for this is Playwright’s own --shard=i/n, and it is genuinely the right answer for a lot of suites. It divides the collected test list into n groups balanced on the number of tests, keeps tests declared in the same file adjacent so file-scoped setup is not paid twice, and needs no infrastructure at all. When every test costs roughly the same — component-level checks, a uniform set of static pages — equal test counts and equal durations are the same thing.

Accessibility route scans are the case where they are not the same thing. A marketing page with 380 DOM nodes scans in 1.9 seconds; an analytics dashboard with 6,400 nodes, four data tables and a colour-heavy chart legend scans in 22 seconds, because axe.run() cost tracks node count and colour-contrast sampling far more than page count does. A balanced-by-count split hands one job three dashboards and another job three marketing pages, and the gate’s wall clock is set by the unluckiest job. Balancing by count when cost varies by an order of magnitude leaves parallel capacity idle by construction.

The second problem is where the split is computed. If each job derives its own slice at run time, the split has to be a pure function of inputs every job sees identically — and it usually is not. Route lists assembled by globbing the filesystem come back in directory order, which differs between the runner image and a developer’s machine. Array.prototype.sort with localeCompare depends on the ICU data compiled into the Node build. A route list gathered by crawling a running app depends on what that app rendered this time. Any of these means a retried shard 2 scans a different set of routes than the shard 2 whose report was thrown away, so the merged report is missing routes that nothing reports as missing.

Slowest shard under three partitioning strategies The same twelve routes totalling 106 seconds are split four ways. Contiguous slicing of a heaviest-first list produces shards of 55, 25, 16 and 10 seconds; round-robin produces 35, 30, 24 and 17; cost-weighted packing produces 27, 27, 26 and 26 against an ideal of 26.5. Four shards, 106 route-seconds, three ways to split them 55 25 16 10 35 30 24 17 27 27 26 26 contiguous slices round-robin cost-weighted packing slowest 55 s slowest 35 s slowest 27 s
The thin horizontal line marks the ideal 26.5 seconds per shard; contiguous slicing doubles the wall clock while round-robin gets two thirds of the way there with no cost data at all.

The numbers in that figure come from twelve routes costing 22, 18, 15, 9, 8, 8, 6, 5, 5, 4, 3 and 3 seconds. Slicing that list contiguously into four groups of three puts the three heaviest routes in the first group: 55 seconds against a 26.5-second ideal. Dealing the same list round-robin — route 0 to shard 0, route 1 to shard 1, route 4 back to shard 0 — gives 35, 30, 24 and 17, because each shard receives one route from each cost tier. Round-robin is the strategy to reach for when there is no cost data, since it only assumes the list is roughly sorted by cost, and even an unsorted list gets shuffled across shards rather than clumped.

Configuration

The partition is computed once, by one job, and published as an artifact that every shard reads. That single decision removes every determinism problem at once: there is nothing left to recompute, so nothing can be recomputed differently.

// a11y/scripts/partition-routes.mjs
// Usage: node a11y/scripts/partition-routes.mjs 4 > a11y/shard-plan.json
import { createHash } from 'node:crypto';
import { readFileSync } from 'node:fs';

const shardTotal = Number(process.argv[2]);
if (!Number.isInteger(shardTotal) || shardTotal < 1) {
  throw new Error(`shard total must be a positive integer, got "${process.argv[2]}"`);
}

const manifest = JSON.parse(readFileSync('a11y/routes.json', 'utf8'));
// Optional weights, keyed by route. A missing entry falls back to 1, so a repo
// with no measurements still gets a deterministic, evenly-sized partition.
const weights = JSON.parse(readFileSync('a11y/route-weights.json', 'utf8'));

// Codepoint comparison, not localeCompare: localeCompare depends on the ICU
// data compiled into the Node build, so it can order ties differently on a
// runner than on a laptop and silently change the partition.
const byPath = (a, b) => (a < b ? -1 : a > b ? 1 : 0);
const ordered = [...new Set(manifest)] // duplicate routes collapse first
  .sort((a, b) => (weights[b] ?? 1) - (weights[a] ?? 1) || byPath(a, b));

const shards = Array.from({ length: shardTotal }, () => []);
ordered.forEach((route, position) => {
  // Round-robin over a heaviest-first list: shard 0 takes positions 0, n, 2n…
  // so every shard gets one route from each cost tier.
  shards[position % shardTotal].push(route);
});

const plan = {
  shardTotal,
  routeCount: ordered.length,
  shards: shards.map((routes, index) => ({
    index,
    routeCount: routes.length,
    weight: routes.reduce((sum, route) => sum + (weights[route] ?? 1), 0),
    routes,
  })),
};

// The hash covers the ordered list and the shard count. Any change to either
// produces a new hash, which lets the merge step reject a stale shard report.
plan.partitionHash = createHash('sha256')
  .update(JSON.stringify({ ordered, shardTotal }))
  .digest('hex')
  .slice(0, 16);

process.stdout.write(`${JSON.stringify(plan, null, 2)}\n`);

The spec reads its own slice out of that plan and writes a report after every route rather than only at the end, so a shard that crashes on route 40 still leaves evidence of the 39 routes it did scan.

// tests/a11y/sharded.spec.ts
import { mkdirSync, readFileSync, writeFileSync } from 'node:fs';
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

const SHARD_INDEX = Number(process.env.SHARD_INDEX ?? 0);
const plan = JSON.parse(readFileSync('a11y/shard-plan.json', 'utf8'));
const shard = plan.shards.find((entry: { index: number }) => entry.index === SHARD_INDEX);
if (!shard) throw new Error(`shard ${SHARD_INDEX} is not in shard-plan.json`);

const scanned: string[] = [];
const findings: unknown[] = [];

// Written after every route. A crashed worker still leaves complete: false,
// which is what stops a half-finished shard from reading as a clean pass.
function persist() {
  mkdirSync('a11y/out', { recursive: true });
  writeFileSync(`a11y/out/shard-${SHARD_INDEX}.json`, `${JSON.stringify({
    shardIndex: SHARD_INDEX,
    shardTotal: plan.shardTotal,
    partitionHash: plan.partitionHash, // echoed so the merge can verify it
    routesPlanned: shard.routes,
    routesScanned: scanned,
    complete: scanned.length === shard.routes.length,
    findings,
  }, null, 2)}\n`);
}

// Tests in this file share module state, so they must share one worker.
// Parallelism comes from shards, not from workers inside a shard.
test.describe.configure({ mode: 'serial' });

for (const route of shard.routes) {
  test(`axe ${route}`, async ({ page }) => {
    await page.goto(route);
    await page.getByRole('main').waitFor();
    const results = await new AxeBuilder({ page })
      .withTags(['wcag2a', 'wcag2aa', 'wcag21aa', 'wcag22aa'])
      .analyze();
    scanned.push(route);
    for (const violation of results.violations) findings.push({ route, ...violation });
    for (const item of results.incomplete) {
      findings.push({ route, incomplete: true, ...item });
    }
    persist();
    // Only critical fails the shard immediately; everything else is merged and
    // judged once, against the whole run, by the verdict job.
    expect(results.violations.filter((v) => v.impact === 'critical')).toEqual([]);
  });
}

The workflow reads the shard count from a repository variable so changing parallelism is a settings edit rather than a code change, and it passes the index through the environment.

name: a11y-route-shards
on:
  pull_request:
    branches: [main]
permissions:
  contents: read
jobs:
  partition:
    runs-on: ubuntu-24.04
    outputs:
      indexes: ${{ steps.split.outputs.indexes }}
      hash: ${{ steps.split.outputs.hash }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - id: split
        env:
          SHARD_TOTAL: ${{ vars.A11Y_SHARD_TOTAL || '4' }}
        run: |
          node a11y/scripts/partition-routes.mjs "$SHARD_TOTAL" > a11y/shard-plan.json
          node a11y/scripts/verify-partition.mjs a11y/shard-plan.json
          echo "indexes=$(jq -c '[.shards[].index]' a11y/shard-plan.json)" >> "$GITHUB_OUTPUT"
          echo "hash=$(jq -r '.partitionHash' a11y/shard-plan.json)" >> "$GITHUB_OUTPUT"
      - uses: actions/upload-artifact@v4
        with:
          name: a11y-partition
          path: a11y/shard-plan.json
          retention-days: 7
  axe-shard:
    needs: partition
    runs-on: ubuntu-24.04
    timeout-minutes: 12
    strategy:
      fail-fast: false # a failing shard must not cancel its siblings
      max-parallel: 4
      matrix:
        shard: ${{ fromJSON(needs.partition.outputs.indexes) }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - uses: actions/download-artifact@v4
        with:
          name: a11y-partition
          path: a11y/
      - run: npx playwright install --with-deps chromium
      - name: Scan the routes assigned to this shard
        env:
          SHARD_INDEX: ${{ matrix.shard }}
        run: npx playwright test tests/a11y/sharded.spec.ts
      - uses: actions/upload-artifact@v4
        if: always() # partial reports are the point of persisting per route
        with:
          name: a11y-shard-${{ matrix.shard }}-${{ needs.partition.outputs.hash }}
          path: a11y/out/shard-${{ matrix.shard }}.json
          retention-days: 7

If the route costs really are uniform, the whole partition file can be dropped in favour of Playwright’s built-in split. Note the base: --shard is one-based while a matrix array of [0,1,2,3] is zero-based, and mixing the two conventions scans shard 1 twice and never scans shard 4.

      - name: Alternative — let Playwright split the collected tests
        env:
          # strategy.job-total is the matrix size; --shard is 1-based, so the
          # matrix must be [1, 2, 3, 4] rather than [0, 1, 2, 3] here.
          SHARD: ${{ matrix.shard }}/${{ strategy.job-total }}
        run: npx playwright test tests/a11y --shard=$SHARD
One job computes the partition, every shard reads it routes.json feeds a partition job that sorts by weight, deals routes round-robin and hashes the result. The plan artifact carries that hash, and each of the four shard jobs downloads the same artifact instead of recomputing its own slice. One job computes the partition, every shard reads it routes.json 480 paths partition job weight sort, round-robin sha256 of the order shard-plan.json hash 9f2c41ab shard 0 · 120 rts shard 1 · 120 rts shard 2 · 120 rts shard 3 · 120 rts a retried shard downloads the same artifact, so it scans the same routes each report echoes the hash, so a stale artifact is detectable rather than merged
Nothing downstream of the partition job recomputes a slice, which is the whole determinism argument in one picture.

Validation

Verify the partition before any browser starts. Three properties matter: every route in the manifest lands in exactly one shard, no route lands in two, and the same inputs produce the same hash twice in a row.

// a11y/scripts/verify-partition.mjs
// Usage: node a11y/scripts/verify-partition.mjs a11y/shard-plan.json
import { readFileSync } from 'node:fs';

const plan = JSON.parse(readFileSync(process.argv[2], 'utf8'));
const manifest = [...new Set(JSON.parse(readFileSync('a11y/routes.json', 'utf8')))];
const assigned = plan.shards.flatMap((shard) => shard.routes);
const problems = [];

if (assigned.length !== new Set(assigned).size) {
  problems.push('at least one route is assigned to two shards');
}
const missing = manifest.filter((route) => !assigned.includes(route));
if (missing.length > 0) {
  problems.push(`${missing.length} route(s) in no shard, first: ${missing[0]}`);
}
if (plan.shards.length !== plan.shardTotal) {
  problems.push(`plan claims ${plan.shardTotal} shards but lists ${plan.shards.length}`);
}

const weights = plan.shards.map((shard) => shard.weight);
const mean = weights.reduce((a, b) => a + b, 0) / weights.length;
const spread = Math.max(...weights) - Math.min(...weights);
const drift = Math.round((spread / mean) * 100);
console.log(`${plan.shardTotal} shards, ${assigned.length} routes, hash ${plan.partitionHash}`);
console.log(`weight spread ${spread} (${drift}% of mean ${Math.round(mean)})`);
for (const problem of problems) console.error(`FAIL ${problem}`);
process.exit(problems.length > 0 ? 1 : 0);

Running it against a healthy plan, then generating the plan twice to confirm the hash is stable:

node a11y/scripts/verify-partition.mjs a11y/shard-plan.json
# 4 shards, 480 routes, hash 9f2c41ab77e0d3b5
# weight spread 13 (4% of mean 295)

node a11y/scripts/partition-routes.mjs 4 | jq -r .partitionHash
node a11y/scripts/partition-routes.mjs 4 | jq -r .partitionHash
# both print 9f2c41ab77e0d3b5 — the partition is a pure function of its inputs

A weight spread above roughly 15% of the mean is the signal that round-robin has run out of road and the plan should be built by cost-weighted packing instead, which is the planner described in the monorepo and parallel test sharding guide. Also assert the count the runner will actually execute, since a testIgnore pattern or a stray test.only can silently shrink a shard:

SHARD_INDEX=2 npx playwright test tests/a11y/sharded.spec.ts --list | tail -n 1
# Total: 120 tests in 1 file

Edge Cases and Conditional Guards

  • Authenticated routes. A shard that has to log in pays the login cost once per worker, not once per route, so generate storageState in a setup project and reuse it. Keep authenticated and anonymous routes in the same partition rather than splitting them into separate matrices; a route’s cost is dominated by rendering, not by which cookie jar it used, and two matrices mean two sets of shard artifacts for the merge to reconcile.
  • More shards than routes. When the shard count exceeds the route count, the trailing shards get empty route arrays. They must still run, still write a report with routesPlanned: [] and complete: true, and still upload an artifact — otherwise the merge sees fewer artifacts than the plan promised and correctly reports an incomplete run. Never filter empty shards out of the matrix.
  • Routes added between the partition job and the shard job. The plan artifact is authoritative for the run, so a route added by a concurrent merge to main is simply not scanned by this pull request. That is the correct behaviour for a gate on a merge base, but it must be visible: record plan.commit from GITHUB_SHA and print it in the step summary so nobody debugs a “missing” route that was never in the plan.

Pipeline Impact

fail-fast: false is the setting with the largest effect on how the gate feels to use. GitHub’s default cancels every in-progress matrix job the moment one job fails, so the first shard to hit a blocking violation destroys the other three reports. The pull request author fixes the two findings they can see, pushes, and discovers the next batch — one CI round trip per shard, on a job that takes seven minutes. With fail-fast: false all four shards run to completion, the merge produces one list, and the author fixes everything in one pass.

What fail-fast does to a sharded scan In the upper timeline shard 1 fails at three minutes ten seconds and the remaining shards are cancelled eight seconds later, yielding one report of four. In the lower timeline the same failure occurs but every shard runs to completion, yielding four reports and one merged verdict. What fail-fast does to a sharded scan fail-fast: true (the default) shard 0 shard 1 shard 2 shard 3 3 shards cancelled at 3:18 1 of 4 reports reaches the merge fail-fast: false shard 0 shard 1 shard 2 shard 3 0 2 min 4 min 6 min 8 min all four finish: 4 of 4 reports, every finding in one merged verdict
The failing shard costs the same time in both timelines; what differs is how much of the run's evidence survives it.

Sharding also changes what a re-run means. Re-running only failed jobs re-executes one shard against the same plan artifact, which is cheap and correct as long as the artifact is still within its retention window — seven days is comfortable, one day is not, because a re-run on Monday of a Friday pull request will find no plan and fail on the download step. Each shard also produces its own check run in the pull request’s checks list, so keep the job name short: axe-shard (2) reads well, accessibility-scan-matrix-execution (2) does not.

Artifact naming carries the partition hash in the examples above, which makes stale artifacts impossible to confuse with current ones when a pull request is re-run after a rebase. If the merge step globs a11y-shard-*, that suffix comes along for free and the merge can compare the hash in each report against the plan it was given. Everything the merge does with those files — deduplication, incomplete reconciliation and the missing-shard check — is covered in merging sharded accessibility reports into one artifact, and the surrounding job structure follows the conventions in GitHub Actions a11y pipeline setup.

Common Pitfalls

  • Slicing the route list contiguously after sorting it by cost, which concentrates every expensive route in shard 0 and leaves the last shard finishing in a fifth of the time.
  • Recomputing the partition inside each shard job, so a retried shard silently scans a different slice than the report it replaced.
  • Using localeCompare in the sort comparator, which orders equal-weight routes differently depending on the ICU data in the runner’s Node build.
  • Mixing Playwright’s one-based --shard=i/n with a zero-based matrix array, scanning one shard twice and skipping the last one entirely.
  • Leaving fail-fast at its default, so the first blocking violation cancels the sibling shards and the author only ever sees a quarter of the findings per push.
  • Dropping empty shards from the matrix when the shard count exceeds the route count, which makes the merge step’s expected-artifact count wrong for a reason nobody remembers a week later.

FAQ

Is round-robin ever worse than contiguous slicing? Only when the route list is deliberately ordered so that adjacent routes share expensive setup — a group of routes behind one login, or a set of pages that hit the same warmed cache. Round-robin scatters those across shards and each shard pays the setup again. In that case group the routes that share setup, treat each group as one unit, and deal the groups round-robin instead of the individual routes.

Should the shard count match the number of CPU cores on the runner? No — those are different axes. Cores decide how many Playwright workers one shard can run in parallel; the shard count decides how many runners the work is spread across. For browser-based accessibility scans, two workers per two-core runner is the practical ceiling before the browsers start competing for CPU and hydration waits get slower and flakier, so extra parallelism has to come from more shards rather than more workers.

How do I keep a route from being scanned twice when it belongs to two packages? Deduplicate the manifest before partitioning, which the example does with new Set(manifest) as its first step. Routes end up duplicated whenever the manifest is assembled by concatenating per-package route files and two packages both claim a shared path like /settings. Deduplicating in the partition job rather than in the merge keeps the cost accounting honest, because a route scanned twice inflates the plan’s total by a route nobody needs to scan again — the package-level view of this is in scanning only affected packages in a Turborepo monorepo.

Back to Monorepo Parallel Test Sharding