Annotating Pull Requests With axe-core Violation Comments

A failing accessibility check that only says “exit 1” forces reviewers to dig through logs. Far better is to surface each axe-core violation exactly where it lives — as an inline annotation on the changed line and a single deduplicated summary comment. This guide is part of GitHub Actions a11y Pipeline Setup, and it shows how to emit ::error file= workflow commands for inline annotations and post a deduplicated review comment via the GitHub API, so repeated runs update one comment instead of spamming the thread.

Key implementation targets:

  • ::error file= workflow commands place annotations on the exact source line.
  • A single bot comment is upserted per PR, so re-runs edit rather than duplicate.
  • Violations map back to source via the axe node’s selector and a source lookup.
axe results to poster script to annotations and one upserted comment axe JSON feeds a Node poster script that emits workflow command annotations and upserts a single PR comment via the API, keyed by a marker. axe results violations.json poster.mjs map + dedupe ::error file= annotations inline on changed line one upserted comment keyed by marker
One poster script emits inline annotations and upserts a single summary comment keyed by a hidden marker.

Root Cause / Context

axe-core reports violations against DOM nodes by CSS selector, not against source files and line numbers. GitHub’s annotation system and review comments, by contrast, are anchored to files and lines in the diff. Bridging the two requires a mapping step: take the failing node’s selector, resolve it to a component file, and emit an annotation for that file. Where a precise line is unavailable, a file-level annotation plus a summary comment still puts the violation in front of the reviewer without log spelunking.

The second concern is noise. A naive script posts a fresh comment on every push, burying the conversation. The fix is an upsert keyed by a hidden HTML marker: on each run the script searches existing comments for the marker, then edits that comment if found or creates it once. The same idea — a stable key — drives annotation dedupe so the same violation does not stack across reruns.

Configuration

Run the scan and write JSON, then run a poster script. The workflow needs pull-requests: write so the token can comment:

# .github/workflows/a11y-annotate.yml
name: a11y-annotate
on: [pull_request]
permissions:
  contents: read
  pull-requests: write   # required for the API to post review comments
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20, cache: npm }
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      # Produce violations.json; '|| true' lets the poster run before we gate.
      - run: npx playwright test tests/a11y.spec.ts || true
      - run: node scripts/poster.mjs   # annotate + upsert one comment, then exit code
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
          REPO: ${{ github.repository }}

Configure the Playwright reporter to emit an axe-shaped JSON file the poster can read. A small custom reporter collects violations across specs:

// tests/a11y.spec.ts (excerpt) — write each violation set to disk
import { test, expect } from '@playwright/test';
import { AxeBuilder } from '@axe-core/playwright';
import { appendFileSync } from 'node:fs';

test('a11y home', async ({ page }, testInfo) => {
  await page.goto('/');
  const { violations } = await new AxeBuilder({ page })
    .withTags(['wcag2a', 'wcag2aa', 'wcag22aa']).analyze();
  for (const v of violations) {
    for (const node of v.nodes) {
      // One JSON line per failing node: id, impact, selector, help text.
      appendFileSync('violations.jsonl', JSON.stringify({
        id: v.id, impact: v.impact, selector: node.target.join(' '),
        help: v.help, html: node.html,
      }) + '\n');
    }
  }
  expect(violations.filter(v => ['serious','critical'].includes(v.impact ?? '')))
    .toEqual([]);
});

The poster reads the lines, emits one ::error annotation per unique violation, then upserts a single summary comment. The hidden marker makes the comment idempotent:

// scripts/poster.mjs
import { readFileSync, existsSync } from 'node:fs';

const MARKER = '<!-- a11y-axe-report -->'; // hidden key for comment upsert
const [owner, repo] = process.env.REPO.split('/');
const pr = process.env.PR_NUMBER;
const token = process.env.GITHUB_TOKEN;
const api = 'https://api.github.com';

const auth = {
  Authorization: `Bearer ${token}`,
  Accept: 'application/vnd.github+json',
};

// 1. Load + dedupe violations by id+selector.
const lines = existsSync('violations.jsonl')
  ? readFileSync('violations.jsonl', 'utf8').trim().split('\n').filter(Boolean)
  : [];
const seen = new Set();
const violations = [];
for (const line of lines) {
  const v = JSON.parse(line);
  const key = `${v.id}::${v.selector}`; // dedupe across reruns/specs
  if (seen.has(key)) continue;
  seen.add(key);
  violations.push(v);
}

// 2. Emit inline annotations (GitHub renders these on the Files tab).
for (const v of violations) {
  // Map selector -> file heuristically; fall back to a workflow-level annotation.
  const file = selectorToFile(v.selector);
  const msg = `${v.id} (${v.impact}): ${v.help} — selector ${v.selector}`;
  if (file) {
    console.log(`::error file=${file}::${escape(msg)}`);
  } else {
    console.log(`::error::${escape(msg)}`); // job-level annotation
  }
}

// 3. Build and upsert the single summary comment.
const body = violations.length
  ? `${MARKER}\n### axe-core found ${violations.length} violation(s)\n\n` +
    violations.map(v => `- **${v.id}** (${v.impact}) \`${v.selector}\` — ${v.help}`).join('\n')
  : `${MARKER}\n### axe-core: no gating violations`;

const existing = await (await fetch(
  `${api}/repos/${owner}/${repo}/issues/${pr}/comments?per_page=100`,
  { headers: auth },
)).json();
const mine = existing.find(c => c.body?.includes(MARKER)); // find prior bot comment

if (mine) {
  await fetch(`${api}/repos/${owner}/${repo}/issues/comments/${mine.id}`, {
    method: 'PATCH', headers: auth, body: JSON.stringify({ body }), // edit in place
  });
} else {
  await fetch(`${api}/repos/${owner}/${repo}/issues/${pr}/comments`, {
    method: 'POST', headers: auth, body: JSON.stringify({ body }),
  });
}

// 4. Gate: non-zero exit if any serious/critical violation remains.
const gating = violations.filter(v => ['serious', 'critical'].includes(v.impact));
if (gating.length) process.exit(1);

function escape(s) { return s.replace(/\r?\n/g, '%0A'); } // newlines in annotations
function selectorToFile(sel) {
  // Project-specific: derive a path from a data-source attr if present.
  const m = sel.match(/\[data-src="([^"]+)"\]/);
  return m ? m[1] : null;
}

Validation

Open a PR that introduces a known violation — for example an <img> without alt. After the workflow runs, the Files tab shows a red annotation on the affected file and the conversation shows exactly one bot comment. Push a second commit and confirm the comment count stays at one:

# Inspect the rendered annotations from the run log:
gh run view --log | grep '::error'
# ::error file=src/components/Hero.tsx::image-alt (critical): Images must have...

# Confirm only one bot comment exists after multiple pushes:
gh pr view --json comments --jq '[.comments[].body | select(contains("a11y-axe-report"))] | length'
# 1

A clean PR shows the same single comment updated to “no gating violations,” so reviewers always see a current status rather than a stale failure.

Edge Cases & Conditional Guards

  • Selector-to-file mapping is heuristic: when a selector cannot be resolved to a source path, emit a job-level ::error (no file=) so the violation still surfaces rather than being dropped.
  • Fork PRs: the default GITHUB_TOKEN is read-only on forked pull requests, so the API comment step will 403. Detect pull_request_target or post annotations only, which still render from forks.
  • Annotation cap: GitHub limits annotations per run (commonly 10 errors shown). Always keep the summary comment as the complete list so nothing is hidden by the cap.

Pipeline Impact

The poster owns the exit code: it emits annotations and upserts the comment first, then exits non-zero if any serious or critical violation remains, so the gate and the reporting happen in one step. Running the scan with || true ensures reporting is not skipped when the scan itself fails an assertion. Artifacts are optional here because the comment and annotations carry the detail, which keeps the PR self-contained and avoids forcing reviewers into the Actions log.

Common Pitfalls

  • Posting a new comment every run instead of upserting by marker, which buries the PR thread.
  • Forgetting pull-requests: write, so the API call returns 403 and the step fails opaquely.
  • Letting the scan’s own non-zero exit short-circuit the job before the poster runs — gate inside the poster instead.
  • Not escaping newlines in ::error messages, which truncates multi-line annotations.
  • Relying on annotations alone and hitting the per-run cap, hiding violations beyond the limit.

FAQ

Should I use review comments on diff lines or issue comments? Issue (conversation) comments are simpler and always available, which is why the summary uses one. True line-anchored review comments require the file path and the diff position, which you only have for changed lines; use them when your selector-to-file mapping is reliable, and keep the summary comment as the catch-all.

How does dedupe survive across multiple workflow runs? Inline annotation dedupe is per-run, handled by the id::selector key. Cross-run dedupe of the comment is handled by the hidden marker: each run finds and edits the same comment, so the thread never accumulates duplicates regardless of how many times CI runs.

Can I post annotations without the API at all? Yes. The ::error file= workflow commands need no token — they are printed to stdout and GitHub renders them. The API token is only required for the persistent summary comment. On fork PRs where the token is read-only, fall back to annotation-only mode.

Back to Github Actions a11y Pipeline Setup