Structuring JSON Violation Output for Slack and GitHub Annotations

Slack wants at most fifty blocks, each a bounded chunk of markdown, addressed to a channel where nobody will scroll. The GitHub Checks API wants an annotations array where every entry names a file path and a line number, delivered fifty at a time, addressed to a diff. The same scan feeds both, and the two formats have almost nothing in common. This guide is part of Reporting, Dashboards & Violation Tracking, and it builds one presentation model with two formatters on top of it, so that the number in the channel and the number on the pull request are the same number.

Root Cause

The failure mode is not that either format is hard. It is that both are usually written as separate scripts, each parsing the scan output its own way, and they immediately disagree. One counts rules and the other counts nodes, so “12 violations” in Slack becomes “47 annotations” on the pull request. One sorts by rule id and the other by document order, so the critical failure is third in one and eleventh in the other. One truncates at ten and the other at fifty, so a reviewer who saw a rule in chat cannot find it on the diff. Every one of those is a trust problem rather than a formatting problem, and no amount of care inside either script fixes it, because the divergence is between them.

The second root cause is that each channel has a hard unit economy that the other does not share. A Slack message accepts a maximum of fifty blocks, a header block’s plain_text is capped at 150 characters, and a section block’s mrkdwn text is capped at 3,000 characters — so a payload with one block per failing node dies at the fortieth node of the first rule. A Checks API request accepts a maximum of fifty annotations, and each annotation requires path, start_line, end_line, annotation_level and message; a DOM scanner produces none of the first three, because a CSS selector is not a source location. More than fifty annotations means additional requests against the same check run, not a bigger array.

One model, two formatters Normalised axe findings enter a channel model that holds run metadata and rule groups with counts and ordering already decided. Two arrows lead from it to a Slack formatter constrained to fifty blocks and three thousand characters per section, and to a GitHub Checks formatter constrained to fifty annotations per request with a required file path, plus a step summary fallback. findings.ndjson one row per node channel model groups, counts, order one run link Slack formatter 50 blocks per message 3000 chars per section one block per rule GitHub Checks formatter 50 annotations per request path and start_line required step summary as fallback Totals and ordering are decided once, in the model, so the channels cannot disagree.
The formatters own layout and limits only; every number they print was computed in one place.

The third pressure is volume. A single color-contrast regression on a shared button component produces forty nodes across six routes, and forty of anything is the wrong thing to send to either channel. Chat needs the fact that one rule broke in forty places plus two examples; a diff needs a handful of annotations on the files somebody can actually edit. Both need a link back to the run, where the full artifact lives, instead of the report itself.

Configuration

The model is the contract. It reads the normalised findings produced in the parent guide, groups them by rule, sorts by impact then node count, keeps a couple of example selectors per rule, caps the number of rules and records how many were dropped. Nothing downstream is allowed to count anything.

// a11y/notify/model.mjs — the only place counts, ordering and caps are decided.
import { readFileSync } from 'node:fs';

const RANK = { critical: 4, serious: 3, moderate: 2, minor: 1 };

export function buildModel(ndjsonPath, { maxRules = 8, samplesPerRule = 2 } = {}) {
  const rows = readFileSync(ndjsonPath, 'utf8').split('\n').filter(Boolean)
    .map(JSON.parse).filter((r) => r.status === 'violation');

  const groups = new Map();
  for (const r of rows) {
    if (!groups.has(r.ruleId)) {
      groups.set(r.ruleId, { ruleId: r.ruleId, impact: r.impact, help: r.help,
        nodes: 0, routes: new Set(), samples: [] });
    }
    const g = groups.get(r.ruleId);
    g.nodes += 1;
    g.routes.add(r.route);
    if (g.samples.length < samplesPerRule) g.samples.push({ route: r.route, sel: r.target });
  }

  const ordered = [...groups.values()]
    .sort((a, b) => RANK[b.impact] - RANK[a.impact] || b.nodes - a.nodes);
  const run = rows[0] ?? {};
  const server = process.env.GITHUB_SERVER_URL ?? 'https://github.com';
  const repo = process.env.GITHUB_REPOSITORY ?? 'org/repo';

  return {
    totals: {
      rules: ordered.length,
      nodes: rows.length,
      critical: rows.filter((r) => r.impact === 'critical').length,
      serious: rows.filter((r) => r.impact === 'serious').length,
    },
    branch: run.branch ?? 'local',
    commit: (run.commit ?? 'local').slice(0, 7),
    // One deep link. The run page already hosts the full findings artifact.
    runUrl: `${server}/${repo}/actions/runs/${process.env.GITHUB_RUN_ID ?? '0'}`,
    shown: ordered.slice(0, maxRules),
    hidden: Math.max(0, ordered.length - maxRules),
  };
}

The Slack formatter turns each rule group into exactly one section block. That is the collapse that keeps the payload legal: forty nodes become one line of prose and two sample selectors, and the reader who needs the other thirty-eight follows the button.

Collapsing forty nodes into one block The left card shows one Slack section block per failing node: four visible rows, thirty-six more, and a rose banner explaining that the payload exceeds the fifty-block limit and is rejected. The right card shows a single block carrying the rule id, impact, node and route counts, two example selectors and a pointer to the run, with a green banner noting that it fits any message. Forty failing nodes of one rule, two ways to send them one block per node color-contrast #hero .cta color-contrast #nav a:nth-child(2) color-contrast .card .price color-contrast footer .legal 36 further blocks 42 blocks with header and context unreadable, and one rule from the cap one block per rule color-contrast — serious 40 nodes on 6 routes #hero .cta #nav a:nth-child(2) open the run for the rest 1 block, 1 count, 1 deep link room for seven more rules The count is the payload; the selectors are only examples of it. Send the shape of the problem to chat and keep the enumeration in the artifact.
Node-per-block payloads fail on the limit long before they fail on readability, and they were never readable.
// a11y/notify/slack.mjs — usage: node a11y/notify/slack.mjs findings.ndjson > slack.json
import { buildModel } from './model.mjs';

const HEADER_MAX = 150;    // header block plain_text limit
const SECTION_MAX = 3000;  // section block mrkdwn text limit
const BLOCK_MAX = 50;      // blocks accepted in one message payload

const clip = (s, max) => (s.length <= max ? s : s.slice(0, max - 1) + '…');
const DOT = { critical: ':red_circle:', serious: ':large_orange_circle:',
              moderate: ':large_yellow_circle:', minor: ':white_circle:' };

const m = buildModel(process.argv[2], { maxRules: 8, samplesPerRule: 2 });

const blocks = [
  { type: 'header',
    text: { type: 'plain_text',
      text: clip(`Accessibility: ${m.totals.nodes} nodes, ${m.totals.rules} rules`,
        HEADER_MAX) } },
  { type: 'context',
    elements: [{ type: 'mrkdwn',
      text: `\`${m.branch}\` at \`${m.commit}\` — ${m.totals.critical} critical, ` +
            `${m.totals.serious} serious` }] },
];

// One section per rule: forty nodes collapse to a count plus two examples.
for (const g of m.shown) {
  const lines = [
    `${DOT[g.impact]} *${g.ruleId}* — ${g.impact}`,
    `${g.nodes} node${g.nodes === 1 ? '' : 's'} on ${g.routes.size} route(s)`,
    ...g.samples.map((s) => `\`${s.route}\` \`${s.sel}\``),
  ];
  blocks.push({ type: 'section',
    text: { type: 'mrkdwn', text: clip(lines.join('\n'), SECTION_MAX) } });
}

if (m.hidden > 0) {
  blocks.push({ type: 'context',
    elements: [{ type: 'mrkdwn', text: `${m.hidden} further rule(s) not shown.` }] });
}
blocks.push({ type: 'actions',
  elements: [{ type: 'button', url: m.runUrl,
    text: { type: 'plain_text', text: 'Open the run' } }] });

if (blocks.length > BLOCK_MAX) throw new Error(`${blocks.length} blocks exceeds 50`);
// `text` is the notification and accessibility fallback, not a copy of the blocks.
process.stdout.write(JSON.stringify({
  text: `Accessibility: ${m.totals.nodes} failing nodes on ${m.branch}`,
  blocks,
}, null, 2));

The GitHub side has to invent the one thing it is required to supply. path must name a file that exists in the head commit, and start_line must be an integer, but a scanner only knows a route and a selector. Fabricating a plausible line number is worse than useless — a reviewer chases a line that has nothing to do with the defect — so map the route to the file that owns it and anchor at line 1, putting the selector in the message where it belongs. A route-owner manifest is cheap and the application usually already has one.

{
  "/": "app/routes/home.tsx",
  "/pricing": "app/routes/pricing.tsx",
  "/reports/quarterly": "app/routes/reports.quarterly.tsx",
  "/settings/notifications": "app/routes/settings.notifications.tsx"
}
// a11y/notify/github.mjs — usage: node a11y/notify/github.mjs findings.ndjson
// Writes one complete Checks API request body per 50-annotation batch.
import { readFileSync, writeFileSync, mkdirSync } from 'node:fs';
import { buildModel } from './model.mjs';

const BATCH = 50;       // annotations accepted per create or update request
const TITLE_MAX = 255;  // annotation title limit
const LEVEL = { critical: 'failure', serious: 'failure',
                moderate: 'warning', minor: 'notice' };

const owners = JSON.parse(readFileSync('a11y/notify/route-owners.json', 'utf8'));
// A wide cap here: the batching, not the model, decides how many are sent.
const m = buildModel(process.argv[2], { maxRules: 200, samplesPerRule: 3 });

const annotations = [];
for (const g of m.shown) {
  for (const s of g.samples) {
    const path = owners[s.route];
    if (!path) continue;          // no source owner: the step summary carries it
    annotations.push({
      path,                        // must exist in the head commit or it is dropped
      start_line: 1,               // required; a DOM defect has no source line
      end_line: 1,
      annotation_level: LEVEL[g.impact],
      title: `${g.ruleId} (${g.impact}) on ${s.route}`.slice(0, TITLE_MAX),
      message: `${g.help}\nSelector: ${s.sel}\n${g.nodes} node(s) of this rule.`,
      raw_details: `route ${s.route}\nrule ${g.ruleId}\nrun ${m.runUrl}`,
    });
  }
}

const batches = [];
for (let i = 0; i < annotations.length; i += BATCH) {
  batches.push(annotations.slice(i, i + BATCH));
}
if (batches.length === 0) batches.push([]);   // still publish a green check run

mkdirSync('checks', { recursive: true });
const blocking = m.totals.critical + m.totals.serious;
batches.forEach((batch, i) => {
  const body = { output: {
    title: `Accessibility: ${m.totals.nodes} failing nodes`,
    summary: `${m.totals.rules} rule(s) failed. [Open the run](${m.runUrl})`,
    annotations: batch,
  } };
  if (i === 0) {                              // the first request creates the run
    body.name = 'accessibility';
    body.head_sha = process.env.GITHUB_SHA ?? 'HEAD';
  }
  // Only the last request completes it; earlier ones leave it in progress.
  if (i === batches.length - 1) body.conclusion = blocking > 0 ? 'failure' : 'neutral';
  else body.status = 'in_progress';
  writeFileSync(`checks/batch-${i}.json`, JSON.stringify(body, null, 2));
});
console.log(`${annotations.length} annotation(s) in ${batches.length} batch(es)`);

Annotations are never the whole story, because any route without a source owner is silently skipped and any run without checks: write cannot publish at all. The markdown fallback is the same model rendered as a table into $GITHUB_STEP_SUMMARY, which needs no permissions, survives on the run page, and holds up to 1 MiB per step.

// a11y/notify/summary.mjs
// usage: node a11y/notify/summary.mjs findings.ndjson >> "$GITHUB_STEP_SUMMARY"
import { buildModel } from './model.mjs';

const m = buildModel(process.argv[2], { maxRules: 25, samplesPerRule: 1 });
// A bare pipe splits a table cell even inside backticks.
const cell = (s) => '`' + String(s).replaceAll('|', '\\|') + '`';

const out = [
  `### Accessibility: ${m.totals.nodes} failing nodes across ${m.totals.rules} rules`,
  '',
  `Branch \`${m.branch}\` at \`${m.commit}\` — [open the run](${m.runUrl}).`,
  '',
  '| Impact | Rule | Nodes | Routes | Example |',
  '|---|---|---|---|---|',
  ...m.shown.map((g) => `| ${g.impact} | ${cell(g.ruleId)} | ${g.nodes} | ` +
    `${g.routes.size} | ${cell(g.samples[0]?.sel ?? 'none')} |`),
];
if (m.hidden > 0) out.push('', `${m.hidden} further rule(s) in the artifact.`);
process.stdout.write(out.join('\n') + '\n');
Limits and reach of the three output channels Three rows compare a Slack message, a Checks API annotation batch and the GitHub step summary across four columns: maximum units per payload, text budget per unit, whether a real file path is required, and where the output remains visible after the run finishes. What each channel will actually accept max units per payload text budget per unit needs a real file path visible after the run Slack message 50 blocks 3000 chars no in the channel Checks annotations 50 per request 64 KB message yes, required on the diff Step summary 1 MiB per step markdown, free no run page only Only annotations reach the diff, and only they need a path that exists in the tree. The step summary is the fallback that needs no token, no webhook and no manifest.
Three channels, three unit economies — which is why the caps live in the formatters and the arithmetic does not.

The workflow wires all three from one findings file. Note that the Slack post swallows its own failure: a webhook outage must never turn a green build red.

- name: Build every channel payload from one model
  if: always()
  run: |
    node a11y/notify/slack.mjs findings.ndjson > slack.json
    node a11y/notify/github.mjs findings.ndjson
    node a11y/notify/summary.mjs findings.ndjson >> "$GITHUB_STEP_SUMMARY"
- name: Post to Slack
  if: always()
  env:
    SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
  run: |
    curl -sS --fail-with-body -X POST -H 'Content-Type: application/json' \
      --data @slack.json "$SLACK_WEBHOOK_URL" \
      || echo 'Slack delivery failed; continuing'   # reporting never blocks
- name: Publish the check run
  if: always()
  env:
    GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}   # needs permissions: checks: write
  run: |
    id=$(gh api --method POST "/repos/$GITHUB_REPOSITORY/check-runs" \
      --input checks/batch-0.json --jq '.id')
    for batch in checks/batch-[1-9]*.json; do
      [ -f "$batch" ] || continue           # the glob stays literal when unmatched
      gh api --method PATCH "/repos/$GITHUB_REPOSITORY/check-runs/$id" \
        --input "$batch" > /dev/null        # each PATCH appends up to 50 more
    done

Validation

Both payloads are checkable offline, and both checks belong in the repository’s own test run rather than in the workflow that sends them. The Slack assertion is a block count and a per-section length; the GitHub assertion is a batch size plus the requirement that every annotated path exists, because an annotation pointing at a file absent from the head commit is discarded without an error.

node a11y/notify/slack.mjs findings.ndjson > slack.json
node a11y/notify/github.mjs findings.ndjson

# 1. Slack: legal block count and no oversized section text.
node -e "
const p = require('./slack.json');
if (p.blocks.length > 50) throw new Error('blocks: ' + p.blocks.length);
if (!p.text) throw new Error('missing notification fallback text');
for (const b of p.blocks) {
  const t = (b.text && b.text.text) || '';
  if (t.length > 3000) throw new Error('over 3000 chars in a ' + b.type);
}
console.log('slack ok:', p.blocks.length, 'blocks');
"

# 2. Checks: batch sizes, required fields, and paths that exist in the tree.
node -e "
const { existsSync, readdirSync } = require('node:fs');
let total = 0;
for (const f of readdirSync('checks')) {
  const a = require('./checks/' + f).output.annotations;
  if (a.length > 50) throw new Error(f + ' has ' + a.length + ' annotations');
  for (const x of a) {
    if (!x.path || !x.start_line) throw new Error('missing path or start_line');
    if (!existsSync(x.path)) throw new Error('path not in tree: ' + x.path);
  }
  total += a.length;
}
console.log('checks ok:', total, 'annotations');
"

# 3. The two channels must agree on the headline number.
diff <(node -e "console.log(require('./slack.json').text.match(/\d+/)[0])") \
     <(node -e "console.log(require('./checks/batch-0.json').output.title.match(/\d+/)[0])")

A healthy run prints three lines and the diff produces no output, which is the whole point of the exercise:

slack ok: 12 blocks
checks ok: 18 annotations

Edge Cases and Conditional Guards

  • Zero violations. The model yields an empty shown array, Slack gets a header and a button, and github.mjs still writes batch-0.json so the check run turns green rather than staying pending forever. Guard the Slack post itself behind a condition if the channel does not want a green heartbeat; leave the check run unconditional, because a missing check run blocks a required status check.
  • Selectors containing backticks or pipes. A selector from a shadow tree or a generated class can carry either. Slack mrkdwn breaks on an unmatched backtick and a markdown table breaks on a bare pipe, which is why the summary formatter escapes pipes and the Slack formatter clips rather than trusting length. Strip backticks from the selector in the model if the design system generates them.
  • Forked pull requests. secrets.GITHUB_TOKEN is read-only on a fork, so the Checks API call fails and the Slack webhook secret is not exposed at all. Fall back to the step summary — which always works — and detect the case with github.event.pull_request.head.repo.fork rather than discovering it from a 403 in the logs.

Pipeline Impact

None of these three steps sets the job status. They run with if: always() so a failed gate is still reported, and the blocking decision stays where it was made, in the assertion or the dedicated gate step described in auto-fail vs warning workflows. The one subtlety is the check run’s own conclusion: publishing failure creates a second signal on the pull request that branch protection can be configured to require, which is useful but easy to double-count. Pick one — either the scan job is the required check or the accessibility check run is — and set the other to neutral. Upload slack.json and the checks/ directory as artifacts; when somebody asks why a rule did not appear in chat, the payload that was actually sent answers it in seconds. If the team prefers a threaded pull-request comment to a check run, the comment-based variant is covered in annotating pull requests with axe-core violation comments.

Common Pitfalls

  • Writing two independent scripts, so the count in chat and the count on the diff diverge and both lose credibility.
  • Emitting one Slack block per failing node, which exceeds the fifty-block limit on the first widespread rule and is unreadable well before that.
  • Posting all 400 annotations in one request and assuming the extras are queued; anything past fifty in a single call is rejected, not deferred.
  • Fabricating a start_line that looks like a real source location, sending reviewers to a line that has nothing to do with the defect.
  • Annotating a path that does not exist in the head commit, which GitHub drops silently so the check run looks empty and correct.
  • Pasting the whole report into the message instead of one deep link to the run, guaranteeing the channel mutes the integration within a week.

FAQ

Why one block per rule rather than one per route? Because a rule is the unit of work and a route is not. A single design-token fix clears color-contrast on every route at once, so grouping by rule shows one actionable item where grouping by route shows six copies of it. Route counts still matter, which is why each block carries the number of affected routes as a measure of blast radius rather than as a separate block.

Can the same model drive Microsoft Teams or a GitLab pipeline? Yes, and that is the reason the model exists. Teams uses Adaptive Cards rather than Block Kit, so it needs a third formatter with its own limits, but it consumes the identical shown, totals and runUrl fields and therefore reports identical numbers. The same is true of a GitLab code-quality report or a Jenkins summary: new formatter, no new arithmetic, and the criterion-level export described in exporting accessibility results to compliance dashboards reads the same source rows.

How many rules should actually be shown in chat? Eight is a good default because it fits comfortably inside the block limit with room for a header, a context line and a button, and because a message longer than one screen is a message nobody reads. What matters more than the number is that the cap is visible: always print how many rules were withheld, or a reader will assume the list is complete and treat a truncated message as the full picture.

Back to Reporting Dashboards Violation Tracking