Visualizing WCAG Compliance Trends with Grafana

A time series of violation counts is only useful if someone looks at it. Grafana turns the persisted series into a live dashboard the whole team sees: WCAG 2.2 AA compliance trending over weeks, with alert thresholds that fire when serious violations climb. This page is part of Reporting, Dashboards & Violation Tracking, and it covers two wiring options — pushing metrics to a Prometheus Pushgateway, or pointing Grafana straight at the SQL store — plus the panel queries and alert rules.

  • Export per-commit severity counts to a Prometheus Pushgateway from CI
  • Or query the SQL trend store directly with a Grafana SQL datasource
  • Build panels for total violations and per-severity breakdown over time
  • Set an alert threshold that fires when serious-or-worse counts rise
Grafana wiring A CI exporter pushes counts to a Pushgateway; Prometheus scrapes it; Grafana queries Prometheus and a SQL store to render panels and fire alerts. CI exporter severity counts Pushgateway push target SQL store trend table Prometheus Grafana panels + alerts
Either path — Pushgateway/Prometheus or direct SQL — feeds Grafana panels and alerts.

Root Cause / Context

CI jobs are short-lived, so Prometheus cannot scrape them directly — by the time Prometheus polls, the runner is gone. The Pushgateway exists exactly for this batch-job case: the job pushes its final metrics to the gateway, which holds them for Prometheus to scrape on its own schedule. This is the infrastructure “cluster” sense of Prometheus and Grafana — a small monitoring stack you run alongside CI, not part of the application.

The alternative avoids Prometheus entirely: point a Grafana SQL datasource at the same trend.db/Postgres written in tracking accessibility violation trends across sprints. SQL is simpler if you already persist there; Pushgateway is better if you already run Prometheus for other services. The compliance figures you chart should match the progressive thresholds your gate enforces, so the dashboard and the build agree on “compliant.”

Configuration

The exporter formats counts in the Prometheus text exposition format and POSTs them to the Pushgateway. Labels carry severity and branch so Grafana can break the series down by bucket.

// grafana-export.js — push severity counts to a Prometheus Pushgateway
const http = require('http');
const findings = require('./normalized.json');

const counts = { critical: 0, serious: 0, moderate: 0, minor: 0 };
for (const f of findings) counts[f.impact] = (counts[f.impact] || 0) + 1;

const branch = process.env.GITHUB_REF_NAME || 'local';
// One gauge per severity; HELP/TYPE lines are required by the exposition format
const body =
  '# HELP a11y_violations Accessibility violations by severity\n' +
  '# TYPE a11y_violations gauge\n' +
  Object.entries(counts)
    .map(([impact, n]) => `a11y_violations{severity="${impact}",branch="${branch}"} ${n}`)
    .join('\n') + '\n';

const gateway = new URL(process.env.PUSHGATEWAY_URL || 'http://localhost:9091');
// Grouping key job/a11y means each push replaces the prior values for that job
const req = http.request(
  `${gateway.origin}/metrics/job/a11y/branch/${branch}`,
  { method: 'POST', headers: { 'Content-Type': 'text/plain' } }
);
req.on('error', e => { console.error('push failed:', e.message); process.exit(0); }); // never block CI
req.write(body);
req.end(() => console.log(`Pushed ${Object.values(counts).reduce((a, b) => a + b, 0)} violations`));

Run it after the scan. The push failure handler exits 0 so a monitoring outage never fails a build.

- name: Push a11y metrics
  if: always()
  env:
    PUSHGATEWAY_URL: ${{ secrets.PUSHGATEWAY_URL }}
  run: node grafana-export.js

In Grafana, add a panel with a Prometheus datasource. This query trends the per-severity gauge; a second panel can show only serious-or-worse for the compliance headline.

# Panel A — all severities over time (Prometheus / PromQL)
a11y_violations{branch="main"}

# Panel B — serious-or-worse headline for WCAG 2.2 AA compliance
sum(a11y_violations{branch="main", severity=~"critical|serious"})

If you skip Prometheus, a Grafana SQL datasource panel reads the trend table directly with a time-bucketed query.

-- Grafana SQL panel — daily serious-or-worse count for the compliance trend
SELECT ts AS "time",
       critical + serious AS "serious_or_worse",
       total AS "all_violations"
FROM trend
WHERE $__timeFilter(ts)          -- Grafana injects the dashboard time range
ORDER BY ts;

Validation

Confirm the Pushgateway holds the metric before trusting the panel. Scrape it and check the gauge is present and non-negative.

# The gateway exposes pushed series at /metrics; the gauge must appear
curl -sS "$PUSHGATEWAY_URL/metrics" | grep '^a11y_violations'
# Expected, e.g.:
# a11y_violations{branch="main",severity="serious"} 3

For the alert, define a Grafana rule that fires when serious-or-worse violations exceed your threshold so a regression pages the team rather than waiting for someone to open the dashboard.

# Grafana alert condition (PromQL) — fire when serious+critical exceed budget
sum(a11y_violations{branch="main", severity=~"critical|serious"}) > 0
# Set "for: 5m" so a single transient scrape does not trigger a false alert.

Edge Cases & Conditional Guards

  • Stale Pushgateway values: the gateway holds the last push indefinitely; if CI stops running, the panel shows a flat line, not zero. Add a push_time_seconds freshness panel so a stalled pipeline is visible.
  • Branch cardinality: pushing a label per feature branch explodes Prometheus cardinality; restrict the pushed branch label to long-lived branches and drop ephemeral ones.
  • Datasource time column: Grafana SQL panels require an aliased time column; an ISO-8601 ts works for SQLite but Postgres may need an explicit cast to timestamp.

Pipeline Impact

The push step uses if: always() and swallows network errors, so neither a gate failure nor a Pushgateway outage affects the build’s exit code. No artifacts are produced by this step — the durable record lives in Prometheus or the SQL store. Keep the alert threshold aligned with your gate policy so the dashboard’s notion of “regression” matches what actually blocks a PR.

Common Pitfalls

  • Letting a Pushgateway timeout fail the build; always wrap the push and exit 0 on error.
  • Pushing without HELP/TYPE lines, which some Pushgateway versions reject, silently dropping the metric.
  • Charting a stale gauge as current state because the gateway never expires values — add freshness monitoring.
  • Using high-cardinality labels (per-commit, per-PR) on a gauge, which degrades Prometheus and clutters the panel legend.

FAQ

Pushgateway or direct SQL datasource — which is right? If you already run Prometheus, push to a Pushgateway and reuse your existing alerting. If you do not, point Grafana’s SQL datasource at the trend store you already persist; it is one fewer moving part and the panel queries are plain SQL.

Why does my Grafana panel show a flat line after CI stops? The Pushgateway retains the last pushed value forever, so the gauge stays at its final reading. Add a panel on push_time_seconds to detect staleness, and treat a flat line plus an old push time as “no recent data,” not “zero violations.”

How do I express WCAG 2.2 AA compliance as a single number? Sum the critical and serious gauges — those are the buckets that map to AA-blocking failures — and chart that as the headline. A value of zero on the default branch is your AA-clean signal; the per-severity panel shows where remaining debt sits.

Back to Reporting Dashboards Violation Tracking