Running lhci autorun Inside a Container

This guide is part of Docker-Based Pipeline Execution, and it is about one command in one environment: lhci autorun executing inside a container on a shared CI runner. Lighthouse behaves differently there than it does on a laptop in three specific ways — it cannot launch Chrome without help, its default throttling turns runner CPU contention into score variance, and everything it writes disappears when the container is removed. Each has a precise fix, and none of them is “retry the job”.

Root Cause

Chrome will not start in a default container job. Its multi-process sandbox needs kernel facilities the runtime does not necessarily grant, /dev/shm is a 64 MB tmpfs that a real page exhausts, and there is no GPU. lhci does not pass any of those flags for you; it forwards what you put in ci.collect.settings.chromeFlags, and if that is empty you get No usable sandbox or, worse, an intermittent renderer crash reported as a Lighthouse run failure. The browser also has to be pinned, because the accessibility category is computed from a live render: contrast comes from composited colours and accessible names come from Chrome’s own name computation, so an unpinned browser makes the number drift on its own schedule.

The throttling problem is subtler and it is the one that gets accessibility gates disabled. Lighthouse’s default throttlingMethod is simulate, which applies a 4× CPU slowdown multiplier while it gathers, then models network latency afterwards. On a two- or four-vCPU runner that is already executing other jobs, that artificial 4× compounds with real contention. Accessibility audits do not measure time, so it is tempting to assume they are immune — but they do not run against the live page. They run against the DOM and accessibility-tree artifacts Lighthouse captured at the end of the gathering pass. When the page has not finished rendering by then, that snapshot contains fewer nodes: a table that had not hydrated, a dialog that had not mounted, a list rendered empty. Audits such as color-contrast, label and aria-allowed-attr therefore see a different node set, and audits with nothing applicable to test drop to notApplicable and leave the weighted average entirely. The score moves because the denominator moved.

The third issue is trivial and catches everyone once. lhci writes its collected runs to .lighthouseci/ inside the working directory, and docker run --rm deletes that directory along with the container the moment the process exits. A failing assertion then produces an exit code with no report to explain it, which is the least useful possible outcome of a gate.

Accessibility score spread across six runs Six paired columns plot the accessibility category score for the same commit. Under simulated throttling with a four times CPU slowdown on a contended runner the scores are 0.93, 0.86, 0.93, 0.89, 0.93 and 0.82. With throttlingMethod set to provided the same commit scores 0.93, 0.93, 0.93, 0.91, 0.93 and 0.93, a spread of two points instead of eleven. One commit, six runs, two throttling settings 1.00 0.95 0.90 0.85 0.80 run 1 run 2 run 3 run 4 run 5 run 6 simulate, 4x CPU slowdown, contended runner throttlingMethod: provided Median of six is 0.91 against 0.93; it is the spread, not the median, that trips a 0.92 gate.
The audits did not change between the two series — what changed is how much of the page had rendered by the time Lighthouse captured its artifacts.

Configuration

Pin Chrome by version number rather than letting lhci find something on PATH, and tell it exactly where that binary is with CHROME_PATH. Installing a specific Chrome for Testing build gives a version string you can read in a bug report, which matters because the accessibility score is partly a property of the browser. The library list below is what that build links against on a -slim Debian base.

# syntax=docker/dockerfile:1.7
# Dockerfile.lhci — pinned Chrome for Testing plus @lhci/cli, nothing else.
FROM node:20.18.1-bookworm-slim AS lhci

ARG CHROME_VERSION=130.0.6723.31
ENV DEBIAN_FRONTEND=noninteractive \
    LANG=en_GB.UTF-8 \
    TZ=UTC

RUN apt-get update && apt-get install -y --no-install-recommends \
      ca-certificates fontconfig fonts-liberation2 fonts-noto-core \
      libasound2 libatk-bridge2.0-0 libatk1.0-0 libcairo2 libcups2 libdbus-1-3 \
      libdrm2 libgbm1 libnspr4 libnss3 libpango-1.0-0 libx11-6 libxcb1 \
      libxcomposite1 libxdamage1 libxext6 libxfixes3 libxkbcommon0 libxrandr2 \
 && fc-cache -f \
 && rm -rf /var/lib/apt/lists/*

# Exact build, its own layer, no floating tag anywhere in the chain.
RUN npx --yes @puppeteer/browsers install "chrome@${CHROME_VERSION}" --path /chrome
ENV CHROME_PATH=/chrome/chrome/linux-${CHROME_VERSION}/chrome-linux64/chrome

WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci --omit=dev  # includes @lhci/cli

# Build the static output that lhci will serve itself. No dev server anywhere.
COPY . .
RUN npm run build

RUN useradd --create-home --uid 10001 lhci \
 && mkdir -p /out && chown lhci:lhci /out /app
USER lhci
CMD ["npx", "lhci", "autorun"]

The configuration file carries the three container fixes. chromeFlags scopes the sandbox and shared-memory workarounds to the audited Chrome process rather than weakening the container; throttlingMethod: 'provided' removes the artificial CPU slowdown that turns runner contention into score variance; and outputDir under the mounted path is what lets the report outlive the container.

// lighthouserc.cjs — .cjs so it loads even when package.json says "type":"module".
module.exports = {
  ci: {
    collect: {
      // lhci serves ./dist itself and audits every HTML file it finds there.
      staticDistDir: './dist',
      // Five runs so a single degraded gather cannot decide the gate.
      numberOfRuns: 5,
      settings: {
        onlyCategories: ['accessibility'], // no perf traces: faster and stabler
        // Flags for the audited Chrome only; the container stays unmodified.
        chromeFlags: [
          '--headless=new',
          '--no-sandbox',              // see the note below before keeping this
          '--disable-dev-shm-usage',   // /dev/shm is 64 MB unless you raise it
          '--disable-gpu',
          '--font-render-hinting=none',
        ].join(' '),
        // No artificial 4x CPU slowdown: the snapshot is taken after the page
        // has actually rendered, even on a contended runner.
        throttlingMethod: 'provided',
        maxWaitForLoad: 45000,         // headroom for a slow shared runner
        locale: 'en-GB',               // fixes the names of native controls
      },
    },
    assert: {
      // Assert against the median run explicitly rather than the default.
      aggregationMethod: 'median-run',
      assertions: {
        'categories:accessibility': ['error', { minScore: 0.95 }],
        // Individual audits pinned to error: a 0.95 average can still hide one.
        'color-contrast': 'error',      // WCAG 2.2 SC 1.4.3 Contrast (Minimum)
        'label': 'error',               // WCAG 2.2 SC 4.1.2 Name, Role, Value
        'image-alt': 'error',           // WCAG 2.2 SC 1.1.1 Non-text Content
        'target-size': 'warn',          // WCAG 2.2 SC 2.5.8, newer and noisier
      },
    },
    upload: {
      target: 'filesystem',
      // The bind-mounted directory, so the report survives docker run --rm.
      outputDir: '/out',
      reportFilenamePattern: '%%PATHNAME%%-%%DATETIME%%.report.%%EXTENSION%%',
    },
  },
};

--no-sandbox is in that list because most runners need it, not because it is safe. It is defensible for a staticDistDir built from your own repository on a trusted branch; it is not defensible for a preview built from a fork’s pull request, where an attacker chooses the JavaScript the renderer executes. If the runner supports unprivileged user namespaces, delete the flag and confirm Chrome still launches — the sandbox postures and how to test them are set out in the Docker-based pipeline execution guide.

Auditing a built directory versus a dev server Five rows compare collect.staticDistDir against collect.startServerCommand running a dev server. The built directory serves minified production CSS, adds no extra DOM nodes, exposes three landmark regions, scores 0.93 on every run and produces the same bytes for a given commit. The dev server serves unprocessed CSS, injects the hot-reload client and an overlay root, exposes four landmarks, scores between 0.89 and 0.93 and tracks the versions of its dev dependencies. What Lighthouse is handed in each case observable property collect.staticDistDir startServerCommand: dev stylesheet served minified production CSS unprocessed dev CSS injected DOM nodes none reload client + overlay root landmark regions 3 4, overlay div counted score over five runs 0.93 every run 0.89 to 0.93 same bytes per commit yes no, tracks dev dependencies Lighthouse audits the DOM it is handed, and a dev server hands it a DOM production never renders.
Serving the built output removes a whole dependency tree from the audited page, which is why the score column on the left has no spread at all.

Validation

Run the image the way CI will, then check three things: the exit code, the pinned Chrome version that actually gathered, and the presence of the report on the host. Setting --user to the invoking account is what keeps the written files usable by later steps.

# Build and run exactly as the pipeline does.
docker build --file Dockerfile.lhci --tag lhci-a11y:ci .
mkdir -p lhci-report
docker run --rm \
  --shm-size=1g \
  --init \
  --user "$(id -u):$(id -g)" \
  -v "$PWD/lhci-report:/out" \
  lhci-a11y:ci; echo "exit=$?"

# Passing run:
#   Automatically determined ./dist as `staticDistDir`.
#   Running Lighthouse 5 time(s) on http://localhost:39121/index.html
#   All results processed!
#   exit=0
#
# Failing assertion:
#   [FAIL] categories:accessibility failure for minScore assertion
#          expected: >=0.95   found: 0.91
#   exit=1

# Which Chrome gathered, and was throttling really disabled?
jq -r '.environment.hostUserAgent, .configSettings.throttlingMethod' \
  lhci-report/*index.html*.report.json | head -2

Then assert on the report itself rather than only on the exit code, because the interesting question during a rollout is how much headroom the gate has. The script below reads every run in the output directory, prints the spread, and fails if the runs disagree by more than two points — a spread that wide means the container is still starved rather than the page being broken.

// scripts/check-lhci-spread.mjs — usage: node scripts/check-lhci-spread.mjs /out
import { readdirSync, readFileSync } from 'node:fs';
import { join } from 'node:path';

const dir = process.argv[2] ?? '/out';
const scores = readdirSync(dir)
  .filter((f) => f.endsWith('.report.json'))
  .map((f) => JSON.parse(readFileSync(join(dir, f), 'utf8')))
  .map((lhr) => lhr.categories.accessibility.score);

if (scores.length === 0) throw new Error(`no lhr JSON found in ${dir}`);
const sorted = [...scores].sort((a, b) => a - b);
const median = sorted[Math.floor(sorted.length / 2)];
const spread = sorted.at(-1) - sorted[0];

console.log(`runs=${scores.length} median=${median.toFixed(2)} ` +
  `spread=${spread.toFixed(2)}`);

// A wide spread means unstable gathering, not an unstable page.
if (spread > 0.02) {
  console.error('spread above 0.02: check throttlingMethod, shm size and CPU');
  process.exit(1);
}
Getting the report out of a container that is about to be removed Inside the container, lhci autorun can write to its default .lighthouseci directory, which exists only on the container filesystem and is destroyed by docker run --rm, or to an outputDir set to a bind-mounted path via the filesystem upload target, which lands the manifest, the JSON reports and the HTML reports on the host owned by the invoking CI user. Two places lhci can write, one of which survives inside the container lhci autorun collect, assert, upload default output .lighthouseci/ container path only --rm wipes it upload.target filesystem outputDir: /out bind-mounted host ./lhci-report/ manifest.json report.json and report.html owned by the CI user docker run --rm -v "$PWD/lhci-report:/out" --user "$(id -u):$(id -g)" lhci-a11y:ci
The bind mount and the filesystem upload target are two halves of one fix: the mount provides a durable path, and outputDir is what makes lhci write there.

Edge Cases and Conditional Guards

  • Content that renders after the load signal. staticDistDir audits a static file, so a single-page application that fetches its real content will be snapshotted nearly empty and score suspiciously well. Raise maxWaitForLoad, and where the content depends on a request, audit a pre-rendered route or switch to collect.url against a seeded preview so the audited DOM is the one users get.
  • Authenticated pages. Lighthouse arrives with no session, so a protected route redirects and the audit describes the login page. Use collect.url with a preview deployment that accepts a scoped token, or audit a fixture route that renders the same components with seeded data rather than trying to script a login inside autorun.
  • A runner too small for five runs. numberOfRuns: 5 on a two-vCPU runner can cost four minutes even with onlyCategories: ['accessibility']. Drop to three runs and keep throttlingMethod: 'provided'; the throttling setting removes far more variance than the extra runs do, and a spread check makes it obvious if three is not enough.

Pipeline Impact

lhci autorun exits non-zero when any assertion fails, so the container is a complete gate with no wrapper script — the exit code of docker run is the exit code of the job step. Keep the artifact upload behind if: always() so the HTML report is attached to exactly the runs that failed, and treat the manifest.json as the machine-readable half: it names the representative run, which is the file worth linking from a pull-request comment.

- name: Lighthouse accessibility gate
  run: |
    mkdir -p lhci-report
    docker run --rm --shm-size=1g --init \
      --user "$(id -u):$(id -g)" \
      -v "$PWD/lhci-report:/out" \
      lhci-a11y:ci
- name: Fail on unstable gathering, not just on a low score
  if: always()
  run: node scripts/check-lhci-spread.mjs "$PWD/lhci-report"
- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: lighthouse-accessibility-report
    path: lhci-report/
    retention-days: 30

A Lighthouse score is a weighted average, so it is a good trend signal and a poor debugging tool: one failing audit on many nodes and one failing audit on a single node can produce the same number. Pair the category threshold with the pinned individual audits above, and when a score moves without an obvious cause, compare rule-level output rather than scores — the mapping between the two is worked through in Lighthouse accessibility score vs axe violation counts. The threshold itself should be chosen deliberately rather than copied, using the baseline method in setting up Lighthouse CI thresholds for WCAG 2.2 AA, and a brand-new threshold should spend a fortnight reporting rather than blocking, following auto-fail vs warning workflows.

Common Pitfalls

  • A single run. One gather decides the gate and any hiccup fails the build. Use three to five runs with aggregationMethod set explicitly so you know which run is being asserted against.
  • Leaving throttlingMethod at its default while gating on accessibility. The 4× CPU slowdown exists to model a slow phone for performance metrics; on a contended runner it only makes the audited snapshot incomplete.
  • Using provided while also gating on performance. With throttling off, the performance metrics describe the runner rather than a user, and comparing them across runs is meaningless. Split the two gates into separate configurations.
  • Auditing a dev server. The hot-reload client and error-overlay root are real DOM nodes with real accessibility properties, and they are not in production.
  • Forgetting the bind mount or outputDir. Either one alone still loses the report: the mount without outputDir leaves lhci writing to .lighthouseci, and outputDir without the mount writes into a filesystem that is about to be deleted.
  • Running as root so the report is root-owned. Later workspace steps then fail to clean up, and some artifact actions cannot read the files. Pass --user.
  • Asserting only the category score. A 0.95 average tolerates a genuine color-contrast failure. Pin the audits that map to the criteria you have committed to.

FAQ

Why does the accessibility score move when only performance is being throttled? Because accessibility audits run against the DOM and accessibility-tree artifacts captured at the end of the gathering pass, not against a live page. Slower gathering means a less complete snapshot, so audits see fewer nodes and some become notApplicable and drop out of the weighted average entirely. Setting throttlingMethod: 'provided' lets the page reach its settled state before the snapshot, which is why the spread collapses.

Should the container run lhci autorun or the separate collect, assert and upload commands? autorun for a gate, because it sequences the three phases and returns the assertion result as the process exit code with nothing to wire together. Split them when you need work between phases — collecting once and then asserting the same runs against two different thresholds, or uploading to a server and a filesystem target in one job. The container and flag setup is identical either way.

Does this replace an axe-core scan? No, it complements it. Lighthouse runs a subset of the axe rule set and folds the results into one weighted number, which makes it a good trend and threshold signal and a weak diagnostic. A rule-level axe run gives per-node output that a developer can act on, which is why many pipelines keep both, and why comparing the two is a rule-ID exercise rather than a score-versus-count one.

Back to Docker Based Pipeline Execution