Internationalization and Localization Testing

A pipeline that scans only the English build ships twenty-nine unverified locales. This guide is part of Custom Rule Development & Context-Aware Testing, and it covers the machinery for running one accessibility suite across every locale a product actually ships: deriving the locale matrix from the message catalogue instead of a hand-maintained array, opening a page per locale in a Playwright project, asserting document lang and dir, comparing every accessible name against its expected translation, and surfacing each locale as its own CI status check so a regression in Arabic does not hide behind a green aggregate.

Problem Statement

Localization breaks accessibility in ways that a single-locale scan structurally cannot see. Four failure families account for almost everything a locale matrix catches in its first month of operation.

The first is untranslated accessible names. A translation pipeline that covers visible text often misses strings that only ever reach the accessibility tree: aria-label on icon-only buttons, alt on decorative-turned-informative images, title on abbreviations, and the visually hidden text inside a <span class="sr-only">. Those strings fall back to the source language. axe-core reports no violation, because the control does have an accessible name — it is simply in the wrong language, and axe carries no dictionary of expected translations. A screen reader in Arabic then announces a Latin string with Arabic phonetics, which is unusable.

The second is direction. dir="rtl" belongs on the root element so that every descendant inherits it and the assistive-technology reading order flips as a unit. Teams routinely set it on a layout wrapper, on <body>, or only in CSS via direction: rtl, which changes visual rendering without changing the DOM attribute that assistive technology and :dir() selectors rely on. Related failures follow: physical CSS margins that refuse to mirror, and embedded Latin or numeric runs inside right-to-left text that reorder because nothing isolates them.

The third is locale-dependent typography. Most design systems ship :lang() overrides — a larger base size for CJK, a different family and line height for Arabic, tighter tracking for Cyrillic. Those overrides change the computed font-size and font-weight that axe-core’s color-contrast check reads to decide whether the 3:1 large-text threshold or the 4.5:1 normal-text threshold applies. A ratio of 3.6:1 that passes as large text in English can fail in Japanese when the fallback stack renders at a lower computed weight. The pass in English is not evidence of anything.

The fourth is expansion and truncation. German and Finnish routinely run 30–40% longer than English; Arabic and Thai change measured line height. Labels that fit in English clip under text-overflow: ellipsis, or wrap into a fixed-height container and get cut. The accessible name survives in the accessibility tree while the visible label no longer matches it, breaking WCAG 2.2 SC 2.5.3 (Label in Name), and reflowed content at 320 CSS pixels starts overlapping, breaking SC 1.4.10 (Reflow).

None of these are exotic. They are the default outcome of a build that scans one locale and assumes the rest are a text substitution.

Key implementation targets:

  • A locale matrix generated from the message catalogue, so adding a locale automatically adds a scan.
  • One Playwright project per shipped locale, carrying locale, direction, and blocking tier as project metadata.
  • Assertions that html[lang] and html[dir] match the requested locale, plus per-element lang on embedded foreign runs.
  • Accessible-name comparison against the catalogue value for that locale, with Unicode normalisation before the string compare.
  • Per-locale CI status checks with a blocking tier and a warning tier, so branch protection can require the locales that matter.

Prerequisites

1. Build the Locale Matrix from the Message Catalogue

Hand-maintained locale arrays rot within one release. A translator adds pt-BR, the catalogue grows a file, and the test matrix keeps scanning the same four locales it had at the start of the project. Make the catalogue the single source of truth: enumerate the files, derive direction from the language subtag, compute a completeness ratio against the source locale, and assign each locale to a blocking or warning tier based on that ratio and an explicit ship list.

Completeness matters because a locale at 62% translated will generate dozens of legitimate untranslated-name failures that nobody intends to fix this sprint. Those locales belong in the warning tier: scanned, reported, and non-blocking. A locale at 100% belongs in the blocking tier, where a single dropped key fails the build.

// scripts/build-locale-matrix.mjs — emits locale-matrix.json from the catalogue
import { readdir, readFile, writeFile } from 'node:fs/promises';

const SOURCE = 'en-US';                       // reference locale for key coverage
const RTL_LANGS = new Set(['ar', 'he', 'fa', 'ur', 'yi', 'dv']);
const SHIPPED = new Set(                      // locales served in production
  (process.env.SHIPPED_LOCALES ?? 'en-US,de-DE,fr-FR,ja-JP,ar-SA,he-IL').split(',')
);

const flatten = (obj, prefix = '') =>
  Object.entries(obj).flatMap(([k, v]) =>
    v && typeof v === 'object'
      ? flatten(v, `${prefix}${k}.`)          // nested namespaces become dotted keys
      : [[`${prefix}${k}`, String(v)]]
  );

const files = (await readdir('locales')).filter((f) => f.endsWith('.json'));
const catalogues = new Map();
for (const file of files) {
  const tag = file.replace(/\.json$/, '');
  const parsed = JSON.parse(await readFile(`locales/${file}`, 'utf8'));
  catalogues.set(tag, new Map(flatten(parsed)));
}

const sourceKeys = [...catalogues.get(SOURCE).keys()];
const locales = [...catalogues.entries()]
  .filter(([tag]) => SHIPPED.has(tag))        // draft locales never enter the matrix
  .map(([tag, entries]) => {
    const lang = tag.split('-')[0];
    const translated = sourceKeys.filter((k) => {
      const value = entries.get(k);
      // A key present but byte-identical to the source is an untranslated fallback.
      return value !== undefined && (tag === SOURCE || value !== catalogues.get(SOURCE).get(k));
    });
    const coverage = translated.length / sourceKeys.length;
    return {
      tag,
      lang,
      dir: RTL_LANGS.has(lang) ? 'rtl' : 'ltr',
      pathPrefix: tag === SOURCE ? '' : `/${lang}`,
      coverage: Number(coverage.toFixed(4)),
      // Blocking tier requires a fully translated catalogue; everything else warns.
      tier: coverage >= 0.999 ? 'blocking' : 'warning',
      missingKeys: sourceKeys.filter((k) => !entries.has(k)),
    };
  })
  .sort((a, b) => a.tag.localeCompare(b.tag));

await writeFile('locale-matrix.json', JSON.stringify({ sourceKeys: sourceKeys.length, locales }, null, 2));
console.log(locales.map((l) => `${l.tag} ${l.dir} ${(l.coverage * 100).toFixed(1)}% ${l.tier}`).join('\n'));

The missingKeys array is the cheapest accessibility signal in the whole system: it is computed without launching a browser, and a non-empty array on a blocking-tier locale means at least one accessible name will fall back. The dedicated technique for turning that array into a failing assertion is covered in the guide on testing internationalized labels, which also explains the pseudo-locale trick for finding strings that never entered the catalogue at all.

Locale matrix build pipeline Locale JSON files feed a matrix builder that emits locale-matrix.json; Playwright turns each entry into a project, and each project reports as its own CI status check. catalogue locales/*.json matrix builder coverage + dir matrix.json tag tier dir Playwright one project each CI checks one per locale SHIPPED_LOCALES filter drafts never scanned context options set here locale · timeZoneId · headers
The catalogue directory, not a hand-edited array, decides which locales get scanned; every downstream stage reads the generated matrix.

2. Load a Page per Locale in a Playwright Project

Playwright projects are the right unit for a locale. Each project gets its own browser context options, its own baseURL, its own retry policy, and its own entry in the HTML and JUnit reports — which is exactly the granularity a per-locale status check needs. Generating the projects from locale-matrix.json means the config file never changes when a locale is added.

Two context options matter beyond locale. Setting timeZoneId freezes date and time formatting, without which a name containing a formatted timestamp differs between a runner in UTC and a developer machine in CET. Setting Accept-Language explicitly stops the framework’s own locale negotiation from overriding the URL prefix; if the app negotiates from the header and the test navigates by prefix, the two can disagree and the page renders in a third locale entirely.

// playwright.config.ts — one project per shipped locale, generated from the matrix
import { defineConfig, devices } from '@playwright/test';
import matrix from './locale-matrix.json' with { type: 'json' };

type LocaleEntry = { tag: string; lang: string; dir: 'ltr' | 'rtl'; pathPrefix: string; tier: string };

export default defineConfig({
  testDir: './tests/i18n-a11y',
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 1 : 0,          // one retry absorbs font-load flake, not real failures
  reporter: [
    ['list'],
    ['junit', { outputFile: 'reports/i18n-a11y.xml' }],
    ['json', { outputFile: 'reports/i18n-a11y.json' }],
  ],
  projects: (matrix.locales as LocaleEntry[]).map((entry) => ({
    name: entry.tag,                        // project name becomes the CI check name
    metadata: { lang: entry.lang, dir: entry.dir, tier: entry.tier },
    use: {
      ...devices['Desktop Chrome'],
      baseURL: `${process.env.BASE_URL ?? 'http://localhost:3000'}${entry.pathPrefix}`,
      locale: entry.tag,                    // drives Intl formatting inside the page
      timeZoneId: 'UTC',                    // deterministic dates in accessible names
      extraHTTPHeaders: { 'Accept-Language': `${entry.tag},${entry.lang};q=0.9` },
      viewport: { width: 1280, height: 900 },
    },
  })),
});

Once a page is open, the same run should collect the geometry and colour evidence that only exists in this locale. Text expansion is measurable: for every element that carries a visible label, compare scrollWidth against clientWidth and scrollHeight against clientHeight. A positive delta on an element with overflow: hidden means the label is clipped, which is where SC 2.5.3 (Label in Name) and SC 1.4.10 (Reflow) start to break even though the accessibility tree still reports the full string.

// tests/i18n-a11y/helpers/clipping.ts — find labels the locale's text length breaks
import type { Page } from '@playwright/test';

export type Clipped = { selector: string; overflowX: number; overflowY: number; text: string };

export async function findClippedLabels(page: Page): Promise<Clipped[]> {
  await page.evaluate(() => document.fonts.ready);   // measure after webfonts swap in
  return page.evaluate(() => {
    const targets = document.querySelectorAll<HTMLElement>(
      'button, a, label, [role="button"], [role="tab"], legend, th'
    );
    const out: Clipped[] = [];
    for (const el of targets) {
      const cs = getComputedStyle(el);
      const hidden = cs.overflowX !== 'visible' || cs.overflowY !== 'visible';
      if (!hidden || !el.textContent?.trim()) continue;
      const overflowX = el.scrollWidth - el.clientWidth;
      const overflowY = el.scrollHeight - el.clientHeight;
      // 1px of subpixel rounding is normal; 2px or more is real clipping.
      if (overflowX < 2 && overflowY < 2) continue;
      out.push({
        selector: el.tagName.toLowerCase() + (el.id ? `#${el.id}` : ''),
        overflowX,
        overflowY,
        text: el.textContent.trim().slice(0, 80),
      });
    }
    return out;
  });
}

Contrast is the third layer. Run the axe scan inside every locale project rather than once, because :lang() rules change the computed font metrics that decide which contrast threshold axe applies. Keep the axe options identical across locales so the only variable is the locale itself; the baseline options belong in shared setup, as described in the axe-core configuration guide.

Assertion layers inside one locale run Each locale run asserts document attributes, then accessible names, then rendered geometry, then computed colour; the cheapest layer runs first and the cost per layer rises down the stack. cheapest and most deterministic layer first 1 · document attributes html[lang] and html[dir] match the requested tag ~5 ms 2 · accessible names tree name equals the catalogue value for this locale ~90 ms 3 · rendered geometry scrollWidth versus clientWidth on every labelled control ~140 ms 4 · computed colour axe color-contrast under this locale's :lang() overrides ~1.4 s
Every locale run walks the same four layers; ordering them by cost means a wrong dir attribute fails in milliseconds instead of after a full axe scan.

3. Assert Document lang and dir

The document attributes are the foundation everything else rests on, and they are trivially assertable. html[lang] must carry a valid BCP 47 tag that matches the requested locale; WCAG 2.2 SC 3.1.1 (Language of Page) requires it, and screen readers use it to pick a speech synthesiser voice. html[dir] must match the script of that language. Because the matrix already carries lang and dir per project, the spec reads them from testInfo.project.metadata and needs no locale-specific branching.

Match policy needs a deliberate decision. If the catalogue declares de-DE but the app renders lang="de", a strict equality assertion fails on a page that is entirely correct. The workable rule is: the primary subtag must match exactly, and the region subtag must match only when the app actually varies content by region. Compare with Intl.Locale rather than string slicing so that zh-Hant-TW and zh-TW do not produce a spurious mismatch on the script subtag.

// tests/i18n-a11y/document-attributes.spec.ts
import { test, expect } from '@playwright/test';

test.describe('document language and direction', () => {
  test('root element declares the requested locale and script direction', async ({
    page,
  }, testInfo) => {
    const { lang, dir } = testInfo.project.metadata as { lang: string; dir: string };
    await page.goto('/checkout');
    // Wait for the app's own translation-applied signal, not networkidle.
    await page.waitForFunction(() => document.documentElement.dataset.i18n === 'ready');

    const html = page.locator('html');
    const declared = (await html.getAttribute('lang')) ?? '';
    // Primary subtag must match; region may be broader than the catalogue tag.
    expect(new Intl.Locale(declared).language).toBe(lang);
    await expect(html).toHaveAttribute('dir', dir);

    // CSS-only direction is not enough: :dir() and AT both read the attribute.
    const computedDir = await html.evaluate((el) => getComputedStyle(el).direction);
    expect(computedDir).toBe(dir);

    // Embedded runs in another language must declare it (SC 3.1.2 Language of Parts).
    const undeclaredForeignRuns = await page.locator('[data-foreign-run]:not([lang])').count();
    expect(undeclaredForeignRuns).toBe(0);
  });
});

The data-foreign-run assertion is worth the small amount of authoring discipline it demands. Marking known foreign-language runs in the templates — a product name, a legal phrase, a quoted review — gives the test something concrete to check for SC 3.1.2 (Language of Parts), which is otherwise unautomatable because no scanner can detect the language of a text node reliably. The right-to-left half of this problem, including bidi isolation and a custom axe check for lang/dir disagreement, is covered in the guide on validating RTL ARIA attributes.

4. Assert Accessible Names Against the Expected Translation

This is the assertion that no off-the-shelf rule provides. The test needs to know what the accessible name should be in this locale, which means the catalogue value has to reach the test. Keep the binding explicit: a manifest that maps a stable test id and role to a catalogue key. Deriving the key from the DOM at runtime seems tempting but couples the test to whichever data-i18n-key attribute the framework happens to emit, and those attributes disappear in production builds.

// tests/i18n-a11y/accessible-names.spec.ts
import { test, expect } from '@playwright/test';
import { readFile } from 'node:fs/promises';

// Stable test id -> role + catalogue key. Reviewed like any other contract.
const NAME_MANIFEST = [
  { testId: 'search-submit', role: 'button' as const, key: 'checkout.search.submit' },
  { testId: 'close-cart', role: 'button' as const, key: 'checkout.cart.close' },
  { testId: 'promo-banner', role: 'img' as const, key: 'checkout.promo.alt' },
  { testId: 'qty-stepper', role: 'spinbutton' as const, key: 'checkout.qty.label' },
];

// Strip formatting controls and normalise so a compare tests meaning, not bytes.
const BIDI_AND_FORMAT = /[\u00AD\u200B-\u200F\u202A-\u202E\u2066-\u2069\uFEFF]/g;
const canonical = (s: string) =>
  s.normalize('NFC').replace(BIDI_AND_FORMAT, '').replace(/\s+/g, ' ').trim();

test('every manifest entry carries its localized accessible name', async ({ page }, testInfo) => {
  const catalogue = JSON.parse(await readFile(`locales/${testInfo.project.name}.json`, 'utf8'));
  const lookup = (key: string) =>
    key.split('.').reduce<unknown>((acc, part) => (acc as Record<string, unknown>)?.[part], catalogue);

  await page.goto('/checkout');
  await page.waitForFunction(() => document.documentElement.dataset.i18n === 'ready');

  for (const { testId, role, key } of NAME_MANIFEST) {
    const expected = lookup(key);
    expect(expected, `missing catalogue key ${key} for ${testInfo.project.name}`).toBeTruthy();
    const node = page.getByTestId(testId);
    const actual = await node.evaluate((el) => el.ariaLabel ?? el.getAttribute('alt') ?? el.textContent ?? '');
    expect(canonical(actual)).toBe(canonical(String(expected)));
    // Role is asserted separately so a name match on the wrong element still fails.
    await expect(page.getByRole(role, { name: String(expected) })).toHaveCount(1);
  }
});

Normalisation before comparison is not optional. Three classes of invisible difference will otherwise produce failures on correct pages. Composed versus decomposed forms differ byte-wise for any accented character, so Ü written as U+00DC and as U+0055 U+0308 compare unequal without normalize('NFC'). Soft hyphens (U+00AD) inserted by translators for German compound words survive into the accessible name. Bidi control characters — U+200E, U+200F, and the isolate pair U+2066–U+2069 — are frequently embedded in Arabic and Hebrew catalogue values to pin embedded Latin runs, and they are part of the accessible name string that the browser reports.

Normalising an accessible name before comparison The raw accessible name is normalised to NFC, stripped of format controls, whitespace-collapsed, and only then compared with the German catalogue value. raw name from tree decomposed U + diaeresis, soft hyphen, 2 spaces normalize('NFC') the umlaut becomes one code point strip format controls soft hyphen, LRM, RLM, isolates removed collapse and trim single spaces, nothing leading or trailing compare to de-DE equals the catalogue value, so the check passes
Without the middle three steps a correct German label fails the comparison on invisible code points alone.

5. Report Per-Locale Results as Separate CI Checks

An aggregate check hides the thing the matrix exists to reveal. If one job runs all locales and reports a11y: failed, the reviewer learns nothing about which locale broke, and a flaky Japanese run blocks a pull request that only touched German copy. Emit one check per locale, named after the project, and let branch protection require only the blocking tier.

Two mechanisms achieve this. The simple one is a GitHub Actions matrix job whose name interpolates the locale, which makes each matrix leg its own status check automatically; requiring a11y (de-DE) in branch protection then works with no extra API calls. The explicit one is a single job that publishes check runs through the Checks API, which is preferable when the locale list changes often, because branch protection rules referencing a removed check name block merges forever.

# .github/workflows/i18n-a11y.yml — matrix legs become individually requirable checks
name: i18n-a11y
on:
  pull_request:
    paths: ['locales/**', 'src/**', 'playwright.config.ts']

jobs:
  matrix:
    runs-on: ubuntu-latest
    outputs:
      blocking: ${{ steps.build.outputs.blocking }}
      warning: ${{ steps.build.outputs.warning }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20', cache: 'npm' }
      - run: npm ci
      - id: build
        run: |
          node scripts/build-locale-matrix.mjs
          jq -c '[.locales[] | select(.tier=="blocking") | .tag]' locale-matrix.json \
            | sed 's/^/blocking=/' >> "$GITHUB_OUTPUT"
          jq -c '[.locales[] | select(.tier=="warning") | .tag]' locale-matrix.json \
            | sed 's/^/warning=/' >> "$GITHUB_OUTPUT"
      - uses: actions/upload-artifact@v4
        with: { name: locale-matrix, path: locale-matrix.json }

  scan:
    needs: matrix
    runs-on: ubuntu-latest
    name: a11y (${{ matrix.locale }})   # this string is the required status check
    strategy:
      fail-fast: false                  # one broken locale must not cancel the rest
      matrix:
        locale: ${{ fromJSON(needs.matrix.outputs.blocking) }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20', cache: 'npm' }
      - run: npm ci
      - uses: actions/download-artifact@v4
        with: { name: locale-matrix }
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test --project="${{ matrix.locale }}"
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: i18n-a11y-${{ matrix.locale }}
          path: reports/
          retention-days: 14

  soak:
    needs: matrix
    runs-on: ubuntu-latest
    name: a11y-soak (${{ matrix.locale }})
    continue-on-error: true             # warning tier reports without blocking merge
    strategy:
      fail-fast: false
      matrix:
        locale: ${{ fromJSON(needs.matrix.outputs.warning) }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20', cache: 'npm' }
      - run: npm ci
      - uses: actions/download-artifact@v4
        with: { name: locale-matrix }
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test --project="${{ matrix.locale }}" || true

Which check names become mandatory is a branch-protection decision, not a test decision; the mechanics of wiring specific names into a protected branch are covered in the guide on requiring accessibility status checks in branch protection.

Serious violations per locale, first matrix run English, French, and German sit at or under the budget of two serious violations, while Japanese, Hebrew, and Arabic exceed it, with Arabic highest at seven. 0 2 4 6 8 dashed line = budget of 2 serious violations per locale 0 en-US 2 de-DE 1 fr-FR 3 ja-JP 5 he-IL 7 ar-SA serious violations found by locale on the first full matrix run
Separate checks make this distribution visible; a single aggregate check would have reported one red bar and hidden which three locales actually regressed.

Pipeline Integration

The matrix produces one JUnit file and one JSON report per locale, which is the wrong shape for a reviewer reading a pull request. Add a summary job that downloads every artifact and writes a locale table into $GITHUB_STEP_SUMMARY, so the pull-request page carries a single table with a row per locale, its violation count by impact, and its clipped-label count. Reviewers then read the summary and open only the failing locale’s artifact.

#!/usr/bin/env bash
# scripts/summarize-locales.sh — one summary table for all downloaded locale reports
set -euo pipefail

printf '| Locale | Critical | Serious | Clipped labels | Result |\n' >> "$GITHUB_STEP_SUMMARY"
printf '|---|---|---|---|---|\n' >> "$GITHUB_STEP_SUMMARY"

exit_code=0
for report in artifacts/i18n-a11y-*/i18n-a11y.json; do
  locale="$(basename "$(dirname "$report")" | sed 's/^i18n-a11y-//')"
  crit=$(jq '[.. | objects | select(.impact=="critical")] | length' "$report")
  serious=$(jq '[.. | objects | select(.impact=="serious")] | length' "$report")
  clipped=$(jq '[.. | objects | select(.overflowX? // 0 > 1)] | length' "$report")
  if [ "$crit" -gt 0 ] || [ "$serious" -gt 2 ]; then
    result='fail'; exit_code=1     # non-zero here is what actually blocks the merge
  else
    result='pass'
  fi
  printf '| %s | %s | %s | %s | %s |\n' \
    "$locale" "$crit" "$serious" "$clipped" "$result" >> "$GITHUB_STEP_SUMMARY"
done
exit "$exit_code"

Runtime is the other integration concern. Six locales multiply wall-clock time by six unless the legs run in parallel, and a matrix of thirty locales against a full page inventory will not fit inside a pull-request budget. Two levers help: scan a representative page set per locale rather than the whole site, and shard the heavy locales across runners using the technique in the guide on sharding axe-core scans across parallel CI jobs. A pragmatic split is a five-page smoke set for every locale on each pull request, and the full inventory for all locales nightly.

Troubleshooting and Flaky-Test Mitigation

Translations applied after the first assertion. Most i18n runtimes fetch the catalogue chunk asynchronously and swap text in after hydration. Waiting on networkidle is not sufficient, because the swap happens in a microtask after the response settles and because analytics beacons keep the network busy. Wait on an explicit application-owned signal — a data-i18n="ready" attribute or a resolved promise exposed on window — and treat its absence as a test failure rather than a timeout. Framework-specific variants of this race, including attribute updates that never fire a mutation the test observes, are covered in the guide on handling dynamic ARIA states in modern JavaScript frameworks.

Font swap changes every measurement. Clipping detection runs on measured geometry, and webfonts land after first paint. A run that measures during the fallback font reports phantom overflow in CJK locales, where fallback metrics differ most. Await document.fonts.ready inside the page before any geometry read, and set font-display: block for the test build if the swap still races.

Locale negotiation disagrees with the URL. When a server negotiates from Accept-Language and the test navigates to /de/checkout, a header of en-US can win and the page renders English at a German URL. The assertion on html[lang] catches it, but the failure reads as a product bug rather than a test misconfiguration. Set the header from the matrix entry, as in the project config above, and assert the negotiated locale once in a smoke test.

Number and date formatting drift. An accessible name built from an interpolated total ("Pay €1.234,56") depends on Intl.NumberFormat output, which varies with the ICU version bundled in the runner’s Node build. Pin timeZoneId, keep a full-ICU Node in CI, and prefer manifest entries whose catalogue values contain no interpolation for the strict-equality assertions; compare interpolated names against a pattern instead.

Retry masking a real regression. One retry absorbs font-load and hydration flake, which is worth having. Three retries hide a genuine intermittent failure in a right-to-left layout. Keep retries: 1 in CI and treat any test that passes only on retry as a defect to investigate, using the per-locale artifact to see which attempt failed.

Pseudo-locale noise in the blocking tier. A pseudo-locale is a diagnostic build, not a shipped locale, and its deliberately mangled strings will trip contrast and clipping checks. Keep it out of SHIPPED_LOCALES and run it in its own job.

Common Pitfalls

  • Scanning only the source locale and treating a green result as coverage for every translated build.
  • Hand-maintaining the locale list in the test config, so new locales ship without ever being scanned.
  • Comparing accessible names byte-for-byte without Unicode normalisation, producing failures on correct pages.
  • Setting direction in CSS only, leaving html[dir] absent so assistive technology and :dir() see the wrong value.
  • Emitting one aggregate status check, which hides which locale regressed and lets a flaky locale block unrelated work.
  • Putting partially translated locales in the blocking tier, which generates unfixable noise and trains reviewers to ignore the gate.
  • Measuring text clipping before document.fonts.ready, which reports overflow that does not exist in the shipped rendering.
  • Asserting only that an accessible name exists, which passes on every source-language fallback string.

FAQ

How many locales belong on the pull-request path versus a nightly run? Put the locales that represent distinct scripts and directions on the pull-request path — typically the source locale, one long-expansion Latin locale such as German, one CJK locale, and one right-to-left locale. That set catches the four failure families at four-way parallel cost. Run the full shipped list nightly, because the remaining locales mostly re-test the same layout with different string lengths.

Does a per-locale scan need to cover every page? No, and trying to is what makes locale matrices get deleted. Direction, lang, and typography failures are layout-wide, so a five-page set covering the shell, a form, a data table, a modal, and a long-content page finds nearly all of them. Accessible-name coverage is the exception: that needs to reach every string, which is why the catalogue-driven approach asserts against the catalogue rather than crawling pages.

Why compare against the catalogue instead of a stored snapshot per locale? A snapshot records what the app rendered last time, so a translation that was wrong when the snapshot was taken stays wrong forever and the test defends the bug. The catalogue is the artifact translators actually edit, so comparing against it means a translator’s fix and the test’s expectation move together, and a dropped key fails immediately instead of quietly matching a stale snapshot.

Can axe-core detect that a label is in the wrong language? No. axe evaluates structure and computed style, so it verifies that a control has a non-empty accessible name and that html[lang] is a valid tag, but it has no view of what the name should say. Language-correctness assertions have to come from outside axe — either from a catalogue comparison or from a custom check that compares an element’s lang against the script of its content.

What is the right failure threshold for a partially translated locale? Gate it on regressions rather than on absolutes. Record the locale’s current violation count as a baseline, fail the job only when the count rises, and ratchet the baseline down as translation coverage improves. That keeps a 60%-translated locale useful as a signal without blocking every merge on work the translation team has not scheduled.

In This Section

Back to Custom Rule Development Context Aware Testing