A11y LLM Eval

Pass@k estimates the probability that at least one of k randomly selected samples passes. This is computed from control samples only.

* Pass rate reflects only this harness's automated checks (a curated set of axe-core WCAG rules plus hand-written assertions per test case). Automated testing can detect only a subset of accessibility issues: 100% here means the sample passed every check that was run, not that the page is WCAG conformant or fully accessible.

This section summarizes where models perform well, where they struggle, and the most frequent types of accessibility issues observed across all samples.

Most common axe WCAG failures

Rule	Impact	Failures	% of failures	Seen in models	Seen in test cases	Description
color-contrast	serious	532	89.1%	8	32	Ensure the contrast between foreground and background colors meets WCAG 2 AA minimum contrast ratio thresholds
aria-required-children	critical	14	2.3%	4	5	Ensure elements with an ARIA role that require child roles contain them
button-name	critical	14	2.3%	3	5	Ensure buttons have discernible text
link-in-text-block	serious	8	1.3%	3	5	Ensure links are distinguished from surrounding text in a way that does not rely on color
link-name	serious	8	1.3%	2	4	Ensure links have discernible text
aria-allowed-attr	critical	5	0.8%	3	4	Ensure an element's role supports its ARIA attributes
listitem	serious	5	0.8%	3	4	Ensure elements are used semantically
aria-prohibited-attr	serious	3	0.5%	2	2	Ensure ARIA attributes are not prohibited for an element's role
image-alt	critical	2	0.3%	1	1	Ensure elements have alternative text or a role of none or presentation
label	critical	2	0.3%	2	2	Ensure every form element has a label

Most common axe best-practice failures

Assertion-level patterns (per test case)

Checkbox Group | React | Dark

Assertion statistics are computed within this test case only and are not compared across different test cases.

Checkbox Group | React | Modern

Assertion statistics are computed within this test case only and are not compared across different test cases.

Checkbox Group | Vanilla JS | Dark

Assertion statistics are computed within this test case only and are not compared across different test cases.

Checkbox Group | Vanilla JS | Modern

Assertion statistics are computed within this test case only and are not compared across different test cases.

Disclosure Widget | React | Dark

Assertion statistics are computed within this test case only and are not compared across different test cases.

Disclosure Widget | React | Modern

Assertion statistics are computed within this test case only and are not compared across different test cases.

Disclosure Widget | Vanilla JS | Dark

Assertion statistics are computed within this test case only and are not compared across different test cases.

Disclosure Widget | Vanilla JS | Modern

Assertion statistics are computed within this test case only and are not compared across different test cases.

Modal Dialog | React | Dark

Assertion statistics are computed within this test case only and are not compared across different test cases.

Modal Dialog | React | Modern

Assertion statistics are computed within this test case only and are not compared across different test cases.

Modal Dialog | Vanilla JS | Dark

Assertion statistics are computed within this test case only and are not compared across different test cases.

Modal Dialog | Vanilla JS | Modern

Assertion statistics are computed within this test case only and are not compared across different test cases.

Radio Button Group | React | Dark

Assertion statistics are computed within this test case only and are not compared across different test cases.

Radio Button Group | React | Modern

Assertion statistics are computed within this test case only and are not compared across different test cases.

Radio Button Group | Vanilla JS | Dark

Assertion statistics are computed within this test case only and are not compared across different test cases.

Radio Button Group | Vanilla JS | Modern

Assertion statistics are computed within this test case only and are not compared across different test cases.

Shopping Home Page | React | Dark

Assertion statistics are computed within this test case only and are not compared across different test cases.

Shopping Home Page | React | Modern

Assertion statistics are computed within this test case only and are not compared across different test cases.

Shopping Home Page | Vanilla JS | Dark

Assertion statistics are computed within this test case only and are not compared across different test cases.

Shopping Home Page | Vanilla JS | Modern

Assertion statistics are computed within this test case only and are not compared across different test cases.

Simple Contact Form | React | Dark | Error Message Present

Assertion statistics are computed within this test case only and are not compared across different test cases.

Simple Contact Form | React | Dark | No Error Message

Assertion statistics are computed within this test case only and are not compared across different test cases.

Simple Contact Form | React | Modern | Error Message Present

Assertion statistics are computed within this test case only and are not compared across different test cases.

Simple Contact Form | React | Modern | No Error Message

Assertion statistics are computed within this test case only and are not compared across different test cases.

Simple Contact Form | Vanilla JS | Dark | Error Message Present

Assertion statistics are computed within this test case only and are not compared across different test cases.

Simple Contact Form | Vanilla JS | Dark | No Error Message

Assertion statistics are computed within this test case only and are not compared across different test cases.

Simple Contact Form | Vanilla JS | Modern | Error Message Present

Assertion statistics are computed within this test case only and are not compared across different test cases.

Simple Contact Form | Vanilla JS | Modern | No Error Message

Assertion statistics are computed within this test case only and are not compared across different test cases.

Single Checkbox | React | Dark

Assertion statistics are computed within this test case only and are not compared across different test cases.

Single Checkbox | React | Modern

Assertion statistics are computed within this test case only and are not compared across different test cases.

Single Checkbox | Vanilla JS | Dark

Assertion statistics are computed within this test case only and are not compared across different test cases.

Single Checkbox | Vanilla JS | Modern

Assertion statistics are computed within this test case only and are not compared across different test cases.

Minimal reminder that all output must be accessible.

Full instruction set

Variant samples per (test, model): 5

Generation mode: copilot_agent

All output MUST be accessible.

Basic reminder that all output must be accessible (includes slightly more instructions than minimal).

Full instruction set

Variant samples per (test, model): 5

Generation mode: copilot_agent

---
description: "Accessibility coding rules: WCAG 2.2 AA conformance, semantic structure, keyboard support, focus management. Apply when writing or modifying UI code."
applyTo: "**"
---

# Accessibility instructions (standard)

Conform to [WCAG 2.2 Level AA](https://www.w3.org/TR/WCAG22/). Do not claim output is "fully accessible" — state what was addressed and any known limitations.

## Implementation priority

Use the first option that fits:

1. Existing accessible component in the project / design system.
2. A component library already in use.
3. Native platform semantics (`<button>`, `<a href>`, `<input>`, `<label>`, `<fieldset>`/`<legend>`, `<dialog>`, `<nav>`, `<main>`, `<h1>`–`<h6>`).
4. Native element + minimum necessary ARIA.
5. Fully custom ARIA widget — only when nothing above fits, and only with complete APG keyboard, focus, and state behavior.

No ARIA is better than bad ARIA. Don't duplicate native semantics (no `role="button"` on `<button>`). Don't use `role="menu"` for site navigation.

## Structure

- Use landmark elements (`<header>`, `<nav>`, `<main>`, `<footer>`). Exactly one `<main>`. Give duplicated landmarks unique accessible names.
- One `<h1>` per view, typically first heading in `<main>`. Don't skip heading levels.
- Set a descriptive `<title>`.
- **Web pages only:** Provide a "Skip to main content" link as the first focusable element.

## Name, role, value

- Every interactive element exposes an accurate accessible name. Role matches purpose. Dynamic states (pressed, expanded, selected, checked, disabled, invalid) stay in sync with visuals.
- The accessible name MUST contain the visible label text. When multiple controls share a label (e.g. many "Remove" buttons), add context: `aria-label="Remove item: Socks"`.

## Keyboard and focus

- All functionality can be achieved by both mouse an keyboard; tab order matches reading/visual order.
- Focus is always visible — do not remove focus outlines without an equal-or-better replacement.
- Avoid keyboard traps.
- Escape should close overlays.
- Static content MUST NOT be sequentially focusable. Use `tabindex="-1"` only for programmatic focus targets.
- Content hidden from AT (`aria-hidden="true"`) MUST NOT be focusable.
- Dialogs move focus in and restore it on close.
- Composite widgets (tabs, listbox, menu, grid): one tab stop total; arrow keys move focus internally via roving `tabindex` or `aria-activedescendant`.

## Forms

- Every form field has a visual and programmatic label (`<label for>` or wrapping `<label>`). Never rely on placeholder alone.
- Associate help/error text via `aria-describedby`.
- Group related options (checkboxes, radios) with `<fieldset>` + `<legend>`.
- Required fields: visible indicator (e.g., `*`) AND `required` / `aria-required="true"`. Never color alone. This is a MUST when the form contains both required and optional fields.
- Invalid fields: `aria-invalid="true"`; remove when corrected. Error messages explain how to fix.
- On submit with invalid input, focus the first invalid control. Don't disable submit solely to prevent submission.

## Contrast and color

- Text contrast ≥ 4.5:1 (≥ 3:1 for large text: 24px regular or 18.66px bold).
- Focus indicators and key control boundaries ≥ 3:1 vs adjacent colors.
- Never use color as the only cue for meaning (error, success, required, selected).
- Use design tokens / CSS custom properties. Avoid `opacity`, `rgba`, `hsla` on text and essential affordances — contrast becomes background-dependent.
- Ensure contrast in all states: default, hover, active, focus, visited, disabled.

## Forced colors / OS settings

- Never override OS high-contrast, reduced-motion, or color-scheme preferences without good reason.
- Do not use `forced-color-adjust: none` without good reason (e.g., data-viz where color needs to remain the same).
- In `@media (forced-colors: active)`, use system color keywords (`ButtonText`, `ButtonBorder`, `CanvasText`, `Canvas`) — never fixed hex/RGB.
- Use `currentColor` for SVG `fill`/`stroke` so icons inherit the foreground.
- If relying on `box-shadow` for focus, add a transparent `outline` so focus renders in forced colors.

## Reflow (SC 1.4.10)

- Content MUST be able to 320 CSS pixels wide without two-dimensional scrolling for multi-line text.
- For multi-column layouts that are not necessary to convey meaning or important to the UX of the interface, content stacks; text wraps; controls remain operable.
- Use fluid `flex`/`grid`. Set `max-width: 100%` on media, `min-width: 0` on flex/grid children, `overflow-wrap: anywhere` for long strings.
- Exception: inherently 2D components (large tables, maps, charts, media, interfaces with toolbars or interfaces that require 2D layout) may scroll horizontally at component level; the surrounding view still reflows.

## Graphics

- Informative `<img>` → meaningful `alt`. Decorative `<img>` → `alt=""`.
- Informative `<svg>` → `role="img"` with `aria-label` / `aria-labelledby`. Decorative SVG/graphics → `aria-hidden="true"`.

## Navigation

- Use `<nav>` with lists and links — not `role="menu"` / `role="menubar"`.
- Expandable navigation: toggle uses `button[aria-expanded]`. Escape MAY close sub-navigations.

## Tables and grids

- Static tabular data: `<table>` with `<th>` for column/row headers.
- Use `role="grid"` only for genuinely interactive tabular UIs, with proper row/cell nesting and arrow-key navigation.

## Status messages

- Announce dynamic updates (loading, success, failure, error, validation summaries) via `aria-live="polite"` or `aria-live="assertive"`.

## Final verification

Before finalizing, verify: landmarks + one `<h1>`; keyboard operability with visible focus and no traps; visible labels included in accessible names; form labels + required + error association + focus-first-invalid; contrast thresholds; forced-colors adaptation; reflow at 320px; image alternatives; table header associations.

This section highlights where each instruction set helped (or hurt) compared to the control, aggregated across all samples for that instruction set.

* Pass rate reflects only this harness's automated checks (a curated set of axe-core WCAG rules plus hand-written assertions per test case). Automated testing can detect only a subset of accessibility issues: 100% here means the sample passed every check that was run, not that the page is WCAG conformant or fully accessible.