Evals in CI

This guide describes how the vally eval pipeline authenticates in CI, how forked pull requests are handled, and how to add a new eval spec for an AI artifact (agent, prompt, instructions file, or skill).

Required Secret

The eval-execute job in .github/workflows/pr-validation.yml runs the vally eval command for each changed AI artifact. The vally CLI delegates to the @github/copilot CLI, which requires a GitHub credential exported as COPILOT_GITHUB_TOKEN.

Configure the secret at the repository (or organization) level:

• Settings -> Secrets and variables -> Actions -> New repository secret
• Name: COPILOT_GITHUB_TOKEN
• Value: a token from one of the accepted token types listed below.

Token-Type Guidance

The @github/copilot CLI accepts the following token prefixes. Classic personal access tokens (ghp_) are rejected at runtime.

Prefix	Token Type	Use in CI
ghs_	GitHub App installation token	Preferred. Short-lived, scoped, auditable
github_pat_	Fine-grained personal access token	Acceptable when a GitHub App is not feasible
gho_, ghu_	OAuth / user-to-server token	Avoid. Tied to a user identity
ghp_	Classic personal access token	Rejected at runtime. The probe fails fast
GITHUB_TOKEN	Actions-issued token	Scope-limited. Not sufficient for vally eval

A GitHub App that mints an installation token in CI is the preferred target state, because a leaked installation token expires in about an hour rather than remaining valid until someone notices.

hve-core has not adopted that pattern for this credential. COPILOT_GITHUB_TOKEN is currently a fine-grained personal access token held as a repository secret, which the table above rates as acceptable where a GitHub App is not feasible. Adopting a GitHub App here requires first confirming that an App can carry the Copilot SDK scopes the CLI needs.

Probe Behavior

scripts/evals/Test-CopilotToken.ps1 runs before any vally eval invocation and exits non-zero with a ::error:: annotation when:

• COPILOT_GITHUB_TOKEN is missing or empty and gh auth token cannot supply a token (fallback for local runs with the GitHub CLI logged in)
• the token begins with ghp_ (classic PAT)
• the optional -SmokeTest switch invokes vally --version and the CLI exits non-zero

The pass-path Reason includes (source: COPILOT_GITHUB_TOKEN) or (source: gh auth token) so contributors can confirm which credential path was used. The smoke test reports a clean skip when vally is not installed locally, so contributors can run the probe outside CI without installing the CLI.

Per-Job COPILOT_HOME Isolation

The @github/copilot CLI persists state (logged-in users, caches) under the directory named by COPILOT_HOME, defaulting to ~/.copilot. CI jobs share runner home directories across steps and can pollute each other when this state leaks.

Export COPILOT_HOME to a job-local path in every workflow job that invokes vally:

env:
  COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN }}
  COPILOT_HOME: ${{ runner.temp }}/copilot-home

This pattern keeps each eval job hermetic, prevents credential bleed-through between matrix legs, and avoids the deprecated --config-dir CLI flag.

Fork PR Policy

GitHub Actions does not expose repository secrets to workflows triggered by pull requests from forks. Without COPILOT_GITHUB_TOKEN, the eval-execute job cannot succeed.

The pipeline clean-skips eval execution for fork PRs rather than failing the check:

jobs:
  eval-execute:
    if: needs.eval-validation.outputs.eval-relevant == 'true' && github.event.pull_request.head.repo.fork == false

The eval-execute job is also skipped for non-eval-relevant PRs (those that change only documentation or other non-AI-artifact paths) through the eval-relevant output gate, independent of the fork policy.

The eval-presence and eval-lint jobs do run on fork PRs because they require no secrets. Structural problems with eval specs (missing coverage, schema violations, profanity in stimulus text) surface immediately. Eval execution itself runs only after a maintainer merges the fork branch into a trusted topic branch on the upstream repository.

Published Artifacts and the Transcript Boundary

Workflow artifacts on a public repository are readable by anyone, and an agent transcript records whatever the agent printed. An agent asked to inspect the workspace can print the job environment, and the eval jobs export COPILOT_GITHUB_TOKEN into that environment. Nothing in the pipeline redacts transcripts.

Eval artifacts therefore publish aggregate results only. These paths are deliberately withheld:

Withheld path	Reason
evals/results/**/results.jsonl	Per-trial trajectories, including raw model output
logs/vally-compare-*.log	Raw judge-run output

What is still published covers the summaries that drive the gate and the per-artifact debugging payloads: logs/eval-summary.json, logs/eval-results-*.json, logs/baseline-equivalence-*.json, and the changed-artifact and stimulus-presence manifests. None of these embed model output.

The cost is that inspecting what an agent actually said during a failed run means re-running it rather than downloading the artifact. Do not re-add the withheld paths to make debugging easier.

Adding a New Eval Spec

When you add or modify an AI artifact under .github/agents/, .github/prompts/, .github/instructions/, or .github/skills/, the eval-presence job fails the PR until a matching eval spec exists.

Steps to add coverage:

1. Create an eval spec under evals/ that follows the structure documented in evals/README.md.

2. Add a stimuli[].tags.<kind> backlink whose value is the artifact slug, where <kind> is one of agent, prompt, instruction, or skill, and the slug is the artifact basename minus its .agent.md, .prompt.md, .instructions.md, or SKILL.md suffix (for example, tags: {agent: code-review-functional} for .github/agents/coding-standards/subagents/code-review-functional.agent.md).

3. Ensure the spec declares an executor compatible with the vally CLI (typically the CopilotSdkExecutor with a model: hint).

4. Run the presence check locally to confirm the artifact is covered:

   pwsh scripts/evals/Get-ChangedAIArtifact.ps1 -BaseRef origin/main -HeadRef HEAD -OutFile logs/changed-ai-artifacts.json
   pwsh scripts/evals/Test-StimulusPresence.ps1 -ManifestPath logs/changed-ai-artifacts.json

5. Run the eval locally (requires COPILOT_GITHUB_TOKEN in your shell environment):

   pwsh scripts/evals/Get-ChangedSpecStimulus.ps1 `
     -BaseRef origin/main `
     -HeadRef HEAD `
     -OutFile logs/changed-spec-stimuli.json
   pwsh scripts/evals/Test-CopilotToken.ps1 -SmokeTest
   pwsh scripts/evals/Invoke-VallyEvals.ps1 `
     -ManifestPath logs/changed-ai-artifacts.json `
     -ChangedSpecManifestPath logs/changed-spec-stimuli.json

Commit the new spec alongside the artifact change. The PR comment summary in eval-execute reports per-artifact pass/fail with links to the captured logs/eval-results-<artifact-id>.json payloads.

Get-ChangedSpecStimulus.ps1 emits a synthetic artifact for every added or modified stimulus in a changed eval spec. Invoke-VallyEvals.ps1 unions those entries with the changed AI artifact manifest and deduplicates them by kind:artifactId. This ensures a changed stimulus runs even when its referenced agent, prompt, instruction, or skill did not change in the same diff.

A single spec can be shared by multiple artifacts: add one stimuli[].tags.<kind> backlink per artifact that the spec covers. When more than one artifact backlinks the same spec, scripts/evals/Invoke-VallyEvals.ps1 runs the spec once per artifact using vally eval --tag kind=slug, so each artifact is scored only on its own stimuli. A spec backlinked by exactly one artifact runs untagged. The per-spec run key reflects this: untagged runs use specRel, and tag-scoped runs use specRel|kind=slug. scripts/evals/Modules/VallyRunner.psm1 computes the backlink count (Get-VallySpecBacklinkCount) and run plan (Get-VallySpecRunPlan).

Stimulus presence linter

scripts/evals/Test-StimulusPresence.ps1 is the gate that fails an eval-presence job when a changed AI artifact lacks an eval spec backlink. It reads the manifest produced by scripts/evals/Get-ChangedAIArtifact.ps1 and builds a coverage index from every evals/**/*.yaml spec.

Each changed artifact is matched against the stimuli[].tags.<kind> = <slug> backlinks in that index. Deleted artifacts (manifest status D) are skipped because coverage cannot be required for removed files.

The script writes a structured report to logs/stimulus-presence.json (covered, missing, errors, skipped) and emits a single ::error file=...:: annotation per missing artifact, so a PR comment names the file that needs coverage.

Exit codes:

Exit	Meaning
0	Every changed artifact is covered, or the manifest is empty or contains only deletions.
1	At least one changed artifact is missing an eval-spec backlink.
2	Invalid input: missing manifest, missing evals/ root, or unrecoverable YAML parse fail.

The -FailOnSpecError switch promotes recoverable YAML parse failures to a hard exit 2 so a malformed spec cannot mask a missing-backlink failure during local hardening sweeps.

Run the linter locally before pushing artifact changes:

pwsh scripts/evals/Get-ChangedAIArtifact.ps1 -BaseRef origin/main -HeadRef HEAD -OutFile logs/changed-ai-artifacts.json
pwsh scripts/evals/Test-StimulusPresence.ps1 -ManifestPath logs/changed-ai-artifacts.json -FailOnSpecError

To add coverage for a missing artifact, create or extend an eval spec under evals/ and set stimuli[].tags.<kind> to the artifact slug (the basename minus the .agent.md, .prompt.md, .instructions.md, or SKILL.md suffix); the next run reports it covered.

Per-Spec Moderation Threshold

The moderation.threshold schema field on an eval spec sets the per-spec Detoxify cutoff (any label score exceeding the value hard-fails the spec):

moderation:
  threshold: 0.7

The validator accepts numeric values in [0.0, 1.0]; out-of-range or non-numeric values emit ModerationThresholdOutOfRange / ModerationThresholdType diagnostics during ci:eval:lint:schema. The default is 0.5 when the field is omitted.

Invoke-VallyEvals.ps1 -ModerationThreshold <value> overrides every spec's threshold for a run. CLI override wins over the per-spec value, which wins over the default.

Content moderation coverage

Content moderation runs in two complementary CI lanes, each scoped to a different surface.

Lane	Job in pr-validation.yml	Script	Toolchain	Surface
Markdown corpus	eval-lint	scripts/evals/Test-EvalSpecText.ps1	Node (retext-equality, retext-profanities)	.github/{agents,prompts,instructions,skills}/**/*.md and docs/**/*.md
Eval-spec stimuli	content-moderation	scripts/evals/Invoke-CorpusModeration.ps1	Python + Detoxify (unitary/toxic-bert)	Stimulus text and expected-output fixtures inside evals/**/*.yaml

The two lanes target different surfaces and do not overlap: the markdown-corpus lane keeps the AI artifacts that ship to contributors free of insensitive or foul language; the eval-spec stimuli lane scores adversarial test inputs against a Detoxify cutoff so a spec that probes a model with toxic content cannot itself ship unredacted.

The content-moderation job is the only path that exercises the real Detoxify model in CI. The job installs the Python dependencies via uv sync --locked in scripts/evals/moderation (declared in its pyproject.toml and uv.lock), caches the Detoxify weights between runs, then invokes Invoke-CorpusModeration.ps1 per spec.

Invoke-CorpusModeration.ps1 shells out to scripts/evals/Invoke-ContentModeration.ps1 for each stimulus. The default Detoxify threshold is 0.5; per-spec overrides come from the moderation.threshold field documented above.

Local opt-in for the Detoxify lane:

uv sync --locked --project scripts/evals/moderation
pwsh scripts/evals/Invoke-CorpusModeration.ps1 -SpecGlob 'evals/**/*.yaml'

Without the Python dependencies installed, Invoke-ContentModeration.ps1 exits 2 with a setup error rather than silently passing. The markdown-corpus lane (Test-EvalSpecText.ps1) requires only Node and runs as the CI-owned ci:eval:lint:text lane without any moderation-environment opt-in.

Eval Lint Scripts

The CI-owned eval-validation workflow runs the static eval-lint lanes. They are not part of validate:local; see Validation Commands and CI-Owned Lanes for local reproduction prerequisites and output handling.

Script	Tool	Purpose
ci:eval:lint:vally	vally lint --eval-spec evals/	Spec validation via the upstream CLI
ci:eval:lint:schema	Test-EvalSpec.ps1	Schema lint, agent-behavior coverage, and orphaned-tag reachability
ci:eval:lint:text	Test-EvalSpecText.ps1	retext-profanities + retext-equality gate on the AI-artifact corpus
ci:eval:lint:safety	Test-VallyTestSafety.ps1	Safety validation for eval stimuli

ci:eval:lint:text scans .github/{agents,prompts,instructions,skills}/**/*.md and docs/**/*.md using separate retext-equality and retext-profanities processors. The alex package is no longer a dependency. Equality findings retain source: alex in the JSON report and emit ::warning annotations by default; the source alias and -FailOnAlex name are retained for existing consumers.

The profanity processor uses sureness: 1, a confidence threshold rather than a severity rating. Rating-0 profanity matches are excluded both with and without -FailOnAlex. Non-allowlisted rating-1 and rating-2 findings are emitted once with source: retext-profanities and cause exit code 1.

Report fields and CLI names remain compatible, but the finding set is intentionally different from the former alex.text() pipeline: profanity findings are no longer duplicated under source: alex, and rating-0 profanity matches are no longer reported. Full historical output equivalence is not claimed.

Pass -FailOnAlex to promote only emitted equality findings to errors. It does not change the profanity threshold or restore excluded matches:

pwsh scripts/evals/Test-EvalSpecText.ps1 -FailOnAlex

Matches admitted by either processor are filtered by the phrase-aware allowlist in scripts/evals/Modules/retext-runner.mjs (PHRASE_ALLOWLIST keyed by retext rule id; ±60-character context window). For example, the allowlist suppresses the equality match in HTTP host and the profanity match in penetration test.

Test-EvalSpecText.ps1 exit codes:

Exit	Meaning
0	No error-level findings; equality findings (source: alex) may still be reported as warnings.
1	At least one emitted profanity finding (source: retext-profanities), or an equality finding with -FailOnAlex.
2	Setup failure (corpus expansion failed, Node shim missing, or node not on PATH).

Schema lint, coverage, and reachability

ci:eval:lint:schema runs three checks in a single pass, and any one of them can fail the lane:

• Schema validation of every spec under evals/, covering required keys, the executor whitelist, moderation.threshold, and stimulus backlink tags.
• Agent-behavior coverage. Every parent (user-invocable) agent under .github/agents/ must have a stimulus partial at evals/agent-behavior/stimuli/<slug>.yml.
• Orphaned-tag reachability. Every agent= and scenario= tag in evals/agent-behavior/eval.yaml must resolve to a slug present in the agent inventory. Unresolvable tags emit ::error annotations and hard-fail the lane.

Test-EvalSpec.ps1 exit codes:

Exit	Meaning
0	Every spec is valid, every parent agent is covered, and no tag is orphaned.
1	A spec failed schema validation, a parent agent lacked a stimulus partial, the inventory was unreadable, or a tag was orphaned.
2	Setup failure (the powershell-yaml module is not installed).

Use -SkipAgentCoverage for fixture-only runs, or -NewAgentsOnly with -BaseRef to enforce coverage incrementally on newly added agents.

Agent inventory

evals/agent-behavior/AGENTS.yml is the source of truth for the orphaned-tag gate. It is generated rather than hand-edited:

pwsh scripts/evals/Build-AgentInventory.ps1 -Force

Enrollment follows two rules:

• Parent agents are always inventoried. An agent file with no user-invocable key is treated as a parent.
• Subagents (user-invocable: false) are inventoried only when a matching stimulus partial exists at evals/agent-behavior/stimuli/<slug>.yml.

When the lane reports an orphaned tag, either the tag is misspelled or the agent it names is not enrolled. Add the agent's stimulus partial when the agent is a subagent, regenerate the inventory, and commit the regenerated AGENTS.yml alongside the change.

Baseline-equivalence specs

ci:eval:lint:vally runs vally lint --eval-spec evals/, which scans recursively to a maximum depth of ten directory levels. The baseline-equivalence suite under evals/baseline-equivalence/ ships its paired specs one level down (baseline/eval.yaml and customized/eval.yaml), so both are discovered by that sweep. Lint either one on its own when iterating on a single spec:

vally lint --eval-spec evals/baseline-equivalence/baseline/eval.yaml
vally lint --eval-spec evals/baseline-equivalence/customized/eval.yaml

scripts/evals/Invoke-BaselineEquivalence.ps1 runs during npm run ci:eval:equivalence and owns environment materialization, seeding, baseline caching, the pinned comparison invocation, and summary generation. It is the only path that materializes the customization surface, so the customized spec must be run through it rather than invoked directly. See evals/baseline-equivalence/README.md for the suite operator guide and driver-output contract.

Matrix, Moderation, and Dashboard Scripts

Beyond the lint lanes, scripts/evals/ holds the scripts that scope runs, moderate artifacts, and render results.

Script	Invoked by	Purpose
Invoke-ArtifactModeration.ps1	ci:eval:moderate:artifacts	Moderates all eval specs plus changed AI artifacts from the changed-artifact manifest
New-AgentMatrixDashboard.ps1	ci:eval:agent:dashboard, ci:eval:agent:report	Renders a self-contained HTML matrix dashboard, one row per inventory agent
New-EquivalenceDashboard.ps1	ci:eval:dashboard	Renders a self-contained HTML dashboard for a baseline-equivalence run
Get-AgentDependencyMap.ps1	Run directly	Builds a JSON map of agent dependencies for the baseline-equivalence dispatcher
Update-AgentMatrixSummariesFromLogs.ps1	Run directly	Rebuilds per-agent matrix summaries from existing vally logs without re-running npx vally

Invoke-ArtifactModeration.ps1 and Invoke-CorpusModeration.ps1 are distinct lanes over the same changed-artifact manifest. Corpus moderation scores stimulus text inside eval specs; artifact moderation covers the specs plus the changed AI artifacts themselves, writing to a separate output file.

ci:eval:agent:report is the end-to-end convenience command: it runs ci:eval:agent:matrix and then ci:eval:agent:dashboard.

Running Pester Tests Locally

npm run test:ps wraps scripts/tests/Invoke-PesterTests.ps1. The default invocation applies ExcludeTag=@('Integration','Slow'):

npm run test:ps                                              # default green-bar (excludes Integration + Slow)
npm run test:ps -- -ExcludeTag Slow                          # include Integration, exclude Slow
npm run test:ps -- -Tag Integration                          # run only Integration-tagged tests
npm run test:ps -- -TestPath scripts/tests/evals/            # scope to one directory

-Tag (with -IncludeTag as an alias) and -ExcludeTag flow through to the inner Pester configuration only when explicitly bound, so omitting them preserves the default exclusion. CI matches the default invocation; opt-in tag overrides are intended for targeted local runs.

Results land in logs/pester-summary.json (overall counts) and logs/pester-failures.json (per-failure detail).

Testing PowerShell Wrappers Around Python Subprocesses

Invoke-ContentModeration.ps1 invokes python through Start-Process in a child pwsh -NoProfile -File boundary. The parent test scope's Mock / function: injections do not cross that boundary, so the test suite at scripts/tests/evals/Invoke-ContentModeration.Tests.ps1 uses a PATH-shimmed stub:

1. Create a temp directory and write python.cmd containing a CMD wrapper that re-launches pwsh against a canned python.ps1.
2. Prepend the temp directory to $env:PATH for the duration of the test.
3. The child process resolves python to the shim, executes python.ps1, and observes real argv (--input, --output, --threshold).

This is the only viable mock boundary for cross-process invocation. Apply the same pattern when adding tests for any PowerShell script that shells out to a Python subprocess.

Test authoring patterns

When authoring new Pester suites for the evals scripts, four patterns recur often enough to call out:

• Define helper functions inside BeforeAll { function ... } so Pester promotes them to the containing Describe scope for all It blocks. Functions defined directly inside Describe (outside BeforeAll) do not survive the fresh runspaces Pester uses for each It.
• When a production script exposes an Invoke-*Core function behind an $MyInvocation.InvocationName -ne '.' guard, dot-source the script in BeforeAll and call the core function directly. This avoids a fresh pwsh startup for every case and lets Pester mocks intercept calls made inside the function.
• When the command under test is invoked through pwsh -File or Start-Process (so the parent runspace cannot install a Mock), declare a bare function at file scope in the test (or in a fixture script the child loads). The PATH-shim pattern above is one instance of this; the scripts/tests/evals/fixtures/stub-vally.ps1 fixture is another.
• When a stub or script under test needs to signal a non-zero exit while $ErrorActionPreference = 'Stop' is in effect, write the diagnostic with [Console]::Error.WriteLine(...) and then call exit <code> explicitly. throw short-circuits the runspace before the intended exit code is set, which causes the parent process to observe exit 1 instead of the contract code.

BeforeAll {
  $script:ScriptPath = Join-Path $PSScriptRoot '../../evals/Build-AgentBehaviorSpec.ps1'
  . $script:ScriptPath
}

It 'reports drift without starting a child pwsh process' {
  $result = Invoke-AgentBehaviorSpecCore -Check
  $result.Outcome | Should -Be 'Drift'
}

The core function returns a PSCustomObject with an Outcome field. Test that result directly; leave exit-code mapping to the guarded main block. Use a child process only when the process boundary itself is part of the behavior under test.

The stub-vally fixture demonstrates the third pattern in practice. scripts/tests/evals/Invoke-VallyEvals.Tests.ps1 drives scripts/evals/Invoke-VallyEvals.ps1 against the fixture by passing -VallyCommand $script:StubPath and setting $env:STUB_VALLY_MODE per scenario. The fixture supports these modes:

• pass - two passing trials, exit 0.
• fail - two failing trials, exit 1.
• fail-noname - two failing trials with no trajectory.stimulus.name, reproducing an empty per-stimulus map, exit 1.
• mixed - one passing and one failing trial, exit 0; the failed trial drives the outer status.
• empty - no trials, exit 0.
• errored - two trials with no gradeResult, exit 1.
• crash - prints an error and exits 99 without writing results.
• per-stim - one trial per STUB_VALLY_STIM_RESULTS_JSON entry.

Per-spec overrides flow through $env:STUB_VALLY_MODES_JSON.

These modes exercise the driver's advisory demotion logic. When vally exits 0, the spec met its aggregate runs/threshold contract, so any per-trial assertion dips counted in assertionsFailed are sub-threshold noise rather than merge blockers; the driver demotes them to advisory, sets status to advisory-fail, and emits a ::warning annotation instead of gating the build. Baseline-equivalence specs resolved via agent tags are likewise advisory (DD-01). A spec gates the build (status fail, promoted to a CI failure) only when it has authoritative grader failures or flagged output moderation.

This lets the stub-mode aggregation tests exercise the real driver code paths (the manifest loop, threshold override, and summary writer) without invoking the vally CLI or paying Copilot SDK costs.

🤖 Crafted with precision by ✨Copilot following brilliant human instruction, then carefully refined by our team of discerning human reviewers.