CLI: grade

Usage

vally eval --output jsonl | vally grade --eval-spec eval.yaml [options]
vally grade --eval-spec eval.yaml [options] < outcomes.jsonl

Description

Grade trajectories read from stdin. Accepts trial-result JSONL from vally eval --output jsonl, legacy EvalOutcome JSONL, a single EvalOutcome JSON object from eval, ATIF, or a session run from an agent like Copilot CLI. The input format is auto-detected from the JSON shape. This lets you iterate on graders without re-running the (expensive) agent execution.

The command reads from stdin and matches each trajectory’s stimulus to the graders defined in the eval spec.

Options

Flag	Type	Required	Description
--eval-spec, -e <path>	string	Yes	Path to eval spec file
--stimulus <name>	string	No	Stimulus name to grade against (ATIF or session run input only; required when prompt matches multiple stimuli)
--workspace <path>	string	No	Path to the agent workspace directory for file-based graders (ATIF or session run input only; defaults to the directory in the session run, or process.cwd() for ATIF)
--run-dir <path>	string	No	Directory of a vally eval --output-dir run. Resolves trajectory.diffPath references in JSONL input so diff graders can re-grade from sidecar files
--artifact-dir <path>	string	No	Stamp trajectory.artifactDir on every input trajectory. Points artifact-reading graders (e.g. custom-metrics) at an output directory that lives outside the graded workspace. Distinct from --run-dir, which resolves diffPath sidecars from an eval --output-dir run
--diff-file <path>	string	No	Read a unified diff from this file and stamp it as trajectory.diff on every input. Injects the workspace diff that ATIF/session-run inputs never carry (and re-graded outcomes may lack), enabling diff-based graders offline
--output jsonl	string	No	Emit graded results as JSONL (one outcome per line)
--judge-model <model>	string	No	Model for LLM judge graders
--judge-reasoning-effort <level>	string	No	Reasoning effort for the judge model: low, medium, high, or xhigh
--grader-plugin <specifier>	string	No	Grader plugin to load (npm package name or local path). Repeatable.
--param <key=value>	string	No	Set a param value (repeatable, e.g. --param MODEL=gpt-4o). Overrides eval param files and .vally.yaml.
--verbose	boolean	No	Show detailed grader evidence
--require-pass	boolean	No	Exit 1 when any aggregate stimulus verdict fails under the eval’s scoring policy. Without this flag, valid passing and failing verdicts both exit 0. Input and grading errors always exit 1.

Offline re-grading

grade is the home for offline re-grading — running graders against a saved trajectory without re-running the (expensive) agent. Pipe a saved EvalOutcome/trial-result stream back in and tweak your graders freely:

vally eval --eval-spec eval.yaml --output jsonl > run.jsonl
# ...edit your graders in eval.yaml...
vally grade --eval-spec eval.yaml < run.jsonl

Output- and trajectory-based graders (output-contains, output-matches, tool-call, skill-invocation, prompt, …) read everything they need from the saved trajectory, so this “just works” with no workspace.

File-based graders (file-exists, file-contains, run-command, …) read the agent’s workspace on disk. Because the agent’s file changes aren’t stored in the trajectory, re-grading them offline requires the workspace to still exist:

• For ATIF and session-run input, pass --workspace <path> to point the graders at a preserved workspace.
• To verify graders against a known-good reference solution instead (harness verification / CI gate), use vally oracle with a golden patch.

Grading pre-hydrated output directories

When a benchmark harness leaves a completed run on disk, the agent’s workspace diff and metric artifacts live outside the trajectory — and ATIF/session-run inputs carry neither. Two flags wire those assets in without re-running the agent:

• --diff-file <path> reads a unified diff (e.g. output/changes.patch) and stamps it as trajectory.diff, so diff-based graders (prompt/panel selecting diff evidence) can judge the agent’s real changes offline.
• --artifact-dir <path> stamps trajectory.artifactDir, so artifact-reading graders such as custom-metrics look up their files (e.g. custom_metrics.json) in that directory. Grading is strict: the artifact directory is authoritative, so a file missing there fails the grader rather than silently falling back to a (possibly stale) workspace copy.

Both are explicit overrides and apply to every input trajectory regardless of input kind (ATIF, session-run, or EvalOutcome/trial-result). They target a single offline re-grade, so grade warns if you pipe multi-record (batch) input while setting either flag — every record would be graded against the same diff or artifact directory:

vally grade --eval-spec eval.yaml \
  --diff-file output/changes.patch \
  --artifact-dir output \
  < output/trajectories/trajectory.json

Both flags are validated before grading begins: once stdin is read, grade exits 1 if --diff-file cannot be read, or if --artifact-dir does not exist or is not a directory — so no trajectory is graded against a bad path. Because these overrides target offline grading — where the workspace is usually absent or stale — a missing path fails fast rather than letting graders silently fall back to a different workspace file and report a misleading result. For the same reason, --artifact-dir disables the artifact-dir-first, workspace-fallback behavior these graders normally use: when you point at an artifact directory explicitly, a metrics file that is absent there fails the grader instead of resolving against the workspace. (The executor-populated trajectory.artifactDir from a live run keeps the lenient fallback.)

--diff-file supplies only a cumulative workspace diff. It cannot synthesize the per-turn evidence required by a multi-turn diff-contains, diff-not-contains, or diff-empty grader. Re-grade those checks from Vally-produced JSONL and pass --run-dir when their turn diffs were written as sidecars.

Reference-based grading with a golden patch

If a stimulus declares a golden_patch (a reference-solution diff), grade resolves it for prompt and panel graders. A grader selecting golden_patch in config.evidence (for example, evidence: [trajectory, golden_patch]) receives it as a Reference Solution section, so the judge compares the agent’s real output against the known-good answer — improving judging consistency on open-ended tasks.

If its path cannot be read, grade warns. A prompt/panel grader selecting golden_patch in config.evidence fails before calling the judge, because selected evidence cannot be silently omitted. (Under vally oracle the same patch is the artifact under test, so a missing patch there is a hard error.)

Exit codes

Code	Meaning
0	All requested graders ran successfully, regardless of whether valid scores passed or failed.
1	An input, configuration, or grading error occurred; or --require-pass was set and an aggregate stimulus verdict failed.

--require-pass requires every aggregate stimulus verdict to pass under the eval’s configured scoring policy. With a scoring threshold, the aggregate passes when its possibly weighted score meets that threshold, even if an individual grader failed. Without a threshold, the aggregate uses its all-graders-passed flag. Grader execution errors are reported as ERROR and exit 1.

Examples

# Pipe directly from eval
vally eval --eval-spec eval.yaml --skip-grade --output jsonl \
  | vally grade --eval-spec eval.yaml

# Grade a saved JSONL file
vally grade --eval-spec eval.yaml < results/outcomes.jsonl

# Grade with verbose output
vally grade --eval-spec eval.yaml --verbose < results/outcomes.jsonl

# Grade and re-emit as JSONL (for downstream processing)
vally grade --eval-spec eval.yaml --output jsonl < results/outcomes.jsonl > graded.jsonl

# Grade an ATIF trajectory (auto-detected from schema_version)
vally grade --eval-spec eval.yaml < trajectory.json

# Grade ATIF with explicit stimulus and workspace
vally grade --eval-spec eval.yaml --stimulus basic --workspace /path/to/workspace < trajectory.json

# Grade a Copilot CLI session run, by id (auto-resolves from ~/.copilot/session-state/)
echo '{"sessionKind":"copilot-cli","sessionId":"abc123"}' | vally grade --eval-spec eval.yaml

# Grade a Copilot CLI session run by explicit path
echo '{"sessionKind":"copilot-cli","sessionPath":"/path/to/events.jsonl"}' | vally grade --eval-spec eval.yaml

# Grade a Copilot CLI session run with explicit stimulus and workspace
echo '{"sessionKind":"copilot-cli","sessionId":"abc123"}' | \
  vally grade --eval-spec eval.yaml --stimulus task-name --workspace /path/to/workspace

Input formats

The grade command auto-detects the input shape from the JSON content and accepts:

1. trial-result JSONL — current vally eval --output jsonl output, including trial-result records and typed metadata such as run-summary
2. Legacy EvalOutcome JSONL — one JSON object per line, as emitted by older eval --output jsonl flows
3. Single EvalOutcome JSON — a whole-file JSON object
4. ATIF JSON — an Agent Trajectory Interchange Format document (identified by schema_version: "ATIF-...")
5. Session Run JSON — an explicit session run reference (sessionKind: "copilot-cli" with either sessionId, set to the session ID or sessionPath, an explicit path to a copilot session log)

Session Run format

A session run provides a reference to Copilot CLI event logs:

{
  "sessionKind": "copilot-cli",
  "sessionId": "abc123"
}

Or with an explicit path:

{
  "sessionKind": "copilot-cli",
  "sessionPath": "/path/to/events.jsonl"
}

When sessionId is provided, the events file is auto-resolved from ~/.copilot/session-state/<sessionId>/events.jsonl. Specify sessionPath to provide an explicit location. Do not set both sessionId and sessionPath.

Mixing formats

ATIF documents and EvalOutcome records must not be mixed in the same stream; grade will exit with an error if both shapes appear together. Session runs cannot be mixed with EvalOutcomes or ATIF.

Output format

━━━ basic-test-generation ━━━
✅ basic-test-generation (2/2 graders passed)
  ✓ [file-exists] Files matching 'add.test.js' found: add.test.js
  ✓ [output-contains] 'test' found in output

Score: 100.0% | PASSED

On failure:

━━━ basic-test-generation ━━━
❌ basic-test-generation (1/2 graders passed, 1 failed)
  ✓ [output-contains] 'test' found in output
  ✗ [file-exists] No files matching 'add.test.js' found

Score: 33.3% | FAILED

Workflow

The typical workflow with grade:

1. Run eval with --skip-grade --output jsonl > outcomes.jsonl to capture trajectories
2. Iterate on your eval.yaml graders
3. Re-grade: vally grade --eval-spec eval.yaml < outcomes.jsonl
4. Repeat until graders work correctly, then run the full eval