CLI: grade

Usage

vally eval --output jsonl | vally grade --eval-spec eval.yaml [options]
vally grade --eval-spec eval.yaml [options] < outcomes.jsonl

Description

Grade trajectories read from stdin. Accepts trial-result JSONL from vally eval --output jsonl, legacy EvalOutcome JSONL, a single EvalOutcome JSON object from eval, ATIF, or a session run from an agent like Copilot CLI. The input format is auto-detected from the JSON shape. This lets you iterate on graders without re-running the (expensive) agent execution.

The command reads from stdin and matches each trajectory’s stimulus to the graders defined in the eval spec.

Options

Flag	Type	Required	Description
--eval-spec, -e <path>	string	Yes	Path to eval spec file
--stimulus <name>	string	No	Stimulus name to grade against (ATIF or session run input only; required when prompt matches multiple stimuli)
--workspace <path>	string	No	Path to the agent workspace directory for file-based graders (ATIF or session run input only; defaults to the directory in the session run, or process.cwd() for ATIF)
--run-dir <path>	string	No	Directory of a vally eval --output-dir run. Resolves trajectory.diffPath references in JSONL input so diff graders can re-grade from sidecar files
--artifact-dir <path>	string	No	Stamp trajectory.artifactDir on every input trajectory. Points artifact-reading graders (e.g. custom-metrics) at an output directory that lives outside the graded workspace. Distinct from --run-dir, which resolves diffPath sidecars from an eval --output-dir run
--diff-file <path>	string	No	Read a unified diff from this file and stamp it as trajectory.diff on every input. Injects the workspace diff that ATIF/session-run inputs never carry (and re-graded outcomes may lack), enabling diff-based graders offline
--output jsonl	string	No	Emit graded results as JSONL (one outcome per line)
--judge-model <model>	string	No	Model for LLM judge graders
--judge-reasoning-effort <level>	string	No	Reasoning effort for the judge model: low, medium, high, or xhigh
--grader-plugin <specifier>	string	No	Grader plugin to load (npm package name or local path). Repeatable.
--param <key=value>	string	No	Set a param value (repeatable, e.g. --param MODEL=gpt-4o). Overrides eval param files and .vally.yaml.
--verbose	boolean	No	Show detailed grader evidence

Offline re-grading

grade is the home for offline re-grading — running graders against a saved trajectory without re-running the (expensive) agent. Pipe a saved EvalOutcome/trial-result stream back in and tweak your graders freely:

vally eval --eval-spec eval.yaml --output jsonl > run.jsonl
# ...edit your graders in eval.yaml...
vally grade --eval-spec eval.yaml < run.jsonl

Output- and trajectory-based graders (output-contains, output-matches, tool-call, skill-invocation, prompt, …) read everything they need from the saved trajectory, so this “just works” with no workspace.

File-based graders (file-exists, file-contains, run-command, …) read the agent’s workspace on disk. Because the agent’s file changes aren’t stored in the trajectory, re-grading them offline requires the workspace to still exist:

• For ATIF and session-run input, pass --workspace <path> to point the graders at a preserved workspace.
• To verify graders against a known-good reference solution instead (harness verification / CI gate), use vally oracle with a golden patch.

Grading pre-hydrated output directories

When a benchmark harness leaves a completed run on disk, the agent’s workspace diff and metric artifacts live outside the trajectory — and ATIF/session-run inputs carry neither. Two flags wire those assets in without re-running the agent:

• --diff-file <path> reads a unified diff (e.g. output/changes.patch) and stamps it as trajectory.diff, so diff-based graders (prompt/panel selecting diff evidence) can judge the agent’s real changes offline.
• --artifact-dir <path> stamps trajectory.artifactDir, so artifact-reading graders such as custom-metrics look up their files (e.g. custom_metrics.json) in that directory. Grading is strict: the artifact directory is authoritative, so a file missing there fails the grader rather than silently falling back to a (possibly stale) workspace copy.

Both are explicit overrides and apply to every input trajectory regardless of input kind (ATIF, session-run, or EvalOutcome/trial-result). They target a single offline re-grade, so grade warns if you pipe multi-record (batch) input while setting either flag — every record would be graded against the same diff or artifact directory:

vally grade --eval-spec eval.yaml \
  --diff-file output/changes.patch \
  --artifact-dir output \
  < output/trajectories/trajectory.json

Both flags are validated before grading begins: once stdin is read, grade exits 1 if --diff-file cannot be read, or if --artifact-dir does not exist or is not a directory — so no trajectory is graded against a bad path. Because these overrides target offline grading — where the workspace is usually absent or stale — a missing path fails fast rather than letting graders silently fall back to a different workspace file and report a misleading result. For the same reason, --artifact-dir disables the artifact-dir-first, workspace-fallback behavior these graders normally use: when you point at an artifact directory explicitly, a metrics file that is absent there fails the grader instead of resolving against the workspace. (The executor-populated trajectory.artifactDir from a live run keeps the lenient fallback.)

--diff-file supplies only a cumulative workspace diff. It cannot synthesize the per-turn evidence required by a multi-turn diff-contains, diff-not-contains, or diff-empty grader. Re-grade those checks from Vally-produced JSONL and pass --run-dir when their turn diffs were written as sidecars.

Reference-based grading with a golden patch

If a stimulus declares a golden_patch (a reference-solution diff), grade resolves it for prompt and panel graders. A grader selecting golden_patch in config.evidence (for example, evidence: [trajectory, golden_patch]) receives it as a Reference Solution section, so the judge compares the agent’s real output against the known-good answer — improving judging consistency on open-ended tasks.

If its path cannot be read, grade warns. A prompt/panel grader selecting golden_patch in config.evidence fails before calling the judge, because selected evidence cannot be silently omitted. (Under vally oracle the same patch is the artifact under test, so a missing patch there is a hard error.)

Exit codes

Code	Meaning
0	All graders passed
1	One or more graders failed, or an error occurred

Examples

# Pipe directly from eval
vally eval --eval-spec eval.yaml --skip-grade --output jsonl \
  | vally grade --eval-spec eval.yaml

# Grade a saved JSONL file
vally grade --eval-spec eval.yaml < results/outcomes.jsonl

# Grade with verbose output
vally grade --eval-spec eval.yaml --verbose < results/outcomes.jsonl

# Grade and re-emit as JSONL (for downstream processing)
vally grade --eval-spec eval.yaml --output jsonl < results/outcomes.jsonl > graded.jsonl

# Grade an ATIF trajectory (auto-detected from schema_version)
vally grade --eval-spec eval.yaml < trajectory.json

# Grade ATIF with explicit stimulus and workspace
vally grade --eval-spec eval.yaml --stimulus basic --workspace /path/to/workspace < trajectory.json

# Grade a Copilot CLI session run, by id (auto-resolves from ~/.copilot/session-state/)
echo '{"sessionKind":"copilot-cli","sessionId":"abc123"}' | vally grade --eval-spec eval.yaml

# Grade a Copilot CLI session run by explicit path
echo '{"sessionKind":"copilot-cli","sessionPath":"/path/to/events.jsonl"}' | vally grade --eval-spec eval.yaml

# Grade a Copilot CLI session run with explicit stimulus and workspace
echo '{"sessionKind":"copilot-cli","sessionId":"abc123"}' | \
  vally grade --eval-spec eval.yaml --stimulus task-name --workspace /path/to/workspace

Input formats

The grade command auto-detects the input shape from the JSON content and accepts:

1. trial-result JSONL — current vally eval --output jsonl output, including trial-result records and typed metadata such as run-summary
2. Legacy EvalOutcome JSONL — one JSON object per line, as emitted by older eval --output jsonl flows
3. Single EvalOutcome JSON — a whole-file JSON object
4. ATIF JSON — an Agent Trajectory Interchange Format document (identified by schema_version: "ATIF-...")
5. Session Run JSON — an explicit session run reference (sessionKind: "copilot-cli" with either sessionId, set to the session ID or sessionPath, an explicit path to a copilot session log)

Session Run format

A session run provides a reference to Copilot CLI event logs:

{
  "sessionKind": "copilot-cli",
  "sessionId": "abc123"
}

Or with an explicit path:

{
  "sessionKind": "copilot-cli",
  "sessionPath": "/path/to/events.jsonl"
}

When sessionId is provided, the events file is auto-resolved from ~/.copilot/session-state/<sessionId>/events.jsonl. Specify sessionPath to provide an explicit location. Do not set both sessionId and sessionPath.

Mixing formats

ATIF documents and EvalOutcome records must not be mixed in the same stream; grade will exit with an error if both shapes appear together. Session runs cannot be mixed with EvalOutcomes or ATIF.

Output format

━━━ basic-test-generation ━━━
✅ basic-test-generation (2/2 graders passed)
  ✓ [file-exists] Files matching 'add.test.js' found: add.test.js
  ✓ [output-contains] 'test' found in output

Score: 100.0% | PASSED

On failure:

━━━ basic-test-generation ━━━
❌ basic-test-generation (1/2 graders passed, 1 failed)
  ✓ [output-contains] 'test' found in output
  ✗ [file-exists] No files matching 'add.test.js' found

Score: 33.3% | FAILED

Workflow

The typical workflow with grade:

1. Run eval with --skip-grade --output jsonl > outcomes.jsonl to capture trajectories
2. Iterate on your eval.yaml graders
3. Re-grade: vally grade --eval-spec eval.yaml < outcomes.jsonl
4. Repeat until graders work correctly, then run the full eval