Grader: prompt (LLM judge)

The prompt grader sends the agent’s trajectory to an LLM (the “judge”) and asks it to evaluate quality against a rubric. This is the most powerful built-in grader — it can assess things no static check can, like whether an explanation is clear or whether code follows best practices.

Taxonomy

Property	Value
Determinism	llm
Cost	high
Reference	reference-free
Temporal scope	trajectory-level
Score kind	llm

Note

Because this grader is llm determinism and high cost, it’s best reserved for nightly / on-demand evaluations rather than included in fast PR-gate runs.

Config

stimuli:
  - name: writes-unit-tests
    prompt: Add unit tests for the parser
    rubric: # the criteria the judge scores
      - Tests cover edge cases (empty input, nulls, errors)
      - Assertions are specific, not just "toBeTruthy"
    graders:
      - type: prompt
        config:
          prompt: Judge only the tests; ignore unrelated refactoring. # supplemental
          model: gpt-5.5 # optional: override judge model
          scoring: scale_1_5 # optional: scoring scale
          threshold: 0.5 # optional: pass threshold
          evidence: [trajectory, diff] # optional judge evidence selector
          output_delivery: workspace # optional: read selected evidence from files

Field	Type	Required	Default	Description
prompt	string	No	(none)	Extra evaluation instructions for the judge. Additive — it does not replace the scored criteria
model	string	No	claude-sonnet-4.6 (or --judge-model)	Which model to use as the judge
scoring	"binary" | "scale_1_5" | "scale_1_10"	No	"scale_1_5"	Scoring scale for the judge’s response
threshold	number	No	0.5	Score threshold for passing (normalized to 0–1)
evidence	("trajectory" | "diff" | "golden_patch" | "repo")[]	No	["trajectory"]	Evidence sections supplied to the judge
output_delivery	"inline" | "workspace"	No	"inline"	How selected evidence is delivered

When evidence is omitted or empty ([]), the judge receives trajectory evidence: agent output, metrics, and session timeline. When evidence is a non-empty list, it is a strict selector:

• trajectory includes Agent Output, Execution Metrics, and Session Timeline.
• diff includes the agent’s cumulative workspace changes as ## Agent Diff.
• golden_patch includes the reference solution declared by the stimulus unless it duplicates the trajectory output, which prevents oracle from judging the applied patch against itself.
• repo includes a bounded snapshot of the final workspace filesystem as ## Repository Files (a file tree plus text-file contents), letting the judge read the agent’s source output directly. Common build artifact and cache directories are excluded by default (.git, __pycache__, node_modules, target, dist, build, .venv, .tox, .gradle, bin, obj, and others), so the snapshot budget is spent on source files. Sensitive filenames (.env*, private keys, package-manager auth configs such as .npmrc) are listed in the tree but their contents are omitted. Per-file, total-content, and file-count caps bound the injected text.

The task is included when the grader has a stimulus, and the rubric is always included. prompt adds optional evaluation instructions. Selected evidence that cannot be supplied fails the grader without calling the judge. Agent Diff and Golden Patch sections each include at most 20,000 characters of patch text, plus a truncation note when needed. Repository Files includes at most 300,000 characters plus the internal per-file, total-content, and file-count caps. Agent Diff, Golden Patch, and Repository Files are nonce-fenced as untrusted judge evidence. Because workspace diffs and the repo snapshot cover the whole run’s final state, diff and repo cannot be combined with grader-level turn or scope.

Note

golden_patch selection is supported by vally grade and vally oracle, where the patch is already resolved for grading. Normal vally eval does not resolve the reference, so this evidence is unavailable there; use diff to judge changes during the inline eval run. repo reads the live workspace, so it works under vally eval and under vally grade when the workspace is on disk.

Evidence delivery

Choose how the selected evidence reaches the judge:

Mode	Best for	Behavior
inline	Most evaluations	Sends evidence in the initial prompt. Lowest latency, but large evidence is truncated to protect the context window.
workspace	Long runs where omitted evidence may matter	Stores complete selected evidence in temporary files and lets the judge retrieve relevant parts with read-only tools.

inline is the default. Its trajectory output is capped at 50,000 characters, long timelines use head/tail windowing and per-event truncation, and selected diff/golden-patch evidence is capped at 20,000 characters.

To opt into workspace delivery:

graders:
  - type: prompt
    config:
      evidence: [trajectory, diff]
      output_delivery: workspace

What workspace mode provides

Vally creates a disposable directory containing only the evidence selected by config.evidence:

evidence/
├── trajectory/            # when trajectory is selected
│   ├── agent-output.txt
│   ├── timeline.jsonl
│   └── metrics.json
├── agent-diff.patch       # when diff is selected
├── golden-patch.patch     # when golden_patch is selected
└── repo-snapshot.txt      # when repo is selected

• agent-output.txt is the complete UTF-8 final output.
• timeline.jsonl contains one complete JSON-compatible trajectory event per line, in execution order.
• metrics.json contains the complete trajectory metrics object.
• Patch files contain complete, untruncated UTF-8 unified diffs.
• repo-snapshot.txt contains the bounded workspace snapshot described above.

The initial prompt still contains the task, rubric, optional evaluation instructions, security guidance, and an exact file map. It does not contain the bulk evidence. The judge uses read-only grep and ranged view calls to find what the rubric requires. A zero-byte diff is valid evidence that the agent made no workspace changes. How much the judge actually retrieved is recorded as metadata.evidence_usage.

The absolute temporary paths are operational metadata sent to the judge and may contain the local account name on platforms whose system temp directory is user-specific. They are deleted after grading — though forced termination of the process may leave one behind — and do not grant access beyond the exact selected files.

The files are complete, but retrieval is bounded so the judge cannot refill its context indiscriminately:

• 12 evidence tool calls per grade;
• 16,000 characters per successful result;
• 64,000 successful evidence characters in total.

Workspace mode improves access to relevant evidence; it does not promise that the judge can ingest every byte.

Caution

Workspace evidence is untrusted agent-produced data. The judge receives only read/search tools scoped to the disposable evidence directory—no shell, writes, network, MCP, or memory—but read-only access does not make the content trustworthy or eliminate prompt-injection risk.

Failure behavior and availability

Workspace setup, serialization, or secured-session failure fails the grader. Vally never silently falls back to truncated inline evidence. The effective mode is recorded as metadata.output_delivery. Cleanup failures preserve a captured verdict, emit a coded warning, and add cleanup metadata.

The mode works anywhere the single-run prompt grader is used, including vally eval, experiment runs, offline vally grade, vally oracle, and direct core-library calls. It is not supported by the panel grader or prompt comparison; comparison rejects output_delivery: workspace rather than silently using inline evidence.

Like inline delivery, a selected golden patch that equals the trajectory output is omitted, so oracle never grades the patch against itself. Workspace mode fails if that leaves nothing to materialize. The judge never receives the agent’s original workspace.

Scoring scales

Scale	Range	Default threshold	When to use
binary	0 or 1	0.5	Simple yes/no judgments
scale_1_5	1–5	0.5 (≈ 3/5)	General-purpose evaluation
scale_1_10	1–10	0.5 (≈ 5.5/10)	Fine-grained quality assessment

Scores are normalized to [0, 1] regardless of scale.

How it works

1. The selected evidence is assembled; trajectory evidence is formatted into a readable timeline with head/tail windowing
2. A system prompt instructs the judge to evaluate against the rubric criteria
3. The judge must call the submit_grade tool exactly once with per-criterion scores and reasoning
4. If the judge fails to call the tool or sends invalid arguments, an in-session reminder nudges it to retry (up to 2 times)
5. Per-criterion results are mapped to GraderResult.details sub-checks
6. The overall score is normalized to [0, 1]

Judge model resolution

The model used for judging follows this priority chain:

1. Grader-level config.model in eval.yaml (most specific)
2. --judge-model CLI flag
3. defaults.judge_model in eval.yaml (global default for all LLM graders)
4. EVAL_JUDGE_MODEL environment variable
5. Default: claude-sonnet-4.6

eval.yaml

defaults:
  model: gpt-5.5 # agent execution model
  judge_model: gpt-5.5 # default judge model for all LLM graders

stimuli:
  - name: test-case
    graders:
      - type: prompt
        config:
          model: o3 # this specific grader uses o3
      - type: prompt # this one uses gpt-5.5 (from judge_model)

Judge reasoning effort

Set defaults.judge_reasoning_effort to control the reasoning effort of the judge model (low, medium, high, or xhigh). It applies only to the eval-level judge_model — graders that pin their own config.model keep the model’s default effort. When unset, the judge runs at whatever the model’s default effort is, which is neither controlled nor recorded. The effective value is recorded in the grader result metadata as reasoning_effort.

eval.yaml

defaults:
  judge_model: claude-opus-4.6
  judge_reasoning_effort: high # judge deliberates harder, reproducibly

A per-grader config.reasoning_effort overrides the eval-level default and is the natural companion to a per-grader model:

eval.yaml

graders:
  - type: prompt
    config:
      model: o3
      reasoning_effort: high # effort for this grader's own judge model

Reasoning effort only takes effect on judge models that support it. vally does not validate this — the value is passed through to the model provider, which may ignore or reject it on an unsupporting model. Consult your model provider’s documentation for which models support reasoning effort and which levels they accept.

Rubric and evaluation criteria

The judge scores the stimulus rubric — one sub-result per criterion. config.prompt does not define those criteria: it is appended as a separate instructions section, so the judge reads it but still scores the rubric.

When a stimulus defines no rubric, the judge falls back to a built-in default:

• The agent completed the requested task correctly
• The output is clear and well-structured

The fallback is reported as rubric_source: "default" in the result metadata, and vally lint warns when it is combined with a config.prompt — that pairing usually means criteria went into the wrong field.

Put the criteria in rubric and use prompt for guidance that spans them:

stimuli:
  - name: writes-unit-tests
    prompt: Add unit tests for the parser
    rubric:
      - Tests cover edge cases (empty input, nulls, errors)
      - Assertions are specific, not just "toBeTruthy"
      - Tests actually run (valid syntax, proper imports)
    graders:
      - type: prompt
        config:
          prompt: Judge only the tests; ignore unrelated refactoring.

Evidence examples

Each rubric criterion becomes a sub-result in details, named after the criterion text (whitespace collapsed, truncated to 50 characters):

prompt (score: 0.75)
  ↳ prompt/Tests cover edge cases: 1.00
  ↳ prompt/Assertions are specific: 0.75
  ↳ prompt/Tests actually run: 0.50

Scores are normalized to 0–1 across the scoring scale’s range, so on the default scale_1_5 a raw 4 becomes (4 - 1) / (5 - 1) = 0.75, and a raw 2 becomes 0.25.

Retry behavior

LLM calls can fail due to rate limits or transient errors. The prompt grader retries with exponential backoff:

• Up to 2 retries (3 total attempts)
• Exponential backoff with jitter (5s → 10s → 20s + random 0–1s)
• 10 minute total budget — won’t retry past this limit
• If all retries fail, the grader returns a failed result (score 0) with the error in evidence, rather than crashing the eval

Tip

If you’re seeing frequent retry failures, check your API rate limits. Consider using a cheaper model for judging (e.g., gpt-5.5-mini) or reducing the number of stimuli that use LLM judges.

Cost considerations

Every trial graded by a prompt grader makes at least one LLM API call. With multi-trial:

Cost ≈ (num_stimuli × runs × prompt_graders_per_stimulus) × per-call cost

For example, 5 stimuli × 5 runs × 1 prompt grader = 25 LLM judge calls per eval.

Use --judge-model to control costs: a smaller model for iteration, a larger model for final evaluation.

Comparison mode

The prompt grader also powers head-to-head comparison: instead of scoring one trajectory, it judges a baseline against a treatment for the same stimulus and rubric, and reports which is better and by how much. Run it with vally compare — any stimulus with a rubric can be compared.

Comparison mode always evaluates both runs’ output, metrics, and session timelines. It does not use diff or golden_patch evidence selections; the CLI warns when either is configured for a compared stimulus. Comparison uses inline evidence. A compared prompt grader configured with output_delivery: workspace is rejected by comparison; remove the field to compare the runs.

How it works

1. A baseline trajectory and a treatment trajectory for the same stimulus are loaded (from an experiment’s variants, or two independent runs).
2. The judge compares them against the rubric and submits a verdict via a submit_comparison_grade tool call.
3. To remove order bias, the comparison runs twice with the two responses swapped (position-swap debiasing). If the directions disagree on the winner, the result is a tie; if they agree on the winner but not the magnitude, the weaker magnitude wins.

Scoring scale

The verdict is signed and treatment-relative, in [-1, 1]:

Verdict	Score	Meaning
much better	+1.0	treatment is clearly better
slightly better	+0.4	treatment is somewhat better
equal / tie	0	no meaningful difference
slightly worse	-0.4	baseline is somewhat better
much worse	-1.0	baseline is clearly better

See the compare CLI reference for usage, statistics, and examples.