Grader: prompt (LLM judge)

The prompt grader sends the agent’s trajectory to an LLM (the “judge”) and asks it to evaluate quality against a rubric. This is the most powerful built-in grader — it can assess things no static check can, like whether an explanation is clear or whether code follows best practices.

Taxonomy

Property	Value
Determinism	llm
Cost	high
Reference	reference-free
Temporal scope	trajectory-level
Score kind	llm

Note

Because this grader is llm determinism and high cost, it’s best reserved for nightly / on-demand evaluations rather than included in fast PR-gate runs.

Authentication

LLM judges use a separate Copilot client from the agent being evaluated. Without an explicit judge provider, they use Copilot authentication.

For headless environments, supply a Copilot-enabled token as described in Add to CI. Credentials configured only for the evaluated agent or its provider do not automatically configure the judge.

Config

stimuli:
  - name: writes-unit-tests
    prompt: Add unit tests for the parser
    rubric: # the criteria the judge scores
      - Tests cover edge cases (empty input, nulls, errors)
      - Assertions are specific, not just "toBeTruthy"
    graders:
      - type: prompt
        config:
          prompt: Judge only the tests; ignore unrelated refactoring. # supplemental
          model: gpt-5.5 # optional: override judge model
          scoring: scale_1_5 # optional: scoring scale
          threshold: 0.5 # optional: pass threshold
          evidence: [trajectory, diff] # optional judge evidence selector

Field	Type	Required	Default	Description
prompt	string	No	(none)	Extra evaluation instructions for the judge. Additive — it does not replace the scored criteria
model	string	No	gpt-5.6-terra (or --judge-model)	Which model to use as the judge
scoring	"binary" | "scale_1_5" | "scale_1_10"	No	"scale_1_5"	Scoring scale for the judge’s response
threshold	number	No	0.5	Score threshold for passing (normalized to 0–1)
evidence	("trajectory" | "diff" | "golden_patch" | "repo")[]	No	["trajectory"]	Evidence sections supplied to the judge

Timeout

Set defaults.judge_timeout to control the timeout for each prompt judge turn, or override it for one eval run with vally eval --judge-timeout <duration>:

defaults:
  judge_timeout: 5m

The default is 2m.

When evidence is omitted or empty ([]), the judge receives trajectory evidence: agent output, metrics, and session timeline. When evidence is a non-empty list, it is a strict selector:

• trajectory includes Agent Output, Execution Metrics, and Session Timeline.
• diff includes the agent’s cumulative workspace changes.
• golden_patch includes the reference solution declared by the stimulus, unless it is identical to the trajectory output — judging a patch against itself is circular, so it is dropped. vally oracle always produces that match when it synthesizes a trajectory from the patch; with a golden_trajectory it matches only when that trajectory’s own output is the patch text. When the patch is dropped and nothing else is selected, the grader fails rather than score the rubric alone; pair golden_patch with trajectory, diff, or repo. Under vally oracle, trajectory is what presents the patch to the judge as the output being graded.
• repo includes a bounded snapshot of the final workspace filesystem (a file tree plus text-file contents), letting the judge read the agent’s source output directly. Common build artifact and cache directories are excluded by default (.git, __pycache__, node_modules, target, dist, build, .venv, .tox, .gradle, bin, obj, and others), so the snapshot budget is spent on source files. Sensitive filenames (.env*, private keys, package-manager auth configs such as .npmrc) are listed in the tree but their contents are omitted. Per-file, total-content, and file-count caps bound the captured snapshot.

The task is included when the grader has a stimulus, and the rubric is always included. prompt adds optional evaluation instructions. Selected evidence that cannot be supplied fails the grader without calling the judge. Selected captured output, events, metrics, and patches are copied in full to temporary evidence files, not truncated to fit the initial prompt. The repo snapshot retains its existing capture limits. All evidence is untrusted. Because workspace diffs and the repo snapshot cover the whole run’s final state, diff and repo cannot be combined with grader-level turn or scope.

Note

golden_patch selection is supported by vally grade and vally oracle, where the patch is already resolved for grading. Normal vally eval does not resolve the reference, so this evidence is unavailable there; use diff to judge changes during the eval run. repo reads the live workspace, so it works under vally eval and under vally grade when the workspace is on disk.

Evidence delivery

Vally automatically gives the judge temporary evidence files to read, for small inputs as well as large ones. Evidence is never pasted into the initial prompt. You choose what evidence to include with config.evidence, not how to deliver it. Remove output_delivery from existing configurations. output_delivery: workspace still works but produces a deprecation warning; output_delivery: inline is rejected because inline delivery is no longer supported.

graders:
  - type: prompt
    config:
      evidence: [trajectory, diff]

What the evidence files provide

Vally creates a separate directory in the system’s temporary directory, outside the agent’s working directory. It contains only the selected evidence:

evidence/
├── trajectory/            # when trajectory is selected
│   ├── agent-output.txt
│   ├── timeline.jsonl
│   └── metrics.json
├── agent-diff.patch       # when diff is selected
├── golden-patch.patch     # when golden_patch is selected
└── repo-snapshot.txt      # when repo is selected

• agent-output.txt is the complete UTF-8 final output.
• timeline.jsonl contains one complete JSON-compatible trajectory event per line, in execution order. These are Vally’s recorded events, such as tool calls and responses, not raw executor logs. A grader scoped to a turn or subagent receives only that scope’s events.
• metrics.json contains the complete trajectory metrics object.
• Patch files contain complete, untruncated UTF-8 unified diffs.
• repo-snapshot.txt contains the bounded workspace snapshot described above.

The initial prompt contains the task, rubric, optional evaluation instructions, security guidance, and the evidence file paths. The judge uses grep to search the files and view to read selected lines. These tools can read only the exact selected files, not the agent’s working directory, and cannot modify files. An empty diff file means the agent made no workspace changes.

Vally deletes the temporary copy after the judge call finishes or fails. It does not save these copies for re-grading or change existing results files. Copying evidence cannot recover anything the executor never captured. Cleanup failures or forced termination can leave temporary files behind. The file paths sent to the judge may contain your local account name if your system uses a user-specific temporary directory.

Read limits and isolation

Storing complete files does not mean the judge reads them in full. Each judge session has these read limits:

• 12 evidence tool calls;
• 16,000 characters per successful result;
• 64,000 successful evidence characters in total.

Each panel judge and each comparison pass gets its own budget. A single very long line can be truncated when read, even though the file itself is complete. Files let the judge find relevant evidence, but do not guarantee it reads every byte. Read usage is recorded as metadata.evidence_usage.

The built-in judge runs separately without loading the agent’s custom instructions or configuration. Reading files adds tool calls and may change latency, token usage, and judgments compared with past versions that pasted evidence into the prompt.

Caution

Evidence files are untrusted agent-produced data. In addition to submitting its verdict, the judge can only read and search the selected files. It has no shell, file-writing, network, external-service, or memory tools. These restrictions do not prevent evidence from containing misleading instructions (prompt injection).

Failure behavior and availability

If Vally cannot prepare the evidence files or start the judge session, the grader fails rather than pasting evidence into the prompt. Cleanup failures produce warnings and result metadata without replacing a verdict already received. If shutting down the judge session fails or times out, Vally still attempts to delete the files; it cannot confirm that the session has stopped.

This delivery applies to vally eval, experiment runs, offline vally grade, vally oracle, and library calls. Panel grading and prompt comparison use it too. A comparison timeout can return while its judge call is still running; its files remain until that call finishes.

Library integrations

Custom LlmClient implementations must set supportsWorkspaceDelivery: true and restrict reads to the supplied evidence files, with no writes or other tools except the verdict tool. This is a capability declaration, not a delivery setting: it does not default to true. The built-in client explicitly declares support; a custom client with a missing or false capability fails before a judge call. There is no inline fallback to enable by disabling this capability.

Direct JavaScript callers should delete output_delivery rather than set it to undefined. Only the legacy value "workspace" is accepted, with a deprecation warning. Results no longer include metadata.output_delivery; file delivery is always used.

Scoring scales

Scale	Range	Default threshold	When to use
binary	0 or 1	0.5	Simple yes/no judgments
scale_1_5	1–5	0.5 (≈ 3/5)	General-purpose evaluation
scale_1_10	1–10	0.5 (≈ 5.5/10)	Fine-grained quality assessment

Scores are normalized to [0, 1] regardless of scale.

How it works

1. The selected captured evidence is copied into private temporary files
2. A system prompt instructs the judge to evaluate against the rubric criteria
3. The judge must call the submit_grade tool exactly once with per-criterion scores and reasoning
4. If the judge fails to call the tool or sends invalid arguments, an in-session reminder nudges it to retry (up to 2 times)
5. Per-criterion results are mapped to GraderResult.details sub-checks
6. The overall score is normalized to [0, 1]

Judge model resolution

The model used for judging follows this priority chain:

1. Grader-level config.model in eval.yaml (most specific)
2. --judge-model CLI flag
3. defaults.judge_model in eval.yaml (global default for all LLM graders)
4. EVAL_JUDGE_MODEL environment variable
5. Default: gpt-5.6-terra

eval.yaml

defaults:
  model: gpt-5.5 # agent execution model
  judge_model: gpt-5.6-terra # default judge model for all LLM graders

stimuli:
  - name: test-case
    graders:
      - type: prompt
        config:
          model: o3 # this specific grader uses o3
      - type: prompt # this one uses gpt-5.6-terra (from judge_model)

Judge reasoning effort

Set defaults.judge_reasoning_effort to control the reasoning effort of the judge model. The value may be any non-empty string supported by the model; common values include none, minimal, low, medium, high, xhigh, and max. It applies only to the eval-level judge_model — graders that pin their own config.model keep the model’s default effort. When unset, the judge runs at whatever the model’s default effort is, which is neither controlled nor recorded. The effective value is recorded in the grader result metadata as reasoning_effort.

eval.yaml

defaults:
  judge_model: gpt-5.6-terra
  judge_reasoning_effort: high # judge deliberates harder, reproducibly

A per-grader config.reasoning_effort overrides the eval-level default and is the natural companion to a per-grader model:

eval.yaml

graders:
  - type: prompt
    config:
      model: o3
      reasoning_effort: high # effort for this grader's own judge model

Reasoning effort only takes effect on judge models that support it. For the default Copilot provider, vally checks explicit model capability metadata and rejects a known-unsupported effort before creating the judge session. Unlisted model IDs and catalog lookup errors or timeouts also stop the call before session creation. BYOK providers are exempt from these checks; incomplete capability metadata for a listed model defers effort validation to the runtime. Consult your model provider’s documentation for supported levels.

Rubric and evaluation criteria

The judge scores the stimulus rubric — one sub-result per criterion. config.prompt does not define those criteria: it is appended as a separate instructions section, so the judge reads it but still scores the rubric.

When a stimulus defines no rubric, the judge falls back to a built-in default:

• The agent completed the requested task correctly
• The output is clear and well-structured

The fallback is reported as rubric_source: "default" in the result metadata, and vally lint warns when it is combined with a config.prompt — that pairing usually means criteria went into the wrong field.

Put the criteria in rubric and use prompt for guidance that spans them:

stimuli:
  - name: writes-unit-tests
    prompt: Add unit tests for the parser
    rubric:
      - Tests cover edge cases (empty input, nulls, errors)
      - Assertions are specific, not just "toBeTruthy"
      - Tests actually run (valid syntax, proper imports)
    graders:
      - type: prompt
        config:
          prompt: Judge only the tests; ignore unrelated refactoring.

Evidence examples

Each rubric criterion becomes a sub-result in details, named after the criterion text (whitespace collapsed, truncated to 50 characters):

prompt (score: 0.75)
  ↳ prompt/Tests cover edge cases: 1.00
  ↳ prompt/Assertions are specific: 0.75
  ↳ prompt/Tests actually run: 0.50

Scores are normalized to 0–1 across the scoring scale’s range, so on the default scale_1_5 a raw 4 becomes (4 - 1) / (5 - 1) = 0.75, and a raw 2 becomes 0.25.

Retry behavior

In ordinary grading, a failed LLM call produces a failed result (score 0) with the error in evidence; the prompt grader does not retry the failed call. In-session reminders for a missing or invalid verdict are separate from failure retries.

In comparison mode, the comparison runner retries only explicitly retryable judge failures, subject to its retry-count and time-budget limits.

Tip

If the reported error indicates rate limiting, check your API limits. Consider using a model with available capacity or reducing the number of concurrent judge calls.

Cost considerations

Every trial graded by a prompt grader makes at least one LLM API call. With multi-trial:

Cost ≈ (num_stimuli × runs × prompt_graders_per_stimulus) × per-call cost

For example, 5 stimuli × 5 runs × 1 prompt grader = 25 LLM judge calls per eval.

Use --judge-model to control costs: a smaller model for iteration, a larger model for final evaluation.

Comparison mode

The prompt grader also powers head-to-head comparison: instead of scoring one trajectory, it judges a baseline against a treatment for the same stimulus and rubric, and reports which is better and by how much. Run it with vally compare — any stimulus with a rubric can be compared.

Comparison mode always evaluates both runs’ output, metrics, and session timelines. It does not use diff, golden_patch, or repo evidence selections; the CLI warns when any of these is configured for a compared stimulus. Both comparison passes share one temporary directory containing each run’s agent-output.txt, timeline.jsonl, and metrics.json. The files use neutral directory names, evidence/run-1/ and evidence/run-2/; each pass’s prompt maps them to A and B. Files are deleted only after both calls finish or fail. The initial prompt contains instructions and file paths, not evidence content.

How it works

1. A baseline trajectory and a treatment trajectory for the same stimulus are loaded (from an experiment’s variants, or two independent runs).
2. The judge compares them against the rubric and submits a verdict via a submit_comparison_grade tool call.
3. To reduce order bias, the comparison runs twice, swapping which response is A and B. If the two passes disagree on the winner, the result is a tie; if they agree on the winner but not the magnitude, the weaker magnitude wins.

Scoring scale

The verdict is signed and treatment-relative, in [-1, 1]:

Verdict	Score	Meaning
much better	+1.0	treatment is clearly better
slightly better	+0.4	treatment is somewhat better
equal / tie	0	no meaningful difference
slightly worse	-0.4	baseline is somewhat better
much worse	-1.0	baseline is clearly better

See the compare CLI reference for usage, statistics, and examples.