Choose a Grader

Start with the evidence that proves the behavior. Prefer a deterministic grader when the requirement can be expressed exactly; use an LLM judge only for qualities that code cannot reliably decide.

Choose by evidence

Grader names match the value used in graders[].type. This guide highlights useful selection paths; the grader catalog remains the exhaustive list.

Responses and conversation

Grader	Use it to verify
output-contains	The final output contains exact text
output-matches	The final output matches a regular expression
transcript-contains	Any assistant message, or one selected turn, contains text
transcript-matches	Any assistant message, or one selected turn, matches a regex

Files and workspace

Grader	Use it to verify
file-exists	A file exists
file-not-exists	No matching file exists
file-contains	A file contains exact text
file-matches	A file’s content matches a regular expression
diff-contains	Workspace changes include a pattern
diff-not-contains	Workspace changes do not include a pattern
diff-empty	The agent made no workspace changes
run-command	A command succeeds or emits expected output

Agent behavior and efficiency

Grader	Use it to verify
tool-calls	Required or forbidden tool calls
skill-invocation	Required or forbidden skill activations
system-event	Required or forbidden runtime events
token-budget	Total token use stays under a limit
tool-call-count	Tool calls stay under a limit
turn-count	Conversation turns stay under a limit
step-count	Agent steps stay under a limit
error-count	Error events stay under a limit
wall-time	Execution time stays under a limit

Custom logic and LLM judgment

Grader	Use it to verify
program	A project-specific script decides the result
prompt	Subjective quality against a rubric
panel	Several judge calls assess the same rubric

Tip

Combine checks that prove different things. For example, file-exists proves an artifact was created, run-command proves it works, and prompt can judge subjective qualities.

A practical selection order

1. Check the final artifact. Prefer file, output, diff, or command graders.
2. Check the path taken only when it matters. Use tool, skill, transcript, or system-event graders.
3. Add budgets. Metric graders turn cost and latency expectations into explicit checks.
4. Use a program for domain logic. Keep deterministic logic deterministic.
5. Use an LLM judge for subjective quality. Put scored criteria in the stimulus rubric.

The grader catalog lists every built-in grader and its complete config.

prompt or panel?

Start with prompt. Use panel when one judge is too inconsistent and the extra calls justify the cost. A panel can repeat or mix models. How strict it is depends on how the judges’ scores are combined. It supports the same evidence selections as prompt, but cannot compare a baseline with a treatment.

See the prompt and panel references for evidence selection, scoring scales, aggregation, retries, and failure behavior.

When built-ins are not enough

Use program when a local script can return an exit code or a GraderResult. Build a custom grader when the check should be packaged, loaded by other projects, or integrated directly with trajectory types.