Grader: program

Taxonomy

Property	Value
Determinism	static
Cost	low
Reference	reference-free
Temporal scope	trajectory-level
Score kind	code

Config

graders:
  - type: program
    config:
      program: "python"
      args: ["graders/check_output.py"]
      timeout: 60s
      sub_path: "src/app"

Field	Type	Required	Default	Description
program	string	Yes	—	The program to run (e.g. python, node, bash)
args	string[]	No	—	Command-line arguments passed to the program
shell	boolean	No	false	Run the program using the system shell (/bin/sh on Unix, cmd.exe on Windows) instead of direct exec
sub_path	string	No	—	Subdirectory under the workspace root to use as the working directory. Must stay within the workspace root.
timeout	duration	No	60s	Maximum time the process can run before being terminated (e.g. 60s, 2m). Must be positive (0 is rejected).
env	Record<string, string>	No	—	Additional environment variables passed to the subprocess. Merged with the process environment. EVALUATE_WORKSPACE and EVALUATE_GRADER_INPUT are always set by the grader and cannot be specified in config.env.

Behavior

Executes the program in a child process and sets two environment variables so your grader can load the full evaluation context:

Environment variable	Value
EVALUATE_WORKSPACE	Path to the workspace root (always the top-level workspace, even when sub_path changes the working directory)
EVALUATE_GRADER_INPUT	Path to a temporary JSON file containing the serialized GraderInput

When sub_path is set, the child process’s working directory (cwd) is the resolved subdirectory, but EVALUATE_WORKSPACE still points to the workspace root.

The reserved variables EVALUATE_WORKSPACE and EVALUATE_GRADER_INPUT are rejected if they appear in env (case-insensitive), because the grader owns their values.

The EVALUATE_GRADER_INPUT payload

The file holds a JSON copy of the same GraderInput an in-process custom grader receives, so your program can grade what the agent did as well as the files it left behind.

Field	Type	Description
trajectory	object	The run itself: events, output, metrics, metadata, and workDir
stimulus	object	The stimulus that produced the run, as declared in eval.yaml
config	object	This grader’s own config block
diffBaseDir	string	Base directory for resolving a relative trajectory.diffPath, set when using --run-dir
goldenPatch	string	Reference-solution diff, when grading with a golden patch

See the trajectory format reference for the event types, metric names, and metadata fields available under trajectory.

Note

Dates (metadata.startedAt, metadata.completedAt, and each event’s timestamp) arrive as ISO-8601 strings, since JSON has no date type. The file is created mode 0600 on POSIX and cleaned up after your program exits, best effort on Windows where a transient lock can defer deletion.

Caution

The grader reads the workspace at grade time and fails when its working directory is missing, so it cannot re-grade a saved results.jsonl once the workspace is gone. Separately, because its evidence is the final workspace state, it cannot be narrowed with turn: or scope:, which are views over the event stream. For trajectory-only checks that survive re-grading, use a built-in grader such as tool-calls, or an in-process custom grader.

Reading the payload

graders/used-create-tool.mjs

import { readFileSync } from "node:fs";

const { trajectory, stimulus } = JSON.parse(
  readFileSync(process.env.EVALUATE_GRADER_INPUT, "utf-8"),
);

const created = trajectory.events.filter(
  (e) => e.type === "tool_call" && e.data.toolName === "create",
);

process.stdout.write(
  JSON.stringify({
    name: "used-create-tool",
    kind: "code",
    passed: created.length > 0,
    score: created.length > 0 ? 1 : 0,
    evidence:
      created.length > 0
        ? `Created ${created.map((e) => e.data.arguments.path).join(", ")} for "${stimulus.name}"`
        : "Agent never called the create tool",
  }),
);

Your program communicates its result back in one of two ways:

Exit-code mode

Print nothing to stdout (stderr is fine). Exit 0 to pass, non-zero to fail. The score is 1 on pass, 0 on fail.

GraderResult JSON mode

Print a JSON object conforming to the GraderResult schema to stdout. This lets you return a custom score (between 0 and 1 inclusive), evidence text, and metadata. Do not print anything else to stdout — send diagnostics to stderr. Diagnostics are kept only when the program fails or uses exit-code mode; when stdout carries a valid GraderResult that becomes the result and stderr is dropped, so put anything you want to keep in its evidence or metadata.

{
  "name": "my-custom-check",
  "passed": true,
  "score": 0.85,
  "evidence": "14 of 16 assertions passed",
  "kind": "code"
}

If stdout contains text that is not valid JSON, or the JSON does not match the GraderResult schema, the grader fails with a 0 score.

Use cases

# Python script that inspects the workspace
- type: program
  config:
    program: "python"
    args: ["graders/validate_schema.py"]

# Bash script using exit-code mode
- type: program
  config:
    program: "bash"
    args: ["graders/check.sh"]
    shell: true

# Node.js script returning a GraderResult JSON
- type: program
  config:
    program: "node"
    args: ["graders/score.js"]
    timeout: 120s

# Pass custom environment variables to the grader
- type: program
  config:
    program: python
    args: [-m, my_checker]
    env:
      API_ENDPOINT: "https://test.example.com"
      EXPECTED_STATUS: "active"

Evidence examples

✔ Grader exited successfully
✘ Grader exited with exit code 1
✘ Grader timed out
✘ Failed to start grader program: ENOENT (spawn)
✘ Grader returned unparseable JSON output on stdout: SyntaxError: …
✘ Grader output did not match schema