Grader: program

Taxonomy

Property	Value
Determinism	static
Cost	low
Reference	reference-free
Temporal scope	trajectory-level
Score kind	code

Config

graders:
  - type: program
    config:
      program: "python"
      args: ["graders/check_output.py"]
      timeout: 60s
      sub_path: "src/app"

Field	Type	Required	Default	Description
program	string	Yes	—	The program to run (e.g. python, node, bash)
args	string[]	No	—	Command-line arguments passed to the program
shell	boolean	No	false	Run the program using the system shell (/bin/sh on Unix, cmd.exe on Windows) instead of direct exec
sub_path	string	No	—	Subdirectory under the workspace root to use as the working directory. Must stay within the workspace root.
timeout	duration	No	60s	Maximum time the process can run before being terminated (e.g. 60s, 2m). Must be positive (0 is rejected) and no longer than ~24.8 days.
env	Record<string, string>	No	—	Additional environment variables passed to the subprocess. Merged with the process environment. The reserved variables (EVALUATE_WORKSPACE, EVALUATE_GRADER_INPUT, EVALUATE_ASSETS, EVALUATE_ARTIFACT_DIR, EVALUATE_ARTIFACT_DIR_STRICT) are always owned by the grader and cannot be specified in config.env.

Behavior

Executes the program in a child process and sets environment variables so your grader can load the full evaluation context:

Environment variable	Value
EVALUATE_WORKSPACE	Path to the workspace root (always the top-level workspace, even when sub_path changes the working directory) — always set
EVALUATE_GRADER_INPUT	Path to a temporary JSON file containing the serialized GraderInput — always set
EVALUATE_ASSETS	Path to the per-trial assets directory (setup inputs staged outside the workspace). Set only when the run has an assets directory
EVALUATE_ARTIFACT_DIR	Path to the run’s output/artifact directory. Set only when the run recorded one
EVALUATE_ARTIFACT_DIR_STRICT	"1" when artifact reads are strict (the artifact dir must contain the referenced path). Set only when strict mode is on

When sub_path is set, the child process’s working directory (cwd) is the resolved subdirectory, but EVALUATE_WORKSPACE still points to the workspace root.

EVALUATE_ASSETS points at setup inputs staged out of the graded workspace, while EVALUATE_ARTIFACT_DIR points at run outputs — they are distinct and never conflated. Both are host paths that may be absent when re-grading a stored trajectory on another host, so treat an unset variable gracefully.

Caution

The assets directory is agent-visible and writable: the agent receives the same path as EVALUATE_ASSETS and can read or modify anything staged there. Keep the grader program — and any script or data it trusts — outside both the workspace and the assets directory, and validate asset contents rather than assuming they survived the run unchanged.

All of the reserved variables above (EVALUATE_WORKSPACE, EVALUATE_GRADER_INPUT, EVALUATE_ASSETS, EVALUATE_ARTIFACT_DIR, and EVALUATE_ARTIFACT_DIR_STRICT) are rejected if they appear in env (case-insensitive), because the grader owns their values.

The payload’s trajectory also carries assetsDir and artifactDir directly, so a grader can read the same paths from the JSON instead of the environment.

The EVALUATE_GRADER_INPUT payload

The file holds a JSON copy of the same GraderInput an in-process custom grader receives, so your program can grade what the agent did as well as the files it left behind.

Field	Type	Description
trajectory	object	The run itself: events, output, metrics, metadata, workDir, and (when present) assetsDir / artifactDir
stimulus	object	The stimulus that produced the run, as declared in eval.yaml
config	object	This grader’s own config block
diffBaseDir	string	Base directory for resolving a relative trajectory.diffPath, set when using --run-dir
goldenPatch	string	Reference-solution diff, when grading with a golden patch

See the trajectory format reference for the event types, metric names, and metadata fields available under trajectory.

Note

Dates (metadata.startedAt, metadata.completedAt, and each event’s timestamp) arrive as ISO-8601 strings, since JSON has no date type. The file is created mode 0600 on POSIX and cleaned up after your program exits, best effort on Windows where a transient lock can defer deletion.

The in-process GraderInput.signal, GraderInput.timeoutMs, and GraderInput.judgeTimeoutMs orchestration fields are omitted from the JSON payload. The program process is terminated directly when the grading budget expires.

Caution

The grader reads the workspace at grade time and fails when its working directory is missing, so it cannot re-grade a saved results.jsonl once the workspace is gone. Separately, because its evidence is the final workspace state, it cannot be narrowed with turn: or scope:, which are views over the event stream. For trajectory-only checks that survive re-grading, use a built-in grader such as tool-calls, or an in-process custom grader.

Reading the payload

graders/used-create-tool.mjs

import { readFileSync } from "node:fs";

const { trajectory, stimulus } = JSON.parse(
  readFileSync(process.env.EVALUATE_GRADER_INPUT, "utf-8"),
);

const created = trajectory.events.filter(
  (e) => e.type === "tool_call" && e.data.toolName === "create",
);

process.stdout.write(
  JSON.stringify({
    name: "used-create-tool",
    kind: "code",
    passed: created.length > 0,
    score: created.length > 0 ? 1 : 0,
    evidence:
      created.length > 0
        ? `Created ${created.map((e) => e.data.arguments.path).join(", ")} for "${stimulus.name}"`
        : "Agent never called the create tool",
  }),
);

Your program communicates its result back in one of two ways:

Exit-code mode

Print nothing to stdout (stderr is fine). Exit 0 to pass, non-zero to fail. The score is 1 on pass, 0 on fail.

GraderResult JSON mode

Print a JSON object conforming to the GraderResult schema to stdout. This lets you return a custom score (between 0 and 1 inclusive), evidence text, and metadata. Do not print anything else to stdout — send diagnostics to stderr. Diagnostics are kept only when the program fails or uses exit-code mode; when stdout carries a valid GraderResult that becomes the result and stderr is dropped, so put anything you want to keep in its evidence or metadata.

{
  "name": "my-custom-check",
  "passed": true,
  "score": 0.85,
  "evidence": "14 of 16 assertions passed",
  "kind": "code"
}

If stdout contains text that is not valid JSON, or the JSON does not match the GraderResult schema, the grader fails with a 0 score.

Use cases

# Python script that inspects the workspace
- type: program
  config:
    program: "python"
    args: ["graders/validate_schema.py"]

# Bash script using exit-code mode
- type: program
  config:
    program: "bash"
    args: ["graders/check.sh"]
    shell: true

# Node.js script returning a GraderResult JSON
- type: program
  config:
    program: "node"
    args: ["graders/score.js"]
    timeout: 120s

# Pass custom environment variables to the grader
- type: program
  config:
    program: python
    args: [-m, my_checker]
    env:
      API_ENDPOINT: "https://test.example.com"
      EXPECTED_STATUS: "active"

Evidence examples

✔ Grader exited successfully
✘ Grader exited with exit code 1
✘ Grader timed out
✘ Failed to start grader program: ENOENT (spawn)
✘ Grader returned unparseable JSON output on stdout: SyntaxError: …
✘ Grader output did not match schema