Graders: custom-metrics

The custom-metrics grader asserts on domain-specific signals a run emits into a JSON file — read from the trajectory artifact dir if present, otherwise the workspace — beyond Vally’s built-in trajectory metrics (token-budget, tool-call-count, wall-time, …).

Use it when your executor, stimulus environment, or a setup/post-run script writes results like a test summary, a quality score from a domain harness, build/runtime stats, or a final state string.

Tip

The run is responsible for writing the metrics file. The grader only reads and asserts on it — it does not compute the metrics.

Metrics file

By default the grader reads custom_metrics.json, looked up artifact dir first, workspace second (see Where the file is read from). The file must contain a JSON object mapping metric names to values (number, boolean, or string):

{
  "tests_failed": 0,
  "quality_score": 0.82,
  "latency_ms": 1500,
  "build_succeeded": true,
  "final_mode": "autopilot"
}

Alternatively, the metrics may be nested under a top-level values object. When a values object is present, the grader reads metrics from it and ignores sibling keys such as schema:

{
  "schema": "v1",
  "values": {
    "tests_failed": 0,
    "quality_score": 0.82
  }
}

Use this envelope when the run writes metrics alongside other bookkeeping (like a schema version) — the grader unwraps values and ignores the sibling keys.

Override the location with config.path (relative to whichever root is used; absolute paths and .. escapes are rejected against both).

Where the file is read from

The grader resolves the metrics file artifact-dir-first, workspace-fallback:

1. If the trajectory has an artifact dir (trajectory.artifactDir, set by executors that keep artifacts outside the workspace) and the file exists there, it’s read from there.
2. Otherwise it’s read from the workspace (trajectory.workDir) — the historical behavior.

Selection is by existence, not parse success: a malformed file in the artifact dir fails there and does not fall back to the workspace. With artifactDir unset the lookup is workspace-only, byte-identical to before. The config.path guards (no absolute paths, no ..) apply to whichever root is used.

Only the root actually read is required to exist. When the metrics live in artifactDir, grading does not need workDir — so re-grading a results.jsonl offline still works after the ephemeral workspace is gone. The evidence names the source it read (… from artifact dir or … from workspace).

Tip

This lets an executor point graders at an artifact directory it owns instead of copying custom_metrics.json into the workspace just so the grader can find it.

Config

graders:
  - type: custom-metrics
    config:
      path: custom_metrics.json # optional, default
      assertions: # required, at least one
        - metric: tests_failed
          equals: 0
        - metric: quality_score
          min: 0.8
        - metric: latency_ms
          max: 2000
        - metric: retries
          gt: 0 # exclusive lower bound (value > 0)
        - metric: error_rate
          lt: 0.05 # exclusive upper bound (value < 0.05)
        - metric: score
          equals: 1.0
          tolerance: 0.01
        - metric: build_succeeded
          equals: true
        - metric: final_mode
          matches: "^autopilot$"
        - metric: summary
          contains: success # case-insensitive substring
        - metric: cache_key
          present: true
        - metric: deprecated_flag
          absent: true

Field	Type	Required	Description
path	string	no	Metrics file path relative to the resolved root (artifact dir if present, else the workspace). Defaults to custom_metrics.json.
assertions	array	yes	One or more per-metric assertions. Must contain at least one entry.

Assertion conditions

Each assertion targets one metric (a key in the JSON object) and specifies one or more conditions. All specified conditions on an assertion must hold for it to pass.

Condition	Applies to	Description
min	number	Value must be >= min.
max	number	Value must be <= max.
gt	number	Value must be > gt (exclusive lower bound).
lt	number	Value must be < lt (exclusive upper bound).
equals	number / boolean / string	Value must equal this (type inferred from the YAML value).
tolerance	number	Optional, pairs with a numeric equals: passes when |value − equals| <= tolerance.
matches	string	Value must match this regular expression.
contains	string / number / boolean	Case-insensitive substring match. The value is string-coerced, so it also works on numbers and booleans. Standalone — cannot combine with other value conditions.
present	—	present: true requires the metric key to exist.
absent	—	absent: true requires the metric key to NOT exist (cannot combine with other conditions).

Scoring

The grader passes only when every assertion passes. The score is the fraction of assertions that passed (passedCount / total), and each per-metric outcome is reported in details.

Error handling

These conditions produce a clear failing result — they never crash the pipeline:

• Missing file → expected metrics file at <path> (not found).
• Malformed JSON → failed to parse <path> as JSON: ….
• Root not a JSON object (array, primitive, null) → <path> must contain a JSON object.
• Missing metric key (for a non-absent assertion) → per-assertion metric '<name>' not found.
• Type mismatch (e.g. a numeric condition on a string value) → per-assertion failure naming both types.

Most invalid configuration — no assertions, an assertion with no condition or no metric, min > max, an invalid regex, tolerance without a numeric equals, incompatible condition combinations — is caught at lint time by vally lint, so mistakes surface before a run. (A config.path that escapes the root is validated at runtime, since it depends on the resolved directory.)

Note

Metrics are run-level, so this grader cannot be scoped to a single turn. Because it reads workspace state, setting a turn on the grader config makes the pipeline fail the grader rather than slice the trajectory — leave turn unset.

Taxonomy

Property	Value
Determinism	static
Cost	low
Reference	reference-free
Temporal scope	trajectory-level
Score kind	code