Graders: custom-metrics

The custom-metrics grader asserts on domain-specific signals a run emits into a JSON file in the workspace — beyond Vally’s built-in trajectory metrics (token-budget, tool-call-count, wall-time, …).

Use it when your executor, stimulus environment, or a setup/post-run script writes results like a test summary, a quality score from a domain harness, build/runtime stats, or a final state string.

Tip

The run is responsible for writing the metrics file. The grader only reads and asserts on it — it does not compute the metrics.

Metrics file

By default the grader reads custom_metrics.json from the workspace root. The file must contain a JSON object mapping metric names to values (number, boolean, or string):

{
  "tests_failed": 0,
  "quality_score": 0.82,
  "latency_ms": 1500,
  "build_succeeded": true,
  "final_mode": "autopilot"
}

Alternatively, the metrics may be nested under a top-level values object. When a values object is present, the grader reads metrics from it and ignores sibling keys such as schema:

{
  "schema": "v1",
  "values": {
    "tests_failed": 0,
    "quality_score": 0.82
  }
}

Use this envelope when the run writes metrics alongside other bookkeeping (like a schema version) — the grader unwraps values and ignores the sibling keys.

Override the location with config.path (relative to the workspace; absolute paths and .. escapes are rejected).

Config

graders:
  - type: custom-metrics
    config:
      path: custom_metrics.json # optional, default
      assertions: # required, at least one
        - metric: tests_failed
          equals: 0
        - metric: quality_score
          min: 0.8
        - metric: latency_ms
          max: 2000
        - metric: retries
          gt: 0 # exclusive lower bound (value > 0)
        - metric: error_rate
          lt: 0.05 # exclusive upper bound (value < 0.05)
        - metric: score
          equals: 1.0
          tolerance: 0.01
        - metric: build_succeeded
          equals: true
        - metric: final_mode
          matches: "^autopilot$"
        - metric: summary
          contains: success # case-insensitive substring
        - metric: cache_key
          present: true
        - metric: deprecated_flag
          absent: true

Field	Type	Required	Description
path	string	no	Metrics file path relative to the workspace. Defaults to custom_metrics.json.
assertions	array	yes	One or more per-metric assertions. Must contain at least one entry.

Assertion conditions

Each assertion targets one metric (a key in the JSON object) and specifies one or more conditions. All specified conditions on an assertion must hold for it to pass.

Condition	Applies to	Description
min	number	Value must be >= min.
max	number	Value must be <= max.
gt	number	Value must be > gt (exclusive lower bound).
lt	number	Value must be < lt (exclusive upper bound).
equals	number / boolean / string	Value must equal this (type inferred from the YAML value).
tolerance	number	Optional, pairs with a numeric equals: passes when |value − equals| <= tolerance.
matches	string	Value must match this regular expression.
contains	string / number / boolean	Case-insensitive substring match. The value is string-coerced, so it also works on numbers and booleans. Standalone — cannot combine with other value conditions.
present	—	present: true requires the metric key to exist.
absent	—	absent: true requires the metric key to NOT exist (cannot combine with other conditions).

Scoring

The grader passes only when every assertion passes. The score is the fraction of assertions that passed (passedCount / total), and each per-metric outcome is reported in details.

Error handling

These conditions produce a clear failing result — they never crash the pipeline:

• Missing file → expected metrics file at <path> (not found).
• Malformed JSON → failed to parse <path> as JSON: ….
• Root not a JSON object (array, primitive, null) → <path> must contain a JSON object.
• Missing metric key (for a non-absent assertion) → per-assertion metric '<name>' not found.
• Type mismatch (e.g. a numeric condition on a string value) → per-assertion failure naming both types.

Most invalid configuration — no assertions, an assertion with no condition or no metric, min > max, an invalid regex, tolerance without a numeric equals, incompatible condition combinations — is caught at lint time by vally lint, so mistakes surface before a run. (A config.path that escapes the workspace is validated at runtime, since it depends on the resolved workspace.)

Note

Metrics are run-level, so this grader cannot be scoped to a single turn. Because it reads workspace state, setting a turn on the grader config makes the pipeline fail the grader rather than slice the trajectory — leave turn unset.

Taxonomy

Property	Value
Determinism	static
Cost	low
Reference	reference-free
Temporal scope	trajectory-level
Score kind	code