Graders: metric thresholds

Five graders that check trajectory metrics against a configurable max budget. All read values that the pipeline already computes — no additional analysis cost.

Tip

These graders are most useful for cost control (token budgets), CI guardrails (wall time, error count), and efficiency tracking (tool calls, turns) in outer-loop evaluations.

token-budget

Checks that total tokens (input + output) do not exceed max.

graders:
  - type: token-budget
    config:
      max: 50000

Evidence: 1500 tokens (within budget of 50000) or 75000 tokens exceeds max of 50000.

---

tool-call-count

Checks that the number of tool calls does not exceed max.

graders:
  - type: tool-call-count
    config:
      max: 3
      tools: ["^web_search$"] # optional; omit to count all tools

Tool filters are regular expressions

Every entry in tools is interpreted as an unanchored regular expression, not a literal tool name. For example, web.search also matches webXsearch because . is a regex wildcard. Escape regex characters and anchor the pattern when matching an exact name, such as ^web\.search$.

When tools is present, only tool calls whose names match at least one pattern count toward the budget. Patterns are unanchored regular expressions; use ^name$ for an exact match. An empty array is rejected at lint time. A pattern that compiles to an empty regex — for example an inline-flags-only pattern like (?i), whose source becomes (?:) — is rejected because it would match every tool.

Filtering uses metrics.toolCallBreakdown, whose values must be non-negative integers summing to metrics.toolCallCount. The grader fails when the two disagree — including when toolCallBreakdown is missing or empty while toolCallCount is nonzero, or when any value is negative or non-integer. A missing or empty breakdown is accepted only when toolCallCount is 0. Evidence includes the configured patterns, for example: 2 tool calls matching ["^web_search$"] (within budget of 3).

To set different budgets for different tool groups, add a tool-call-count grader for each group and give each one a distinct name: override, such as name: web-search-budget.

---

turn-count

Checks that the number of agent turns does not exceed max.

graders:
  - type: turn-count
    config:
      max: 10

---

error-count

Checks that the number of error events does not exceed max. Use max: 0 to require zero errors.

graders:
  - type: error-count
    config:
      max: 0

---

wall-time

Checks that wall-clock execution time does not exceed max. Accepts duration strings ("30s", "2m", "1h").

graders:
  - type: wall-time
    config:
      max: "2m"

---

Shared behavior

Config

All five require a max field:

Field	Type	Required	Description
max	integer or duration	yes	Upper bound. wall-time requires a duration string (e.g. "30s", "2m"); the rest require non-negative integers.

max must be a finite non-negative value. Omitting it is a validation error.

Scoring

When within budget (value ≤ max): score = 1.

When over budget: score degrades linearly — score = max(0, 1 − (value − max) / max(max, 1)), floored at 0. This means:

• At exactly the threshold → score 1 (pass)
• At 1.5× the threshold → score 0.5 (for thresholds ≥ 1)
• At 2× the threshold or above → score 0 (for thresholds ≥ 1)
• For max: 0, any value above 0 immediately scores 0

Taxonomy

All five share identical taxonomy metadata:

Property	Value
Determinism	static
Cost	free
Reference	reference-free
Temporal scope	trajectory-level
Score kind	code

Metadata

Every result includes structured metadata:

{
  "value": 1500,
  "max": 50000
}

tool-call-count additionally includes a tools key — null when tools is omitted from config, otherwise the configured patterns. value is null when per-tool aggregation is unavailable (see above):

{
  "value": 2,
  "max": 3,
  "tools": ["^web_search$"]
}