Graders: metric thresholds

Six graders that check trajectory metrics against a configurable max budget. Five read a value the pipeline already computes — no additional analysis cost. step-count is the exception: it scans the trajectory’s event stream (and, when tools is set, applies regex matching) rather than reading a precomputed metric.

Tip

These graders are most useful for cost control (token budgets), CI guardrails (wall time, error count), and efficiency tracking (tool calls, steps, turns) in outer-loop evaluations.

token-budget

Checks that total tokens (input + output) do not exceed max.

graders:
  - type: token-budget
    config:
      max: 50000

Evidence: 1500 tokens (within budget of 50000) or 75000 tokens exceeds max of 50000.

---

tool-call-count

Checks that the number of tool calls does not exceed max.

graders:
  - type: tool-call-count
    config:
      max: 3
      tools: ["^web_search$"] # optional; omit to count all tools

Tool filters are regular expressions

Every entry in tools is interpreted as an unanchored regular expression, not a literal tool name. For example, web.search also matches webXsearch because . is a regex wildcard. Escape regex characters and anchor the pattern when matching an exact name, such as ^web\.search$.

When tools is present, only tool calls whose names match at least one pattern count toward the budget. Patterns are unanchored regular expressions; use ^name$ for an exact match. An empty array is rejected at lint time. A pattern that compiles to an empty regex — for example an inline-flags-only pattern like (?i), whose source becomes (?:) — is rejected because it would match every tool.

Filtering uses metrics.toolCallBreakdown, whose values must be non-negative integers summing to metrics.toolCallCount. The grader fails when the two disagree — including when toolCallBreakdown is missing or empty while toolCallCount is nonzero, or when any value is negative or non-integer. A missing or empty breakdown is accepted only when toolCallCount is 0. Evidence includes the configured patterns, for example: 2 tool calls matching ["^web_search$"] (within budget of 3).

To set different budgets for different tool groups, add a tool-call-count grader for each group and give each one a distinct name: override, such as name: web-search-budget.

---

turn-count

Checks that the number of agent turns does not exceed max.

graders:
  - type: turn-count
    config:
      max: 10

---

step-count

Checks that the number of agent steps does not exceed max. A step is a single tool call, or an agent message that issued no tool call — counted across the parent agent and any subagents. Use it when you need a total-activity budget rather than the tool-only tool-call-count or the turn-only turn-count.

graders:
  - type: step-count
    config:
      max: 40
      tools: ["^bash(\\.|$)"] # optional; omit to count every step

What counts:

Counted	Not counted
Each tool call (parent and subagent)	User messages
An agent message that issued no tool call	An agent message that did issue tool calls (its calls count instead)
A system event (e.g. compaction)	Turn boundaries, token usage, tool results, reasoning, skill activations

An agent response is either one message or its tool calls, never both: a response with three tool calls counts three, not four. Derived and bookkeeping events are excluded because a single source step expands into several of them, which would inflate the total. Subagent steps are included automatically — inlined subagent events live in the same event stream. To budget one agent’s steps only, add scope: to the grader.

Rare undercount with same-named subagents

Same-named subagent instances launched back to back with no intervening events can be conflated — one instance’s tool call can be attributed to another’s pending step, undercounting by one. This is rare (it requires two instances of the same subagent launched with nothing between them) but can flip a result at the exact threshold.

When tools is present, only steps whose name matches at least one pattern count. A tool-call step is named after its tool, a message step is named response, and a system event step is named after its event type. The same regex rules as tool-call-count apply — patterns are unanchored, so anchor them (^task$) when a prefix would over-match. Unlike tool-call-count, filtering reads the trajectory’s events directly rather than metrics.toolCallBreakdown. Evidence: 12 steps (within budget of 40) or 3 steps matching ["^bash$"] exceeds max of 2.

To set different budgets for different tool groups, add a step-count grader for each group and give each one a distinct name: override.

---

error-count

Checks that the number of error events does not exceed max. Use max: 0 to require zero errors.

graders:
  - type: error-count
    config:
      max: 0

---

wall-time

Checks that wall-clock execution time does not exceed max. Accepts duration strings ("30s", "2m", "1h").

graders:
  - type: wall-time
    config:
      max: "2m"

---

Shared behavior

Config

All six require a max field:

Field	Type	Required	Description
max	integer or duration	yes	Upper bound. wall-time requires a duration string (e.g. "30s", "2m"); the rest require non-negative integers.

max must be a finite non-negative value. Omitting it is a validation error.

Scoring

When within budget (value ≤ max): score = 1.

When over budget: score degrades linearly — score = max(0, 1 − (value − max) / max(max, 1)), floored at 0. This means:

• At exactly the threshold → score 1 (pass)
• At 1.5× the threshold → score 0.5 (for thresholds ≥ 1)
• At 2× the threshold or above → score 0 (for thresholds ≥ 1)
• For max: 0, any value above 0 immediately scores 0

Taxonomy

All six share identical taxonomy metadata:

Property	Value
Determinism	static
Cost	free
Reference	reference-free
Temporal scope	trajectory-level
Score kind	code

Metadata

Every result includes structured metadata:

{
  "value": 1500,
  "max": 50000
}

tool-call-count and step-count additionally include a tools key — null when tools is omitted from config, otherwise the configured patterns. For tool-call-count, value is null when per-tool aggregation is unavailable (see above):

{
  "value": 2,
  "max": 3,
  "tools": ["^web_search$"]
}