Grader: max-repeat

Taxonomy

Property	Value
Determinism	static
Cost	free
Reference	reference-free
Temporal scope	trajectory-level
Score kind	code

Config

graders:
  - type: max-repeat
    config:
      max: 3 # fail when an action repeats more than 3 times in a row
      match: observation # "observation" (default) | "name" | "name-args"
      tools: ["^bash$"] # optional regex patterns; default watches all tools

Field	Type	Required	Default	Description
max	number	Yes	—	Maximum allowed consecutive repeats. Fails when an action repeats more than max times in a row. Integer >= 1.
match	string	No	"observation"	What counts as “the same” action: observation, name, or name-args (see below).
tools	string[]	No	(all tools)	Regex patterns selecting which tools to watch. When omitted, every tool is watched.

Match modes

Mode	Two calls are “the same” when…
observation	they invoke the same tool and produce the same result/output (the default).
name	they invoke the same tool, regardless of arguments or output.
name-args	they invoke the same tool with the same arguments (deep-equal).

All three modes are per-tool — two different tools are never considered the same action, even if they produce identical output.

Watching specific tools

tools is a list of unanchored regex patterns matched against the tool name; use ^bash$ for an exact match. A call is watched if it matches any pattern. Omit tools to watch every tool. An empty array, or a pattern that is empty or only whitespace, is rejected at lint time. A pattern that compiles to an empty regex — for example an inline-flags-only pattern like (?i), whose source becomes (?:) — is rejected at runtime instead, since it would match every tool.

To enforce different thresholds per tool, add multiple max-repeat graders — one per tool with its own max.

Behavior

This is a post-hoc grader: it runs after a trajectory is collected and never intervenes during execution. Stopping a live loop is the executor’s job (max_turns); this grader is loop detection for scoring and regression coverage.

1. Filter trajectory events to the watched tool(s).
2. Group by step. A step is one agent turn (delimited by turn_start / turn_end). Every watched call inside a turn collapses into one step, so parallel or duplicate calls within a single turn count once — not as a loop. Watched calls outside any turn each form their own step.
3. Fingerprint each step from its calls according to match. In observation mode a missing or empty observation produces an empty fingerprint.
4. Count the longest run of consecutive identical, non-empty step fingerprints. An empty fingerprint breaks a run rather than extending it, so unknown or missing observations can’t fake a loop. A trajectory with no watched calls has a max run of 0.

Passes when the longest run is <= max. Fails otherwise, and the evidence names the repeated tool(s) and a preview of the repeated key.

Consecutive only

This grader detects consecutive repeats of the same action (A A A A). It does not detect higher-period cycles where different actions alternate (A B A B …) — those never produce consecutive identical fingerprints, so they pass.

Use cases

# Fail if the agent re-runs the same action (same output) more than 3 times in a row
- type: max-repeat
  config:
    max: 3

# Only watch bash; fail on 3+ identical bash invocations in a row (ignore arguments/output)
- type: max-repeat
  config:
    max: 2
    match: name
    tools: ["^bash$"]

# Fail if the agent calls the same tool with the same arguments repeatedly
- type: max-repeat
  config:
    max: 2
    match: name-args

# Different thresholds per tool: stricter on web_search than on bash
- type: max-repeat
  config:
    max: 2
    tools: ["^web_search$"]
- type: max-repeat
  config:
    max: 5
    tools: ["^bash$"]