Grader: max-repeat

Taxonomy

Property	Value
Determinism	static
Cost	free
Reference	reference-free
Temporal scope	trajectory-level
Score kind	code

Config

graders:
  - type: max-repeat
    config:
      max: 3 # fail when an action or pattern repeats more than 3 times
      max_cycle_period: 2 # optionally detect A B A B cycles
      match: observation # "observation" (default) | "name" | "name-args"
      tools: ["^bash$"] # optional regex patterns; default watches all tools

Field	Type	Required	Default	Description
max	number	Yes	—	Maximum allowed complete occurrences of the same action or pattern. Fails above max. Integer >= 1.
max_cycle_period	number	No	1	Longest pattern period to detect. 1 detects only consecutive repeats; 2 also detects A B A B; longer patterns are not detected. Integer 1–10 (inclusive).
match	string	No	"observation"	What counts as “the same” action: observation, name, or name-args (see below).
tools	string[]	No	(all tools)	Regex patterns selecting which tools to watch. When omitted, every tool is watched.

Match modes

Mode	Two calls are “the same” when…
observation	they invoke the same tool and produce the same result/output (the default).
name	they invoke the same tool, regardless of arguments or output.
name-args	they invoke the same tool with the same arguments (deep-equal).

All three modes are per-tool — two different tools are never considered the same action, even if they produce identical output.

Watching specific tools

tools is a list of unanchored regex patterns matched against the tool name; use ^bash$ for an exact match. A call is watched if it matches any pattern. Omit tools to watch every tool. An empty array, or a pattern that is empty or only whitespace, is rejected at lint time. A pattern that compiles to an empty regex — for example an inline-flags-only pattern like (?i), whose source becomes (?:) — is rejected at runtime instead, since it would match every tool.

To enforce different thresholds per tool, add multiple max-repeat graders — one per tool with its own max.

Behavior

This is a post-hoc grader: it runs after a trajectory is collected and never intervenes during execution. Stopping a live loop is the executor’s job (max_turns); this grader is loop detection for scoring and regression coverage.

1. Filter trajectory events to the watched tool(s).
2. Group by step. A step is one agent turn (delimited by turn_start / turn_end). Every watched call inside a turn collapses into one step, so parallel or duplicate calls within a single turn count once — not as a loop. Watched calls outside any turn each form their own step.
3. Fingerprint each step from its calls according to match. In observation mode, a step containing only missing or empty watched observations has an empty fingerprint. When periodic detection is enabled (max_cycle_period > 1), a missing or empty observation from any watched parallel call makes the entire step fingerprint empty so partial step data cannot form a cycle.
4. Find repeated patterns. The grader finds the longest contiguous sequence of complete repeated patterns with a period up to max_cycle_period. A period-one pattern is a consecutive repeat (A A A); a period-two pattern is an alternating cycle (A B A B). An empty fingerprint breaks a pattern rather than extending it, so unknown or missing observations cannot bridge a loop. A trajectory with no watched calls has zero occurrences.

max counts complete pattern occurrences, not individual steps. For example, A B A B A B contains three occurrences of the period-two pattern A B: it passes at max: 3 and fails at max: 2. Incomplete leading or trailing fragments do not count as an additional occurrence.

Passes when the maximum occurrence count is <= max. Fails otherwise. When a period-two-or-higher pattern triggers the failure, evidence identifies its pattern, period, and occurrence count. For a plain consecutive (period-one) failure, it identifies the repeated action and count. Metadata always includes the effective period bound and a pattern preview.

Opt-in periodic detection

max_cycle_period defaults to 1, so existing configs retain consecutive-only behavior. Raise it only for scenarios where repeating short action cycles is known to be invalid: a larger period can classify a legitimate multi-step workflow as a loop when the fingerprints do not capture progress. The maximum value is 10 to keep detection bounded and diagnostics readable.

Use cases

# Fail if the agent re-runs the same action (same output) more than 3 times in a row
- type: max-repeat
  config:
    max: 3

# Only watch bash; fail on 3+ identical bash invocations in a row (ignore arguments/output)
- type: max-repeat
  config:
    max: 2
    match: name
    tools: ["^bash$"]

# Fail if the agent calls the same tool with the same arguments repeatedly
- type: max-repeat
  config:
    max: 2
    match: name-args

# Fail if the agent alternates between two actions more than twice
- type: max-repeat
  config:
    max: 2
    max_cycle_period: 2
    match: name-args

# Different thresholds per tool: stricter on web_search than on bash
- type: max-repeat
  config:
    max: 2
    tools: ["^web_search$"]
- type: max-repeat
  config:
    max: 5
    tools: ["^bash$"]