Grader: loop-outcome

Taxonomy

Property	Value
Determinism	static
Cost	free
Reference	reference-free
Temporal scope	trajectory-level
Score kind	code

Config

graders:
  - type: loop-outcome
    config:
      should_loop: false # the agent should recover, not spin (default)
      max_acceptable_retries: 3 # >3 retries in a row = a loop (a run of N identical calls is N-1 retries)
      match: name-args # "name-args" (default) | "name" | "observation"
      tools: ["^bash$"] # optional regex patterns; default watches all tools

Field	Type	Required	Default	Description
should_loop	boolean	No	false	Whether looping/retrying is the expected behavior. false expects the agent to recognize the dead end and stop.
max_acceptable_retries	number	Yes	—	Maximum consecutive identical re-attempts (retries) still considered acceptable. Repeating more than this counts as a loop. Integer >= 0.
match	string	No	"name-args"	What counts as “the same” retried action: name-args, name, or observation (see below).
tools	string[]	No	(all tools)	Regex patterns selecting which tools the retry signal watches. When omitted, every tool is watched.

A couple of retries is normal recovery, not a loop — max_acceptable_retries is the line between the two. Use max_acceptable_retries: 0 for zero tolerance (any repeat is a loop). It’s required (like max-repeat’s max) so the threshold is always an explicit author choice, never a silent default.

Match modes

Mode	Two attempts are “the same” retry when…
name-args	they invoke the same tool with the same arguments (deep-equal) — the default.
name	they invoke the same tool, regardless of arguments or output.
observation	they invoke the same tool and produce the same result/output.

The default is name-args (not observation, as in max-repeat) — a “retry” is naturally the same action re-attempted. Use observation to catch an agent that keeps getting the same result even as it varies its inputs. All modes are per-tool.

tools is a list of unanchored regex patterns matched against the tool name (^bash$ for an exact match); omit it to watch every tool. It scopes only the retry signal — the simulation-cap signal below is unaffected.

Behavior

Like max-repeat, this is a post-hoc loop detector — it walks the collected trajectory after the run and never intervenes during execution. The difference is that max-repeat only fails on loops, while loop-outcome compares the observed behavior to an expected outcome, so a scenario can also assert that a loop was expected (should_loop: true) — useful for probing recovery/detection tooling.

Two independent signals count as looping:

1. Excess retries. The agent re-issued the same action more than max_acceptable_retries times in a row. What counts as “the same” is set by match (default name-args), and tools scopes which tools are watched. Identical calls emitted within one model step (parallel calls in a single turn) count once, mirroring max-repeat. A run of N identical calls is N - 1 re-attempts. This works on any trajectory.
2. Iteration cap. The run ended with endReason: "simulation_cap" — the agent exhausted the tool-call simulation’s max_iterations budget. Only runs that use stimulus.simulation can produce this signal. This signal is independent of match/tools.

The verdict compares the observation to should_loop:

• should_loop: false (default) — passes when the agent did not loop.
• should_loop: true — passes when the agent did loop.

The cap always counts as a loop

A run that ends with simulation_cap is counted as looping regardless of max_acceptable_retries — the two signals are OR’d. The cap also bounds the retry signal: a non-capped run reaches at most simulation.max_iterations assistant turns, and a run of N identical turns is N - 1 retries, so the observed retry count tops out at max_iterations - 1. Setting max_acceptable_retries >= simulation.max_iterations - 1 therefore makes the retry signal unreachable — it can never exceed that threshold. vally lint warns when it detects this.

Consecutive only

Retry detection counts consecutive identical actions (A A A A). Higher-period cycles where different actions alternate (A B A B …) are not counted as retries.

Use cases

# The agent should recover from a failure instead of spinning on it
- type: loop-outcome
  config:
    should_loop: false
    max_acceptable_retries: 3

# A scenario that intentionally induces a loop to verify it's detected.
- type: loop-outcome
  name: environment-loop check
  config:
    should_loop: true
    max_acceptable_retries: 3