Grader: assistant-contains-tool-output

Taxonomy

Property	Value
Determinism	static
Cost	free
Reference	reference-free
Temporal scope	trajectory-level
Score kind	code

Config

graders:
  - type: assistant-contains-tool-output
    config:
      tools: [tool-catalog-lookup_release_train] # one or more tool names
      pattern: '"release_id":"(rt_[0-9a-f]{16})"' # regex; group 1 (or full match) is the needle

Field	Type	Required	Default	Description
tools	string[]	Yes	—	Tool name(s) whose output is captured. Matches exactly or on a name. prefix (namespaced tools)
pattern	string	Yes	—	Regex matched against the tool output; capture group 1 (or the full match if no group) is the needle

Turn scoping

Use the pipeline’s standard turn: field on the grader wrapper to restrict the check to a specific conversation turn. When turn: 0 is set, the pipeline slices the trajectory to that turn before the grader runs.

graders:
  - type: assistant-contains-tool-output
    turn: 0
    config:
      tools: [tool-catalog-lookup_release_train]
      pattern: '"release_id":"(rt_[0-9a-f]{16})"'

What it does

Unlike transcript-contains — which matches an author-supplied literal — this grader is dynamic: the needle is whatever a tool actually returned at runtime. It runs the pattern regex against a tool’s result to capture a value, then asserts that a subsequent assistant message contains that exact value.

A typical use: a tool returns a computed identifier (a release id, an owner, a token) and you want to assert the agent surfaced that exact value to the user rather than inventing a plausible-looking one.

Behavior

The grader walks the trajectory in order:

1. For each tool result whose tool name matches tools (exactly, or as a name. prefix), it applies the pattern regex to the result. When the regex has a capture group, group 1 is the captured value; otherwise the full match is used. An empty capture (e.g. an unmatched optional group) is skipped.
2. It then looks for an assistant message after that tool result whose content contains the captured value. Matching is a plain case-sensitive substring test.

Passes when some captured value is echoed by a later assistant message.

Fails when:

• no tool output matched the pattern regex (nothing was captured), or
• a value was captured but no subsequent assistant message echoed it, or
• the pattern regex is invalid (reported as failing evidence rather than throwing).

Evidence examples

✔ assistant echoed captured value "rt_0123456789abcdef"
✘ no "lookup_release_train" output matched "\"release_id\":\"(rt_[0-9a-f]{16})\""
✘ assistant did not echo 2 captured value(s)