Grader: transcript-contains

Taxonomy

Property	Value
Determinism	static
Cost	free
Reference	reference-free
Temporal scope	trajectory-level
Score kind	code

Config

graders:
  - type: transcript-contains
    config:
      substring: "could you clarify"
      case_sensitive: false # optional, default: false

Field	Type	Required	Default	Description
substring	string	Yes	—	The text to search for across all assistant messages
case_sensitive	boolean	No	false	Whether the match is case-sensitive
negate	boolean	No	false	When true, passes when NO assistant message contains it

transcript-contains vs. output-contains

output-contains checks only the final output (trajectory.output). transcript-contains checks every assistant message produced during the run, so it can assert on intermediate steps — a clarifying question, an interim plan, or a mid-run refusal — that never appear in the final answer. It passes when any assistant message contains the substring.

Negated variant

Use transcript-not-contains as a shorthand for negate: true — it passes only when no assistant message contains the substring (and fails if the run produced no assistant messages):

graders:
  # Assert the agent never mentioned a forbidden command anywhere in the run.
  - type: transcript-not-contains
    config:
      substring: "rm -rf"

This is equivalent to type: transcript-contains with negate: true.

Empty transcripts

The negated variant does not pass vacuously. When the trajectory contains zero assistant messages, there is nothing to inspect, so transcript-not-contains (and transcript-contains with negate: true) fails rather than reporting a misleading success. This prevents a run where the agent produced no output from silently satisfying a “must not contain” assertion. The evidence in that case is:

✘ no assistant messages in transcript to check

Behavior

Scans every assistant message in the trajectory for the specified substring. By default, comparison is case-insensitive.

Passes when the substring is found in any assistant message (or NOT found in any if negated). Fails otherwise — including when a negated check runs against a transcript with no assistant messages (see Empty transcripts).

Evidence examples

✔ 'could you clarify' found in transcript
✘ 'could you clarify' NOT found in transcript