Grader: transcript-matches

Taxonomy

Property	Value
Determinism	static
Cost	free
Reference	reference-free
Temporal scope	trajectory-level
Score kind	code

Config

graders:
  - type: transcript-matches
    config:
      pattern: "(?i)i will not"

Field	Type	Required	Description
pattern	string	Yes	Regular expression pattern to match against assistant messages
negate	boolean	No	When true, passes when NO assistant message matches

transcript-matches vs. output-matches

output-matches tests the pattern against only the final output (trajectory.output). transcript-matches tests it against every assistant message produced during the run, so it can assert on intermediate steps that never appear in the final answer. It passes when the pattern matches any assistant message.

Negated variant

Use transcript-not-matches as a shorthand for negate: true — it passes only when no assistant message matches (and fails if the run produced no assistant messages):

graders:
  - type: transcript-not-matches
    config:
      pattern: "(?i)error|exception"

This is equivalent to type: transcript-matches with negate: true.

Empty transcripts

The negated variant does not pass vacuously. When the trajectory contains zero assistant messages, there is nothing to inspect, so transcript-not-matches (and transcript-matches with negate: true) fails rather than reporting a misleading success. This prevents a run where the agent produced no output from silently satisfying a “must not match” assertion. The evidence in that case is:

✘ no assistant messages in transcript to check

Behavior

Constructs a RegExp from the pattern and tests it against each assistant message. This is a regex search: it matches when the pattern appears anywhere in a message. To require an entire message to match, anchor the pattern with ^ and $.

Inline flags: Use the (?flags) prefix syntax to set regex flags:

• (?i)pattern — case-insensitive matching
• (?m)pattern — multiline (^/$ match line start/end)
• (?s)pattern — dotAll (. matches newlines)
• (?ims)pattern — combine multiple flags

Passes when the pattern matches any assistant message (or matches none if negated). Fails otherwise — including when a negated check runs against a transcript with no assistant messages (see Empty transcripts).

Evidence examples

✔ transcript matches pattern /(?i)i will not/
✘ transcript does not match pattern /(?i)i will not/