Grader: output-contains

Taxonomy

Property	Value
Determinism	static
Cost	low
Portability	t1-universal
Reference	reference-free
Temporal scope	trajectory-level
Score kind	code

Config

graders:
  - type: output-contains
    config:
      substring: "hello world"
      case_sensitive: false # optional, default: false

Field	Type	Required	Default	Description
substring	string	Yes	—	The text to search for in the agent’s output
case_sensitive	boolean	No	false	Whether the match is case-sensitive
negate	boolean	No	false	When true, passes when the substring is NOT found

Negated variant

Use output-not-contains as a shorthand for negate: true:

graders:
  - type: output-not-contains
    config:
      substring: "error"

This is equivalent to type: output-contains with negate: true.

Behavior

Checks whether trajectory.output contains the specified substring. By default, comparison is case-insensitive.

Passes when the substring is found (or NOT found if negated). Fails otherwise.

Evidence examples

✔ 'hello world' found in output
✘ 'hello world' NOT found in output

Source

async grade(input: GraderInput): Promise<GraderResult> {
  const caseSensitive = input.config.case_sensitive ?? false;
  const output = caseSensitive
    ? input.trajectory.output
    : input.trajectory.output.toLowerCase();
  const substring = caseSensitive
    ? input.config.substring
    : input.config.substring.toLowerCase();
  const passed = output.includes(substring);

  return {
    name: this.metadata.name,
    kind: "code",
    passed,
    score: passed ? 1 : 0,
    evidence: passed
      ? `'${input.config.substring}' found in output`
      : `'${input.config.substring}' NOT found in output`,
    label: passed ? "correct" : "incorrect",
  };
}