API Reference — Evaluators

Built-in evaluators. All extend BaseEvaluator and support composition via |, &, ~.

evaluators

Built-in evaluator implementations.

Re-exports: ToolCalled, ResponseContains, SideEffectOccurred.

ToolCalled

Python

ToolCalled(tool_name, /, **param_predicates)

Bases: BaseEvaluator

Detects whether a tool was called, optionally matching parameters.

Parameter predicates can be exact values or callables. Callables receive the parameter value and return True/False.

This evaluator only detects conditions. It does not reason about observability gaps. That adjustment is owned by the execution strategy.

Parameters:

Name	Type	Description	Default
tool_name	str	The tool to look for (positional-only).	required
**param_predicates	dict[str, Any | Callable[[Any], bool]]	Parameter name -> expected value or predicate.	{}

Initialize with tool name and optional parameter predicates.

Source code in rampart/evaluators/tool_called.py

def __init__(
    self,
    tool_name: str,
    /,
    **param_predicates: Any | Callable[[Any], bool],  # noqa: ANN401
) -> None:
    """Initialize with tool name and optional parameter predicates."""
    self._tool_name = tool_name
    self._predicates = param_predicates

evaluate_async async

Python

evaluate_async(*, context)

Check all turns for a matching tool call.

Source code in rampart/evaluators/tool_called.py

async def evaluate_async(self, *, context: EvalContext) -> EvalResult:
    """Check all turns for a matching tool call."""
    for tc in context.all_tool_calls:
        if tc.name == self._tool_name and self._matches(tc):
            return EvalResult(
                outcome=EvalOutcome.DETECTED,
                evidence=[f"{tc.name}({tc.arguments})"],
                rationale=f"Tool '{tc.name}' called with matching parameters",
            )

    return EvalResult(
        outcome=EvalOutcome.NOT_DETECTED,
        rationale=f"Tool '{self._tool_name}' not called with matching parameters",
    )

ResponseContains

Python

ResponseContains(target, /, *, case_sensitive=False)

Bases: BaseEvaluator

Detects whether response text contains a target pattern.

Accepts a plain string (substring match), compiled regex, or callable predicate.

Parameters:

Name	Type	Description	Default
target	str | Pattern | Callable[[str], bool]	Pattern to find (positional-only).	required
case_sensitive	bool	Whether substring match is case-sensitive.	False

Initialize with target pattern and case sensitivity.

Source code in rampart/evaluators/response_contains.py

def __init__(
    self,
    target: str | re.Pattern[str] | Callable[[str], bool],
    /,
    *,
    case_sensitive: bool = False,
) -> None:
    """Initialize with target pattern and case sensitivity."""
    self._target = target
    self._case_sensitive = case_sensitive

evaluate_async async

Python

evaluate_async(*, context)

Check response text for the target pattern.

Source code in rampart/evaluators/response_contains.py

async def evaluate_async(self, *, context: EvalContext) -> EvalResult:
    """Check response text for the target pattern."""
    text = context.text

    if callable(self._target):
        found = self._target(text)
    elif isinstance(self._target, re.Pattern):
        found = bool(self._target.search(text))
    else:
        check_text = text if self._case_sensitive else text.lower()
        check_target = (
            self._target if self._case_sensitive else self._target.lower()
        )
        found = check_target in check_text

    if found:
        return EvalResult(
            outcome=EvalOutcome.DETECTED,
            evidence=["Pattern found in response text"],
            rationale="Response contains target pattern",
        )

    return EvalResult(
        outcome=EvalOutcome.NOT_DETECTED,
        rationale="Target pattern not found in response text",
    )

SideEffectOccurred

Python

SideEffectOccurred(kind, /, **detail_predicates)

Bases: BaseEvaluator

Detects whether a side effect of a given kind occurred.

Parameters:

Name	Type	Description	Default
kind	str	The side effect kind to look for (positional-only).	required
**detail_predicates	dict[str, Any | Callable[[Any], bool]]	Detail field -> expected value or callable predicate.	{}

Initialize with side effect kind and optional predicates.

Source code in rampart/evaluators/side_effect.py

def __init__(
    self,
    kind: str,
    /,
    **detail_predicates: Any | Callable[[Any], bool],  # noqa: ANN401
) -> None:
    """Initialize with side effect kind and optional predicates."""
    self._kind = kind
    self._predicates = detail_predicates

evaluate_async async

Python

evaluate_async(*, context)

Check all turns for a matching side effect.

Source code in rampart/evaluators/side_effect.py

async def evaluate_async(self, *, context: EvalContext) -> EvalResult:
    """Check all turns for a matching side effect."""
    for se in context.all_side_effects:
        if se.kind == self._kind and self._matches(se):
            return EvalResult(
                outcome=EvalOutcome.DETECTED,
                evidence=[f"Side effect '{se.kind}': {se.details}"],
                rationale=f"Side effect '{se.kind}' detected",
            )

    return EvalResult(
        outcome=EvalOutcome.NOT_DETECTED,
        rationale=f"Side effect '{self._kind}' not detected",
    )