Skip to content

CI Integration

RAMPART tests run as standard pytest tests. This guide covers patterns for CI pipelines.

Running in CI

Bash
pytest tests/ -v --tb=short

RAMPART tests interact with real or simulated agents and may take longer than unit tests. Set appropriate timeouts:

Bash
pytest tests/ -v --timeout=300

Parallel Execution

For faster CI runs, use pytest-xdist:

Bash
pip install pytest-xdist
pytest tests/ -n auto

RAMPART aggregates results across worker processes and emits a single unified report under any --dist mode. Trial markers do not affect xdist scheduling because they do not clone tests.

Trial Markers for Statistical Confidence

Use @pytest.mark.trial(n=, threshold=) for tests where a single run is not conclusive:

Python
from rampart import Attacks, execute_trials_async

@pytest.mark.trial(n=10, threshold=0.8)
async def test_injection_resistance(adapter, trial_config):
    population = await execute_trials_async(
        execution_factory=lambda: Attacks.xpia(...),
        adapter=adapter,
        n=trial_config.n,
        threshold=trial_config.threshold,
    )
    assert population, population.summary

The test controls population execution. CI can change its depth with --rampart-trials=N without changing the declared threshold.

Structured Reports

Register the pytest_rampart_sinks hook to write JSON reports for downstream processing:

Python
# conftest.py
from pathlib import Path

from rampart.reporting import JsonFileReportSink


def pytest_rampart_sinks(config):
    return [JsonFileReportSink(output_dir=Path(".report"))]

The JSON file contains aggregate statistics and per-result data that CI dashboards can consume. The hook is resolved on the controller, so it behaves identically in single-process and pytest-xdist CI runs. See Registering Sinks.

Pytest Options

RAMPART is configured via pytest options and Python (sinks, adapters, payloads).

--rampart-xdist-max-bytes

Maximum size in bytes of each serialized Result when running under pytest-xdist. Defaults to 16777216 (16 MiB). Oversized Results are replaced by truncation markers and the controller marks the run as incomplete. Also configurable via the rampart_xdist_max_bytes ini option.

Bash
pytest -n auto --rampart-xdist-max-bytes=134217728   # 128 MB

Environment Variables

Your adapter and test configuration typically read environment variables. Setting them locally for ad-hoc runs:

Bash
export AGENT_API_KEY="..."
export AGENT_ENDPOINT="https://..."
pytest tests/
PowerShell
$env:AGENT_API_KEY = "..."
$env:AGENT_ENDPOINT = "https://..."
pytest tests/

Then consume them in your adapter and configuration:

Python
import os
from rampart.core.llm import LLMConfig

@pytest.fixture
def adapter():
    return MyAdapter(
        api_key=os.environ["AGENT_API_KEY"],
        endpoint=os.environ["AGENT_ENDPOINT"],
    )

# For LLM-driven attacks
llm = LLMConfig(
    model="gpt-4o",
    endpoint=os.environ["OPENAI_ENDPOINT"],
    api_key=os.environ.get("OPENAI_API_KEY"),  # None → azure-identity
    deployment=os.environ.get("OPENAI_DEPLOYMENT"),
)

Exit Codes

RAMPART does not alter pytest's exit codes:

Exit Code Meaning
0 All tests passed
1 Some tests failed
2 Test execution interrupted
5 No tests collected