PyRIT Scan - PyRIT Documentation

pyrit_scan is the primary command-line tool for running automated security assessments and red teaming attacks against AI systems. It leverages scenarios to define attack strategies and supports flexible configuration for targeting different AI endpoints.

For configuration setup, see Configuration.

For scenario-specific examples, see AIRT, Foundry, and Garak.

Note in this doc the ! prefaces all commands in the terminal so we can run in a Jupyter Notebook.

Quick Start¶

For help:

!pyrit_scan --help

Discovery¶

List all available scenarios:

!pyrit_scan --list-scenarios

Tip: You can also discover user-defined scenarios by providing initialization scripts:

pyrit_scan --list-scenarios --initialization-scripts ./my_custom_initializer.py

This will load your custom scenario definitions and include them in the list.

Initializers¶

PyRITInitializers are how you can configure the CLI scanner. PyRIT includes several built-in initializers you can use with the --initializers flag.

The --list-initializers command shows all available initializers. Initializers are referenced by their filename (e.g., target, scorer, simple) regardless of which subdirectory they’re in.

List the available initializers using the --list-initializers flag.

!pyrit_scan --list-initializers

Running Scenarios¶

You need a single scenario to run, you need two things:

A Scenario. Many are defined in pyrit.scenario.scenarios. But you can also define your own in initialization_scripts.
Initializers (which can be supplied via --initializers or --initialization-scripts or initializers section of config file (see here)). Scenarios often don’t need many arguments, but they can be configured in different ways. And at the very least, most need an objective_target (the thing you’re running a scan against) which you can configure by using the --target flag if your initializer registers targets (e.g. target initializer)
Scenario Strategies (optional). These are supplied by the --scenario-strategies flag and tell the scenario what to test, but they are always optional. Also note you can obtain these by running --list-scenarios

Basic usage will look something like:

pyrit_scan <scenario> --target <target_name> --initializers <initializer1> <initializer2> --scenario-strategies <strategy1> <strategy2>

You can also override scenario parameters directly from the CLI:

pyrit_scan <scenario> --max-concurrency 10 --max-retries 3 --memory-labels '{"experiment": "test1", "version": "v2"}'

Or concretely:

!pyrit_scan foundry.red_team_agent --target openai_chat --initializers load_default_datasets target --scenario-strategies base64

Example with a basic configuration that runs the Foundry scenario against the objective target defined in the target initializer.

!pyrit_scan foundry.red_team_agent --target openai_chat --initializers load_default_datasets target --strategies base64

Or with all options and multiple initializers and multiple strategies:

pyrit_scan foundry.red_team_agent --target openai_chat --initializers load_default_datasets target --strategies easy crescendo

You can also override scenario execution parameters:

# Override concurrency and retry settings
pyrit_scan foundry.red_team_agent --target openai_chat --initializers load_default_datasets target --max-concurrency 10 --max-retries 3

# Add custom memory labels for tracking (must be valid JSON)
pyrit_scan foundry.red_team_agent --target openai_chat --initializers load_default_datasets target --memory-labels '{"experiment": "test1", "version": "v2", "researcher": "alice"}'

Available CLI parameter overrides:

--max-concurrency <int>: Maximum number of concurrent attack executions
--max-retries <int>: Maximum number of automatic retries if the scenario raises an exception
--memory-labels <json>: Additional labels to apply to all attack runs (must be a JSON string with string keys and values)

You can also use custom initialization scripts by passing file paths. It is relative to your current working directory, but to avoid confusion, full paths are always better:

pyrit_scan garak.encoding --initialization-scripts ./my_custom_config.py

Using Custom Scenarios¶

You can define your own scenarios in initialization scripts. The CLI will automatically discover any Scenario subclasses and make them available:

# my_custom_scenarios.py

from pyrit.common import apply_defaults
from pyrit.prompt_target.openai.openai_chat_target import OpenAIChatTarget
from pyrit.scenario import DatasetConfiguration, Scenario, ScenarioStrategy
from pyrit.score import SelfAskRefusalScorer, TrueFalseInverterScorer
from pyrit.setup import initialize_pyrit_async


class MyCustomStrategy(ScenarioStrategy):
    """Strategies for my custom scenario."""

    ALL = ("all", {"all"})
    Strategy1 = ("strategy1", set[str]())
    Strategy2 = ("strategy2", set[str]())


class MyCustomScenario(Scenario):
    """My custom scenario that does XYZ."""

    @classmethod
    def get_strategy_class(cls):
        return MyCustomStrategy

    @classmethod
    def get_default_strategy(cls):
        return MyCustomStrategy.ALL

    @classmethod
    def default_dataset_config(cls) -> DatasetConfiguration:
        # Return default dataset configuration for this scenario
        return DatasetConfiguration(dataset_names=["harmbench"])

    @apply_defaults
    def __init__(self, *, scenario_result_id=None, **kwargs):
        # Scenario-specific configuration only - no runtime parameters
        super().__init__(
            name="My Custom Scenario",
            version=1,
            objective_scorer=TrueFalseInverterScorer(scorer=SelfAskRefusalScorer(chat_target=OpenAIChatTarget())),
            strategy_class=MyCustomStrategy,
            scenario_result_id=scenario_result_id,
        )
        # ... your scenario-specific initialization code

    async def _get_atomic_attacks_async(self):
        # Override only if your scenario needs custom attack construction.
        # The base class provides a default that uses the factory/registry pattern.
        # Example: create attacks for each strategy composite
        return []


await initialize_pyrit_async(memory_db_type="InMemory")  # type: ignore
MyCustomScenario()

Then discover and run it:

# List to see it's available
pyrit_scan --list-scenarios --initialization-scripts ./my_custom_scenarios.py

# Run it with parameter overrides
pyrit_scan my_custom_scenario --initialization-scripts ./my_custom_scenarios.py --max-concurrency 10

The scenario name is automatically converted from the class name (e.g., MyCustomScenario becomes my_custom_scenario).