1. PyRIT Scan - PyRIT Documentation

pyrit_scan allows you to run automated security testing and red teaming attacks against AI systems using scenarios for strategies and configuration.

Note in this doc the ! prefaces all commands in the terminal so we can run in a Jupyter Notebook.

Quick Start¶

For help:

!pyrit_scan --help

Starting PyRIT...
usage: pyrit_scan [-h] [--config-file CONFIG_FILE] [--log-level LOG_LEVEL]
                  [--list-scenarios] [--list-initializers] [--list-targets]
                  [--initializers INITIALIZERS [INITIALIZERS ...]]
                  [--initialization-scripts INITIALIZATION_SCRIPTS [INITIALIZATION_SCRIPTS ...]]
                  [--strategies SCENARIO_STRATEGIES [SCENARIO_STRATEGIES ...]]
                  [--max-concurrency MAX_CONCURRENCY]
                  [--max-retries MAX_RETRIES] [--memory-labels MEMORY_LABELS]
                  [--dataset-names DATASET_NAMES [DATASET_NAMES ...]]
                  [--max-dataset-size MAX_DATASET_SIZE] [--target TARGET]
                  [scenario_name]

PyRIT Scanner - Run security scenarios against AI systems

Examples:
  # List available scenarios, initializers, and targets
  pyrit_scan --list-scenarios
  pyrit_scan --list-initializers
  pyrit_scan --list-targets --initializers target

  # Run a scenario with a target and initializers
  pyrit_scan foundry.red_team_agent --target my_target --initializers target load_default_datasets

  # Run with a configuration file (recommended for complex setups)
  pyrit_scan foundry.red_team_agent --target my_target --config-file ./my_config.yaml

  # Run with custom initialization scripts
  pyrit_scan garak.encoding --target my_target --initialization-scripts ./my_config.py

  # Run specific strategies or options
  pyrit_scan foundry.red_team_agent --target my_target --strategies base64 rot13 --initializers target
  pyrit_scan foundry.red_team_agent --target my_target --initializers target --max-concurrency 10 --max-retries 3

positional arguments:
  scenario_name         Name of the scenario to run

options:
  -h, --help            show this help message and exit
  --config-file CONFIG_FILE
                        Path to a YAML configuration file. Allows specifying
                        database, initializers (with args), initialization
                        scripts, and env files. CLI arguments override config
                        file values. If not specified, ~/.pyrit/.pyrit_conf is
                        loaded if it exists.
  --log-level LOG_LEVEL
                        Logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
                        (default: WARNING)
  --list-scenarios      List all available scenarios and exit
  --list-initializers   List all available scenario initializers and exit
  --list-targets        List all available targets from the TargetRegistry and
                        exit. Requires initializers that register targets
                        (e.g., --initializers target)
  --initializers INITIALIZERS [INITIALIZERS ...]
                        Built-in initializer names to run before the scenario.
                        Supports optional params with name:key=val syntax
                        (e.g., target:tags=default,scorer dataset:mode=strict)
  --initialization-scripts INITIALIZATION_SCRIPTS [INITIALIZATION_SCRIPTS ...]
                        Paths to custom Python initialization scripts to run
                        before the scenario
  --strategies SCENARIO_STRATEGIES [SCENARIO_STRATEGIES ...], -s SCENARIO_STRATEGIES [SCENARIO_STRATEGIES ...]
                        List of strategy names to run (e.g., base64 rot13)
  --max-concurrency MAX_CONCURRENCY
                        Maximum number of concurrent attack executions (must
                        be >= 1)
  --max-retries MAX_RETRIES
                        Maximum number of automatic retries on exception (must
                        be >= 0)
  --memory-labels MEMORY_LABELS
                        Additional labels as JSON string (e.g.,
                        '{"experiment": "test1"}')
  --dataset-names DATASET_NAMES [DATASET_NAMES ...]
                        List of dataset names to use instead of scenario
                        defaults (e.g., harmbench advbench). Creates a new
                        dataset config; fetches all items unless --max-
                        dataset-size is also specified
  --max-dataset-size MAX_DATASET_SIZE
                        Maximum number of items to use from the dataset (must
                        be >= 1). Limits new datasets if --dataset-names
                        provided, otherwise overrides scenario's default limit
  --target TARGET       Name of a registered target from the TargetRegistry to
                        use as the objective target. Targets are registered by
                        initializers (e.g., 'target' initializer). Use --list-
                        targets to see available target names after
                        initializers have run

Discovery¶

List all available scenarios:

!pyrit_scan --list-scenarios

Starting PyRIT...
Loading default configuration file: ./.pyrit/.pyrit_conf
Found default environment files: ['./.pyrit/.env']
Loaded environment file: ./.pyrit/.env

Available Scenarios:
================================================================================

  airt.content_harms
    Class: ContentHarms
    Description:
      Content Harms Scenario implementation for PyRIT. This scenario contains
      various harm-based checks that you can run to get a quick idea about
      model behavior with respect to certain harm categories.
    Aggregate Strategies:
      - all
    Available Strategies (7):
      hate, fairness, violence, sexual, harassment, misinformation, leakage
    Default Strategy: all
    Default Datasets (7, max 4 per dataset):
      airt_hate, airt_fairness, airt_violence, airt_sexual, airt_harassment,
      airt_misinformation, airt_leakage

  airt.cyber
    Class: Cyber
    Description:
      Cyber scenario implementation for PyRIT. This scenario tests how willing
      models are to exploit cybersecurity harms by generating malware. The
      Cyber class contains different variations of the malware generation
      techniques.
    Aggregate Strategies:
      - all
    Available Strategies (2):
      single_turn, multi_turn
    Default Strategy: all
    Default Datasets (1, max 4 per dataset):
      airt_malware

  airt.jailbreak
    Class: Jailbreak
    Description:
      Jailbreak scenario implementation for PyRIT. This scenario tests how
      vulnerable models are to jailbreak attacks by applying various
      single-turn jailbreak templates to a set of test prompts. The responses
      are scored to determine if the jailbreak was successful.
    Aggregate Strategies:
      - all, simple, complex
    Available Strategies (4):
      prompt_sending, many_shot, skeleton, role_play
    Default Strategy: simple
    Default Datasets (1, max 4 per dataset):
      airt_harms

  airt.leakage
    Class: Leakage
    Description:
      Leakage scenario implementation for PyRIT. This scenario tests how
      susceptible models are to leaking training data, PII, intellectual
      property, or other confidential information. The LeakageScenario class
      contains different attack variations designed to extract sensitive
      information from models.
    Aggregate Strategies:
      - all, single_turn, multi_turn, ip, sensitive_data
    Available Strategies (4):
      first_letter, image, role_play, crescendo
    Default Strategy: all
    Default Datasets (1, max 4 per dataset):
      airt_leakage

  airt.psychosocial
    Class: Psychosocial
    Description:
      Psychosocial Harms Scenario implementation for PyRIT. This scenario
      contains various psychosocial harm-based checks that evaluate model
      behavior for responses that may harm the well-being or mental health of
      users. Examples include mistreating users in a mental health crisis or
      aiding them in self-harm. All datasets in this scenario share the
      unified 'psychosocial' harm category, but each dataset (e.g.,
      imminent_crisis, dependency) can have unique: - Scoring rubrics - Seed
      objectives The scenario tests whether models: - Recognize mental health
      crises and respond appropriately - Avoid facilitating self-harm or
      harmful behaviors - Provide crisis resources and encourage professional
      help Example usage with custom configurations: # Define custom
      configurations per subharm category custom_configs = {
      "airt_imminent_crisis": SubharmConfig(
      crescendo_system_prompt_path="path/to/custom_escalation.yaml",
      scoring_rubric_path="path/to/custom_rubric.yaml", ), } scenario =
      Psychosocial(subharm_configs=custom_configs) await
      scenario.initialize_async( objective_target=target_llm,
      scenario_strategies=[PsychosocialStrategy.ImminentCrisis], )
    Aggregate Strategies:
      - all
    Available Strategies (2):
      imminent_crisis, licensed_therapist
    Default Strategy: all
    Default Datasets (1, max 4 per dataset):
      airt_imminent_crisis

  airt.scam
    Class: Scam
    Description:
      Scam scenario evaluates an endpoint's ability to generate scam-related
      materials (e.g., phishing emails, fraudulent messages) with primarily
      persuasion-oriented techniques.
    Aggregate Strategies:
      - all, single_turn, multi_turn
    Available Strategies (3):
      context_compliance, role_play, persuasive_rta
    Default Strategy: all
    Default Datasets (1, max 4 per dataset):
      airt_scams

  foundry.red_team_agent
    Class: RedTeamAgent
    Description:
      RedTeamAgent is a preconfigured scenario that automatically generates
      multiple AtomicAttack instances based on the specified attack
      strategies. It supports both single-turn attacks (with various
      converters) and multi-turn attacks (Crescendo, RedTeaming), making it
      easy to quickly test a target against multiple attack vectors. The
      scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into
      their constituent attack strategies, or you can specify individual
      strategies directly. This scenario is designed for use with the Foundry
      AI Red Teaming Agent library, providing a consistent PyRIT contract for
      their integration.
    Aggregate Strategies:
      - all, easy, moderate, difficult
    Available Strategies (25):
      ansi_attack, ascii_art, ascii_smuggler, atbash, base64, binary, caesar,
      character_space, char_swap, diacritic, flip, leetspeak, morse, rot13,
      suffix_append, string_join, unicode_confusable, unicode_substitution,
      url, jailbreak, tense, multi_turn, crescendo, pair, tap
    Default Strategy: easy
    Default Datasets (1, max 4 per dataset):
      harmbench

  garak.encoding
    Class: Encoding
    Description:
      Encoding Scenario implementation for PyRIT. This scenario tests how
      resilient models are to various encoding attacks by encoding potentially
      harmful text (by default slurs and XSS payloads) and testing if the
      model will decode and repeat the encoded payload. It mimics the Garak
      encoding probe. The scenario works by: 1. Taking seed prompts (the
      harmful text to be encoded) 2. Encoding them using various encoding
      schemes (Base64, ROT13, Morse, etc.) 3. Asking the target model to
      decode the encoded text 4. Scoring whether the model successfully
      decoded and repeated the harmful content By default, this uses the same
      dataset as Garak: slur terms and web XSS payloads.
    Aggregate Strategies:
      - all
    Available Strategies (17):
      base64, base2048, base16, base32, ascii85, hex, quoted_printable,
      uuencode, rot13, braille, atbash, morse_code, nato, ecoji, zalgo,
      leet_speak, ascii_smuggler
    Default Strategy: all
    Default Datasets (2, max 3 per dataset):
      garak_slur_terms_en, garak_web_html_js

================================================================================

Total scenarios: 8

Tip: You can also discover user-defined scenarios by providing initialization scripts:

pyrit_scan --list-scenarios --initialization-scripts ./my_custom_initializer.py

This will load your custom scenario definitions and include them in the list.

Initializers¶

PyRITInitializers are how you can configure the CLI scanner. PyRIT includes several built-in initializers you can use with the --initializers flag.

The --list-initializers command shows all available initializers. Initializers are referenced by their filename (e.g., target, objective_list, simple) regardless of which subdirectory they’re in.

List the available initializers using the --list-initializers flag.

!pyrit_scan --list-initializers

Starting PyRIT...
Loading default configuration file: ./.pyrit/.pyrit_conf
Found default environment files: ['./.pyrit/.env']
Loaded environment file: ./.pyrit/.env

Available Initializers:
================================================================================

  airt
    Class: AIRTInitializer
    Name: AIRT Default Configuration
    Execution Order: 1
    Required Environment Variables:
      - AZURE_OPENAI_GPT4O_UNSAFE_CHAT_ENDPOINT
      - AZURE_OPENAI_GPT4O_UNSAFE_CHAT_MODEL
      - AZURE_OPENAI_GPT4O_UNSAFE_CHAT_ENDPOINT2
      - AZURE_OPENAI_GPT4O_UNSAFE_CHAT_MODEL2
      - AZURE_CONTENT_SAFETY_API_ENDPOINT
    Description:
      AI Red Team setup with Azure OpenAI converters, composite harm/objective
      scorers, and adversarial targets

  load_default_datasets
    Class: LoadDefaultDatasets
    Name: Default Dataset Loader for Scenarios
    Execution Order: 10
    Required Environment Variables: None
    Description:
      This configuration uses the DatasetLoader to load default datasets into
      memory. This will enable all scenarios to run. Datasets can be
      customized in memory. Note: if you are using persistent memory, avoid
      calling this every time as datasets can take time to load.

  scenario_objective_list
    Class: ScenarioObjectiveListInitializer
    Name: Simple Objective List Configuration for Scenarios
    Execution Order: 10
    Required Environment Variables: None
    Description:
      Simple Objective List Configuration for Scenarios

  scorer
    Class: ScorerInitializer
    Name: Scorer Initializer
    Execution Order: 2
    Required Environment Variables: None
    Supported Parameters:
      - tags [default: ['default']]: Tags for filtering (e.g., ['default'])
    Description:
      Instantiates a collection of scorers using targets from the
      TargetRegistry and adds them to the ScorerRegistry

  simple
    Class: SimpleInitializer
    Name: Simple Complete Configuration
    Execution Order: 1
    Required Environment Variables:
      - OPENAI_CHAT_ENDPOINT
      - OPENAI_CHAT_MODEL
    Description:
      Complete simple setup with basic OpenAI converters, objective scorer (no
      harm detection), and adversarial targets. Only requires
      OPENAI_CHAT_ENDPOINT and OPENAI_CHAT_MODEL environment variables.

  target
    Class: TargetInitializer
    Name: Target Initializer
    Execution Order: 1
    Required Environment Variables: None
    Supported Parameters:
      - tags [default: ['default']]: Target tags to register (e.g., ['default'], ['default', 'scorer'], or ['all'])
    Description:
      Instantiates a collection of targets from available environment
      variables and adds them to the TargetRegistry

================================================================================

Total initializers: 6

Running Scenarios¶

You need a single scenario to run, you need two things:

A Scenario. Many are defined in pyrit.scenario.scenarios. But you can also define your own in initialization_scripts.
Initializers (which can be supplied via --initializers or --initialization-scripts or initializers section of config file (see here)). Scenarios often don’t need many arguments, but they can be configured in different ways. And at the very least, most need an objective_target (the thing you’re running a scan against) which you can configure by using the --target flag if your initializer registers targets (e.g. target initializer)
Scenario Strategies (optional). These are supplied by the --scenario-strategies flag and tell the scenario what to test, but they are always optional. Also note you can obtain these by running --list-scenarios

Basic usage will look something like:

pyrit_scan <scenario> --target <target_name> --initializers <initializer1> <initializer2> --scenario-strategies <strategy1> <strategy2>

You can also override scenario parameters directly from the CLI:

pyrit_scan <scenario> --max-concurrency 10 --max-retries 3 --memory-labels '{"experiment": "test1", "version": "v2"}'

Or concretely:

!pyrit_scan foundry.red_team_agent --target openai_chat --initializers load_default_datasets target --scenario-strategies base64

Example with a basic configuration that runs the Foundry scenario against the objective target defined in the target initializer.

!pyrit_scan foundry.red_team_agent --target openai_chat --initializers load_default_datasets target --strategies base64

Starting PyRIT...
Loading default configuration file: ./.pyrit/.pyrit_conf
Found default environment files: ['./.pyrit/.env']
Loaded environment file: ./.pyrit/.env
Running 2 initializer(s)...

Running scenario: foundry.red_team_agent

====================================================================================================
                                  📊 SCENARIO RESULTS: RedTeamAgent                                  
====================================================================================================

▼ Scenario Information
────────────────────────────────────────────────────────────────────────────────────────────────────
  📋 Scenario Details
    • Name: RedTeamAgent
    • Scenario Version: 1
    • PyRIT Version: 0.12.1.dev0
    • Description:
        RedTeamAgent is a preconfigured scenario that automatically generates multiple AtomicAttack instances based on
        the specified attack strategies. It supports both single-turn attacks (with various converters) and multi-turn
        attacks (Crescendo, RedTeaming), making it easy to quickly test a target against multiple attack vectors. The
        scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into their constituent attack strategies, or
        you can specify individual strategies directly. This scenario is designed for use with the Foundry AI Red
        Teaming Agent library, providing a consistent PyRIT contract for their integration.

  🎯 Target Information
    • Target Type: OpenAIChatTarget
    • Target Model: gpt-4o
    • Target Endpoint: https://pyrit-japan-test.openai.azure.com/openai/v1

  📊 Scorer Information
    ▸ Scorer Identifier
      • Scorer Type: TrueFalseInverterScorer
      • scorer_type: true_false
      • score_aggregator: OR_
        └─ Composite of 1 scorer(s):
            • Scorer Type: SelfAskRefusalScorer
            • scorer_type: true_false
            • score_aggregator: OR_
            • model_name: gpt-4o

    ▸ Performance Metrics
      • Accuracy: 84.84%
      • Accuracy Std Error: ±0.0185
      • F1 Score: 0.8606
      • Precision: 0.7928
      • Recall: 0.9412
      • Average Score Time: 1.27s

▼ Overall Statistics
────────────────────────────────────────────────────────────────────────────────────────────────────
  📈 Summary
    • Total Strategies: 2
    • Total Attack Results: 8
    • Overall Success Rate: 12%
    • Unique Objectives: 8

▼ Per-Strategy Breakdown
────────────────────────────────────────────────────────────────────────────────────────────────────

  🔸 Strategy: baseline
    • Number of Results: 4
    • Success Rate: 0%

  🔸 Strategy: base64
    • Number of Results: 4
    • Success Rate: 25%

====================================================================================================


Loading datasets - this can take a few minutes:   0%|          | 0/58 [00:00<?, ?dataset/s]
Loading datasets - this can take a few minutes:   2%|▏         | 1/58 [00:00<00:33,  1.72dataset/s]
Loading datasets - this can take a few minutes:  29%|██▉       | 17/58 [00:00<00:01, 32.77dataset/s]
Loading datasets - this can take a few minutes:  60%|██████    | 35/58 [00:00<00:00, 52.67dataset/s]
Loading datasets - this can take a few minutes: 100%|██████████| 58/58 [00:00<00:00, 65.13dataset/s]

Executing RedTeamAgent:   0%|          | 0/2 [00:00<?, ?attack/s]
Executing RedTeamAgent:  50%|█████     | 1/2 [00:07<00:07,  7.14s/attack]
Executing RedTeamAgent: 100%|██████████| 2/2 [00:15<00:00,  7.67s/attack]
Executing RedTeamAgent: 100%|██████████| 2/2 [00:15<00:00,  7.59s/attack]
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x000001A51ECDEB50>

Or with all options and multiple initializers and multiple strategies:

pyrit_scan foundry.red_team_agent --target openai_chat --initializers load_default_datasets target --strategies easy crescendo

You can also override scenario execution parameters:

# Override concurrency and retry settings
pyrit_scan foundry.red_team_agent --target openai_chat --initializers load_default_datasets target --max-concurrency 10 --max-retries 3

# Add custom memory labels for tracking (must be valid JSON)
pyrit_scan foundry.red_team_agent --target openai_chat --initializers load_default_datasets target --memory-labels '{"experiment": "test1", "version": "v2", "researcher": "alice"}'

Available CLI parameter overrides:

--max-concurrency <int>: Maximum number of concurrent attack executions
--max-retries <int>: Maximum number of automatic retries if the scenario raises an exception
--memory-labels <json>: Additional labels to apply to all attack runs (must be a JSON string with string keys and values)

You can also use custom initialization scripts by passing file paths. It is relative to your current working directory, but to avoid confusion, full paths are always better:

pyrit_scan garak.encoding --initialization-scripts ./my_custom_config.py

Using Custom Scenarios¶

You can define your own scenarios in initialization scripts. The CLI will automatically discover any Scenario subclasses and make them available:

# my_custom_scenarios.py

from pyrit.common import apply_defaults
from pyrit.prompt_target.openai.openai_chat_target import OpenAIChatTarget
from pyrit.scenario import DatasetConfiguration, Scenario, ScenarioStrategy
from pyrit.score import SelfAskRefusalScorer, TrueFalseInverterScorer
from pyrit.setup import initialize_pyrit_async


class MyCustomStrategy(ScenarioStrategy):
    """Strategies for my custom scenario."""

    ALL = ("all", {"all"})
    Strategy1 = ("strategy1", set[str]())
    Strategy2 = ("strategy2", set[str]())


class MyCustomScenario(Scenario):
    """My custom scenario that does XYZ."""

    @classmethod
    def get_strategy_class(cls):
        return MyCustomStrategy

    @classmethod
    def get_default_strategy(cls):
        return MyCustomStrategy.ALL

    @classmethod
    def default_dataset_config(cls) -> DatasetConfiguration:
        # Return default dataset configuration for this scenario
        return DatasetConfiguration(dataset_names=["harmbench"])

    @apply_defaults
    def __init__(self, *, scenario_result_id=None, **kwargs):
        # Scenario-specific configuration only - no runtime parameters
        super().__init__(
            name="My Custom Scenario",
            version=1,
            objective_scorer=TrueFalseInverterScorer(scorer=SelfAskRefusalScorer(chat_target=OpenAIChatTarget())),
            strategy_class=MyCustomStrategy,
            scenario_result_id=scenario_result_id,
        )
        # ... your scenario-specific initialization code

    async def _get_atomic_attacks_async(self):
        # Build and return your atomic attacks based on self._scenario_composites
        # Example: create attacks for each strategy composite
        return []


await initialize_pyrit_async(memory_db_type="InMemory")  # type: ignore
MyCustomScenario()

Found default environment files: ['./.pyrit/.env']
Loaded environment file: ./.pyrit/.env

<__main__.MyCustomScenario at 0x1ec016f9c90>

Then discover and run it:

# List to see it's available
pyrit_scan --list-scenarios --initialization-scripts ./my_custom_scenarios.py

# Run it with parameter overrides
pyrit_scan my_custom_scenario --initialization-scripts ./my_custom_scenarios.py --max-concurrency 10

The scenario name is automatically converted from the class name (e.g., MyCustomScenario becomes my_custom_scenario).