Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

PyRIT Scan

pyrit_scan is the primary command-line tool for running automated security assessments and red teaming attacks against AI systems. It leverages scenarios to define attack techniques and supports flexible configuration for targeting different AI endpoints.

For configuration setup, see Configuration.

For scenario-specific examples, see AIRT, Foundry, and Garak.

Note in this doc the ! prefaces all commands in the terminal so we can run in a Jupyter Notebook.

Starting a Backend Server

pyrit_scan is a thin client that talks to a PyRIT backend server (by default at http://localhost:8000). Before running any command that reaches the backend (listing scenarios, running a scan, etc.) you need a server. Start a local one with --start-server; it launches a detached pyrit_backend process that stays up and is reused by every command below. We stop it again at the end of the notebook.

!pyrit_scan start-server
Server is running at http://localhost:8000

Quick Start

For help:

!pyrit_scan --help
usage: pyrit_scan [-h] <command> ...

PyRIT Scanner - Run AI security scenarios from the command line.

Requires a running PyRIT backend server. Use 'start-server' to launch one,
or connect to an existing server with --server-url.

Global options (usable with any command, before or after the verb):
  --server-url  --config-file  --log-level  --request-timeout  --start-server  --startup-timeout
Run 'pyrit_scan <command> --help' for full option descriptions and a command's arguments.

Examples:
  # Start the backend server
  pyrit_scan start-server

  # List scenarios, targets, or converters
  pyrit_scan list-scenarios
  pyrit_scan list-targets

  # Run single-turn cyber attacks against a target
  pyrit_scan run airt.cyber --target openai_chat --techniques single_turn

  # Run rapid response with specific datasets and concurrency
  pyrit_scan run airt.rapid_response --target openai_chat
    --techniques role_play_movie_script --dataset-names airt_hate
    --max-dataset-size 5 --max-concurrency 4

  # Attach registered converters to a technique (repeatable, applied in order)
  pyrit_scan run airt.rapid_response --target openai_chat
    --techniques role_play_movie_script:converter.translation_spanish:converter.leetspeak

  # List recent runs, then inspect one (overview by default; --view attacks for per-attack rows)
  pyrit_scan scenario-history 20
  pyrit_scan scenario-results 605d715b-7c07-4bde-a8f9-22fea0b50c4f --view attacks

  # Register a custom initializer from a Python script
  pyrit_scan add-initializer ./my_custom_init.py

  # Connect to a remote server
  pyrit_scan list-scenarios --server-url http://remote:8000

  # Stop the server
  pyrit_scan stop-server

options:
  -h, --help         show this help message and exit

commands:
  <command>
    run              Run a scenario against a target
    list-scenarios   List all available scenarios
    list-initializers
                     List all available initializers
    list-targets     List all available targets
    list-converters  List all registered converter instances
    list-datasets    List all available datasets
    add-initializer  Register initializer(s) from Python script file(s)
    scenario-results
                     Inspect the results of a completed scenario run
    scenario-history
                     List recent scenario runs
    start-server     Start a local backend server
    stop-server      Stop the backend server

Discovery

List all available scenarios:

!pyrit_scan list-scenarios
Fetching long content....

Tip: You can also surface user-defined scenarios. List your initializer script in the initialization_scripts section of the config file the backend loads (see here). The backend runs those scripts at startup and auto-discovers any Scenario subclasses they define, so start the server with that config, then list:

pyrit_scan --config-file ./my_pyrit_conf.yaml start-server
pyrit_scan list-scenarios

Initializers

PyRITInitializers are how you can configure the CLI scanner. PyRIT includes several built-in initializers you can use with the --initializers flag.

The --list-initializers command shows all available initializers. Initializers are referenced by their filename (e.g., target, scorer) regardless of which subdirectory they’re in.

List the available initializers using the --list-initializers flag.

!pyrit_scan list-initializers

Available Initializers:
================================================================================

  load_default_datasets
    Class: LoadDefaultDatasets
    Required Environment Variables: None
    Supported Parameters:
      - dataset_names: Explicit dataset names to load. Overrides the scenario-default selection.
      - tags: Load datasets whose metadata matches these tags. Overrides scenario-default selection.
    Description:
      Load datasets into memory so scenarios can run.

  preload_scenario_metadata
    Class: PreloadScenarioMetadata
    Required Environment Variables: None
    Description:
      Instantiate every registered scenario once to warm the metadata cache.

  refresh_datasets
    Class: RefreshDatasets
    Required Environment Variables: None
    Supported Parameters:
      - days [default: 30]: Refresh only datasets whose newest seed is older than this many days. 0 refreshes every selected dataset regardless of age.
      - dataset_names: Explicit dataset names to refresh; refreshes all in-memory datasets if omitted.
    Description:
      Refresh datasets already loaded in memory from their registered
      providers.

  scorer
    Class: ScorerInitializer
    Required Environment Variables: None
    Supported Parameters:
      - tags [default: ['default']]: Tags for filtering (e.g., ['default'])
    Description:
      Instantiates a collection of scorers using targets from the
      TargetRegistry and adds them to the ScorerRegistry.

  target
    Class: TargetInitializer
    Required Environment Variables: None
    Supported Parameters:
      - tags [default: ['default']]: Target tags to register (e.g., ['default'], ['default', 'scorer'], or ['all'])
      - auto_group [default: True]: Auto-create round-robin groups from targets with matching behavioral eval params
    Description:
      Target Initializer for registering pre-configured targets.

  technique
    Class: TechniqueInitializer
    Required Environment Variables: None
    Supported Parameters:
      - tags [default: ['core']]: Technique groups to register (e.g., ['core'], ['core', 'extra'], or ['all'])
    Description:
      Register scenario attack technique factories into the
      AttackTechniqueRegistry.

================================================================================

Total initializers: 6

Running Scenarios

You need a single scenario to run, you need two things:

  1. A Scenario. Many are defined in pyrit.scenario.scenarios. But you can also define your own in initialization_scripts.

  2. Initializers (which can be supplied via the --initializers flag on run, or the initializers / initialization_scripts sections of a config file (see here)). Scenarios often don’t need many arguments, but they can be configured in different ways. And at the very least, most need an objective_target (the thing you’re running a scan against) which you can configure by using the --target flag if your initializer registers targets (e.g. target initializer)

  3. Scenario Techniques (optional). These are supplied by the --techniques flag and tell the scenario what to test, but they are always optional. Also note you can obtain these by running --list-scenarios

Basic usage will look something like:

pyrit_scan run <scenario> --target <target_name> --initializers <initializer1> <initializer2> --techniques <technique1> <technique2>

You can also override scenario parameters directly from the CLI:

pyrit_scan run <scenario> --max-concurrency 10 --max-retries 3 --memory-labels '{"experiment": "test1", "version": "v2"}'

Or concretely:

!pyrit_scan run foundry.red_team_agent --target openai_chat --initializers target --techniques base64

Example with a basic configuration that runs the Foundry scenario against the objective target defined in the target initializer.

!pyrit_scan run foundry.red_team_agent --target openai_chat --initializers target --techniques base64

Running scenario: foundry.red_team_agent

  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS
  [██████████████████████████████] techniques: 2/2 (100%) | success rate: 0% | IN_PROGRESS
  [██████████████████████████████] techniques: 2/2 (100%) | success rate: 0% | COMPLETED
====================================================================================================
                                  📊 SCENARIO RESULTS: RedTeamAgent                                  
====================================================================================================

▼ Scenario Information
────────────────────────────────────────────────────────────────────────────────────────────────────
  📋 Scenario Details
    • Name: RedTeamAgent
    • Result ID: 10840c79-567c-4ecd-a3c8-fb7e9349bd57
    • Scenario Version: 1
    • PyRIT Version: 1.1.0.dev0
    • Description:
        RedTeamAgent is a preconfigured scenario that automatically generates multiple AtomicAttack instances based on
        the specified attack techniques. It supports both single-turn attacks (with various converters) and multi-turn
        attacks (Crescendo, RedTeaming), making it easy to quickly test a target against multiple attack vectors. The
        scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into their constituent attack techniques, or
        you can specify individual techniques directly. This scenario is designed for use with the Foundry AI Red
        Teaming Agent library, providing a consistent PyRIT contract for their integration.

  🎯 Target Information
    • Target Type: OpenAIChatTarget
    • Target Model: gpt-4o
    • Target Endpoint: https://pyrit-japan-test.openai.azure.com/openai/v1

  📊 Scorer Information
    ▸ Scorer Identifier
      • Scorer Type: FloatScaleThresholdScorer
      • scorer_type: true_false
      • score_aggregator: OR_
        └─ Composite of 1 scorer(s):
            • Scorer Type: AzureContentFilterScorer
            • scorer_type: float_scale

    ▸ Performance Metrics
      • Accuracy: 59.24%
      • Accuracy Std Error: ±0.0247
      • F1 Score: 0.5306
      • Precision: 0.5987
      • Recall: 0.4764
      • Average Score Time: 0.04s

▼ Overall Statistics
────────────────────────────────────────────────────────────────────────────────────────────────────
  📈 Summary
    • Total Techniques: 2
    • Total Attack Results: 8
    • Overall Success Rate: 0%
    • Unique Objectives: 4

▼ Per-Group Breakdown
────────────────────────────────────────────────────────────────────────────────────────────────────

  🔸 Group: base64
    • Number of Results: 4
    • Success Rate: 0%

  🔸 Group: baseline
    • Number of Results: 4
    • Success Rate: 0%

====================================================================================================

Or with all options and multiple techniques:

pyrit_scan run foundry.red_team_agent --target openai_chat --initializers target --techniques easy crescendo

You can also override scenario execution parameters:

# Override concurrency and retry settings
pyrit_scan run foundry.red_team_agent --target openai_chat --initializers target --max-concurrency 10 --max-retries 3

# Add custom memory labels for tracking (must be valid JSON)
pyrit_scan run foundry.red_team_agent --target openai_chat --initializers target --memory-labels '{"experiment": "test1", "version": "v2", "researcher": "alice"}'

Available CLI parameter overrides:

  • --max-concurrency <int>: Maximum number of concurrent attack executions

  • --max-retries <int>: Maximum number of automatic retries if the scenario raises an exception

  • --memory-labels <json>: Additional labels to apply to all attack runs (must be a JSON string with string keys and values)

Dataset-backed scenarios can also select a supported dataset. Requested datasets are fetched on demand, so a full dataset preload is not required:

pyrit_scan run garak.figstep --target openai_chat --dataset-names figstep_pro --max-dataset-size 1

Custom initialization scripts are loaded by the backend at startup: list them in the initialization_scripts section of the config the server loads (paths are relative to your working directory, but full paths avoid confusion). Once the server is running with that config, they apply to every run:

pyrit_scan --config-file ./my_pyrit_conf.yaml start-server
pyrit_scan run garak.encoding

Attaching Converters to a Technique

Techniques (techniques) can have a registered converter instance appended to them with the <technique>:converter.<name> syntax. The converter is added to the request side of every attack the technique produces, on top of any converters the technique already bakes in. This also works on aggregate techniques (the converter is applied to every technique the aggregate expands to).

First discover the registered converter instances with list-converters. Converters are registered by initializers that the backend runs at startup, so the initializer that registers them must be part of the server’s configuration — a built-in in the initializers section, or a custom script in the initialization_scripts section of the config the server loads (see here). Start the server with that config, then list:

pyrit_scan --config-file ./my_pyrit_conf.yaml start-server
pyrit_scan list-converters

Then reference a converter by name in --techniques:

# Add the registered "translation_spanish" converter to role_play_movie_script only
pyrit_scan run airt.rapid_response --target openai_chat --initializers target my_converters --techniques role_play_movie_script:converter.translation_spanish

Chain multiple converters (applied in order) and combine with plain techniques

pyrit_scan run airt.rapid_response --target openai_chat --initializers load_default_datasets target my_converters --techniques role_play_movie_script:converter.translation_spanish:converter.base64 many_shot

Using Custom Scenarios

You can define your own scenarios in initialization scripts. The CLI will automatically discover any Scenario subclasses and make them available:

# my_custom_scenarios.py

from pyrit.common import apply_defaults
from pyrit.prompt_target.openai.openai_chat_target import OpenAIChatTarget
from pyrit.scenario import DatasetAttackConfiguration, Scenario, ScenarioTechnique
from pyrit.score import SelfAskRefusalScorer, TrueFalseInverterScorer
from pyrit.setup import initialize_pyrit_async


class MyCustomTechnique(ScenarioTechnique):
    """Techniques for my custom scenario."""

    ALL = ("all", {"all"})
    Technique1 = ("technique1", set[str]())
    Technique2 = ("technique2", set[str]())


class MyCustomScenario(Scenario):
    """My custom scenario that does XYZ."""

    @apply_defaults
    def __init__(self, *, scenario_result_id=None, **kwargs):
        # Scenario-specific configuration only - no runtime parameters
        super().__init__(
            name="My Custom Scenario",
            version=1,
            objective_scorer=TrueFalseInverterScorer(scorer=SelfAskRefusalScorer(chat_target=OpenAIChatTarget())),
            technique_class=MyCustomTechnique,
            default_dataset_config=DatasetAttackConfiguration(dataset_names=["harmbench"]),
            scenario_result_id=scenario_result_id,
        )
        # ... your scenario-specific initialization code

    async def _build_atomic_attacks_async(self, *, context):
        # The single abstract extension point every scenario implements.
        # Read runtime inputs from `context`; return the list of AtomicAttack to run.
        # Matrix-shaped scenarios can delegate to build_matrix_atomic_attacks(context=...).
        # Example: create attacks for each technique composite
        return []


await initialize_pyrit_async(memory_db_type="InMemory")  # type: ignore
MyCustomScenario()
Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']
Loaded environment file: ./.pyrit/.env
Loaded environment file: ./.pyrit/.env.local

[pyrit:alembic] No new upgrade operations detected.
<__main__.MyCustomScenario at 0x1d2391a9d30>

Then discover and run it:

# Start the backend with a config whose initialization_scripts lists my_custom_scenarios.py
pyrit_scan --config-file ./my_pyrit_conf.yaml start-server

# List to confirm it's available
pyrit_scan list-scenarios

# Run it with parameter overrides
pyrit_scan run my_custom_scenario --max-concurrency 10

```The scenario name is automatically converted from the class name (e.g., `MyCustomScenario` becomes `my_custom_scenario`).

Inspecting Results

Each run is saved under a scenario_result_id (printed when the run finishes), so you can revisit it later without re-running. Use scenario-history to find recent run ids:

pyrit_scan scenario-history          # last 10 runs
pyrit_scan scenario-history 25       # last 25

Then inspect a run with scenario-results at the granularity you want via --view:

# Aggregate stats + per-group success rates (the default; same as the run summary)
pyrit_scan scenario-results <scenario_result_id>

# One row per attack: id, technique, objective, outcome, turns, score
pyrit_scan scenario-results <scenario_result_id> --view attacks

# Full message transcripts with the objective score per turn
pyrit_scan scenario-results <scenario_result_id> --view conversations

# Both: the attacks table followed by the transcripts
pyrit_scan scenario-results <scenario_result_id> --view full

--view picks how much detail per attack; --attack-result-ids picks which attacks (ids come from the attacks view); the two combine. Use --limit to cap how many attacks are shown; the conversations and full views default to 5 when you scope neither:

pyrit_scan scenario-results <id> --view conversations --attack-result-ids <attack_id> <attack_id>
pyrit_scan scenario-results <id> --view conversations --limit 3

Stopping the Backend Server

When you’re done, stop the local backend that we started at the top of the notebook.

!pyrit_scan stop-server
Server on port 8000 stopped.
PyRIT Documentation
PyRIT Scanner
PyRIT Documentation
PyRIT Shell