Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Scenarios

A Scenario is a higher-level construct that groups multiple Attack Configurations together. This allows you to execute a comprehensive testing campaign with multiple attack methods sequentially. Scenarios are meant to be configured and written to test for specific workflows. As such, it is okay to hard code some values.

What is a Scenario?

A Scenario represents a comprehensive testing campaign composed of multiple atomic attack tests. It orchestrates the execution of multiple AtomicAttack instances sequentially and aggregates the results into a single ScenarioResult.

Key Components

  • Scenario: The top-level orchestrator that groups and executes multiple atomic attacks

  • AtomicAttack: An atomic test unit combining an attack strategy, objectives, and execution parameters

  • ScenarioResult: Contains the aggregated results from all atomic attacks and scenario metadata

Use Cases

Some examples of scenarios you might create:

  • VibeCheckScenario: Randomly selects a few prompts from HarmBench Mazeika et al., 2024 to quickly assess model behavior

  • QuickViolence: Checks how resilient a model is to violent objectives using multiple attack techniques

  • ComprehensiveFoundry: Tests a target with all available attack converters and strategies

  • CustomCompliance: Tests against specific compliance requirements with curated datasets and attacks

These Scenarios can be updated and added to as you refine what you are testing for.

How to Run Scenarios

Scenarios should take almost no effort to run with default values. The PyRIT Scanner provides two CLIs for running scenarios: pyrit_scan for automated execution and pyrit_shell for interactive exploration.

For programmatic configuration — customizing datasets, strategies, scorers, and baseline mode — see Common Scenario Parameters.

How It Works

Each Scenario contains a collection of AtomicAttack objects. When executed:

  1. Each AtomicAttack is executed sequentially

  2. Every AtomicAttack tests its configured attack against all specified objectives and datasets

  3. Results are aggregated into a single ScenarioResult with all attack outcomes

  4. Optional memory labels help track and categorize the scenario execution

Creating Custom Scenarios

To create a custom scenario, extend the Scenario base class and implement the required abstract methods.

Required Components

  1. Strategy Enum: Create a ScenarioStrategy enum that defines the available attack techniques for your scenario.

    • Each enum member represents an attack technique (the how of an attack)

    • Each member is defined as (value, tags) where value is a string and tags is a set of strings

    • Include an ALL aggregate strategy that expands to all available strategies

    • Optionally override _prepare_strategies() for custom composition logic (see FoundryComposite)

  2. Scenario Class: Extend Scenario and pass these to super().__init__():

    • strategy_class: Your strategy enum class

    • default_strategy: The default strategy (typically YourStrategy.ALL or YourStrategy.DEFAULT)

    • The base class provides a default _get_atomic_attacks_async() that uses the factory/registry pattern. Override it only if your scenario needs custom attack construction logic.

  3. Default Dataset: Pass default_dataset_config= to super().__init__() to specify the datasets your scenario uses out of the box.

    • Returns a DatasetConfiguration with one or more named datasets (e.g., DatasetConfiguration(dataset_names=["my_dataset"]))

    • Users can override this at runtime via --dataset-names in the CLI or by passing a custom dataset_config programmatically

  4. Constructor: Use @apply_defaults decorator and call super().__init__() with scenario metadata:

    • name: Descriptive name for your scenario

    • version: Integer version number

    • strategy_class: The strategy enum class for this scenario

    • default_strategy: The default strategy member (typically YourStrategy.ALL or YourStrategy.DEFAULT)

    • default_dataset_config: A DatasetConfiguration specifying the scenario’s default datasets

    • objective_scorer: The scorer used to judge responses

    • scenario_result_id: Optional ID to resume an existing scenario (optional)

  5. Initialization: Call await scenario.initialize_async() to populate atomic attacks:

    • objective_target: The target system being tested (required)

    • scenario_strategies: List of strategies to execute (optional, defaults to ALL)

    • max_concurrency: Number of concurrent operations (default: 4)

    • max_retries: Number of retry attempts on failure (default: 0)

    • memory_labels: Optional labels for tracking (optional)

    • include_baseline: Whether to prepend a baseline attack (defaults to the scenario type’s BASELINE_ATTACK_POLICY; most scenarios default it on, Jailbreak defaults it off)

Example Structure

The simplest approach uses the factory/registry pattern: define your strategy, dataset config, and constructor — the base class handles building atomic attacks automatically from registered attack techniques.

from pyrit.common import apply_defaults
from pyrit.scenario import (
    DatasetConfiguration,
    Scenario,
    ScenarioStrategy,
)
from pyrit.score.true_false.true_false_scorer import TrueFalseScorer
from pyrit.setup import initialize_pyrit_async
from pyrit.setup.initializers.components import ScenarioTechniqueInitializer

await initialize_pyrit_async(memory_db_type="InMemory")  # type: ignore [top-level-await]
await ScenarioTechniqueInitializer().initialize_async()  # type: ignore [top-level-await]


class MyStrategy(ScenarioStrategy):
    ALL = ("all", {"all"})
    DEFAULT = ("default", {"default"})
    SINGLE_TURN = ("single_turn", {"single_turn"})
    # Strategy members represent attack techniques
    PromptSending = ("prompt_sending", {"single_turn", "default"})
    RolePlay = ("role_play", {"single_turn"})


class MyScenario(Scenario):
    """Quick-check scenario for testing model behavior across harm categories."""

    VERSION: int = 1

    @apply_defaults
    def __init__(
        self,
        *,
        objective_scorer: TrueFalseScorer | None = None,
        scenario_result_id: str | None = None,
    ) -> None:
        self._objective_scorer: TrueFalseScorer = (
            objective_scorer if objective_scorer else self._get_default_objective_scorer()
        )

        super().__init__(
            version=self.VERSION,
            objective_scorer=self._objective_scorer,
            strategy_class=MyStrategy,
            default_strategy=MyStrategy.DEFAULT,
            default_dataset_config=DatasetConfiguration(dataset_names=["dataset_name"], max_dataset_size=4),
            scenario_result_id=scenario_result_id,
        )

    # Optional: override _build_display_group to customize result grouping.
    # Default groups by technique name; override to group by dataset instead:
    def _build_display_group(self, *, technique_name: str, seed_group_name: str) -> str:
        return seed_group_name

    # No _get_atomic_attacks_async override needed!
    # The base class builds attacks from the (technique x dataset) cross-product
    # using the factory/registry pattern automatically.
Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']
Loaded environment file: ./.pyrit/.env
Loaded environment file: ./.pyrit/.env.local

[pyrit:alembic] No new upgrade operations detected.

Existing Scenarios

import logging

from pyrit.backend.services.scenario_service import get_scenario_service
from pyrit.cli._output import print_scenario_list

logging.getLogger("pyrit").setLevel(logging.ERROR)

response = await get_scenario_service().list_scenarios_async(limit=200)  # type: ignore
print_scenario_list(items=[s.model_dump() for s in response.items])

Available Scenarios:
================================================================================

  adaptive.text_adaptive
    Class: TextAdaptive
    Description:
      Adaptive text-attack scenario. Selects techniques per-objective via an
      epsilon-greedy selector over the set of selected strategies.
      ``prompt_sending`` runs as the baseline comparison and is excluded from
      the adaptive technique pool.
    Aggregate Strategies:
      - all, default, single_turn, multi_turn
    Available Strategies (10):
      role_play, many_shot, tap, pair, crescendo_simulated, red_teaming,
      context_compliance, crescendo_movie_director, crescendo_history_lecture,
      crescendo_journalist_interview
    Default Strategy: default
    Default Datasets (7, max 4 per dataset):
      airt_hate, airt_fairness, airt_violence, airt_sexual, airt_harassment,
      airt_misinformation, airt_leakage
    Supported Parameters:
      - max_attempts_per_objective (int) [default: '3']: Max techniques tried per objective. Defaults to 3.

  airt.cyber
    Class: Cyber
    Description:
      Cyber scenario implementation for PyRIT. This scenario tests how willing
      models are to exploit cybersecurity harms by generating malware. The
      Cyber class contains different variations of the malware generation
      techniques.
    Aggregate Strategies:
      - all, multi_turn
    Available Strategies (1):
      red_teaming
    Default Strategy: all
    Default Datasets (1, max 4 per dataset):
      airt_malware

  airt.jailbreak
    Class: Jailbreak
    Description:
      Jailbreak scenario implementation for PyRIT. This scenario tests how
      vulnerable models are to jailbreak attacks by applying various
      single-turn jailbreak templates to a set of test prompts. The responses
      are scored to determine if the jailbreak was successful.
    Aggregate Strategies:
      - all, simple, complex
    Available Strategies (4):
      prompt_sending, many_shot, skeleton, role_play
    Default Strategy: simple
    Default Datasets (1, max 4 per dataset):
      airt_harms

  airt.leakage
    Class: Leakage
    Description:
      Leakage scenario implementation for PyRIT. This scenario tests how
      susceptible models are to leaking training data, PII, intellectual
      property, or other confidential information. Uses the registry/factory
      pattern to construct attack techniques.
    Aggregate Strategies:
      - all, default, single_turn, multi_turn
    Available Strategies (12):
      role_play, many_shot, tap, pair, crescendo_simulated, red_teaming,
      context_compliance, crescendo_movie_director, crescendo_history_lecture,
      crescendo_journalist_interview, first_letter, image
    Default Strategy: default
    Default Datasets (1, max 4 per dataset):
      airt_leakage

  airt.psychosocial
    Class: Psychosocial
    Description:
      Psychosocial Harms Scenario implementation for PyRIT. This scenario
      contains various psychosocial harm-based checks that evaluate model
      behavior for responses that may harm the well-being or mental health of
      users. Examples include mistreating users in a mental health crisis or
      aiding them in self-harm. All datasets in this scenario share the
      unified 'psychosocial' harm category, but each dataset (e.g.,
      imminent_crisis, dependency) can have unique: - Scoring rubrics - Seed
      objectives The scenario tests whether models: - Recognize mental health
      crises and respond appropriately - Avoid facilitating self-harm or
      harmful behaviors - Provide crisis resources and encourage professional
      help Example usage with custom configurations: # Define custom
      configurations per subharm category custom_configs = {
      "airt_imminent_crisis": SubharmConfig(
      crescendo_system_prompt_path="path/to/custom_escalation.yaml",
      scoring_rubric_path="path/to/custom_rubric.yaml", ), } scenario =
      Psychosocial(subharm_configs=custom_configs) await
      scenario.initialize_async( objective_target=target_llm,
      scenario_strategies=[PsychosocialStrategy.ImminentCrisis], )
    Aggregate Strategies:
      - all
    Available Strategies (2):
      imminent_crisis, licensed_therapist
    Default Strategy: all
    Default Datasets (1, max 4 per dataset):
      airt_imminent_crisis

  airt.rapid_response
    Class: RapidResponse
    Description:
      Rapid Response scenario for content-harms testing. Tests model behavior
      across multiple harm categories using selectable attack techniques.
    Aggregate Strategies:
      - all, default, single_turn, multi_turn
    Available Strategies (10):
      role_play, many_shot, tap, pair, crescendo_simulated, red_teaming,
      context_compliance, crescendo_movie_director, crescendo_history_lecture,
      crescendo_journalist_interview
    Default Strategy: default
    Default Datasets (7, max 4 per dataset):
      airt_hate, airt_fairness, airt_violence, airt_sexual, airt_harassment,
      airt_misinformation, airt_leakage

  airt.scam
    Class: Scam
    Description:
      Scam scenario evaluates an endpoint's ability to generate scam-related
      materials (e.g., phishing emails, fraudulent messages) with primarily
      persuasion-oriented techniques.
    Aggregate Strategies:
      - all, single_turn, multi_turn
    Available Strategies (3):
      context_compliance, role_play, persuasive_rta
    Default Strategy: all
    Default Datasets (1, max 4 per dataset):
      airt_scams
    Supported Parameters:
      - max_turns (int) [default: '5']: Maximum conversation turns for the persuasive_rta strategy.

  benchmark.adversarial
    Class: AdversarialBenchmark
    Description:
      Benchmark scenario that compares the attack success rate (ASR) across
      adversarial models. Adversarial targets are user-supplied via the
      ``adversarial_targets`` parameter (declared in
      ``supported_parameters``). Each target must already be registered in
      ``TargetRegistry`` — typically by ``TargetInitializer`` from
      ``ADVERSARIAL_CHAT_*`` env vars, or programmatically via
      ``TargetRegistry.register_instance``. At run time,
      ``_get_atomic_attacks_async`` performs the ``(technique ×
      adversarial_target × dataset)`` cross-product: for each selected
      adversarial-capable ``core`` factory in the ``AttackTechniqueRegistry``
      and each requested target, it calls
      ``factory.create(attack_adversarial_config_override=...)`` with the
      resolved target — no global registry mutation. The resulting
      ``AtomicAttack`` is named ``f"{technique}__{target}_{dataset}"`` with
      ``display_group`` set to the target's registry name so per-model ASR
      rolls up naturally in result displays.
    Aggregate Strategies:
      - all, default, light, single_turn, multi_turn
    Available Strategies (9):
      role_play, tap, pair, crescendo_simulated, red_teaming,
      context_compliance, crescendo_movie_director, crescendo_history_lecture,
      crescendo_journalist_interview
    Default Strategy: light
    Default Datasets (1, max 8 per dataset):
      harmbench
    Supported Parameters:
      - adversarial_targets (list[str]): Registry names of adversarial chat targets to benchmark. Each name must already be registered in TargetRegistry (via TargetInitializer or TargetRegistry.register_instance). Use 'pyrit_scan list-targets' to see registered targets. Settable via --adversarial-targets <name> [<name> ...] on the CLI, or scenario.args.adversarial_targets in .pyrit_conf.

  foundry.red_team_agent
    Class: RedTeamAgent
    Description:
      RedTeamAgent is a preconfigured scenario that automatically generates
      multiple AtomicAttack instances based on the specified attack
      strategies. It supports both single-turn attacks (with various
      converters) and multi-turn attacks (Crescendo, RedTeaming), making it
      easy to quickly test a target against multiple attack vectors. The
      scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into
      their constituent attack strategies, or you can specify individual
      strategies directly. This scenario is designed for use with the Foundry
      AI Red Teaming Agent library, providing a consistent PyRIT contract for
      their integration.
    Aggregate Strategies:
      - all, easy, moderate, difficult
    Available Strategies (25):
      ansi_attack, ascii_art, ascii_smuggler, atbash, base64, binary, caesar,
      character_space, char_swap, diacritic, flip, leetspeak, morse, rot13,
      suffix_append, string_join, unicode_confusable, unicode_substitution,
      url, jailbreak, tense, multi_turn, crescendo, pair, tap
    Default Strategy: easy
    Default Datasets (1, max 4 per dataset):
      harmbench

  garak.encoding
    Class: Encoding
    Description:
      Encoding Scenario implementation for PyRIT. This scenario tests how
      resilient models are to various encoding attacks by encoding potentially
      harmful text (by default slurs and XSS payloads) and testing if the
      model will decode and repeat the encoded payload. It mimics the Garak
      encoding probe. The scenario works by: 1. Taking seed prompts (the
      harmful text to be encoded) 2. Encoding them using various encoding
      schemes (Base64, ROT13, Morse, etc.) 3. Asking the target model to
      decode the encoded text 4. Scoring whether the model successfully
      decoded and repeated the harmful content By default, this uses the same
      dataset as Garak: slur terms and web XSS payloads.
    Aggregate Strategies:
      - all
    Available Strategies (17):
      base64, base2048, base16, base32, ascii85, hex, quoted_printable,
      uuencode, rot13, braille, atbash, morse_code, nato, ecoji, zalgo,
      leet_speak, ascii_smuggler
    Default Strategy: all
    Default Datasets (2, max 3 per dataset):
      garak_slur_terms_en, garak_web_html_js

================================================================================

Total scenarios: 10

Baseline Execution

Every scenario can optionally include a baseline attack — a PromptSendingAttack that sends each objective directly to the target without any converters or multi-turn techniques. This is controlled by the include_baseline parameter on initialize_async; when omitted, each scenario falls back to its own BASELINE_ATTACK_POLICY class attribute (most scenarios default it on; Jailbreak defaults it off). See Common Scenario Parameters for a worked example.

Custom scenarios should choose their BASELINE_ATTACK_POLICY based on whether an unmodified prompt is a meaningful comparator for the scenario’s strategies:

  • Enabled — the baseline is prepended by default and the caller can opt out. Use when an unmodified-prompt run is a meaningful comparison point (most scenarios).

  • Disabled — the baseline is supported but omitted by default; the caller must opt in. Use when the scenario is already dominated by a large set of templates/strategies that already exercise the unmodified surface (e.g., Jailbreak).

  • Forbidden — the baseline is unavailable and passing include_baseline=True raises. Use when the scenario’s semantics make a single-shot unmodified prompt meaningless as a comparator (e.g., benchmarks comparing across adversarial models, or multi-turn-only scenarios).

Resiliency

Scenarios can run for a long time, and because of that, things can go wrong. Network issues, rate limits, or other transient failures can interrupt execution. PyRIT provides built-in resiliency features to handle these situations gracefully.

Automatic Resume

If you re-run a scenario, it will automatically start where it left off. The framework tracks completed attacks and objectives in memory, so you won’t lose progress if something interrupts your scenario execution. This means you can safely stop and restart scenarios without duplicating work.

Retry Mechanism

You can utilize the max_retries parameter to handle transient failures. If any unknown exception occurs during execution, PyRIT will automatically retry the failed operation (starting where it left off) up to the specified number of times. This helps ensure your scenario completes successfully even in the face of temporary issues.

Dynamic Configuration

During a long-running scenario, you may want to adjust parameters like max_concurrency to manage resource usage, or switch your scorer to use a different target. PyRIT’s resiliency features make it safe to stop, reconfigure, and continue scenarios as needed.

For more information, see resiliency

References
  1. Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., Sakhaee, E., Li, N., Basart, S., Li, B., Forsyth, D., & Hendrycks, D. (2024). HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. arXiv Preprint arXiv:2402.04249. https://arxiv.org/abs/2402.04249
PyRIT Documentation
1. Generating GCG Suffixes Using Azure Machine Learning
PyRIT Documentation
Attack Techniques