Common Scenario Parameters - PyRIT Documentation

This guide covers the key parameters for configuring scenarios programmatically: datasets, strategies, baseline execution, and custom scorers. All examples use RedTeamAgent but the patterns apply to any scenario.

Two selection axes: Strategies select attack techniques (how attacks run — e.g., prompt sending, role play, TAP). Datasets select objectives (what is tested — e.g., harm categories, compliance topics). Use --dataset-names on the CLI to filter by content category.

Running scenarios from the command line? See the Scanner documentation.

Setup¶

Initialize PyRIT and create the target we want to test.

from pathlib import Path

from pyrit.registry import TargetRegistry
from pyrit.scenario.printer.console_printer import ConsoleScenarioResultPrinter
from pyrit.scenario.scenarios.foundry import FoundryStrategy, RedTeamAgent
from pyrit.setup import initialize_from_config_async

await initialize_from_config_async(config_path=Path("../../scanner/pyrit_conf.yaml"))  # type: ignore

objective_target = TargetRegistry.get_registry_singleton().get_instance_by_name("openai_chat")
printer = ConsoleScenarioResultPrinter()

Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']
Loaded environment file: ./.pyrit/.env
Loaded environment file: ./.pyrit/.env.local

Dataset Configuration¶

DatasetConfiguration controls which prompts (objectives) are sent to the target. The simplest approach uses dataset_names to load datasets by name from memory. By default, RedTeamAgent loads four random objectives from HarmBench Mazeika et al., 2024.

from pyrit.scenario import DatasetConfiguration

dataset_config = DatasetConfiguration(dataset_names=["harmbench"], max_dataset_size=2)

For more control, use SeedDatasetProvider to fetch datasets and pass explicit seed_groups. This is useful when you need to filter, combine, or inspect the prompts before running.

from pyrit.datasets import SeedDatasetProvider
from pyrit.models import SeedGroup

datasets = await SeedDatasetProvider.fetch_datasets_async(dataset_names=["harmbench"])  # type: ignore
seed_groups: list[SeedGroup] = datasets[0].seed_groups  # type: ignore

# Pass explicit seed_groups instead of dataset_names
dataset_config = DatasetConfiguration(seed_groups=seed_groups, max_dataset_size=2)

Strategy Selection and Composition¶

FoundryStrategy is an enum that defines which attack strategies the scenario runs. There are three ways to specify strategies:

Individual strategies — a single converter or multi-turn attack:

single_strategy = [FoundryStrategy.Base64]

Aggregate strategies — tag-based groups that expand to all matching strategies. For example, EASY expands to all strategies tagged as easy (Base64, Binary, CharSwap, etc.):

aggregate_strategy = [FoundryStrategy.EASY]

Composite strategies — pair an attack with one or more converters using FoundryComposite. For example, to run Crescendo with Base64 encoding applied:

from pyrit.scenario.scenarios.foundry import FoundryComposite

composite_strategy = [FoundryComposite(attack=FoundryStrategy.Crescendo, converters=[FoundryStrategy.Base64])]

You can mix all three types in a single list:

scenario_strategies = [
    FoundryStrategy.Base64,
    FoundryStrategy.Binary,
    FoundryComposite(attack=FoundryStrategy.Crescendo, converters=[FoundryStrategy.Caesar]),
]

Baseline Execution¶

The baseline sends each objective directly to the target without any converters or multi-turn strategies. It is included automatically when include_baseline=True (the default). This is useful for:

Measuring default defenses — how does the target respond to unmodified harmful prompts?
Establishing comparison points — compare baseline refusal rates against attack-enhanced runs
Calculating attack lift — how much does each strategy improve over the baseline?

baseline_scenario = RedTeamAgent()
await baseline_scenario.initialize_async(  # type: ignore
    objective_target=objective_target,
    scenario_strategies=None,  # Uses default strategies; baseline is prepended automatically
    dataset_config=dataset_config,
)
baseline_result = await baseline_scenario.run_async()  # type: ignore
await printer.print_summary_async(baseline_result)  # type: ignore


====================================================================================================
                                  📊 SCENARIO RESULTS: RedTeamAgent                                  
====================================================================================================

▼ Scenario Information
────────────────────────────────────────────────────────────────────────────────────────────────────
  📋 Scenario Details
    • Name: RedTeamAgent
    • Scenario Version: 1
    • PyRIT Version: 0.12.1.dev0
    • Description:
        RedTeamAgent is a preconfigured scenario that automatically generates multiple AtomicAttack instances based on
        the specified attack strategies. It supports both single-turn attacks (with various converters) and multi-turn
        attacks (Crescendo, RedTeaming), making it easy to quickly test a target against multiple attack vectors. The
        scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into their constituent attack strategies, or
        you can specify individual strategies directly. This scenario is designed for use with the Foundry AI Red
        Teaming Agent library, providing a consistent PyRIT contract for their integration.

  🎯 Target Information
    • Target Type: OpenAIChatTarget
    • Target Model: gpt-4o
    • Target Endpoint: https://pyrit-dev.openai.azure.com/openai/v1

  📊 Scorer Information
    ▸ Scorer Identifier
      • Scorer Type: TrueFalseInverterScorer
      • scorer_type: true_false
      • score_aggregator: OR_
        └─ Composite of 1 scorer(s):
            • Scorer Type: SelfAskRefusalScorer
            • scorer_type: true_false
            • score_aggregator: OR_
            • model_name: gpt-5.4

    ▸ Performance Metrics
      • Accuracy: 87.85%
      • Accuracy Std Error: ±0.0164
      • F1 Score: 0.8750
      • Precision: 0.8705
      • Recall: 0.8796
      • Average Score Time: 0.66s

▼ Overall Statistics
────────────────────────────────────────────────────────────────────────────────────────────────────
  📈 Summary
    • Total Strategies: 1
    • Total Attack Results: 2
    • Overall Success Rate: 0%
    • Unique Objectives: 2

▼ Per-Strategy Breakdown
────────────────────────────────────────────────────────────────────────────────────────────────────

  🔸 Strategy: baseline
    • Number of Results: 2
    • Success Rate: 0%

====================================================================================================

To disable the automatic baseline entirely (e.g., when you only want attack strategies with no comparison), set include_baseline=False in the constructor:

scenario = RedTeamAgent(include_baseline=False)
await scenario.initialize_async(
    objective_target=objective_target,
    scenario_strategies=[FoundryStrategy.Base64],
)

Custom Scorers¶

By default, RedTeamAgent uses a composite scorer with Azure Content Filter and SelfAsk Refusal scorers. You can override this by passing your own AttackScoringConfig with a custom objective_scorer.

For example, to use an inverted refusal scorer (where “True” means the target refused):

from pyrit.executor.attack import AttackScoringConfig
from pyrit.prompt_target import OpenAIChatTarget
from pyrit.score import SelfAskRefusalScorer, TrueFalseInverterScorer

refusal_scorer = SelfAskRefusalScorer(chat_target=OpenAIChatTarget())
inverted_scorer = TrueFalseInverterScorer(scorer=refusal_scorer)

custom_scenario = RedTeamAgent(
    attack_scoring_config=AttackScoringConfig(objective_scorer=inverted_scorer),
)
await custom_scenario.initialize_async(  # type: ignore
    objective_target=objective_target,
    scenario_strategies=[FoundryStrategy.Base64],
    dataset_config=dataset_config,
)
custom_result = await custom_scenario.run_async()  # type: ignore
await printer.print_summary_async(custom_result)  # type: ignore


====================================================================================================
                                  📊 SCENARIO RESULTS: RedTeamAgent                                  
====================================================================================================

▼ Scenario Information
────────────────────────────────────────────────────────────────────────────────────────────────────
  📋 Scenario Details
    • Name: RedTeamAgent
    • Scenario Version: 1
    • PyRIT Version: 0.12.1.dev0
    • Description:
        RedTeamAgent is a preconfigured scenario that automatically generates multiple AtomicAttack instances based on
        the specified attack strategies. It supports both single-turn attacks (with various converters) and multi-turn
        attacks (Crescendo, RedTeaming), making it easy to quickly test a target against multiple attack vectors. The
        scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into their constituent attack strategies, or
        you can specify individual strategies directly. This scenario is designed for use with the Foundry AI Red
        Teaming Agent library, providing a consistent PyRIT contract for their integration.

  🎯 Target Information
    • Target Type: OpenAIChatTarget
    • Target Model: gpt-4o
    • Target Endpoint: https://pyrit-dev.openai.azure.com/openai/v1

  📊 Scorer Information
    ▸ Scorer Identifier
      • Scorer Type: TrueFalseInverterScorer
      • scorer_type: true_false
      • score_aggregator: OR_
        └─ Composite of 1 scorer(s):
            • Scorer Type: SelfAskRefusalScorer
            • scorer_type: true_false
            • score_aggregator: OR_
            • model_name: gpt-4o

    ▸ Performance Metrics
      Official evaluation has not been run yet for this specific configuration

▼ Overall Statistics
────────────────────────────────────────────────────────────────────────────────────────────────────
  📈 Summary
    • Total Strategies: 2
    • Total Attack Results: 4
    • Overall Success Rate: 0%
    • Unique Objectives: 4

▼ Per-Strategy Breakdown
────────────────────────────────────────────────────────────────────────────────────────────────────

  🔸 Strategy: baseline
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Strategy: base64
    • Number of Results: 2
    • Success Rate: 0%

====================================================================================================

References¶

Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., Sakhaee, E., Li, N., Basart, S., Li, B., Forsyth, D., & Hendrycks, D. (2024). HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. arXiv Preprint arXiv:2402.04249. https://arxiv.org/abs/2402.04249