Common Scenario Parameters - PyRIT Documentation

This guide covers the key parameters for configuring scenarios programmatically: datasets, techniques, baseline execution, and custom scorers. All examples use RedTeamAgent but the patterns apply to any scenario.

Two selection axes: Techniques select attack techniques (how attacks run — e.g., prompt sending, role play, TAP). Datasets select objectives (what is tested — e.g., harm categories, compliance topics). Use --dataset-names on the CLI to filter by content category.

Running scenarios from the command line? See the Scanner documentation.

Setup¶

Initialize PyRIT and create the target we want to test.

from pathlib import Path

from pyrit.output import output_scenario_async
from pyrit.registry import TargetRegistry
from pyrit.scenario.foundry import FoundryTechnique, RedTeamAgent
from pyrit.setup import initialize_from_config_async

await initialize_from_config_async(config_path=Path("../../scanner/pyrit_conf.yaml"))  # type: ignore

objective_target = TargetRegistry.get_registry_singleton().instances.get("openai_chat")

Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']
Loaded environment file: ./.pyrit/.env
Loaded environment file: ./.pyrit/.env.local

[pyrit:alembic] No new upgrade operations detected.

TextAdaptive: _EXCLUDED_TECHNIQUES entries ['prompt_sending'] are not in the current scenario-techniques catalog ['context_compliance', 'crescendo_history_lecture', 'crescendo_journalist_interview', 'crescendo_movie_director', 'crescendo_simulated', 'many_shot', 'pair', 'red_teaming', 'role_play', 'tap', 'violent_durian']; the exclusion is a no-op for those entries. Remove stale entries or update the catalog.

Dataset Configuration¶

DatasetAttackConfiguration controls which prompts (objectives) are sent to the target. The simplest approach uses dataset_names to load datasets by name from memory. By default, RedTeamAgent loads four random objectives from HarmBench Mazeika et al., 2024.

from pyrit.scenario import DatasetAttackConfiguration

dataset_config = DatasetAttackConfiguration(dataset_names=["harmbench"], max_dataset_size=2)

For more control, use SeedDatasetProvider to fetch datasets and pass explicit seed_groups. This is useful when you need to filter, combine, or inspect the prompts before running.

from pyrit.datasets import SeedDatasetProvider
from pyrit.models import SeedGroup

datasets = await SeedDatasetProvider.fetch_datasets_async(dataset_names=["harmbench"])  # type: ignore
seed_groups: list[SeedGroup] = datasets[0].seed_groups  # type: ignore

# Pass explicit seed_groups instead of dataset_names
dataset_config = DatasetAttackConfiguration(seed_groups=seed_groups, max_dataset_size=2)

Technique Selection and Composition¶

FoundryTechnique is an enum that defines which attack techniques the scenario runs. There are three ways to specify techniques:

Individual techniques — a single converter or multi-turn attack:

single_technique = [FoundryTechnique.Base64]

Aggregate techniques — tag-based groups that expand to all matching techniques. For example, EASY expands to all techniques tagged as easy (Base64, Binary, CharSwap, etc.):

aggregate_technique = [FoundryTechnique.EASY]

Composite techniques — pair an attack with one or more converters using FoundryComposite. For example, to run Crescendo with Base64 encoding applied:

from pyrit.scenario.foundry import FoundryComposite

composite_technique = [FoundryComposite(attack=FoundryTechnique.Crescendo, converters=[FoundryTechnique.Base64])]

You can mix all three types in a single list:

scenario_techniques = [
    FoundryTechnique.Base64,
    FoundryTechnique.Binary,
    FoundryComposite(attack=FoundryTechnique.Crescendo, converters=[FoundryTechnique.Caesar]),
]

Baseline Execution¶

The baseline sends each objective directly to the target without any converters or multi-turn techniques. It is included automatically when include_baseline=True (the default for scenarios that support a baseline). This is useful for:

Measuring default defenses — how does the target respond to unmodified harmful prompts?
Establishing comparison points — compare baseline refusal rates against attack-enhanced runs
Calculating attack lift — how much does each technique improve over the baseline?

baseline_scenario = RedTeamAgent()
baseline_scenario.set_params_from_args(  # type: ignore
    args={
        "objective_target": objective_target,
        "scenario_techniques": None,  # Uses default techniques; baseline is prepended automatically
        "dataset_config": dataset_config,
    }
)
await baseline_scenario.initialize_async()  # type: ignore
baseline_result = await baseline_scenario.run_async()  # type: ignore
await output_scenario_async(baseline_result)  # type: ignore [top-level-await]


====================================================================================================
                                  📊 SCENARIO RESULTS: RedTeamAgent                                  
====================================================================================================

▼ Scenario Information
────────────────────────────────────────────────────────────────────────────────────────────────────
  📋 Scenario Details
    • Name: RedTeamAgent
    • Scenario Version: 1
    • PyRIT Version: 0.15.0.dev0
    • Description:
        RedTeamAgent is a preconfigured scenario that automatically generates multiple AtomicAttack instances based on
        the specified attack techniques. It supports both single-turn attacks (with various converters) and multi-turn
        attacks (Crescendo, RedTeaming), making it easy to quickly test a target against multiple attack vectors. The
        scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into their constituent attack techniques, or
        you can specify individual techniques directly. This scenario is designed for use with the Foundry AI Red
        Teaming Agent library, providing a consistent PyRIT contract for their integration.

  🎯 Target Information
    • Target Type: OpenAIChatTarget
    • Target Model: gpt-4o-japan-nilfilter
    • Target Endpoint: https://pyrit-japan-test.openai.azure.com/openai/v1

  📊 Scorer Information
    ▸ Scorer Identifier
      • Scorer Type: FloatScaleThresholdScorer
      • scorer_type: true_false
      • score_aggregator: OR_
        └─ Composite of 1 scorer(s):
            • Scorer Type: AzureContentFilterScorer
            • scorer_type: float_scale

    ▸ Performance Metrics
      • Accuracy: 59.24%
      • Accuracy Std Error: ±0.0247
      • F1 Score: 0.5306
      • Precision: 0.5987
      • Recall: 0.4764
      • Average Score Time: 0.04s

▼ Overall Statistics
────────────────────────────────────────────────────────────────────────────────────────────────────
  📈 Summary
    • Total Techniques: 21
    • Total Attack Results: 42
    • Overall Success Rate: 28%
    • Unique Objectives: 2

▼ Per-Group Breakdown
────────────────────────────────────────────────────────────────────────────────────────────────────

  🔸 Group: baseline
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: ansi_attack
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Group: ascii_art
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Group: ascii_smuggler
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Group: base64
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Group: atbash
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Group: caesar
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: binary
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: character_space
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: char_swap
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: diacritic
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: flip
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: leetspeak
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: morse
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: suffix_append
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: rot13
    • Number of Results: 2
    • Success Rate: 100%

  🔸 Group: string_join
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Group: unicode_confusable
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Group: unicode_substitution
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Group: url
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Group: jailbreak
    • Number of Results: 2
    • Success Rate: 0%

====================================================================================================

Sorting the Per-Group Breakdown by Success Rate¶

By default, the Per-Group Breakdown lists groups in the order they were executed. The baseline run above produces a row for every default technique, which makes it hard to spot the most successful ones at a glance. Pass sort_groups_by_success_rate=True to output_scenario_async to re-render the same result with the highest success rates at the top (groups with equal rates keep their original relative order):

await output_scenario_async(baseline_result, sort_groups_by_success_rate=True)


====================================================================================================
                                  📊 SCENARIO RESULTS: RedTeamAgent                                  
====================================================================================================

▼ Scenario Information
────────────────────────────────────────────────────────────────────────────────────────────────────
  📋 Scenario Details
    • Name: RedTeamAgent
    • Scenario Version: 1
    • PyRIT Version: 0.15.0.dev0
    • Description:
        RedTeamAgent is a preconfigured scenario that automatically generates multiple AtomicAttack instances based on
        the specified attack techniques. It supports both single-turn attacks (with various converters) and multi-turn
        attacks (Crescendo, RedTeaming), making it easy to quickly test a target against multiple attack vectors. The
        scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into their constituent attack techniques, or
        you can specify individual techniques directly. This scenario is designed for use with the Foundry AI Red
        Teaming Agent library, providing a consistent PyRIT contract for their integration.

  🎯 Target Information
    • Target Type: OpenAIChatTarget
    • Target Model: gpt-4o-japan-nilfilter
    • Target Endpoint: https://pyrit-japan-test.openai.azure.com/openai/v1

  📊 Scorer Information
    ▸ Scorer Identifier
      • Scorer Type: FloatScaleThresholdScorer
      • scorer_type: true_false
      • score_aggregator: OR_
        └─ Composite of 1 scorer(s):
            • Scorer Type: AzureContentFilterScorer
            • scorer_type: float_scale

    ▸ Performance Metrics
      • Accuracy: 59.24%
      • Accuracy Std Error: ±0.0247
      • F1 Score: 0.5306
      • Precision: 0.5987
      • Recall: 0.4764
      • Average Score Time: 0.04s

▼ Overall Statistics
────────────────────────────────────────────────────────────────────────────────────────────────────
  📈 Summary
    • Total Techniques: 21
    • Total Attack Results: 42
    • Overall Success Rate: 28%
    • Unique Objectives: 2

▼ Per-Group Breakdown
────────────────────────────────────────────────────────────────────────────────────────────────────

  🔸 Group: rot13
    • Number of Results: 2
    • Success Rate: 100%

  🔸 Group: baseline
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: caesar
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: binary
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: character_space
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: char_swap
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: diacritic
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: flip
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: leetspeak
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: morse
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: suffix_append
    • Number of Results: 2
    • Success Rate: 50%

  🔸 Group: ansi_attack
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Group: ascii_art
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Group: ascii_smuggler
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Group: base64
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Group: atbash
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Group: string_join
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Group: unicode_confusable
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Group: unicode_substitution
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Group: url
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Group: jailbreak
    • Number of Results: 2
    • Success Rate: 0%

====================================================================================================

To disable the automatic baseline entirely (e.g., when you only want attack techniques with no comparison), set include_baseline=False in the run params:

scenario = RedTeamAgent()
scenario.set_params_from_args(
    args={
        "objective_target": objective_target,
        "scenario_techniques": [FoundryTechnique.Base64],
        "include_baseline": False,
    }
)
await scenario.initialize_async()

Custom Scorers¶

By default, RedTeamAgent uses a composite scorer with Azure Content Filter and SelfAsk Refusal scorers. You can override this by passing your own AttackScoringConfig with a custom objective_scorer.

For example, to use an inverted refusal scorer (where “True” means the target refused):

from pyrit.executor.attack import AttackScoringConfig
from pyrit.prompt_target import OpenAIChatTarget
from pyrit.score import SelfAskRefusalScorer, TrueFalseInverterScorer

refusal_scorer = SelfAskRefusalScorer(chat_target=OpenAIChatTarget())
inverted_scorer = TrueFalseInverterScorer(scorer=refusal_scorer)

custom_scenario = RedTeamAgent(
    attack_scoring_config=AttackScoringConfig(objective_scorer=inverted_scorer),
)
custom_scenario.set_params_from_args(  # type: ignore
    args={
        "objective_target": objective_target,
        "scenario_techniques": [FoundryTechnique.Base64],
        "dataset_config": dataset_config,
    }
)
await custom_scenario.initialize_async()  # type: ignore

custom_result = await custom_scenario.run_async()  # type: ignore
await output_scenario_async(custom_result)


====================================================================================================
                                  📊 SCENARIO RESULTS: RedTeamAgent                                  
====================================================================================================

▼ Scenario Information
────────────────────────────────────────────────────────────────────────────────────────────────────
  📋 Scenario Details
    • Name: RedTeamAgent
    • Scenario Version: 1
    • PyRIT Version: 0.15.0.dev0
    • Description:
        RedTeamAgent is a preconfigured scenario that automatically generates multiple AtomicAttack instances based on
        the specified attack techniques. It supports both single-turn attacks (with various converters) and multi-turn
        attacks (Crescendo, RedTeaming), making it easy to quickly test a target against multiple attack vectors. The
        scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into their constituent attack techniques, or
        you can specify individual techniques directly. This scenario is designed for use with the Foundry AI Red
        Teaming Agent library, providing a consistent PyRIT contract for their integration.

  🎯 Target Information
    • Target Type: OpenAIChatTarget
    • Target Model: gpt-4o-japan-nilfilter
    • Target Endpoint: https://pyrit-japan-test.openai.azure.com/openai/v1

  📊 Scorer Information
    ▸ Scorer Identifier
      • Scorer Type: TrueFalseInverterScorer
      • scorer_type: true_false
      • score_aggregator: OR_
        └─ Composite of 1 scorer(s):
            • Scorer Type: SelfAskRefusalScorer
            • scorer_type: true_false
            • score_aggregator: OR_
            • model_name: gpt-4o-japan-nilfilter

    ▸ Performance Metrics
      Official evaluation has not been run yet for this specific configuration

▼ Overall Statistics
────────────────────────────────────────────────────────────────────────────────────────────────────
  📈 Summary
    • Total Techniques: 2
    • Total Attack Results: 4
    • Overall Success Rate: 0%
    • Unique Objectives: 2

▼ Per-Group Breakdown
────────────────────────────────────────────────────────────────────────────────────────────────────

  🔸 Group: baseline
    • Number of Results: 2
    • Success Rate: 0%

  🔸 Group: base64
    • Number of Results: 2
    • Success Rate: 0%

====================================================================================================

References¶

Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., Sakhaee, E., Li, N., Basart, S., Li, B., Forsyth, D., & Hendrycks, D. (2024). HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. arXiv Preprint arXiv:2402.04249. https://arxiv.org/abs/2402.04249