pyrit.scenario

High-level scenario classes for running attack configurations.

AtomicAttack¶

Represents a single atomic attack test combining an attack strategy and dataset.

An AtomicAttack is an executable unit that executes a configured attack against all objectives in a dataset. Multiple AtomicAttacks can be grouped together into larger test scenarios for comprehensive security testing and evaluation.

The AtomicAttack uses SeedAttackGroups as the single source of truth for objectives, prepended conversations, and next messages. Each SeedAttackGroup must have an objective set.

An AttackTechnique bundles the attack strategy with an optional SeedAttackTechniqueGroup, cleanly separating “how to attack” from “what to attack” (the objective).

Constructor Parameters:

Methods:

drop_seed_groups_with_hashes¶

drop_seed_groups_with_hashes(hashes: set[str]) → None

Drop seed groups whose objective_sha256 is in hashes.

This is the resume filter: within an atomic attack, objective_sha256 is the stable identity (enforced unique by __init__). Content-derived keys are robust to reordering and resampling, so resume produces the right remaining-work set even when get_seed_groups() is rebuilt from scratch on each run_async().

filter_seed_groups_by_objectives¶

filter_seed_groups_by_objectives(remaining_objectives: list[str]) → None

Filter seed groups to only those with objectives in the remaining list.

.. deprecated:: Use drop_seed_groups_with_hashes (or keep_seed_groups_with_hashes) which keys on content-addressed objective_sha256 instead of objective text. Scheduled for removal in 0.16.0.

keep_seed_groups_with_hashes¶

keep_seed_groups_with_hashes(hashes: set[str]) → set[str]

Keep only seed groups whose objective_sha256 is in hashes.

Inverse of drop_seed_groups_with_hashes: used on resume to replay the originally-sampled subset and ignore any seed groups that were added since (or that landed in this run’s fresh random.sample draw and are no longer in the persisted set).

Returns:

• set[str] — set[str]: The hashes that were actually retained (intersection of

• set[str] — hashes and the current seed_groups’ hashes). The caller can

• set[str] — union these across atomic attacks to detect persisted hashes that

• set[str] — no longer exist in the dataset.

run_async¶

run_async(max_concurrency: int = 1, return_partial_on_failure: bool = True, attack_params: Any = {}) → AttackExecutorResult[AttackResult]

Execute the atomic attack against all seed groups.

This method uses AttackExecutor to run the configured attack against all seed groups.

When return_partial_on_failure=True (default), this method will return an AttackExecutorResult containing both completed results and incomplete objectives (those that didn’t finish execution due to exceptions). This allows scenarios to save progress and retry only the incomplete objectives.

Note: “completed” means the execution finished, not that the attack objective was achieved. “incomplete” means execution didn’t finish (threw an exception).

Returns:

• AttackExecutorResult[AttackResult] — AttackExecutorResult[AttackResult]: Result containing completed attack results and incomplete objectives (those that didn’t finish execution).

Raises:

• ValueError — If the attack execution fails completely and return_partial_on_failure=False.

set_scenario_result_id¶

set_scenario_result_id(scenario_result_id: str | None) → None

Bind this atomic attack to a scenario result for attribution.

Called by Scenario._execute_scenario_async before each run_async so persisted AttackResult rows carry the attribution_parent_id foreign key back to the scenario. Pass None to clear the binding (e.g. when running an atomic attack outside of a scenario).

AttackTechnique¶

Bases: Identifiable

Bundles an attack strategy with an optional technique seed group.

An AttackTechnique encapsulates the full attack configuration — the strategy (including its target, converters, and scorer) plus any reusable technique seeds (e.g. jailbreak templates). The objectives that define which weaknesses to probe live separately on the SeedAttackGroup / AtomicAttack.

AttackTechniqueFactory¶

Bases: Identifiable

A factory that produces AttackTechnique instances on demand.

Captures technique-specific configuration (converters, adversarial config, tree depth, etc.) at registration time. Produces fresh, fully-constructed attacks by calling the real constructor with the captured params plus scenario-specific objective_target and scoring config.

Validates kwargs against the attack class constructor signature at construction time, catching typos and incompatible parameter names early.

Constructor Parameters:

Methods:

create¶

create(objective_target: PromptTarget, attack_scoring_config: AttackScoringConfig, attack_adversarial_config_override: AttackAdversarialConfig | None = None, attack_converter_config_override: AttackConverterConfig | None = None) → AttackTechnique

Create a fresh AttackTechnique bound to the given target.

Each call produces a fully independent attack instance by calling the real constructor. Config objects frozen at factory construction time are deep-copied into every new instance.

The *_override parameters let a caller replace a config that was baked into the factory at construction time. When None (the default), the factory’s original config is kept as-is — so baked-in converters, adversarial targets, etc. are preserved automatically.

Override configs are only forwarded when the attack class constructor declares a matching parameter (without the _override suffix). This allows a single call site to safely pass all available overrides without breaking attacks that don’t support them.

Returns:

• AttackTechnique — A fresh AttackTechnique with a newly-constructed attack strategy.

Raises:

• ValueError — If scorer_override_policy is RAISE and the override config is incompatible with the attack’s type annotation.

BaselineAttackPolicy¶

Bases: Enum

Declares how a scenario type treats the default baseline atomic attack.

The baseline is a plain PromptSendingAttack that sends each objective unmodified, used as a comparison point against the scenario’s strategies. Each scenario class declares its policy via Scenario.BASELINE_ATTACK_POLICY; callers can still override at runtime via initialize_async(include_baseline=...) for the Enabled and Disabled states.

DatasetConfiguration¶

Configuration for scenario datasets.

This class provides a unified way to specify the dataset source for scenarios. Only ONE of seed_groups or dataset_names can be set.

Constructor Parameters:

Methods:

get_all_seed_attack_groups¶

get_all_seed_attack_groups() → list[SeedAttackGroup]

Resolve and return all seed groups as SeedAttackGroups in a flat list.

This is a convenience method that calls get_seed_attack_groups() and flattens the results into a single list. Use this for attack scenarios that need SeedAttackGroup functionality.

Returns:

• list[SeedAttackGroup] — List[SeedAttackGroup]: All resolved seed attack groups from all datasets.

Raises:

• ValueError — If no seed groups could be resolved from the configuration.

get_all_seed_groups¶

get_all_seed_groups() → list[SeedGroup]

Resolve and return all seed groups as a flat list.

This is a convenience method that calls get_seed_groups() and flattens the results into a single list. Use this when you don’t need to track which dataset each seed group came from.

Returns:

• list[SeedGroup] — List[SeedGroup]: All resolved seed groups from all datasets, with max_dataset_size applied per dataset.

Raises:

• ValueError — If no seed groups could be resolved from the configuration.

get_all_seeds¶

get_all_seeds() → list[Seed]

Load all seed prompts from memory for all configured datasets.

This is a convenience method that retrieves SeedPrompt objects directly from memory for all configured datasets. If max_dataset_size is set, randomly samples up to that many prompts per dataset (without replacement).

Returns:

• list[Seed] — List[SeedPrompt]: List of SeedPrompt objects from all configured datasets. Returns an empty list if no prompts are found.

Raises:

• ValueError — If no dataset names are configured.

get_default_dataset_names¶

get_default_dataset_names() → list[str]

Get the list of default dataset names for this configuration.

This is used by the CLI to display what datasets the scenario uses by default.

Returns:

• list[str] — List[str]: List of dataset names, or empty list if using explicit seed_groups.

get_seed_attack_groups¶

get_seed_attack_groups() → dict[str, list[SeedAttackGroup]]

Resolve and return seed groups as SeedAttackGroups, grouped by dataset.

This wraps get_seed_groups() and converts each SeedGroup to a SeedAttackGroup. Use this when you need attack-specific functionality like objectives, prepended conversations, or simulated conversation configuration.

Returns:

• dict[str, list[SeedAttackGroup]] — Dict[str, List[SeedAttackGroup]]: Dictionary mapping dataset names to their seed attack groups.

Raises:

• ValueError — If no seed groups could be resolved from the configuration.

get_seed_groups¶

get_seed_groups() → dict[str, list[SeedGroup]]

Resolve and return seed groups based on the configuration.

This method handles all resolution logic:

1. If seed_groups is set, use those directly (under key ‘_explicit_seed_groups’)

2. If dataset_names is set, load from memory using those names

In all cases, max_dataset_size is applied per dataset if set.

Subclasses can override this to filter or customize which seed groups are loaded based on the stored scenario_composites.

Returns:

• dict[str, list[SeedGroup]] — Dict[str, List[SeedGroup]]: Dictionary mapping dataset names to their seed groups. When explicit seed_groups are provided, the key is ‘_explicit_seed_groups’. Each dataset’s seed groups are potentially sampled down to max_dataset_size.

Raises:

• ValueError — If no seed groups could be resolved from the configuration.

has_data_source¶

has_data_source() → bool

Check if this configuration has a data source configured.

Returns:

• bool — True if seed_groups or dataset_names is configured.

Parameter¶

Describes a parameter that a PyRIT component (initializer or scenario) accepts.

Scenario¶

Bases: ABC

Groups and executes multiple AtomicAttack instances sequentially.

A Scenario represents a comprehensive testing campaign composed of multiple atomic attack tests (AtomicAttacks). It executes each AtomicAttack in sequence and aggregates the results into a ScenarioResult.

Constructor Parameters:

Methods:

default_dataset_config¶

default_dataset_config() → DatasetConfiguration

Return the default dataset configuration for this scenario.

This abstract method must be implemented by all scenario subclasses to return a DatasetConfiguration specifying the default datasets to use when no dataset_config is provided by the user.

Returns:

• DatasetConfiguration — The default dataset configuration.

get_default_strategy¶

get_default_strategy() → ScenarioStrategy

Get the default strategy used when no strategies are specified.

This abstract method must be implemented by all scenario subclasses to return the default aggregate strategy (like EASY, ALL) used when scenario_strategies parameter is None.

Returns:

• ScenarioStrategy — The default aggregate strategy (e.g., FoundryStrategy.EASY, EncodingStrategy.ALL).

get_strategy_class¶

get_strategy_class() → type[ScenarioStrategy]

Get the strategy enum class for this scenario.

This abstract method must be implemented by all scenario subclasses to return the ScenarioStrategy enum class that defines the available attack strategies for the scenario.

Returns:

• type[ScenarioStrategy] — Type[ScenarioStrategy]: The strategy enum class (e.g., FoundryStrategy, EncodingStrategy).

initialize_async¶

initialize_async(objective_target: PromptTarget = REQUIRED_VALUE, scenario_strategies: Optional[Sequence[ScenarioStrategy]] = None, dataset_config: Optional[DatasetConfiguration] = None, max_concurrency: int = 10, max_retries: int = 0, memory_labels: Optional[dict[str, str]] = None, include_baseline: bool | None = None) → None

Initialize the scenario by populating self._atomic_attacks and creating the ScenarioResult.

This method allows scenarios to be initialized with atomic attacks after construction, which is useful when atomic attacks require async operations to be built.

If a scenario_result_id was provided in init, this method will check if it exists in memory and validate that the stored scenario matches the current configuration. If it matches, the scenario will resume from prior progress. If it doesn’t match or doesn’t exist, a new scenario result will be created.

Raises:

• ValueError — If no objective_target is provided, or if include_baseline=True is passed to a scenario whose BASELINE_ATTACK_POLICY is Forbidden.

run_async¶

run_async() → ScenarioResult

Execute all atomic attacks in the scenario sequentially.

Each AtomicAttack is executed in order, and all results are aggregated into a ScenarioResult containing the scenario metadata and all attack results. This method supports resumption - if the scenario raises an exception partway through, calling run_async again will skip already-completed objectives.

If max_retries is set, the scenario will automatically retry after an exception up to the specified number of times. Each retry will resume from where it left off, skipping completed objectives.

Returns:

• ScenarioResult — Contains scenario identifier and aggregated list of all attack results from all atomic attacks.

Raises:

• ValueError — If the scenario has no atomic attacks configured. If your scenario requires initialization, call await scenario.initialize() first.

• ValueError — If the scenario raises an exception after exhausting all retry attempts.

• RuntimeError — If the scenario fails for any other reason while executing.

set_params_from_args¶

set_params_from_args(args: dict[str, Any]) → None

Populate self.params from merged CLI / config arguments.

Coerces each value to its declared param_type, validates, and materializes declared defaults for params not in args. Every declared parameter is guaranteed a key in self.params after this call; params without a declared default land as None.

Raises:

• ValueError — Invalid declaration, unknown parameter, coercion failure, or value not in choices.

supported_parameters¶

supported_parameters() → list[Parameter]

Override to declare custom parameters this scenario accepts.

Declared parameters flow from CLI/config through set_params_from_args into self.params before initialize_async() runs. Implemented as a classmethod so --list-scenarios can introspect without instantiating.

Note: PyRITInitializer.supported_parameters is an instance @property; this asymmetry is intentional pending a future alignment.

Returns:

• list[Parameter] — list[Parameter]: Declared parameters (default: empty list).

ScenarioCompositeStrategy¶

Represents a composition of one or more attack strategies.

This class encapsulates a collection of ScenarioStrategy instances along with an auto-generated descriptive name, making it easy to represent both single strategies and composed multi-strategy attacks.

The name is automatically derived from the strategies:

• Single strategy: Uses the strategy’s value (e.g., “base64”)

• Multiple strategies: Generates “ComposedStrategy(base64, rot13)”

Constructor Parameters:

ScenarioIdentifier¶

Scenario result class for aggregating results from multiple AtomicAttacks.

Constructor Parameters:

Methods:

from_dict¶

from_dict(data: dict[str, Any]) → ScenarioIdentifier

Reconstruct a ScenarioIdentifier from a dictionary.

Returns:

• ScenarioIdentifier — Reconstructed instance.

to_dict¶

to_dict() → dict[str, Any]

Serialize to a JSON-compatible dictionary.

Returns:

• dict[str, Any] — dict[str, Any]: Serialized payload.

ScenarioResult¶

Scenario result class for aggregating scenario results.

Constructor Parameters:

Methods:

from_dict¶

from_dict(data: dict[str, Any]) → ScenarioResult

Reconstruct a ScenarioResult from a dictionary.

Returns:

• ScenarioResult — Reconstructed instance.

get_display_groups¶

get_display_groups() → dict[str, list[AttackResult]]

Aggregate attack results by display group.

When a display_group_map was provided, results from multiple atomic_attack_name keys that share the same display group are merged into a single list. When no map was provided, this returns the same structure as attack_results (identity mapping).

Returns:

• dict[str, list[AttackResult]] — dict[str, list[AttackResult]]: Results grouped by display label.

get_objectives¶

get_objectives(atomic_attack_name: str | None = None) → list[str]

Get the list of unique objectives for this scenario.

Returns:

• list[str] — List[str]: Deduplicated list of objectives.

get_scorer_evaluation_metrics¶

get_scorer_evaluation_metrics() → ScorerMetrics | None

Get the evaluation metrics for the scenario’s scorer from the scorer evaluation registry.

Returns:

• ScorerMetrics | None — The evaluation metrics object, or None if not found.

get_strategies_used¶

get_strategies_used() → list[str]

Get the list of strategies used in this scenario.

Returns:

• list[str] — List[str]: Atomic attack strategy names present in the results.

normalize_scenario_name¶

normalize_scenario_name(scenario_name: str) → str

Normalize a scenario name to match the stored class name format.

Converts CLI-style snake_case names (e.g., “foundry” or “content_harms”) to PascalCase class names (e.g., “Foundry” or “ContentHarms”) for database queries. If the input is already in PascalCase or doesn’t match the snake_case pattern, it is returned unchanged.

This is the inverse of ScenarioRegistry._class_name_to_scenario_name().

Returns:

• str — The normalized scenario name suitable for database queries.

objective_achieved_rate¶

objective_achieved_rate(atomic_attack_name: str | None = None) → int

Get the success rate of this scenario.

Returns:

• int — Success rate as a percentage (0-100).

to_dict¶

to_dict() → dict[str, Any]

Serialize this scenario result to a JSON-compatible dictionary.

Returns:

• dict[str, Any] — dict[str, Any]: Serialized payload suitable for REST APIs or persistence.

ScenarioStrategy¶

Bases: Enum

Base class for attack strategies with tag-based categorization and aggregation.

This class provides a pattern for defining attack strategies as enums where each strategy has a set of tags for flexible categorization. It supports aggregate tags (like “easy”, “moderate”, “difficult” or “fast”, “medium”) that automatically expand to include all strategies with that tag.

Convention: Strategy enum members should map 1:1 to selectable attack techniques (e.g., PromptSending, RolePlay, TAP) or to aggregates of techniques (e.g., DEFAULT, SINGLE_TURN). Datasets control what content or objectives are tested; strategies control how attacks are executed. Avoid encoding dataset or category selection into the strategy enum — use DatasetConfiguration and the --dataset-names CLI flag for that axis.

Tags: Flexible categorization system where strategies can have multiple tags (e.g., {“easy”, “converter”}, {“difficult”, “multi_turn”})

Subclasses should define their enum members with (value, tags) tuples and override the get_aggregate_tags() classmethod to specify which tags represent aggregates that should expand.

Convention: All subclasses should include ALL = ("all", {"all"}) as the first aggregate member. The base class automatically handles expanding “all” to include all non-aggregate strategies.

The normalization process automatically:

1. Expands aggregate tags into their constituent strategies

2. Excludes the aggregate tag enum members themselves from the final set

3. Handles the special “all” tag by expanding to all non-aggregate strategies

Methods:

expand¶

expand(strategies: set[T]) → list[T]

Expand a set of strategies (including aggregates) into an ordered, deduplicated list.

Aggregate markers (like EASY, ALL) are expanded into their constituent concrete strategies. The result is sorted by enum definition order for determinism.

Returns:

• list[T] — list[T]: Ordered list of concrete strategies with aggregates expanded.

get_aggregate_strategies¶

get_aggregate_strategies() → list[T]

Get all aggregate strategies for this strategy enum.

This method returns only the aggregate markers (like ALL, EASY, MODERATE, DIFFICULT) that are used to group concrete strategies by tags.

Returns:

• list[T] — list[T]: List of all aggregate strategies.

get_aggregate_tags¶

get_aggregate_tags() → set[str]

Get the set of tags that represent aggregate categories.

Subclasses should override this method to specify which tags are aggregate markers (e.g., {“easy”, “moderate”, “difficult”} for complexity-based scenarios or {“fast”, “medium”} for speed-based scenarios).

The base class automatically includes “all” as an aggregate tag that expands to all non-aggregate strategies.

Returns:

• set[str] — Set[str]: Set of tags that represent aggregates.

get_all_strategies¶

get_all_strategies() → list[T]

Get all non-aggregate strategies for this strategy enum.

This method returns all concrete attack strategies, excluding aggregate markers (like ALL, EASY, MODERATE, DIFFICULT) that are used for grouping.

Returns:

• list[T] — list[T]: List of all non-aggregate strategies.

get_strategies_by_tag¶

get_strategies_by_tag(tag: str) → set[T]

Get all attack strategies that have a specific tag.

This method returns concrete attack strategies (not aggregate markers) that include the specified tag.

Returns:

• set[T] — Set[T]: Set of strategies that include the specified tag, excluding any aggregate markers.

normalize_strategies¶

normalize_strategies(strategies: set[T]) → set[T]

Normalize a set of attack strategies by expanding aggregate tags.

This method processes a set of strategies and expands any aggregate tags (like EASY, MODERATE, DIFFICULT or FAST, MEDIUM) into their constituent concrete strategies. The aggregate tag markers themselves are removed from the result.

The special “all” tag is automatically supported and expands to all non-aggregate strategies.

Returns:

• set[T] — Set[T]: The normalized set of concrete attack strategies with aggregate tags expanded and removed.

resolve¶

resolve(strategies: Sequence[Any] | None, default: T) → list[T]

Resolve strategy inputs into a concrete, ordered, deduplicated list.

Handles None (returns expanded default), plain strategies, and aggregate strategies. Non-cls items (e.g., ScenarioCompositeStrategy) are silently skipped for backward compatibility.

Returns:

• list[T] — list[T]: Ordered, deduplicated list of concrete strategies.