Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

pyrit.scenario

High-level scenario classes for running attack configurations.

AtomicAttack

Represents a single atomic attack test combining an attack strategy and dataset.

An AtomicAttack is an executable unit that executes a configured attack against all objectives in a dataset. Multiple AtomicAttacks can be grouped together into larger test scenarios for comprehensive security testing and evaluation.

The AtomicAttack uses AttackSeedGroups as the single source of truth for objectives, prepended conversations, and next messages. Each AttackSeedGroup must have an objective set.

An AttackTechnique bundles the attack strategy with an optional AttackTechniqueSeedGroup, cleanly separating “how to attack” from “what to attack” (the objective).

Constructor Parameters:

ParameterTypeDescription
atomic_attack_namestrUnique key for this atomic attack. Used for resume tracking and result persistence — must be unique across all AtomicAttack instances in a scenario.
display_group`strNone`
attack_techniqueAttackTechniqueAn AttackTechnique bundling the attack strategy and optional technique seeds.
seed_groupslist[AttackSeedGroup]List of seed attack groups. Each must be a AttackSeedGroup (which guarantees exactly one objective).
adversarial_chat`PromptTargetNone`
objective_scorer`TrueFalseScorerNone`
memory_labels`dict[str, str]None`
**attack_execute_paramsAnyAdditional parameters to pass to the attack execution method. Defaults to {}.

Methods:

drop_seed_groups_with_hashes

drop_seed_groups_with_hashes(hashes: set[str]) → None

Drop seed groups whose objective_sha256 is in hashes.

This is the resume filter: within an atomic attack, objective_sha256 is the stable identity (enforced unique by __init__). Content-derived keys are robust to reordering and resampling, so resume produces the right remaining-work set even when get_seed_groups() is rebuilt from scratch on each run_async().

ParameterTypeDescription
hashesset[str]SHA256 hashes of objective text for seed groups to drop (typically those that have already produced a non-error AttackResult).

keep_seed_groups_with_hashes

keep_seed_groups_with_hashes(hashes: set[str]) → set[str]

Keep only seed groups whose objective_sha256 is in hashes.

Inverse of drop_seed_groups_with_hashes: used on resume to replay the originally-sampled subset and ignore any seed groups that were added since (or that landed in this run’s fresh random.sample draw and are no longer in the persisted set).

ParameterTypeDescription
hashesset[str]SHA256 hashes of objective text for seed groups to keep.

Returns:

run_async

run_async(executor: AttackExecutor | None = None, return_partial_on_failure: bool = True, attack_params: Any = {}) → AttackExecutorResult[AttackResult]

Execute the atomic attack against all seed groups.

This method uses AttackExecutor to run the configured attack against all seed groups. Concurrency is owned by the executor: pass a shared AttackExecutor instance to share a single budget across multiple atomic attacks (this is how Scenario parallelizes them).

When return_partial_on_failure=True (default), this method will return an AttackExecutorResult containing both completed results and incomplete objectives (those that didn’t finish execution due to exceptions). This allows scenarios to save progress and retry only the incomplete objectives.

Note: “completed” means the execution finished, not that the attack objective was achieved. “incomplete” means execution didn’t finish (threw an exception).

ParameterTypeDescription
executor`AttackExecutorNone`
return_partial_on_failureboolIf True, returns partial results even when some objectives don’t complete execution. If False, raises an exception on any execution failure. Defaults to True. Defaults to True.
**attack_paramsAnyAdditional parameters to pass to the attack strategy. Defaults to {}.

Returns:

Raises:

set_scenario_result_id

set_scenario_result_id(scenario_result_id: str | None) → None

Bind this atomic attack to a scenario result for attribution.

Called by Scenario._execute_scenario_async before each run_async so persisted AttackResult rows carry the attribution_parent_id foreign key back to the scenario. Pass None to clear the binding (e.g. when running an atomic attack outside of a scenario).

ParameterTypeDescription
scenario_result_id`strNone`

AttackTechnique

Bases: Identifiable

Bundles an attack technique with an optional technique seed group.

An AttackTechnique encapsulates the full attack configuration — the technique (including its target, converters, and scorer) plus any reusable technique seeds (e.g. jailbreak templates). The objectives that define which weaknesses to probe live separately on the AttackSeedGroup / AtomicAttack.

AttackTechniqueFactory

Bases: Identifiable

A self-describing factory that produces AttackTechnique instances on demand.

Captures technique-specific configuration (name, technique tags, converters, adversarial config, tree depth, etc.) at construction time. Produces fresh, fully-constructed attacks by calling the real constructor with the captured params plus scenario-specific objective_target and scoring config.

Validates kwargs against the attack class constructor signature at construction time, catching typos and incompatible parameter names early.

Constructor Parameters:

ParameterTypeDescription
namestrRegistry name for this technique. This is used as the scenario technique name.
attack_classtype[AttackStrategy[Any, Any]]The AttackStrategy subclass to instantiate.
technique_tags`list[str]None`
attack_kwargs`dict[str, Any]None`
adversarial_chat`PromptTargetNone`
adversarial_system_prompt`strSeedPrompt
adversarial_seed_prompt`SeedPromptstr
seed_technique`AttackTechniqueSeedGroupNone`
uses_adversarial`boolNone`
scorer_override_policyScorerOverridePolicyWhat to do when a scenario’s scorer is incompatible with the attack’s attack_scoring_config type annotation. Defaults to WARN. Defaults to ScorerOverridePolicy.WARN.

Methods:

add_technique_tags

add_technique_tags(tags: str = ()) → None

Append technique tags, skipping any already present.

ParameterTypeDescription
*tagsstrTechnique tags to add to this factory. Defaults to ().

create

create(objective_target: PromptTarget, attack_scoring_config: AttackScoringConfig, adversarial_chat: PromptTarget | None = None, adversarial_system_prompt: str | SeedPrompt | None = None, adversarial_seed_prompt: SeedPrompt | str | None = None, attack_converter_config_override: AttackConverterConfig | None = None, extra_request_converters: list[ConverterConfiguration] | None = None) → AttackTechnique

Create a fresh AttackTechnique bound to the given target.

Each call produces a fully independent attack instance by calling the real constructor. Config objects frozen at factory construction time are deep-copied into every new instance.

Create-time adversarial_chat mirrors the constructor’s adversarial target slot: pass it to supply the adversarial target for techniques that resolve it lazily (i.e. that did not bake one in). Supplying adversarial_chat when the factory already baked one is a conflict and raises — create() fills the lazy slot, it does not overwrite a technique’s own adversarial target. (The custom adversarial prompts remain construction-time only.) Like a baked target, a create-time adversarial_chat only reaches attacks whose constructor accepts attack_adversarial_config.

Override configs are only forwarded when the attack class constructor declares a matching parameter (without the _override suffix). This allows a single call site to safely pass all available overrides without breaking attacks that don’t support them.

ParameterTypeDescription
objective_targetPromptTargetThe target to attack (always required at create time).
attack_scoring_configAttackScoringConfigThe scoring config to use for the attack. This is important for attacks like TAP that may need a more specific scorer than the scorer the scenario provides.
adversarial_chat`PromptTargetNone`
adversarial_system_prompt`strSeedPrompt
adversarial_seed_prompt`SeedPromptstr
attack_converter_config_override`AttackConverterConfigNone`
extra_request_converters`list[ConverterConfiguration]None`

Returns:

Raises:

with_simulated_conversation

with_simulated_conversation(name: str, attack_class: type[AttackStrategy[Any, Any]] | None = None, adversarial_chat_system_prompt_path: str | Path | None = None, next_message_system_prompt_path: str | Path | None = None, num_turns: int = 3, technique_tags: list[str] | None = None, attack_kwargs: dict[str, Any] | None = None, adversarial_chat: PromptTarget | None = None, uses_adversarial: bool | None = None, scorer_override_policy: ScorerOverridePolicy = ScorerOverridePolicy.WARN) → AttackTechniqueFactory

Alternative constructor that builds a SeedSimulatedConversation inline.

Wraps a single SeedSimulatedConversation in a AttackTechniqueSeedGroup and assigns it as seed_technique so callers don’t have to construct both manually. All other parameters are forwarded to __init__.

ParameterTypeDescription
namestrRegistry name for this technique. When other defaults are used, name also picks the canonical YAML at EXECUTOR_SEED_PROMPT_PATH/red_teaming/{name}.yaml.
attack_class`type[AttackStrategy[Any, Any]]None`
adversarial_chat_system_prompt_path`strPath
next_message_system_prompt_path`strPath
num_turnsintNumber of simulated conversation turns. Defaults to 3. Defaults to 3.
technique_tags`list[str]None`
attack_kwargs`dict[str, Any]None`
adversarial_chat`PromptTargetNone`
uses_adversarial`boolNone`
scorer_override_policyScorerOverridePolicyPolicy applied when a scenario’s scorer is incompatible with the attack’s attack_scoring_config type annotation. Defaults to WARN. Forwarded to the factory constructor. Defaults to ScorerOverridePolicy.WARN.

Returns:

BaselineAttackPolicy

Bases: Enum

Declares how a scenario type treats the default baseline atomic attack.

The baseline is a plain PromptSendingAttack that sends each objective unmodified, used as a comparison point against the scenario’s techniques. Each scenario class declares its policy via Scenario.BASELINE_ATTACK_POLICY; callers can still override at runtime via initialize_async(include_baseline=...) for the Enabled and Disabled states.

CompoundDatasetAttackConfiguration

Bases: DatasetAttackConfiguration

A DatasetAttackConfiguration composed of child configurations.

Each child resolves, validates, and samples itself (with its own max_dataset_size); this compound concatenates the results. Use it to combine datasets that need independent budgets or shaping -- for example “up to 4 attack groups from each of several datasets” (see per_dataset), or pairing one dataset’s objectives with another dataset’s prompts.

A single DatasetAttackConfiguration applies max_dataset_size as one global budget; per-dataset budgets are expressed by composing one child per dataset rather than by special-casing dataset names inside a single configuration. An optional compound-level max_dataset_size caps the combined result on top of each child’s own sampling.

Constructor Parameters:

ParameterTypeDescription
configurationsSequence[DatasetAttackConfiguration]The child configurations to combine; each resolves and samples independently. Must be non-empty.
max_dataset_size`intNone`
validators`Sequence[Callable[[ResolvedDataset], None]]None`

Methods:

get_attack_groups_by_dataset_async

get_attack_groups_by_dataset_async(apply_sampling: bool = True) → dict[str, list[AttackSeedGroup]]

Merge each child’s by-dataset result, validate, then apply the global cap across the union.

ParameterTypeDescription
apply_samplingboolWhen True (default), sample both each child and the merged union under max_dataset_size. Pass False to resolve the full, deterministic dataset with no sampling at any level -- used on resume (propagated to children). Defaults to True.

Returns:

Raises:

get_attack_seed_groups_async

get_attack_seed_groups_async(apply_sampling: bool = True) → list[AttackSeedGroup]

Concatenate every child’s flat result, then validate and apply the global cap.

Each child validates and samples itself; the combined result is validated against this compound’s validators and capped by an optional compound max_dataset_size.

ParameterTypeDescription
apply_samplingboolWhen True (default), sample both each child and the combined result under max_dataset_size. Pass False to resolve the full, deterministic dataset with no sampling at any level -- used on resume (propagated to children). Defaults to True.

Returns:

Raises:

per_dataset

per_dataset(dataset_names: Sequence[str], max_dataset_size: int | None = None, auto_fetch: bool = True, filters: dict[str, list[str]] | None = None, validators: Sequence[Callable[[ResolvedDataset], None]] | None = None) → CompoundDatasetAttackConfiguration

Build a compound that draws up to max_dataset_size from each dataset name.

Creates one single-dataset DatasetAttackConfiguration child per name, so the budget applies independently to each -- the explicit, composable form of “N per dataset”.

ParameterTypeDescription
dataset_namesSequence[str]The dataset names; one child is built per name.
max_dataset_size`intNone`
auto_fetchboolPassed to each child (fetch missing datasets into memory). Defaults to True.
filters`dict[str, list[str]]None`
validators`Sequence[Callable[[ResolvedDataset], None]]None`

Returns:

Raises:

update_filters

update_filters(filters: dict[str, list[str]]) → None

Merge filters into the compound and propagate them to every child configuration.

The children run the actual get_seeds queries, so run-time filter overrides must reach each child to take effect.

ParameterTypeDescription
filtersdict[str, list[str]]Filters to merge, keyed by get_seeds kwarg name.

DatasetAttackConfiguration

Bases: DatasetConfiguration

A DatasetConfiguration that groups resolved seeds into attack groups.

This is the default most scenarios use: scenarios run over AttackSeedGroup s (each carrying exactly one objective plus optional prompts). max_dataset_size is a single global budget for the configuration; both resolvers apply it the same way:

To draw an independent budget from each of several datasets (the old “N per dataset” behavior), compose one child per dataset with CompoundDatasetAttackConfiguration -- e.g. CompoundDatasetAttackConfiguration.per_dataset(dataset_names=[...], max_dataset_size=4) -- rather than relying on a single config to special-case dataset names.

Both run validators against the full resolved seed set before sampling.

Override _build_attack_groups to change how raw seeds become attack groups (e.g. synthesizing a per-prompt objective). The default regroups by prompt_group_id via group_seeds_into_attack_groups.

Methods:

get_attack_groups_by_dataset_async

get_attack_groups_by_dataset_async(apply_sampling: bool = True) → dict[str, list[AttackSeedGroup]]

Resolve attack groups keyed by dataset name, globally sampled.

Inline configs resolve under the INLINE_DATASET_NAME label. Builds attack groups (auto-fetching missing datasets), validates the full resolved seed set, then applies max_dataset_size as one global budget across all datasets -- the survivors stay keyed by their originating dataset. For an independent budget per dataset, compose CompoundDatasetAttackConfiguration.per_dataset(...) instead.

ParameterTypeDescription
apply_samplingboolWhen True (default), apply max_dataset_size sampling. Pass False to resolve the full, deterministic dataset with no random.sample draw -- used on resume so the persisted objective subset can be reconstructed exactly rather than intersected against a fresh (divergent) sample. Defaults to True.

Returns:

Raises:

get_attack_seed_groups_async

get_attack_seed_groups_async(apply_sampling: bool = True) → list[AttackSeedGroup]

Resolve the configured dataset into a flat list[AttackSeedGroup].

Builds attack groups (inline or from memory, auto-fetching missing datasets), validates the full resolved seed set, then samples max_dataset_size globally over all built groups.

ParameterTypeDescription
apply_samplingboolWhen True (default), apply max_dataset_size sampling. Pass False to resolve the full, deterministic dataset with no random.sample draw -- used on resume so the persisted objective subset can be reconstructed exactly rather than intersected against a fresh (divergent) sample. Defaults to True.

Returns:

Raises:

DatasetConfiguration

Configuration describing where a scenario’s seeds come from.

This base class handles resolution, fetching, validation, and sampling. DatasetAttackConfiguration is the concrete subclass most scenarios use; it groups the resolved seeds into AttackSeedGroup s. A configuration draws from exactly one source:

Resolution reads memory (the source of truth) and, per dataset name, fetches from the provider when missing and auto_fetch is set. If a configured name still yields no seeds, _collect_seeds_for_dataset_async raises DatasetConstraintError -- failures are loud, not silently skipped.

Constraints are expressed through a single mechanism -- validators -- so there is one place to look. Customize behavior through small seams without re-implementing sampling/fetching:

Constructor Parameters:

ParameterTypeDescription
seeds`Sequence[Seed]None`
seed_groups`list[SeedGroup]None`
dataset_names`list[str]None`
max_dataset_size`intNone`
filters`dict[str, list[str]]None`
validators`Sequence[Callable[[ResolvedDataset], None]]None`
auto_fetchboolWhen True (default), a configured dataset name that is not in memory is fetched from the registered SeedDatasetProvider into memory before resolving. Set False for strict “must already be in memory”. Defaults to True.

Methods:

update_filters

update_filters(filters: dict[str, list[str]]) → None

Merge additional get_seeds filters into this configuration (run-time override).

Used when a run overrides dataset selection without rebuilding the configuration -- the provided filters take precedence over any already configured with the same key.

ParameterTypeDescription
filtersdict[str, list[str]]Filters to merge, keyed by get_seeds kwarg name.

validate

validate(resolved: ResolvedDataset) → None

Validate the resolved dataset against every configured validator.

Runs the defaults from _default_validators (non-emptiness, plus any seed-type constraint a subclass imposes) followed by any validators passed to validators=. Prefer adding a validator over overriding this method.

ParameterTypeDescription
resolvedResolvedDatasetThe resolved seeds and their source kind.

Raises:

DatasetSourceKind

Bases: Enum

How a DatasetConfiguration’s seeds were sourced.

Only two cases matter to validators: seeds supplied inline by the caller, versus seeds loaded from memory by dataset name (auto-fetched into memory first when missing). This lets a constraint require or forbid inline data -- e.g. a CLI --objectives flag that must be passed inline rather than via a named dataset.

Parameter

Bases: BaseModel

Describes a parameter that a PyRIT component accepts.

This is the single JSON-serializable parameter descriptor reused across the registry, scenarios, the backend API, and the CLI. param_type carries the value’s live Python type and its allowed set (a Literal[...] or Enum is the allowed set) and drives coerce_value / validate; it is not serialized. Serialization instead projects the type into the display fields type_name, choices, and is_list (plus required from the REQUIRED_VALUE sentinel), so a consumer can rebuild a usable contract from the registry without the live type travelling on the wire.

reference, when set, marks the parameter as a registry reference: its value is supplied by name and resolved to a registered instance by the registry layer (Parameter itself never resolves references). It is also excluded from serialization.

coerce_value and validate are the only public behaviors; all coercion branching lives behind them so callers never touch a free function.

Methods:

coerce_value

coerce_value(raw_value: Any) → Any

Coerce raw_value to this parameter’s declared type.

An opaque or reference parameter passes its value through unchanged (by identity — the registry layer resolves a reference by name; an opaque value is a live object owned by the caller). Otherwise it branches by shape: None passes through (deep-copied), a list coerces per element, and a scalar form (including Literal/Enum) coerces and validates membership. Arbitrary defaulted types pass through unchanged.

ParameterTypeDescription
raw_valueAnyThe raw value to coerce.

Returns:

Raises:

is_reference_to

is_reference_to(component_type: ComponentType) → bool

Whether this parameter is a registry reference to the given component family.

A reference parameter is supplied by name and resolved to a registered instance by the registry layer. This is the single source of truth for “does this parameter point at a TARGET / CONVERTER / SCORER”, so callers never re-derive it from reference internals.

ParameterTypeDescription
component_typeComponentTypeThe component family to test against.

Returns:

validate

validate() → None

Reject a declaration with an unsupported param_type.

Supported forms are a plain scalar, a constrained scalar (Literal/Enum), a list of any of those, a registry reference, an opaque passthrough, or None. An otherwise-unsupported type is tolerated only when the parameter declares a default (the builder simply does not supply it, and the value passes through unchanged).

Raises:

ResolvedDataset

The fully resolved seeds plus the source they came from.

Passed to every validator so a constraint can inspect the seeds, how they were supplied (inline vs named dataset), and which dataset names contributed.

Scenario

Bases: ABC

Groups and executes multiple AtomicAttack instances sequentially.

A Scenario represents a comprehensive testing campaign composed of multiple atomic attack tests (AtomicAttacks). It executes each AtomicAttack in sequence and aggregates the results into a ScenarioResult.

Subclasses must use the keyword-only constructor shape (def __init__(self, *, ...)); the contract is enforced at class-definition time via enforce_keyword_only_init. See .github/instructions/scenarios.instructions.md for the full contract.

Constructor Parameters:

ParameterTypeDescription
namestrDescriptive name for the scenario. Defaults to ''.
versionintVersion number of the scenario.
technique_classtype[ScenarioTechnique]The technique enum class for this scenario.
default_techniqueScenarioTechniqueThe default technique member used when no scenario_techniques are passed to initialize_async. Usually an aggregate member like MyTechnique.ALL or MyTechnique.DEFAULT.
default_dataset_configDatasetAttackConfigurationThe default dataset configuration used when no dataset_config is passed to initialize_async.
objective_scorerScorerThe objective scorer used to evaluate attack results.
scenario_result_id`uuid.UUIDstr

Methods:

additional_parameters

additional_parameters() → list[Parameter]

Declare the scenario-specific parameters this scenario accepts, beyond the common run inputs.

This is the extension point for the common case: override it to add parameters without repeating the common inputs. The base supported_parameters composes _common_scenario_parameters() + additional_parameters(), so overrides never need to call super() or risk dropping a common input. To remove or replace a common input instead, override supported_parameters directly.

Returns:

initialize_async

initialize_async() → None

Initialize the scenario by populating self._atomic_attacks and creating the ScenarioResult.

All run inputs are read from the parameter bag (self.params), which is populated by set_params_from_args from the merged CLI / config / programmatic arguments. Callers fill the bag then initialize:

.. code-block:: python

scenario.set_params_from_args(args={"objective_target": target, "max_concurrency": 8})
await scenario.initialize_async()

This method allows scenarios to be initialized with atomic attacks after construction, which is useful when atomic attacks require async operations to be built.

If a scenario_result_id was provided in init, this method will check if it exists in memory and validate that the stored scenario matches the current configuration. If it matches, the scenario will resume from prior progress. If it doesn’t match or doesn’t exist, a new scenario result will be created.

The common run inputs read from the bag are objective_target (a PromptTarget instance or a registered target name resolved against TargetRegistry), scenario_techniques, technique_converters, dataset_config, max_concurrency, max_retries, memory_labels, and include_baseline (see _common_scenario_parameters). A subclass that removes a common input via supported_parameters falls back to that input’s default here.

Raises:

run_async

run_async() → ScenarioResult

Execute all atomic attacks in the scenario sequentially.

Each AtomicAttack is executed in order, and all results are aggregated into a ScenarioResult containing the scenario metadata and all attack results. This method supports resumption - if the scenario raises an exception partway through, calling run_async again will skip already-completed objectives.

If max_retries is set, the scenario will automatically retry after an exception up to the specified number of times. Each retry will resume from where it left off, skipping completed objectives.

Returns:

Raises:

set_params_from_args

set_params_from_args(args: dict[str, Any]) → None

Populate self.params from merged CLI / config arguments.

The scenario only declares its parameters via supported_parameters(); the coerce / validate / inject-defaults mapping is owned by the registry layer (pyrit.registry.resolution.resolve_declared_params) so there is a single implementation shared by the programmatic, CLI, and registry paths. Every declared parameter is guaranteed a key in self.params after this call; params without a declared default land as None.

ParameterTypeDescription
argsdict[str, Any]Map of parameter name to raw value. Keys with None values are treated as absent (YAML null). Argparse callers should use argparse.SUPPRESS.

Raises:

supported_parameters

supported_parameters() → list[Parameter]

Declare the parameters this scenario accepts, resolved into self.params before initialize_async() runs. The base returns the common run inputs (see _common_scenario_parameters) plus whatever additional_parameters declares.

To add scenario-specific parameters, override additional_parameters (the common case). Override this method only to remove or replace a common input, composing against super().supported_parameters().

Implemented as a classmethod so --list-scenarios can introspect without instantiating.

Note: PyRITInitializer.supported_parameters is an instance @property; this asymmetry is intentional pending a future alignment.

Returns:

ScenarioIdentifier

Bases: ComponentIdentifier

Strongly-typed projection of a Scenario’s ComponentIdentifier.

Like the sibling projections (TargetIdentifier / ScorerIdentifier), this is produced by the scenario registry when a scenario is built. It is also the canonical per-run identity carried on the ScenarioResult aggregate and persisted with it: the scenario class name (class_name), definition version, resolved techniques / datasets, the resolved scenario params, and the objective_target / objective_scorer child references all live here rather than as separate denormalized fields. Its eval hash (via ScenarioEvaluationIdentifier) backs resume drift detection.

Promotes the scenario’s behavioral identity to typed params fields that feed both the content and eval hash: the definition version and the resolved techniques / datasets the scenario runs (a v1 vs a v2, or a different technique / dataset selection, is a different identity). The two run-resolved reference slots — objective_target (a PromptTarget) and objective_scorer (a Scorer) — are promoted children the registry resolves by name from the target / scorer registries when building a scenario.

ScenarioResult

Bases: BaseModel

Scenario result class for aggregating scenario results.

Methods:

get_display_groups

get_display_groups() → dict[str, list[AttackResult]]

Aggregate attack results by display group.

When a display_group_map was provided, results from multiple atomic_attack_name keys that share the same display group are merged into a single list. When no map was provided, this returns the same structure as attack_results (identity mapping).

Returns:

get_objectives

get_objectives(atomic_attack_name: str | None = None) → list[str]

Get the list of unique objectives for this scenario.

ParameterTypeDescription
atomic_attack_name`strNone`

Returns:

get_techniques_used

get_techniques_used() → list[str]

Get the list of techniques used in this scenario.

Returns:

normalize_scenario_name

normalize_scenario_name(scenario_name: str) → str

Normalize a scenario name to match the stored class name format.

Converts CLI-style snake_case names (e.g., “foundry” or “content_harms”) to PascalCase class names (e.g., “Foundry” or “ContentHarms”) for database queries. If the input is already in PascalCase or doesn’t match the snake_case pattern, it is returned unchanged.

This is the inverse of the snake_case registry-name conversion (class_name_to_snake_case) applied to scenario class names during discovery.

ParameterTypeDescription
scenario_namestrThe scenario name to normalize.

Returns:

objective_achieved_rate

objective_achieved_rate(atomic_attack_name: str | None = None) → int

Get the success rate of this scenario.

ParameterTypeDescription
atomic_attack_name`strNone`

Returns:

ScenarioTechnique

Bases: Enum

Base class for attack techniques with tag-based categorization and aggregation.

This class provides a pattern for defining attack techniques as enums where each technique has a set of tags for flexible categorization. It supports aggregate tags (like “easy”, “moderate”, “difficult” or “fast”, “medium”) that automatically expand to include all techniques with that tag.

Convention: Technique enum members should map 1:1 to selectable attack techniques (e.g., PromptSending, RolePlay, TAP) or to aggregates of techniques (e.g., DEFAULT, SINGLE_TURN). Datasets control what content or objectives are tested; techniques control how attacks are executed. Avoid encoding dataset or category selection into the technique enum — use DatasetConfiguration and the --dataset-names CLI flag for that axis.

Tags: Flexible categorization system where techniques can have multiple tags (e.g., {“easy”, “converter”}, {“difficult”, “multi_turn”})

Subclasses should define their enum members with (value, tags) tuples and override the get_aggregate_tags() classmethod to specify which tags represent aggregates that should expand.

Convention: All subclasses should include ALL = ("all", {"all"}) as the first aggregate member. The base class automatically handles expanding “all” to include all non-aggregate techniques.

The normalization process automatically:

  1. Expands aggregate tags into their constituent techniques

  2. Excludes the aggregate tag enum members themselves from the final set

  3. Handles the special “all” tag by expanding to all non-aggregate techniques

Methods:

expand

expand(techniques: set[T]) → list[T]

Expand a set of techniques (including aggregates) into an ordered, deduplicated list.

Aggregate markers (like EASY, ALL) are expanded into their constituent concrete techniques. The result is sorted by enum definition order for determinism.

ParameterTypeDescription
techniquesset[T]Set of techniques, which may include aggregate markers.

Returns:

get_aggregate_tags

get_aggregate_tags() → set[str]

Get the set of tags that represent aggregate categories.

Subclasses should override this method to specify which tags are aggregate markers (e.g., {“easy”, “moderate”, “difficult”} for complexity-based scenarios or {“fast”, “medium”} for speed-based scenarios).

The base class automatically includes “all” as an aggregate tag that expands to all non-aggregate techniques.

Returns:

get_aggregate_techniques

get_aggregate_techniques() → list[T]

Get all aggregate techniques for this technique enum.

This method returns only the aggregate markers (like ALL, EASY, MODERATE, DIFFICULT) that are used to group concrete techniques by tags.

Returns:

get_all_techniques

get_all_techniques() → list[T]

Get all non-aggregate techniques for this technique enum.

This method returns all concrete attack techniques, excluding aggregate markers (like ALL, EASY, MODERATE, DIFFICULT) that are used for grouping.

Returns:

get_techniques_by_tag

get_techniques_by_tag(tag: str) → set[T]

Get all attack techniques that have a specific tag.

This method returns concrete attack techniques (not aggregate markers) that include the specified tag.

ParameterTypeDescription
tagstrThe tag to filter by (e.g., “easy”, “converter”, “multi_turn”).

Returns:

resolve

resolve(techniques: Sequence[Any] | None, default: T) → list[T]

Resolve technique inputs into a concrete, ordered, deduplicated list.

Handles None (returns expanded default), plain techniques, and aggregate techniques. Non-cls items (e.g., FoundryComposite) are silently skipped for backward compatibility.

ParameterTypeDescription
techniques`Sequence[Any]None`
defaultTDefault aggregate technique to use when techniques is None or empty.

Returns: