High-level scenario classes for running attack configurations.
AtomicAttack¶
Represents a single atomic attack test combining an attack strategy and dataset.
An AtomicAttack is an executable unit that executes a configured attack against all objectives in a dataset. Multiple AtomicAttacks can be grouped together into larger test scenarios for comprehensive security testing and evaluation.
The AtomicAttack uses AttackSeedGroups as the single source of truth for objectives, prepended conversations, and next messages. Each AttackSeedGroup must have an objective set.
An AttackTechnique bundles the attack strategy with an optional
AttackTechniqueSeedGroup, cleanly separating “how to attack” from
“what to attack” (the objective).
Constructor Parameters:
| Parameter | Type | Description |
|---|---|---|
atomic_attack_name | str | Unique key for this atomic attack. Used for resume tracking and result persistence — must be unique across all AtomicAttack instances in a scenario. |
display_group | `str | None` |
attack_technique | AttackTechnique | An AttackTechnique bundling the attack strategy and optional technique seeds. |
seed_groups | list[AttackSeedGroup] | List of seed attack groups. Each must be a AttackSeedGroup (which guarantees exactly one objective). |
adversarial_chat | `PromptTarget | None` |
objective_scorer | `TrueFalseScorer | None` |
memory_labels | `dict[str, str] | None` |
**attack_execute_params | Any | Additional parameters to pass to the attack execution method. Defaults to {}. |
Methods:
drop_seed_groups_with_hashes¶
drop_seed_groups_with_hashes(hashes: set[str]) → NoneDrop seed groups whose objective_sha256 is in hashes.
This is the resume filter: within an atomic attack, objective_sha256
is the stable identity (enforced unique by __init__). Content-derived
keys are robust to reordering and resampling, so resume produces the
right remaining-work set even when get_seed_groups() is rebuilt
from scratch on each run_async().
| Parameter | Type | Description |
|---|---|---|
hashes | set[str] | SHA256 hashes of objective text for seed groups to drop (typically those that have already produced a non-error AttackResult). |
keep_seed_groups_with_hashes¶
keep_seed_groups_with_hashes(hashes: set[str]) → set[str]Keep only seed groups whose objective_sha256 is in hashes.
Inverse of drop_seed_groups_with_hashes: used on resume to
replay the originally-sampled subset and ignore any seed groups that
were added since (or that landed in this run’s fresh random.sample
draw and are no longer in the persisted set).
| Parameter | Type | Description |
|---|---|---|
hashes | set[str] | SHA256 hashes of objective text for seed groups to keep. |
Returns:
set[str]— set[str]: The hashes that were actually retained (intersection ofset[str]—hashesand the current seed_groups’ hashes). The caller canset[str]— union these across atomic attacks to detect persisted hashes thatset[str]— no longer exist in the dataset.
run_async¶
run_async(executor: AttackExecutor | None = None, return_partial_on_failure: bool = True, attack_params: Any = {}) → AttackExecutorResult[AttackResult]Execute the atomic attack against all seed groups.
This method uses AttackExecutor to run the configured attack against
all seed groups. Concurrency is owned by the executor: pass a shared
AttackExecutor instance to share a single budget across multiple
atomic attacks (this is how Scenario parallelizes them).
When return_partial_on_failure=True (default), this method will return an AttackExecutorResult containing both completed results and incomplete objectives (those that didn’t finish execution due to exceptions). This allows scenarios to save progress and retry only the incomplete objectives.
Note: “completed” means the execution finished, not that the attack objective was achieved. “incomplete” means execution didn’t finish (threw an exception).
| Parameter | Type | Description |
|---|---|---|
executor | `AttackExecutor | None` |
return_partial_on_failure | bool | If True, returns partial results even when some objectives don’t complete execution. If False, raises an exception on any execution failure. Defaults to True. Defaults to True. |
**attack_params | Any | Additional parameters to pass to the attack strategy. Defaults to {}. |
Returns:
AttackExecutorResult[AttackResult]— AttackExecutorResult[AttackResult]: Result containing completed attack results and incomplete objectives (those that didn’t finish execution).
Raises:
ValueError— If the attack execution fails completely and return_partial_on_failure=False.
set_scenario_result_id¶
set_scenario_result_id(scenario_result_id: str | None) → NoneBind this atomic attack to a scenario result for attribution.
Called by Scenario._execute_scenario_async before each
run_async so persisted AttackResult rows carry the
attribution_parent_id foreign key back to the scenario. Pass
None to clear the binding (e.g. when running an atomic attack
outside of a scenario).
| Parameter | Type | Description |
|---|---|---|
scenario_result_id | `str | None` |
AttackTechnique¶
Bases: Identifiable
Bundles an attack technique with an optional technique seed group.
An AttackTechnique encapsulates the full attack configuration — the technique (including its target, converters, and scorer) plus any reusable technique seeds (e.g. jailbreak templates). The objectives that define which weaknesses to probe live separately on the AttackSeedGroup / AtomicAttack.
AttackTechniqueFactory¶
Bases: Identifiable
A self-describing factory that produces AttackTechnique instances on demand.
Captures technique-specific configuration (name, technique tags, converters, adversarial config, tree depth, etc.) at construction time. Produces fresh, fully-constructed attacks by calling the real constructor with the captured params plus scenario-specific objective_target and scoring config.
Validates kwargs against the attack class constructor signature at construction time, catching typos and incompatible parameter names early.
Constructor Parameters:
| Parameter | Type | Description |
|---|---|---|
name | str | Registry name for this technique. This is used as the scenario technique name. |
attack_class | type[AttackStrategy[Any, Any]] | The AttackStrategy subclass to instantiate. |
technique_tags | `list[str] | None` |
attack_kwargs | `dict[str, Any] | None` |
adversarial_chat | `PromptTarget | None` |
adversarial_system_prompt | `str | SeedPrompt |
adversarial_seed_prompt | `SeedPrompt | str |
seed_technique | `AttackTechniqueSeedGroup | None` |
uses_adversarial | `bool | None` |
scorer_override_policy | ScorerOverridePolicy | What to do when a scenario’s scorer is incompatible with the attack’s attack_scoring_config type annotation. Defaults to WARN. Defaults to ScorerOverridePolicy.WARN. |
Methods:
add_technique_tags¶
add_technique_tags(tags: str = ()) → NoneAppend technique tags, skipping any already present.
| Parameter | Type | Description |
|---|---|---|
*tags | str | Technique tags to add to this factory. Defaults to (). |
create¶
create(objective_target: PromptTarget, attack_scoring_config: AttackScoringConfig, adversarial_chat: PromptTarget | None = None, adversarial_system_prompt: str | SeedPrompt | None = None, adversarial_seed_prompt: SeedPrompt | str | None = None, attack_converter_config_override: AttackConverterConfig | None = None, extra_request_converters: list[ConverterConfiguration] | None = None) → AttackTechniqueCreate a fresh AttackTechnique bound to the given target.
Each call produces a fully independent attack instance by calling the real constructor. Config objects frozen at factory construction time are deep-copied into every new instance.
Create-time adversarial_chat mirrors the constructor’s adversarial
target slot: pass it to supply the adversarial target for techniques that
resolve it lazily (i.e. that did not bake one in). Supplying
adversarial_chat when the factory already baked one is a conflict and
raises — create() fills the lazy slot, it does not overwrite a technique’s
own adversarial target. (The custom adversarial prompts remain
construction-time only.) Like a baked target, a create-time
adversarial_chat only reaches attacks whose constructor accepts
attack_adversarial_config.
Override configs are only forwarded when the attack class constructor
declares a matching parameter (without the _override suffix).
This allows a single call site to safely pass all available overrides
without breaking attacks that don’t support them.
| Parameter | Type | Description |
|---|---|---|
objective_target | PromptTarget | The target to attack (always required at create time). |
attack_scoring_config | AttackScoringConfig | The scoring config to use for the attack. This is important for attacks like TAP that may need a more specific scorer than the scorer the scenario provides. |
adversarial_chat | `PromptTarget | None` |
adversarial_system_prompt | `str | SeedPrompt |
adversarial_seed_prompt | `SeedPrompt | str |
attack_converter_config_override | `AttackConverterConfig | None` |
extra_request_converters | `list[ConverterConfiguration] | None` |
Returns:
AttackTechnique— A fresh AttackTechnique with a newly-constructed attack technique.
Raises:
ValueError— If a create-time adversarial chat is supplied while the factory already baked one, or ifscorer_override_policyis RAISE and the scenario scorer is incompatible with the attack’s type annotation.
with_simulated_conversation¶
with_simulated_conversation(name: str, attack_class: type[AttackStrategy[Any, Any]] | None = None, adversarial_chat_system_prompt_path: str | Path | None = None, next_message_system_prompt_path: str | Path | None = None, num_turns: int = 3, technique_tags: list[str] | None = None, attack_kwargs: dict[str, Any] | None = None, adversarial_chat: PromptTarget | None = None, uses_adversarial: bool | None = None, scorer_override_policy: ScorerOverridePolicy = ScorerOverridePolicy.WARN) → AttackTechniqueFactoryAlternative constructor that builds a SeedSimulatedConversation inline.
Wraps a single SeedSimulatedConversation in a AttackTechniqueSeedGroup
and assigns it as seed_technique so callers don’t have to construct
both manually. All other parameters are forwarded to __init__.
| Parameter | Type | Description |
|---|---|---|
name | str | Registry name for this technique. When other defaults are used, name also picks the canonical YAML at EXECUTOR_SEED_PROMPT_PATH/red_teaming/{name}.yaml. |
attack_class | `type[AttackStrategy[Any, Any]] | None` |
adversarial_chat_system_prompt_path | `str | Path |
next_message_system_prompt_path | `str | Path |
num_turns | int | Number of simulated conversation turns. Defaults to 3. Defaults to 3. |
technique_tags | `list[str] | None` |
attack_kwargs | `dict[str, Any] | None` |
adversarial_chat | `PromptTarget | None` |
uses_adversarial | `bool | None` |
scorer_override_policy | ScorerOverridePolicy | Policy applied when a scenario’s scorer is incompatible with the attack’s attack_scoring_config type annotation. Defaults to WARN. Forwarded to the factory constructor. Defaults to ScorerOverridePolicy.WARN. |
Returns:
AttackTechniqueFactory— A new factory whoseseed_techniqueis the wrapped simulated conversation.
BaselineAttackPolicy¶
Bases: Enum
Declares how a scenario type treats the default baseline atomic attack.
The baseline is a plain PromptSendingAttack that sends each objective unmodified,
used as a comparison point against the scenario’s techniques. Each scenario class
declares its policy via Scenario.BASELINE_ATTACK_POLICY; callers can still override
at runtime via initialize_async(include_baseline=...) for the Enabled and
Disabled states.
CompoundDatasetAttackConfiguration¶
Bases: DatasetAttackConfiguration
A DatasetAttackConfiguration composed of child configurations.
Each child resolves, validates, and samples itself (with its own max_dataset_size);
this compound concatenates the results. Use it to combine datasets that need independent
budgets or shaping -- for example “up to 4 attack groups from each of several datasets”
(see per_dataset), or pairing one dataset’s objectives with another dataset’s prompts.
A single DatasetAttackConfiguration applies max_dataset_size as one global budget;
per-dataset budgets are expressed by composing one child per dataset rather than by
special-casing dataset names inside a single configuration. An optional compound-level
max_dataset_size caps the combined result on top of each child’s own sampling.
Constructor Parameters:
| Parameter | Type | Description |
|---|---|---|
configurations | Sequence[DatasetAttackConfiguration] | The child configurations to combine; each resolves and samples independently. Must be non-empty. |
max_dataset_size | `int | None` |
validators | `Sequence[Callable[[ResolvedDataset], None]] | None` |
Methods:
get_attack_groups_by_dataset_async¶
get_attack_groups_by_dataset_async(apply_sampling: bool = True) → dict[str, list[AttackSeedGroup]]Merge each child’s by-dataset result, validate, then apply the global cap across the union.
| Parameter | Type | Description |
|---|---|---|
apply_sampling | bool | When True (default), sample both each child and the merged union under max_dataset_size. Pass False to resolve the full, deterministic dataset with no sampling at any level -- used on resume (propagated to children). Defaults to True. |
Returns:
dict[str, list[AttackSeedGroup]]— dict[str, list[AttackSeedGroup]]: Combined groups keyed by dataset name.
Raises:
DatasetConstraintError— If a child yields nothing, or the combined result fails validation.
get_attack_seed_groups_async¶
get_attack_seed_groups_async(apply_sampling: bool = True) → list[AttackSeedGroup]Concatenate every child’s flat result, then validate and apply the global cap.
Each child validates and samples itself; the combined result is validated against this
compound’s validators and capped by an optional compound max_dataset_size.
| Parameter | Type | Description |
|---|---|---|
apply_sampling | bool | When True (default), sample both each child and the combined result under max_dataset_size. Pass False to resolve the full, deterministic dataset with no sampling at any level -- used on resume (propagated to children). Defaults to True. |
Returns:
list[AttackSeedGroup]— list[AttackSeedGroup]: The combined, validated attack groups (capped whenapply_samplingis True).
Raises:
DatasetConstraintError— If a child yields nothing, or the combined result fails validation.
per_dataset¶
per_dataset(dataset_names: Sequence[str], max_dataset_size: int | None = None, auto_fetch: bool = True, filters: dict[str, list[str]] | None = None, validators: Sequence[Callable[[ResolvedDataset], None]] | None = None) → CompoundDatasetAttackConfigurationBuild a compound that draws up to max_dataset_size from each dataset name.
Creates one single-dataset DatasetAttackConfiguration child per name, so the budget
applies independently to each -- the explicit, composable form of “N per dataset”.
| Parameter | Type | Description |
|---|---|---|
dataset_names | Sequence[str] | The dataset names; one child is built per name. |
max_dataset_size | `int | None` |
auto_fetch | bool | Passed to each child (fetch missing datasets into memory). Defaults to True. |
filters | `dict[str, list[str]] | None` |
validators | `Sequence[Callable[[ResolvedDataset], None]] | None` |
Returns:
CompoundDatasetAttackConfiguration— The composed configuration.
Raises:
ValueError— Ifdataset_namesis empty.
update_filters¶
update_filters(filters: dict[str, list[str]]) → NoneMerge filters into the compound and propagate them to every child configuration.
The children run the actual get_seeds queries, so run-time filter overrides must
reach each child to take effect.
| Parameter | Type | Description |
|---|---|---|
filters | dict[str, list[str]] | Filters to merge, keyed by get_seeds kwarg name. |
DatasetAttackConfiguration¶
Bases: DatasetConfiguration
A DatasetConfiguration that groups resolved seeds into attack groups.
This is the default most scenarios use: scenarios run over AttackSeedGroup s
(each carrying exactly one objective plus optional prompts). max_dataset_size is a
single global budget for the configuration; both resolvers apply it the same way:
get_attack_seed_groups_async-- a flatlist[AttackSeedGroup], sampled globally over all built groups.get_attack_groups_by_dataset_async-- the same globally sampled groups, keyed by dataset name, used when a scenario fans atomic attacks out per (technique, dataset).
To draw an independent budget from each of several datasets (the old “N per dataset”
behavior), compose one child per dataset with CompoundDatasetAttackConfiguration --
e.g. CompoundDatasetAttackConfiguration.per_dataset(dataset_names=[...], max_dataset_size=4)
-- rather than relying on a single config to special-case dataset names.
Both run validators against the full resolved seed set before sampling.
Override _build_attack_groups to change how raw seeds become attack groups
(e.g. synthesizing a per-prompt objective). The default regroups by
prompt_group_id via group_seeds_into_attack_groups.
Methods:
get_attack_groups_by_dataset_async¶
get_attack_groups_by_dataset_async(apply_sampling: bool = True) → dict[str, list[AttackSeedGroup]]Resolve attack groups keyed by dataset name, globally sampled.
Inline configs resolve under the INLINE_DATASET_NAME label. Builds attack groups
(auto-fetching missing datasets), validates the full resolved seed set, then applies
max_dataset_size as one global budget across all datasets -- the survivors stay
keyed by their originating dataset. For an independent budget per dataset, compose
CompoundDatasetAttackConfiguration.per_dataset(...) instead.
| Parameter | Type | Description |
|---|---|---|
apply_sampling | bool | When True (default), apply max_dataset_size sampling. Pass False to resolve the full, deterministic dataset with no random.sample draw -- used on resume so the persisted objective subset can be reconstructed exactly rather than intersected against a fresh (divergent) sample. Defaults to True. |
Returns:
dict[str, list[AttackSeedGroup]]— dict[str, list[AttackSeedGroup]]: Dataset name -> attack groups (sampled whenapply_samplingis True, otherwise the full resolved set).
Raises:
DatasetConstraintError— If a configured dataset yields no seeds, the resolved dataset fails validation, or no attack groups could be built.
get_attack_seed_groups_async¶
get_attack_seed_groups_async(apply_sampling: bool = True) → list[AttackSeedGroup]Resolve the configured dataset into a flat list[AttackSeedGroup].
Builds attack groups (inline or from memory, auto-fetching missing datasets),
validates the full resolved seed set, then samples max_dataset_size globally
over all built groups.
| Parameter | Type | Description |
|---|---|---|
apply_sampling | bool | When True (default), apply max_dataset_size sampling. Pass False to resolve the full, deterministic dataset with no random.sample draw -- used on resume so the persisted objective subset can be reconstructed exactly rather than intersected against a fresh (divergent) sample. Defaults to True. |
Returns:
list[AttackSeedGroup]— list[AttackSeedGroup]: The validated attack groups (sampled whenapply_samplingis True, otherwise the full resolved set).
Raises:
DatasetConstraintError— If a configured dataset yields no seeds, the resolved dataset fails validation, or no attack groups could be built.
DatasetConfiguration¶
Configuration describing where a scenario’s seeds come from.
This base class handles resolution, fetching, validation, and sampling.
DatasetAttackConfiguration is the concrete subclass most scenarios use; it groups
the resolved seeds into AttackSeedGroup s. A configuration draws from exactly one
source:
seeds-- an explicit, inline list of seeds (never touches memory).seed_groups-- explicit, inline seed groups (never touches memory).dataset_names-- names looked up in memory; missing names are fetched from the registeredSeedDatasetProviderwhenauto_fetchis enabled.
Resolution reads memory (the source of truth) and, per dataset name, fetches from the
provider when missing and auto_fetch is set. If a configured name still yields no
seeds, _collect_seeds_for_dataset_async raises DatasetConstraintError -- failures
are loud, not silently skipped.
Constraints are expressed through a single mechanism -- validators -- so there is
one place to look. Customize behavior through small seams without re-implementing
sampling/fetching:
_default_validators-- validators a subclass always applies (e.g. a seed-type check). The preferred way to enforce a constraint type-wide._collect_seeds_for_dataset_async-- the per-dataset memory query (override for richer filters).
Constructor Parameters:
| Parameter | Type | Description |
|---|---|---|
seeds | `Sequence[Seed] | None` |
seed_groups | `list[SeedGroup] | None` |
dataset_names | `list[str] | None` |
max_dataset_size | `int | None` |
filters | `dict[str, list[str]] | None` |
validators | `Sequence[Callable[[ResolvedDataset], None]] | None` |
auto_fetch | bool | When True (default), a configured dataset name that is not in memory is fetched from the registered SeedDatasetProvider into memory before resolving. Set False for strict “must already be in memory”. Defaults to True. |
Methods:
update_filters¶
update_filters(filters: dict[str, list[str]]) → NoneMerge additional get_seeds filters into this configuration (run-time override).
Used when a run overrides dataset selection without rebuilding the configuration -- the provided filters take precedence over any already configured with the same key.
| Parameter | Type | Description |
|---|---|---|
filters | dict[str, list[str]] | Filters to merge, keyed by get_seeds kwarg name. |
validate¶
validate(resolved: ResolvedDataset) → NoneValidate the resolved dataset against every configured validator.
Runs the defaults from _default_validators (non-emptiness, plus any seed-type
constraint a subclass imposes) followed by any validators passed to validators=.
Prefer adding a validator over overriding this method.
| Parameter | Type | Description |
|---|---|---|
resolved | ResolvedDataset | The resolved seeds and their source kind. |
Raises:
DatasetConstraintError— If any constraint is violated.
DatasetSourceKind¶
Bases: Enum
How a DatasetConfiguration’s seeds were sourced.
Only two cases matter to validators: seeds supplied inline by the caller, versus
seeds loaded from memory by dataset name (auto-fetched into memory first when
missing). This lets a constraint require or forbid inline data -- e.g. a CLI
--objectives flag that must be passed inline rather than via a named dataset.
Parameter¶
Bases: BaseModel
Describes a parameter that a PyRIT component accepts.
This is the single JSON-serializable parameter descriptor reused across the
registry, scenarios, the backend API, and the CLI. param_type carries the
value’s live Python type and its allowed set (a Literal[...] or Enum
is the allowed set) and drives coerce_value / validate; it is not
serialized. Serialization instead projects the type into the display fields
type_name, choices, and is_list (plus required from the
REQUIRED_VALUE sentinel), so a consumer can rebuild a usable contract from
the registry without the live type travelling on the wire.
reference, when set, marks the parameter as a registry reference: its value
is supplied by name and resolved to a registered instance by the registry
layer (Parameter itself never resolves references). It is also excluded
from serialization.
coerce_value and validate are the only public behaviors; all coercion
branching lives behind them so callers never touch a free function.
Methods:
coerce_value¶
coerce_value(raw_value: Any) → AnyCoerce raw_value to this parameter’s declared type.
An opaque or reference parameter passes its value through unchanged (by
identity — the registry layer resolves a reference by name; an opaque
value is a live object owned by the caller). Otherwise it branches by
shape: None passes through (deep-copied), a list coerces per
element, and a scalar form (including Literal/Enum) coerces and
validates membership. Arbitrary defaulted types pass through unchanged.
| Parameter | Type | Description |
|---|---|---|
raw_value | Any | The raw value to coerce. |
Returns:
Any— The coerced value (the raw value unchanged for opaque/reference/ arbitrary types, a deep copy for theNonepassthrough, a coerced list for list types, or a coerced scalar for scalar types).
Raises:
ValueError— If the value cannot be coerced to a constrained scalar or list element type.
is_reference_to¶
is_reference_to(component_type: ComponentType) → boolWhether this parameter is a registry reference to the given component family.
A reference parameter is supplied by name and resolved to a registered
instance by the registry layer. This is the single source of truth for
“does this parameter point at a TARGET / CONVERTER / SCORER”,
so callers never re-derive it from reference internals.
| Parameter | Type | Description |
|---|---|---|
component_type | ComponentType | The component family to test against. |
Returns:
bool— True when this parameter is a reference tocomponent_type.
validate¶
validate() → NoneReject a declaration with an unsupported param_type.
Supported forms are a plain scalar, a constrained scalar
(Literal/Enum), a list of any of those, a registry reference,
an opaque passthrough, or None. An otherwise-unsupported type is
tolerated only when the parameter declares a default (the builder simply
does not supply it, and the value passes through unchanged).
Raises:
ValueError— Ifparam_typeis unsupported and no default is declared.
ResolvedDataset¶
The fully resolved seeds plus the source they came from.
Passed to every validator so a constraint can inspect the seeds, how they were supplied (inline vs named dataset), and which dataset names contributed.
Scenario¶
Bases: ABC
Groups and executes multiple AtomicAttack instances sequentially.
A Scenario represents a comprehensive testing campaign composed of multiple atomic attack tests (AtomicAttacks). It executes each AtomicAttack in sequence and aggregates the results into a ScenarioResult.
Subclasses must use the keyword-only constructor shape (def __init__(self, *, ...));
the contract is enforced at class-definition time via
enforce_keyword_only_init. See
.github/instructions/scenarios.instructions.md for the full contract.
Constructor Parameters:
| Parameter | Type | Description |
|---|---|---|
name | str | Descriptive name for the scenario. Defaults to ''. |
version | int | Version number of the scenario. |
technique_class | type[ScenarioTechnique] | The technique enum class for this scenario. |
default_technique | ScenarioTechnique | The default technique member used when no scenario_techniques are passed to initialize_async. Usually an aggregate member like MyTechnique.ALL or MyTechnique.DEFAULT. |
default_dataset_config | DatasetAttackConfiguration | The default dataset configuration used when no dataset_config is passed to initialize_async. |
objective_scorer | Scorer | The objective scorer used to evaluate attack results. |
scenario_result_id | `uuid.UUID | str |
Methods:
additional_parameters¶
additional_parameters() → list[Parameter]Declare the scenario-specific parameters this scenario accepts, beyond the common run inputs.
This is the extension point for the common case: override it to add parameters
without repeating the common inputs. The base supported_parameters composes
_common_scenario_parameters() + additional_parameters(), so overrides never need
to call super() or risk dropping a common input. To remove or replace a common
input instead, override supported_parameters directly.
Returns:
list[Parameter]— list[Parameter]: The scenario-specific parameters (default: none).
initialize_async¶
initialize_async() → NoneInitialize the scenario by populating self._atomic_attacks and creating the ScenarioResult.
All run inputs are read from the parameter bag (self.params), which is populated by
set_params_from_args from the merged CLI / config / programmatic arguments. Callers
fill the bag then initialize:
.. code-block:: python
scenario.set_params_from_args(args={"objective_target": target, "max_concurrency": 8})
await scenario.initialize_async()This method allows scenarios to be initialized with atomic attacks after construction, which is useful when atomic attacks require async operations to be built.
If a scenario_result_id was provided in init, this method will check if it exists in memory and validate that the stored scenario matches the current configuration. If it matches, the scenario will resume from prior progress. If it doesn’t match or doesn’t exist, a new scenario result will be created.
The common run inputs read from the bag are objective_target (a PromptTarget
instance or a registered target name resolved against TargetRegistry),
scenario_techniques, technique_converters, dataset_config,
max_concurrency, max_retries, memory_labels, and include_baseline
(see _common_scenario_parameters). A subclass that removes a common input via
supported_parameters falls back to that input’s default here.
Raises:
ValueError— Ifobjective_targetis declared but not resolvable (neither supplied nor registered as a default), if a supplied target name is not registered inTargetRegistry, or ifinclude_baseline=Trueis set for a scenario whoseBASELINE_ATTACK_POLICYisForbidden.
run_async¶
run_async() → ScenarioResultExecute all atomic attacks in the scenario sequentially.
Each AtomicAttack is executed in order, and all results are aggregated into a ScenarioResult containing the scenario metadata and all attack results. This method supports resumption - if the scenario raises an exception partway through, calling run_async again will skip already-completed objectives.
If max_retries is set, the scenario will automatically retry after an exception up to the specified number of times. Each retry will resume from where it left off, skipping completed objectives.
Returns:
ScenarioResult— Contains scenario identifier and aggregated list of all attack results from all atomic attacks.
Raises:
ValueError— If the scenario has no atomic attacks configured. If your scenario requires initialization, call await scenario.initialize() first.ValueError— If the scenario raises an exception after exhausting all retry attempts.RuntimeError— If the scenario fails for any other reason while executing.
set_params_from_args¶
set_params_from_args(args: dict[str, Any]) → NonePopulate self.params from merged CLI / config arguments.
The scenario only declares its parameters via supported_parameters();
the coerce / validate / inject-defaults mapping is owned by the registry
layer (pyrit.registry.resolution.resolve_declared_params) so there is a
single implementation shared by the programmatic, CLI, and registry paths.
Every declared parameter is guaranteed a key in self.params after this
call; params without a declared default land as None.
| Parameter | Type | Description |
|---|---|---|
args | dict[str, Any] | Map of parameter name to raw value. Keys with None values are treated as absent (YAML null). Argparse callers should use argparse.SUPPRESS. |
Raises:
ValueError— Invalid declaration, unknown parameter, coercion failure, value not inchoices, or a declared parameter using a reserved scenario identity name (e.g.version).
supported_parameters¶
supported_parameters() → list[Parameter]Declare the parameters this scenario accepts, resolved into self.params before
initialize_async() runs. The base returns the common run inputs (see
_common_scenario_parameters) plus whatever additional_parameters declares.
To add scenario-specific parameters, override additional_parameters (the
common case). Override this method only to remove or replace a common input,
composing against super().supported_parameters().
Implemented as a classmethod so --list-scenarios can introspect without
instantiating.
Note: PyRITInitializer.supported_parameters is an instance @property;
this asymmetry is intentional pending a future alignment.
Returns:
list[Parameter]— list[Parameter]: Declared parameters (default: common run inputs + additional).
ScenarioIdentifier¶
Bases: ComponentIdentifier
Strongly-typed projection of a Scenario’s ComponentIdentifier.
Like the sibling projections (TargetIdentifier / ScorerIdentifier),
this is produced by the scenario registry when a scenario is built. It is also
the canonical per-run identity carried on the ScenarioResult aggregate and
persisted with it: the scenario class name (class_name), definition
version, resolved techniques / datasets, the resolved scenario
params, and the objective_target / objective_scorer child
references all live here rather than as separate denormalized fields. Its eval
hash (via ScenarioEvaluationIdentifier) backs resume drift detection.
Promotes the scenario’s behavioral identity to typed params fields that
feed both the content and eval hash: the definition version and the
resolved techniques / datasets the scenario runs (a v1 vs a v2, or a
different technique / dataset selection, is a different identity). The two
run-resolved reference slots — objective_target (a PromptTarget) and
objective_scorer (a Scorer) — are promoted children the registry
resolves by name from the target / scorer registries when building a scenario.
ScenarioResult¶
Bases: BaseModel
Scenario result class for aggregating scenario results.
Methods:
get_display_groups¶
get_display_groups() → dict[str, list[AttackResult]]Aggregate attack results by display group.
When a display_group_map was provided, results from multiple
atomic_attack_name keys that share the same display group are
merged into a single list. When no map was provided, this returns
the same structure as attack_results (identity mapping).
Returns:
dict[str, list[AttackResult]]— dict[str, list[AttackResult]]: Results grouped by display label.
get_objectives¶
get_objectives(atomic_attack_name: str | None = None) → list[str]Get the list of unique objectives for this scenario.
| Parameter | Type | Description |
|---|---|---|
atomic_attack_name | `str | None` |
Returns:
list[str]— list[str]: Deduplicated list of objectives.
get_techniques_used¶
get_techniques_used() → list[str]Get the list of techniques used in this scenario.
Returns:
list[str]— list[str]: Atomic attack technique names present in the results.
normalize_scenario_name¶
normalize_scenario_name(scenario_name: str) → strNormalize a scenario name to match the stored class name format.
Converts CLI-style snake_case names (e.g., “foundry” or “content_harms”) to PascalCase class names (e.g., “Foundry” or “ContentHarms”) for database queries. If the input is already in PascalCase or doesn’t match the snake_case pattern, it is returned unchanged.
This is the inverse of the snake_case registry-name conversion
(class_name_to_snake_case) applied to scenario class names during
discovery.
| Parameter | Type | Description |
|---|---|---|
scenario_name | str | The scenario name to normalize. |
Returns:
str— The normalized scenario name suitable for database queries.
objective_achieved_rate¶
objective_achieved_rate(atomic_attack_name: str | None = None) → intGet the success rate of this scenario.
| Parameter | Type | Description |
|---|---|---|
atomic_attack_name | `str | None` |
Returns:
int— Success rate as a percentage (0-100).
ScenarioTechnique¶
Bases: Enum
Base class for attack techniques with tag-based categorization and aggregation.
This class provides a pattern for defining attack techniques as enums where each technique has a set of tags for flexible categorization. It supports aggregate tags (like “easy”, “moderate”, “difficult” or “fast”, “medium”) that automatically expand to include all techniques with that tag.
Convention: Technique enum members should map 1:1 to selectable attack techniques
(e.g., PromptSending, RolePlay, TAP) or to aggregates of techniques
(e.g., DEFAULT, SINGLE_TURN). Datasets control what content or objectives
are tested; techniques control how attacks are executed. Avoid encoding dataset or
category selection into the technique enum — use DatasetConfiguration and the
--dataset-names CLI flag for that axis.
Tags: Flexible categorization system where techniques can have multiple tags (e.g., {“easy”, “converter”}, {“difficult”, “multi_turn”})
Subclasses should define their enum members with (value, tags) tuples and override the get_aggregate_tags() classmethod to specify which tags represent aggregates that should expand.
Convention: All subclasses should include ALL = ("all", {"all"}) as the first
aggregate member. The base class automatically handles expanding “all” to
include all non-aggregate techniques.
The normalization process automatically:
Expands aggregate tags into their constituent techniques
Excludes the aggregate tag enum members themselves from the final set
Handles the special “all” tag by expanding to all non-aggregate techniques
Methods:
expand¶
expand(techniques: set[T]) → list[T]Expand a set of techniques (including aggregates) into an ordered, deduplicated list.
Aggregate markers (like EASY, ALL) are expanded into their constituent concrete techniques. The result is sorted by enum definition order for determinism.
| Parameter | Type | Description |
|---|---|---|
techniques | set[T] | Set of techniques, which may include aggregate markers. |
Returns:
list[T]— list[T]: Ordered list of concrete techniques with aggregates expanded.
get_aggregate_tags¶
get_aggregate_tags() → set[str]Get the set of tags that represent aggregate categories.
Subclasses should override this method to specify which tags are aggregate markers (e.g., {“easy”, “moderate”, “difficult”} for complexity-based scenarios or {“fast”, “medium”} for speed-based scenarios).
The base class automatically includes “all” as an aggregate tag that expands to all non-aggregate techniques.
Returns:
set[str]— set[str]: Set of tags that represent aggregates.
get_aggregate_techniques¶
get_aggregate_techniques() → list[T]Get all aggregate techniques for this technique enum.
This method returns only the aggregate markers (like ALL, EASY, MODERATE, DIFFICULT) that are used to group concrete techniques by tags.
Returns:
list[T]— list[T]: List of all aggregate techniques.
get_all_techniques¶
get_all_techniques() → list[T]Get all non-aggregate techniques for this technique enum.
This method returns all concrete attack techniques, excluding aggregate markers (like ALL, EASY, MODERATE, DIFFICULT) that are used for grouping.
Returns:
list[T]— list[T]: List of all non-aggregate techniques.
get_techniques_by_tag¶
get_techniques_by_tag(tag: str) → set[T]Get all attack techniques that have a specific tag.
This method returns concrete attack techniques (not aggregate markers) that include the specified tag.
| Parameter | Type | Description |
|---|---|---|
tag | str | The tag to filter by (e.g., “easy”, “converter”, “multi_turn”). |
Returns:
set[T]— set[T]: Set of techniques that include the specified tag, excluding any aggregate markers.
resolve¶
resolve(techniques: Sequence[Any] | None, default: T) → list[T]Resolve technique inputs into a concrete, ordered, deduplicated list.
Handles None (returns expanded default), plain techniques, and aggregate techniques. Non-cls items (e.g., FoundryComposite) are silently skipped for backward compatibility.
| Parameter | Type | Description |
|---|---|---|
techniques | `Sequence[Any] | None` |
default | T | Default aggregate technique to use when techniques is None or empty. |
Returns:
list[T]— list[T]: Ordered, deduplicated list of concrete techniques.