Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Benchmark Scenarios

Benchmark scenarios compare attack effectiveness across an axis that varies within the scenario itself. Currently the only benchmark variant is the adversarial benchmark, whose axis of change is the adversarial chat helper model used in attacks. For full configuration options see pyrit_scan --help and the Scenarios Programming Guide.

Adversarial Benchmark

AdversarialBenchmark holds the objective target and dataset constant and varies the adversarial chat model used to drive multi-turn attacks. Useful for evaluating which adversarial helper models produce stronger or weaker attack success rates against the same target.

Adversarial targets are user-provided via the adversarial_targets scenario parameter. Each name must already be registered in TargetRegistry — typically by TargetInitializer from the ADVERSARIAL_CHAT_* env vars (see .env_example). Use pyrit_scan list-targets to see every target currently registered.

pyrit_scan run benchmark.adversarial \
  --initializers target \
  --target openai_chat \
  --adversarial-targets adversarial_chat \
  --techniques role_play_video_game \
  --max-total 1

Pass multiple --adversarial-targets values to compare across models in a single run.

Default techniques: role_play_video_game, crescendo_simulated, and tap. TAP’s branching search makes this default slower and more expensive than the former light default. Every selected adversarial technique prepends one shared benchmark guidance layer to its native system prompt, including explicitly selected techniques such as red_teaming. The native strategy, template parameters, and response format remain the source of truth. Global registered factories and techniques used outside this benchmark remain unchanged. For a cheaper run, explicitly pass --techniques light.

Other available selections: light, single_turn, multi_turn, plus one member per adversarial-capable source technique (e.g. red_teaming, tap, crescendo_simulated).

Setup

from pyrit.output import output_scenario_async
from pyrit.prompt_target import OpenAIChatTarget
from pyrit.scenario import DatasetAttackConfiguration, DatasetSource
from pyrit.scenario.benchmark import AdversarialBenchmark
from pyrit.setup import IN_MEMORY, initialize_pyrit_async
from pyrit.setup.initializers import ScorerInitializer, TargetInitializer, TechniqueInitializer

await initialize_pyrit_async(  # type: ignore
    memory_db_type=IN_MEMORY,
    initializers=[TargetInitializer(), ScorerInitializer(), TechniqueInitializer()],
)

objective_target = OpenAIChatTarget()
Auto-discovered plaintext environment file ./.pyrit/.env will be loaded. Azure Key Vault through env_akv_ref is more secure for shared or deployed secrets; use .env.local only for deliberate local overrides. To inspect a resolved AKV-only configuration from a source checkout, run `python -m build_scripts.export_akv_environment`; it writes ~/.pyrit/.env_akv.

WARNING: Auto-discovered plaintext environment file ./.pyrit/.env will be loaded. Azure Key Vault through env_akv_ref is more secure for shared or deployed secrets; use .env.local only for deliberate local overrides. To inspect a resolved AKV-only configuration from a source checkout, run `python -m build_scripts.export_akv_environment`; it writes ~/.pyrit/.env_akv.
Found default environment files: ['./.pyrit/.env']
Loaded environment file: ./.pyrit/.env

[pyrit:alembic] Scored expectation migration: adding scored_expectation column.
[pyrit:alembic] Scored expectation backfill: processing rows in batches of 500.
[pyrit:alembic] Scored expectation backfill: updated 0 row(s).
[pyrit:alembic] Scored expectation migration: dropping legacy objective column.
[pyrit:alembic] Scored expectation migration: upgrade completed.
[pyrit:alembic] Attack history migration: adding attribution columns.
[pyrit:alembic] Attack history migration: moving attribution values from labels.
[pyrit:alembic] Attack attribution backfill: processing 0 row(s) in 0 batch(es).
[pyrit:alembic] Attack attribution backfill: updated 0 row(s).
[pyrit:alembic] Attack history migration: validating and bounding indexed text columns.
[pyrit:alembic] Attack history migration: replacing AttackResultEntries indexes.
[pyrit:alembic] Attack history migration: creating ix_AttackResultEntries_conversation_timestamp_id.
[pyrit:alembic] Attack history migration: creating ix_AttackResultEntries_operator_timestamp_id.
[pyrit:alembic] Attack history migration: creating ix_AttackResultEntries_operation_timestamp_id.
[pyrit:alembic] Attack history migration: replacing PromptMemoryEntries indexes.
[pyrit:alembic] Attack history migration: creating ix_PromptMemoryEntries_conversation_sequence_id.
[pyrit:alembic] Attack history migration: creating ScenarioResultEntries indexes.
[pyrit:alembic] Attack history migration: creating ix_ScenarioResultEntries_scenario_name_timestamp_id.
[pyrit:alembic] Attack history migration: creating ix_ScenarioResultEntries_scenario_run_state_timestamp_id.
[pyrit:alembic] Attack history migration: upgrade completed.

from pyrit.scenario.benchmark import AdversarialBenchmarkTechnique

dataset_config = DatasetAttackConfiguration(sources=[DatasetSource(name="harmbench")], max_total=1)

scenario = AdversarialBenchmark()
scenario.set_params_from_args(
    args={
        "adversarial_targets": ["adversarial_chat"],
        "objective_target": objective_target,
        "scenario_techniques": [AdversarialBenchmarkTechnique.role_play_video_game],
        "dataset_config": dataset_config,
    }
)
await scenario.initialize_async()  # type: ignore

scenario_result = await scenario.run_async()  # type: ignore




await output_scenario_async(scenario_result)

====================================================================================================
                              📊 SCENARIO RESULTS: AdversarialBenchmark                              
====================================================================================================

▼ Scenario Information
────────────────────────────────────────────────────────────────────────────────────────────────────
  📋 Scenario Details
    • Name: AdversarialBenchmark
    • Result ID: 7690eb39-ec4d-45a8-9721-4e7eb3e8739b
    • Scenario Version: 6
    • PyRIT Version: 1.2.0.dev0
    • Description:
        Benchmark scenario that compares the attack success rate (ASR) across adversarial models. Adversarial targets
        are user-supplied via the ``adversarial_targets`` parameter (declared in ``supported_parameters``). Each target
        must already be registered in ``TargetRegistry`` — typically by ``TargetInitializer`` from
        ``ADVERSARIAL_CHAT_*`` env vars, or programmatically via
        ``TargetRegistry.get_registry_singleton().instances.register``. Every selected adversarial technique prepends
        one shared benchmark guidance layer to its native adversarial system prompt at creation time; global factories
        and canonical prompt files are left unchanged. At run time, ``_build_atomic_attacks_async`` performs the
        ``(technique × adversarial_target × dataset)`` cross-product: for each selected adversarial-capable factory in
        the ``AttackTechniqueRegistry`` and each requested target, it calls ``factory.create(adversarial_chat=...)``
        with the resolved target — no global registry mutation. The resulting ``AtomicAttack`` is named
        ``f"{technique}__{target}_{dataset}"`` with ``display_group`` set to the target's registry name so per-model ASR
        rolls up naturally in result displays.

  🎯 Target Information
    • Target Type: OpenAIChatTarget
    • Target Model: gpt-4o-japan-nilfilter
    • Target Endpoint: https://pyrit-japan-test.openai.azure.com/openai/v1

  📊 Scorer Information
    ▸ Scorer Identifier
      • Scorer Type: TrueFalseCompositeScorer
        Configuration:
          score_aggregator=AND_
          sub_scorers_order_independent=True
          scorer_type=true_false
        ▸ sub_scorers (2 components)
          • Component 1: SelfAskTrueFalseScorer
            Configuration:
              score_aggregator=OR_
              scorer_type=true_false
              user_prompt_template=<43 chars>
              response_json_schema=<270 chars>
              question=<1257 chars>
              system_prompt_template=<2358 chars>
            • prompt_target: OpenAIChatTarget
              Configuration:
                underlying_model_name=gpt-5.4
          • Component 2: TrueFalseInverterScorer
            Configuration:
              score_aggregator=OR_
              scorer_type=true_false
            ▸ sub_scorers (1 components)
              • Component 1: SelfAskRefusalScorer
                Configuration:
                  score_aggregator=OR_
                  scorer_type=true_false
                  user_prompt_template=<74 chars>
                  response_json_schema=<461 chars>
                  system_prompt_template=<5139 chars>
                • prompt_target: OpenAIChatTarget
                  Configuration:
                    underlying_model_name=gpt-5.4

    ▸ Performance Metrics
      Official evaluation has not been run yet for this specific configuration

▼ Overall Statistics
────────────────────────────────────────────────────────────────────────────────────────────────────
  📈 Summary
    • Total Techniques: 1
    • Total Objective Executions: 1
    • Total Attempts: 1
    • Overall Success Rate: 0%
    • Unique Objectives: 1

▼ Per-Group Breakdown
────────────────────────────────────────────────────────────────────────────────────────────────────

  🔸 Group: adversarial_chat
    • Objective Executions: 1
    • Attempts: 1
    • Success Rate: 0%

====================================================================================================
PyRIT Documentation
AIRT Scenarios
PyRIT Documentation
Foundry Scenarios