Benchmark Scenarios - PyRIT Documentation

Benchmark scenarios compare attack effectiveness across an axis that varies within the scenario itself. Currently the only benchmark variant is the adversarial benchmark, whose axis of change is the adversarial chat helper model used in attacks. For full configuration options see pyrit_scan --help and the Scenarios Programming Guide.

Adversarial Benchmark¶

AdversarialBenchmark holds the objective target and dataset constant and varies the adversarial chat model used to drive multi-turn attacks. Useful for evaluating which adversarial helper models produce stronger or weaker attack success rates against the same target.

Adversarial targets are user-provided via the adversarial_targets scenario parameter. Each name must already be registered in TargetRegistry — typically by TargetInitializer from the ADVERSARIAL_CHAT_* env vars (see .env_example). Use pyrit_scan --list-targets to see every target currently registered.

pyrit_scan benchmark.adversarial \
  --initializers target \
  --target openai_chat \
  --adversarial-targets adversarial_chat_singleturn adversarial_chat_multiturn \
  --max-dataset-size 4

Pass multiple --adversarial-targets values to compare across models in a single run.

Available techniques: light (default — a quick snapshot using the cheaper techniques), single_turn, multi_turn, plus one member per adversarial-capable source technique (e.g. red_teaming, tap, crescendo_simulated). The light aggregate excludes tap and crescendo_simulated, which can take hours.

Setup¶

from pyrit.output import output_scenario_async
from pyrit.prompt_target import OpenAIChatTarget
from pyrit.scenario import DatasetAttackConfiguration
from pyrit.scenario.benchmark import AdversarialBenchmark
from pyrit.setup import IN_MEMORY, initialize_pyrit_async
from pyrit.setup.initializers import (
    LoadDefaultDatasets,
    ScorerInitializer,
    TargetInitializer,
    TechniqueInitializer,
)

await initialize_pyrit_async(  # type: ignore
    memory_db_type=IN_MEMORY,
    initializers=[TargetInitializer(), ScorerInitializer(), TechniqueInitializer(), LoadDefaultDatasets()],
)

objective_target = OpenAIChatTarget()

Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']
Loaded environment file: ./.pyrit/.env
Loaded environment file: ./.pyrit/.env.local

[pyrit:alembic] No new upgrade operations detected.

TextAdaptive: _EXCLUDED_TECHNIQUES entries ['prompt_sending'] are not in the current scenario-techniques catalog ['context_compliance', 'crescendo_history_lecture', 'crescendo_journalist_interview', 'crescendo_movie_director', 'crescendo_simulated', 'many_shot', 'pair', 'red_teaming', 'role_play', 'tap', 'violent_durian']; the exclusion is a no-op for those entries. Remove stale entries or update the catalog.

dataset_config = DatasetAttackConfiguration(dataset_names=["harmbench"], max_dataset_size=4)

scenario = AdversarialBenchmark()
scenario.set_params_from_args(
    args={
        "adversarial_targets": ["adversarial_chat_singleturn", "adversarial_chat_multiturn"],
        "objective_target": objective_target,
        "dataset_config": dataset_config,
    }
)
await scenario.initialize_async()  # type: ignore

scenario_result = await scenario.run_async()  # type: ignore

await output_scenario_async(scenario_result)


====================================================================================================
                              📊 SCENARIO RESULTS: AdversarialBenchmark                              
====================================================================================================

▼ Scenario Information
────────────────────────────────────────────────────────────────────────────────────────────────────
  📋 Scenario Details
    • Name: AdversarialBenchmark
    • Scenario Version: 2
    • PyRIT Version: 0.15.0.dev0
    • Description:
        Benchmark scenario that compares the attack success rate (ASR) across adversarial models. Adversarial targets
        are user-supplied via the ``adversarial_targets`` parameter (declared in ``supported_parameters``). Each target
        must already be registered in ``TargetRegistry`` — typically by ``TargetInitializer`` from
        ``ADVERSARIAL_CHAT_*`` env vars, or programmatically via
        ``TargetRegistry.get_registry_singleton().instances.register``. At run time, ``_build_atomic_attacks_async``
        performs the ``(technique × adversarial_target × dataset)`` cross-product: for each selected adversarial-capable
        ``core`` factory in the ``AttackTechniqueRegistry`` and each requested target, it calls
        ``factory.create(adversarial_chat=...)`` with the resolved target — no global registry mutation. The resulting
        ``AtomicAttack`` is named ``f"{technique}__{target}_{dataset}"`` with ``display_group`` set to the target's
        registry name so per-model ASR rolls up naturally in result displays.

  🎯 Target Information
    • Target Type: OpenAIChatTarget
    • Target Model: gpt-4o-japan-nilfilter
    • Target Endpoint: https://pyrit-japan-test.openai.azure.com/openai/v1

  📊 Scorer Information
    ▸ Scorer Identifier
      • Scorer Type: FloatScaleThresholdScorer
      • scorer_type: true_false
      • score_aggregator: OR_
        └─ Composite of 1 scorer(s):
            • Scorer Type: AzureContentFilterScorer
            • scorer_type: float_scale

    ▸ Performance Metrics
      • Accuracy: 59.24%
      • Accuracy Std Error: ±0.0247
      • F1 Score: 0.5306
      • Precision: 0.5987
      • Recall: 0.4764
      • Average Score Time: 0.04s

▼ Overall Statistics
────────────────────────────────────────────────────────────────────────────────────────────────────
  📈 Summary
    • Total Techniques: 6
    • Total Attack Results: 24
    • Overall Success Rate: 54%
    • Unique Objectives: 4

▼ Per-Group Breakdown
────────────────────────────────────────────────────────────────────────────────────────────────────

  🔸 Group: adversarial_chat_singleturn
    • Number of Results: 12
    • Success Rate: 66%

  🔸 Group: adversarial_chat_multiturn
    • Number of Results: 12
    • Success Rate: 41%

====================================================================================================