Adaptive Scenarios - PyRIT Documentation

Adaptive scenarios select attack techniques for each objective from prior success data. They can spend more attempts on techniques that have worked well while retaining some exploration.

TextAdaptive¶

TextAdaptive uses an epsilon-greedy selector over text-compatible attack techniques. Each objective stops after the first successful technique or after max_attempts_per_objective. The direct prompt is available as a baseline comparison.

pyrit_scan run adaptive.text_adaptive \
  --initializers target \
  --target openai_chat \
  --dataset-names airt_hate \
  --max-dataset-size 2 \
  --max-attempts-per-objective 2

from pathlib import Path

from pyrit.output import output_scenario_async
from pyrit.registry import TargetRegistry
from pyrit.scenario import DatasetAttackConfiguration
from pyrit.scenario.adaptive import TextAdaptive
from pyrit.setup import initialize_from_config_async

await initialize_from_config_async(config_path=Path("pyrit_conf.yaml"))  # type: ignore

objective_target = TargetRegistry.get_registry_singleton().instances.get("openai_chat")

dataset_config = DatasetAttackConfiguration(dataset_names=["airt_hate"], max_dataset_size=2)

scenario = TextAdaptive()
scenario.set_params_from_args(  # type: ignore
    args={
        "objective_target": objective_target,
        "dataset_config": dataset_config,
        "max_attempts_per_objective": 2,
    }
)
await scenario.initialize_async()  # type: ignore

scenario_result = await scenario.run_async()  # type: ignore

Auto-discovered plaintext environment file ./.pyrit/.env will be loaded. Azure Key Vault through env_akv_ref is more secure for shared or deployed secrets; use .env.local only for deliberate local overrides. To inspect a resolved AKV-only configuration from a source checkout, run `python -m build_scripts.export_akv_environment`; it writes ~/.pyrit/.env_akv.

WARNING: Auto-discovered plaintext environment file ./.pyrit/.env will be loaded. Azure Key Vault through env_akv_ref is more secure for shared or deployed secrets; use .env.local only for deliberate local overrides. To inspect a resolved AKV-only configuration from a source checkout, run `python -m build_scripts.export_akv_environment`; it writes ~/.pyrit/.env_akv.
Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']
Loaded environment file: ./.pyrit/.env
Loaded environment file: ./.pyrit/.env.local

[pyrit:alembic] No new upgrade operations detected.

TextAdaptive: _EXCLUDED_TECHNIQUES entries ['prompt_sending'] are not in the current scenario-techniques catalog ['best_of_n', 'bijection', 'code_attack', 'context_compliance', 'crescendo_history_lecture', 'crescendo_journalist_interview', 'crescendo_movie_director', 'crescendo_simulated', 'flip', 'many_shot', 'pair', 'red_teaming', 'role_play_movie_script', 'role_play_persuasion', 'role_play_persuasion_written', 'role_play_trivia_game', 'role_play_video_game', 'skeleton_key', 'split_payload', 'tap', 'violent_durian']; the exclusion is a no-op for those entries. Remove stale entries or update the catalog.

await output_scenario_async(scenario_result)


====================================================================================================
                                  📊 SCENARIO RESULTS: TextAdaptive                                  
====================================================================================================

▼ Scenario Information
────────────────────────────────────────────────────────────────────────────────────────────────────
  📋 Scenario Details
    • Name: TextAdaptive
    • Result ID: 1d3c86d4-53b0-4943-9def-085206a8d38e
    • Scenario Version: 1
    • PyRIT Version: 1.1.0.dev0
    • Description:
        Selects an attack technique for each objective using an epsilon-greedy strategy informed by prior success rates.
        The scenario stops after a successful attack or after ``max_attempts_per_objective`` attempts.

  🎯 Target Information
    • Target Type: OpenAIChatTarget
    • Target Model: gpt-4o
    • Target Endpoint: https://pyrit-japan-test.openai.azure.com/openai/v1

  📊 Scorer Information
    ▸ Scorer Identifier
      • Scorer Type: FloatScaleThresholdScorer
      • scorer_type: true_false
      • score_aggregator: OR_
        └─ Composite of 1 scorer(s):
            • Scorer Type: AzureContentFilterScorer
            • scorer_type: float_scale

    ▸ Performance Metrics
      • Accuracy: 59.24%
      • Accuracy Std Error: ±0.0247
      • F1 Score: 0.5306
      • Precision: 0.5987
      • Recall: 0.4764
      • Average Score Time: 0.04s

▼ Overall Statistics
────────────────────────────────────────────────────────────────────────────────────────────────────
  📈 Summary
    • Total Techniques: 2
    • Total Attack Results: 6
    • Overall Success Rate: 50%
    • Unique Objectives: 2

▼ Per-Group Breakdown
────────────────────────────────────────────────────────────────────────────────────────────────────

  🔸 Group: airt_hate
    • Number of Results: 4
    • Success Rate: 50%

  🔸 Group: baseline
    • Number of Results: 2
    • Success Rate: 50%

====================================================================================================

For more details, see the Scenarios Programming Guide and Configuration.