Adaptive Scenarios - PyRIT Documentation

Adaptive scenarios select attack techniques for each objective from prior success data. They can spend more attempts on techniques that have worked well while retaining some exploration.

TextAdaptive¶

TextAdaptive uses an epsilon-greedy selector over text-compatible attack techniques. Each objective stops after the first successful technique or after max_attempts_per_objective. The direct prompt is available as a baseline comparison.

pyrit_scan run adaptive.text_adaptive \
  --initializers target \
  --target openai_chat \
  --datasets airt_hate \
  --max-total 2 \
  --max-attempts-per-objective 2

from pathlib import Path

from pyrit.output import output_scenario_async
from pyrit.registry import TargetRegistry
from pyrit.scenario import DatasetAttackConfiguration, DatasetSource
from pyrit.scenario.adaptive import ImageTechniqueAdaptive, TextAdaptive
from pyrit.setup import initialize_from_config_async

await initialize_from_config_async(config_path=Path("pyrit_conf.yaml"))  # type: ignore

objective_target = TargetRegistry.get_registry_singleton().instances.get("openai_chat")

dataset_config = DatasetAttackConfiguration(sources=[DatasetSource(name="airt_hate")], max_total=2)

scenario = TextAdaptive()
scenario.set_params_from_args(  # type: ignore
    args={
        "objective_target": objective_target,
        "dataset_config": dataset_config,
        "max_attempts_per_objective": 2,
    }
)
await scenario.initialize_async()  # type: ignore

scenario_result = await scenario.run_async()  # type: ignore

No default environment files found. Using system environment variables only.

Skipping scorer task_achieved_microsoft_decision: required target not found in TargetRegistry
Skipping scorer task_achieved_microsoft_decision_threshold_07: required target not found in TargetRegistry
TextAdaptive: _EXCLUDED_TECHNIQUES entries ['prompt_sending'] are not in the current scenario-techniques catalog ['best_of_n', 'bijection', 'code_attack', 'code_attack_framed', 'context_compliance', 'crescendo', 'crescendo_history_lecture', 'crescendo_journalist_interview', 'crescendo_movie_director', 'crescendo_simulated', 'flip', 'goat', 'many_shot', 'pair', 'red_teaming', 'role_play_movie_script', 'role_play_persuasion', 'role_play_persuasion_written', 'role_play_trivia_game', 'role_play_video_game', 'skeleton_key', 'split_payload', 'tap', 'violent_durian']; the exclusion is a no-op for those entries. Remove stale entries or update the catalog.
Technique 'many_shot', dataset 'airt_hate': skipped 0 seed group(s); 1 require objective-only adaptation for execution.
Technique 'role_play_movie_script', dataset 'airt_hate': skipped 1 seed group(s); 0 require objective-only adaptation for execution. Reasons: {'Technique accepts only an objective; dataset prompts and simulated conversations are not accepted.': 1}

await output_scenario_async(scenario_result)


====================================================================================================
                                  📊 SCENARIO RESULTS: TextAdaptive                                  
====================================================================================================

▼ Scenario Information
────────────────────────────────────────────────────────────────────────────────────────────────────
  📋 Scenario Details
    • Name: TextAdaptive
    • Result ID: 0ab03cdd-3cc8-43af-bfbd-f460ce5bf10b
    • Scenario Version: 1
    • PyRIT Version: 1.2.0.dev0
    • Description:
        Selects an ordered technique list for each objective before execution, using historical success rates. The
        scenario stops after a successful attack or after ``max_attempts_per_objective`` attempts.

  🎯 Target Information
    • Target Type: OpenAIChatTarget
    • Target Model: gpt-4o
    • Target Endpoint: https://pyrit-japan-test.openai.azure.com/openai/v1

  📊 Scorer Information
    ▸ Scorer Identifier
      • Scorer Type: TrueFalseInverterScorer
        Configuration:
          score_aggregator=OR_
          scorer_type=true_false
        ▸ sub_scorers (1 components)
          • Component 1: SelfAskRefusalScorer
            Configuration:
              score_aggregator=OR_
              scorer_type=true_false
              user_prompt_template=<74 chars>
              response_json_schema=<461 chars>
              system_prompt_template=<5139 chars>
            • prompt_target: OpenAIChatTarget
              Configuration:
                underlying_model_name=gpt-5.4

    ▸ Performance Metrics
      • Accuracy: 89.62%
      • Accuracy Std Error: ±0.0153
      • F1 Score: 0.8935
      • Precision: 0.8866
      • Recall: 0.9005
      • Average Score Time: 1.18s

▼ Overall Statistics
────────────────────────────────────────────────────────────────────────────────────────────────────
  📈 Summary
    • Total Techniques: 2
    • Total Objective Executions: 4
    • Total Attempts: 5
    • Overall Success Rate: 25%
    • Unique Objectives: 2

▼ Per-Group Breakdown
────────────────────────────────────────────────────────────────────────────────────────────────────

  🔸 Group: airt_hate
    • Objective Executions: 2
    • Attempts: 3
    • Success Rate: 50%

  🔸 Group: baseline
    • Objective Executions: 2
    • Attempts: 2
    • Success Rate: 0%

====================================================================================================

ImageTechniqueAdaptive¶

ImageTechniqueAdaptive is the image sibling of TextAdaptive. It renders each text objective into an image (blank canvas, QR code, grid composite, comic panel, ...) and sends it to a vision-capable target, then scores the text response. The objective target must accept both text and image input (so the direct-text baseline stays a valid comparison) and return text — use a multimodal target such as openai_chat (e.g. gpt-4o). See the Adaptive Scenarios programming guide for a full walkthrough.

pyrit_scan run adaptive.image_technique_adaptive \
  --initializers target \
  --target openai_chat

image_dataset_config = DatasetAttackConfiguration(sources=[DatasetSource(name="airt_hate")], max_total=1)

image_scenario = ImageTechniqueAdaptive()
image_scenario.set_params_from_args(  # type: ignore
    args={
        "objective_target": objective_target,
        "dataset_config": image_dataset_config,
        "max_attempts_per_objective": 1,
    }
)
await image_scenario.initialize_async()  # type: ignore

image_scenario_result = await image_scenario.run_async()  # type: ignore

await output_scenario_async(image_scenario_result)


====================================================================================================
                             📊 SCENARIO RESULTS: ImageTechniqueAdaptive                             
====================================================================================================

▼ Scenario Information
────────────────────────────────────────────────────────────────────────────────────────────────────
  📋 Scenario Details
    • Name: ImageTechniqueAdaptive
    • Result ID: 67b753b1-bd65-415c-80de-4f7517045b3d
    • Scenario Version: 1
    • PyRIT Version: 1.2.0.dev0
    • Description:
        Selects an image-rendering attack technique for each objective using an epsilon-greedy strategy informed by
        prior success rates. Each technique renders the text objective as an image for a vision-capable target, and the
        target's text response is scored. The scenario stops after a successful attack or after
        ``max_attempts_per_objective`` attempts.

  🎯 Target Information
    • Target Type: OpenAIChatTarget
    • Target Model: gpt-4o
    • Target Endpoint: https://pyrit-japan-test.openai.azure.com/openai/v1

  📊 Scorer Information
    ▸ Scorer Identifier
      • Scorer Type: TrueFalseInverterScorer
        Configuration:
          score_aggregator=OR_
          scorer_type=true_false
        ▸ sub_scorers (1 components)
          • Component 1: SelfAskRefusalScorer
            Configuration:
              score_aggregator=OR_
              scorer_type=true_false
              user_prompt_template=<74 chars>
              response_json_schema=<461 chars>
              system_prompt_template=<5139 chars>
            • prompt_target: OpenAIChatTarget
              Configuration:
                underlying_model_name=gpt-5.4

    ▸ Performance Metrics
      • Accuracy: 89.62%
      • Accuracy Std Error: ±0.0153
      • F1 Score: 0.8935
      • Precision: 0.8866
      • Recall: 0.9005
      • Average Score Time: 1.18s

▼ Overall Statistics
────────────────────────────────────────────────────────────────────────────────────────────────────
  📈 Summary
    • Total Techniques: 2
    • Total Objective Executions: 2
    • Total Attempts: 2
    • Overall Success Rate: 100%
    • Unique Objectives: 1

▼ Per-Group Breakdown
────────────────────────────────────────────────────────────────────────────────────────────────────

  🔸 Group: airt_hate
    • Objective Executions: 1
    • Attempts: 1
    • Success Rate: 100%

  🔸 Group: baseline
    • Objective Executions: 1
    • Attempts: 1
    • Success Rate: 100%

====================================================================================================

For more details, see the Scenarios Programming Guide and Configuration.