Modality feedback in multi-turn attacks (Crescendo example)

This notebook shows how attacks such as CrescendoAttack, RedTeamingAttack, and TAPAttack use target capabilities to decide whether media should be forwarded turn-to-turn.

We use a two-seed image-editing setup:

seed 1: roakey.png
seed 2: a real photo of a three-masted ship

and an intentionally detailed objective:

show the character from seed 1 boarding the three-masted ship from seed 2 at night, hanging upside down from a scarlet rope and yelling, while a turquoise clockwork albatross with brass wings holds a red key in its beak atop the center mast.

The Crescendo image-generation system prompt starts with a benign base composition without previewing later details, then introduces bounded refinements in later turns. The same modality wiring applies across Crescendo, Red Teaming, and TAP; we run Crescendo end-to-end here.

import os
from pathlib import Path

from pyrit.auth import get_azure_openai_auth
from pyrit.common.path import EXECUTOR_SEED_PROMPT_PATH
from pyrit.executor.attack import (
    AttackAdversarialConfig,
    AttackScoringConfig,
    CrescendoAttack,
)
from pyrit.models import Message, MessagePiece, SeedPrompt
from pyrit.output import output_attack_async
from pyrit.prompt_target import OpenAIChatTarget, OpenAIImageTarget
from pyrit.prompt_target.common.target_capabilities import TargetCapabilities
from pyrit.prompt_target.common.target_configuration import TargetConfiguration
from pyrit.score import SelfAskTrueFalseScorer, TrueFalseQuestion
from pyrit.setup import IN_MEMORY, initialize_pyrit_async

await initialize_pyrit_async(memory_db_type=IN_MEMORY)  # type: ignore

Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']
Loaded environment file: ./.pyrit/.env
Loaded environment file: ./.pyrit/.env.local

[pyrit:alembic] No new upgrade operations detected.

1) Choose objective-target capability profile¶

This controls how media is handled in the attack loop:

"edit-only": requires text + image_path every turn.
"hybrid": accepts either text-only or text + image_path; this seeded example uses media.

OBJECTIVE_CAPABILITY_PROFILE = "hybrid"  # "edit-only" or "hybrid"

profile_to_input_modalities = {
    "edit-only": frozenset({frozenset({"text", "image_path"})}),
    "hybrid": frozenset({frozenset({"text"}), frozenset({"text", "image_path"})}),
}
if OBJECTIVE_CAPABILITY_PROFILE not in profile_to_input_modalities:
    raise ValueError(f"Unsupported OBJECTIVE_CAPABILITY_PROFILE: {OBJECTIVE_CAPABILITY_PROFILE}")

objective_target = OpenAIImageTarget(
    custom_configuration=TargetConfiguration(
        capabilities=TargetCapabilities(
            # Crescendo requires a multi-turn + editable-history objective target.
            # The image target still receives the latest multimodal turn payload.
            supports_multi_turn=True,
            supports_editable_history=True,
            supports_multi_message_pieces=True,
            input_modalities=profile_to_input_modalities[OBJECTIVE_CAPABILITY_PROFILE],
            output_modalities=frozenset({frozenset({"image_path"})}),
        )
    )
)

print(f"Objective capability profile: {OBJECTIVE_CAPABILITY_PROFILE}")
print(f"Objective input modalities: {objective_target.configuration.capabilities.input_modalities}")

Objective capability profile: hybrid
Objective input modalities: frozenset({frozenset({'image_path', 'text'}), frozenset({'text'})})

2) Build adversarial target and inspect whether it can receive image feedback¶

The modality router checks this up front. If the adversarial target advertises {"text", "image_path"} input, the objective image output can be forwarded along with score feedback; otherwise only text feedback is sent.

adversarial_endpoint = os.environ["AZURE_OPENAI_GPT4O_UNSAFE_CHAT_ENDPOINT"]
adversarial_chat = OpenAIChatTarget(
    endpoint=adversarial_endpoint,
    api_key=get_azure_openai_auth(adversarial_endpoint),
    model_name=os.environ["AZURE_OPENAI_GPT4O_UNSAFE_CHAT_MODEL"],
)

adversarial_input_modalities = adversarial_chat.configuration.capabilities.input_modalities
adversarial_accepts_text_plus_image = frozenset({"text", "image_path"}) in adversarial_input_modalities

print(f"Adversarial input modalities: {adversarial_input_modalities}")
print(f"Adversarial accepts text+image feedback: {adversarial_accepts_text_plus_image}")

Adversarial input modalities: frozenset({frozenset({'image_path', 'text'}), frozenset({'text'}), frozenset({'image_path'})})
Adversarial accepts text+image feedback: True

3) Prepare the two seed images¶

roakey.png is loaded from the docs root.
A modern color photo of a three-masted ship is loaded from a checked-in asset.
Ship photo source: Gorch Fock unter Segeln Kieler Foerde 2006 (Wikimedia Commons), licensed under CC BY-SA 2.5.

roakey_seed_path = (Path(".") / ".." / ".." / "roakey.png").resolve()
ship_seed_path = (Path(".") / "assets" / "three_masted_ship_color.jpg").resolve()

if not roakey_seed_path.exists():
    raise FileNotFoundError(f"Missing roakey seed image: {roakey_seed_path}")

print(f"Seed 1 (roakey): {roakey_seed_path}")
print(f"Seed 2 (ship photo): {ship_seed_path}")

Seed 1 (roakey): ./git/copilot-worktrees/PyRIT/romanlutz-bookish-disco/doc/roakey.png
Seed 2 (ship photo): ./git/copilot-worktrees/PyRIT/romanlutz-bookish-disco/doc/code/executor/assets/three_masted_ship_color.jpg

4) Create a seed message with adversarial placeholder + two images¶

The first text piece is an adversarial placeholder. The attack fills it with the generated prompt, while preserving the two seed images.

shared_conversation_id = "roakey-ship-seeds"
next_message = Message(
    message_pieces=[
        MessagePiece(
            role="user",
            original_value="",
            original_value_data_type="text",
            conversation_id=shared_conversation_id,
            prompt_metadata={"adversarial_placeholder": True},
        ),
        MessagePiece(
            role="user",
            original_value=str(roakey_seed_path),
            original_value_data_type="image_path",
            conversation_id=shared_conversation_id,
        ),
        MessagePiece(
            role="user",
            original_value=str(ship_seed_path),
            original_value_data_type="image_path",
            conversation_id=shared_conversation_id,
        ),
    ]
)

5) Define objective + scorer and execute Crescendo¶

The scorer checks visible final-image criteria jointly. It does not receive the seed images, so it deliberately avoids claiming exact pixel-level identity with either reference. The unusual albatross-and-key detail makes accidental first-turn success unlikely while giving Crescendo several independent details to introduce progressively. The objective target receives its previously generated image as the edit input rather than receiving the two original seeds again.

objective = (
    "Create a moonlit image where the same anthropomorphic raccoon from seed image 1, with the same pirate "
    "attire, boards the three-masted ship from seed image 2 while hanging unmistakably upside down from a "
    "scarlet rope and visibly yelling. A turquoise clockwork albatross with brass wings must be perched atop "
    "the center mast, clearly holding a small red key in its beak."
)

scoring_config = AttackScoringConfig(
    objective_scorer=SelfAskTrueFalseScorer.from_question(
        chat_target=OpenAIChatTarget(),
        question=TrueFalseQuestion(
            true_description=(
                "Evaluate only details visible in the generated image; the seed images are not available "
                "to this scorer, so do not claim or require an exact identity comparison with them. "
                "The generated image satisfies ALL seven of the following criteria simultaneously: "
                "(1) an anthropomorphic raccoon is present in pirate attire, and is not replaced by "
                "a human or anime pirate; "
                "(2) a large three-masted sailing ship is present; "
                "(3) the scene is visibly moonlit or nighttime; "
                "(4) the raccoon is unmistakably upside down while suspended from a scarlet rope; "
                "(5) the raccoon's mouth is clearly open in a visible yell; "
                "(6) a turquoise mechanical albatross with clearly visible brass wings is perched "
                "atop the center mast; "
                "(7) a small red key is clearly visible in the albatross's beak. "
                "ALL seven criteria must be met. If any one is absent or ambiguous, score False."
            )
        ),
    )
)

crescendo_attack = CrescendoAttack(
    objective_target=objective_target,
    attack_adversarial_config=AttackAdversarialConfig(
        target=adversarial_chat,
        system_prompt=SeedPrompt.from_yaml_file(EXECUTOR_SEED_PROMPT_PATH / "crescendo" / "image_generation.yaml"),
    ),
    attack_scoring_config=scoring_config,
    max_turns=6,
    max_backtracks=2,
)

result = await crescendo_attack.execute_async(  # type: ignore
    objective=objective,
    next_message=next_message,
)

await output_attack_async(  # type: ignore
    result,
    include_pruned_conversations=True,
    include_adversarial_conversation=True,
)

Fetching long content....

6) The same pattern for Red Teaming and TAP¶

To run this with RedTeamingAttack or TAPAttack, keep:

the same objective_target capability profile,
the same next_message with adversarial placeholder + two seeds,
an image-capable scoring setup.

Then swap only the attack class and (optionally) the adversarial system prompt.