Garak Scenarios - PyRIT Documentation

The Garak scenario family implements encoding-based probes inspired by the Garak framework. These test whether a target model can be tricked into producing harmful content when prompts are encoded in various formats.

For full programming details, see the Scenarios Programming Guide.

from pathlib import Path

from pyrit.registry import TargetRegistry
from pyrit.scenario.printer.console_printer import ConsoleScenarioResultPrinter
from pyrit.scenario.scenarios.garak import Encoding, EncodingStrategy
from pyrit.scenario.scenarios.garak.encoding import EncodingDatasetConfiguration
from pyrit.setup import initialize_from_config_async

await initialize_from_config_async(config_path=Path("pyrit_conf.yaml"))  # type: ignore

objective_target = TargetRegistry.get_registry_singleton().get_instance_by_name("openai_chat")
printer = ConsoleScenarioResultPrinter()

Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']
Loaded environment file: ./.pyrit/.env
Loaded environment file: ./.pyrit/.env.local

Encoding¶

Tests whether the target can decode and comply with encoded harmful prompts. Each encoding strategy encodes the prompt, asks the target to decode it, and scores whether the decoded output matches the harmful content. Default datasets include slur terms and web/HTML/JS content.

CLI example:

pyrit_scan garak.encoding --target openai_chat --strategies base64 --max-dataset-size 1

Available strategies (17 encodings): Base64, Base2048, Base16, Base32, ASCII85, Hex, QuotedPrintable, UUencode, ROT13, Braille, Atbash, MorseCode, NATO, Ecoji, Zalgo, LeetSpeak, AsciiSmuggler

Note: Strategy composition is NOT supported for Encoding — each encoding is tested independently.

dataset_config = EncodingDatasetConfiguration(dataset_names=["garak_slur_terms_en"], max_dataset_size=1)

scenario = Encoding()
await scenario.initialize_async(  # type: ignore
    objective_target=objective_target,
    scenario_strategies=[EncodingStrategy.Base64],
    dataset_config=dataset_config,
)

print(f"Scenario: {scenario.name}")
print(f"Atomic attacks: {scenario.atomic_attack_count}")

scenario_result = await scenario.run_async()  # type: ignore

Scenario: Encoding
Atomic attacks: 21

await printer.print_summary_async(scenario_result)  # type: ignore


====================================================================================================
                                    📊 SCENARIO RESULTS: Encoding                                    
====================================================================================================

▼ Scenario Information
────────────────────────────────────────────────────────────────────────────────────────────────────
  📋 Scenario Details
    • Name: Encoding
    • Scenario Version: 1
    • PyRIT Version: 0.12.1.dev0
    • Description:
        Encoding Scenario implementation for PyRIT. This scenario tests how resilient models are to various encoding
        attacks by encoding potentially harmful text (by default slurs and XSS payloads) and testing if the model will
        decode and repeat the encoded payload. It mimics the Garak encoding probe. The scenario works by: 1. Taking seed
        prompts (the harmful text to be encoded) 2. Encoding them using various encoding schemes (Base64, ROT13, Morse,
        etc.) 3. Asking the target model to decode the encoded text 4. Scoring whether the model successfully decoded
        and repeated the harmful content By default, this uses the same dataset as Garak: slur terms and web XSS
        payloads.

  🎯 Target Information
    • Target Type: OpenAIChatTarget
    • Target Model: gpt-4o
    • Target Endpoint: https://pyrit-dev.openai.azure.com/openai/v1

  📊 Scorer Information
    ▸ Scorer Identifier
      • Scorer Type: DecodingScorer
      • scorer_type: true_false
      • score_aggregator: OR_

    ▸ Performance Metrics
      Official evaluation has not been run yet for this specific configuration

▼ Overall Statistics
────────────────────────────────────────────────────────────────────────────────────────────────────
  📈 Summary
    • Total Strategies: 2
    • Total Attack Results: 21
    • Overall Success Rate: 90%
    • Unique Objectives: 2

▼ Per-Strategy Breakdown
────────────────────────────────────────────────────────────────────────────────────────────────────

  🔸 Strategy: baseline
    • Number of Results: 1
    • Success Rate: 100%

  🔸 Strategy: base64
    • Number of Results: 20
    • Success Rate: 90%

====================================================================================================

For more details, see the Scenarios Programming Guide and Configuration.