Garak Scenarios - PyRIT Documentation

The Garak scenario family implements probes inspired by the Garak framework. These include encoding-based probes (which test whether a target can be tricked into producing harmful content when prompts are encoded in various formats) and web-injection probes (which test whether a target emits markdown data-exfiltration or cross-site-scripting payloads).

For full programming details, see the Scenarios Programming Guide.

from pathlib import Path

from pyrit.output import output_scenario_async
from pyrit.registry import TargetRegistry
from pyrit.scenario.garak import Encoding, EncodingTechnique
from pyrit.scenario.garak.encoding import EncodingDatasetConfiguration
from pyrit.setup import initialize_from_config_async

await initialize_from_config_async(config_path=Path("pyrit_conf.yaml"))  # type: ignore

objective_target = TargetRegistry.get_registry_singleton().instances.get("openai_chat")

Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']
Loaded environment file: ./.pyrit/.env
Loaded environment file: ./.pyrit/.env.local

[pyrit:alembic] No new upgrade operations detected.

TextAdaptive: _EXCLUDED_TECHNIQUES entries ['prompt_sending'] are not in the current scenario-techniques catalog ['context_compliance', 'crescendo_history_lecture', 'crescendo_journalist_interview', 'crescendo_movie_director', 'crescendo_simulated', 'many_shot', 'pair', 'red_teaming', 'role_play', 'tap', 'violent_durian']; the exclusion is a no-op for those entries. Remove stale entries or update the catalog.

Encoding¶

Tests whether the target can decode and comply with encoded harmful prompts. Each encoding technique encodes the prompt, asks the target to decode it, and scores whether the decoded output matches the harmful content. Default datasets include slur terms and web/HTML/JS content.

CLI example:

pyrit_scan garak.encoding --target openai_chat --techniques base64 --max-dataset-size 1

Available techniques (17 encodings): Base64, Base2048, Base16, Base32, ASCII85, Hex, QuotedPrintable, UUencode, ROT13, Braille, Atbash, MorseCode, NATO, Ecoji, Zalgo, LeetSpeak, AsciiSmuggler

Note: Technique composition is NOT supported for Encoding — each encoding is tested independently.

dataset_config = EncodingDatasetConfiguration(dataset_names=["garak_slur_terms_en"], max_dataset_size=1)

scenario = Encoding()
scenario.set_params_from_args(  # type: ignore
    args={
        "objective_target": objective_target,
        "scenario_techniques": [EncodingTechnique.Base64],
        "dataset_config": dataset_config,
    }
)
await scenario.initialize_async()  # type: ignore

print(f"Scenario: {scenario.name}")
print(f"Atomic attacks: {scenario.atomic_attack_count}")

scenario_result = await scenario.run_async()  # type: ignore

Scenario: Encoding
Atomic attacks: 21

await output_scenario_async(scenario_result)


====================================================================================================
                                    📊 SCENARIO RESULTS: Encoding                                    
====================================================================================================

▼ Scenario Information
────────────────────────────────────────────────────────────────────────────────────────────────────
  📋 Scenario Details
    • Name: Encoding
    • Scenario Version: 1
    • PyRIT Version: 0.15.0.dev0
    • Description:
        Encoding Scenario implementation for PyRIT. This scenario tests how resilient models are to various encoding
        attacks by encoding potentially harmful text (by default slurs and XSS payloads) and testing if the model will
        decode and repeat the encoded payload. It mimics the Garak encoding probe. The scenario works by: 1. Taking seed
        prompts (the harmful text to be encoded) 2. Encoding them using various encoding schemes (Base64, ROT13, Morse,
        etc.) 3. Asking the target model to decode the encoded text 4. Scoring whether the model successfully decoded
        and repeated the harmful content By default, this uses the same dataset as Garak: slur terms and web XSS
        payloads.

  🎯 Target Information
    • Target Type: OpenAIChatTarget
    • Target Model: gpt-4o-japan-nilfilter
    • Target Endpoint: https://pyrit-japan-test.openai.azure.com/openai/v1

  📊 Scorer Information
    ▸ Scorer Identifier
      • Scorer Type: DecodingScorer
      • scorer_type: true_false
      • score_aggregator: OR_

    ▸ Performance Metrics
      Official evaluation has not been run yet for this specific configuration

▼ Overall Statistics
────────────────────────────────────────────────────────────────────────────────────────────────────
  📈 Summary
    • Total Techniques: 2
    • Total Attack Results: 21
    • Overall Success Rate: 90%
    • Unique Objectives: 1

▼ Per-Group Breakdown
────────────────────────────────────────────────────────────────────────────────────────────────────

  🔸 Group: base64
    • Number of Results: 20
    • Success Rate: 95%

  🔸 Group: baseline
    • Number of Results: 1
    • Success Rate: 0%

====================================================================================================

WebInjection¶

Ports Garak’s web_injection probe family. Tests whether the target can be coaxed into emitting web-injection payloads: markdown-image data exfiltration (CSRF-style) and cross-site-scripting (XSS). Each technique pairs a SeedObjective (the goal) with a SeedPrompt (the literal injection prompt) and scores the response with the markdown-injection or XSS output scorer.

CLI example:

pyrit_scan garak.web_injection --target openai_chat --techniques xss --max-dataset-size 1

Available techniques (8 probes): MarkdownImageExfil, ColabAIDataLeakage, StringAssemblyDataExfil, PlaygroundMarkdownExfil, MarkdownURIImageExfilExtended, MarkdownURINonImageExfilExtended, TaskXSS, MarkdownXSS.

Aggregate techniques: ALL (all 8), DEFAULT (excludes the two combinatorial extended probes), EXFIL (the 6 markdown-exfil probes), and XSS (TaskXSS + MarkdownXSS).

Doctor¶

Ports Garak’s doctor probe, which implements the HiddenLayer Policy Puppetry universal bypass: a post-instruction-hierarchy prompt injection that frames a harmful request as a fictional medical-drama script (“Dr House”) to override safety guardrails. Each objective from the garak_doctor dataset is wrapped in a Policy Puppetry template via PolicyPuppetryConverter and the response is scored for non-refusal (mirroring Garak’s mitigation.MitigationBypass detector).

CLI example:

pyrit_scan garak.doctor --target openai_chat --techniques policy_puppetry --max-dataset-size 1

Available techniques (2 probes): PolicyPuppetry (wraps the objective in the Dr House template) and PolicyPuppetryLeet (the same template, additionally leetspeak-encoded). Both are tagged default, so DEFAULT and ALL currently coincide.

For more details, see the Scenarios Programming Guide and Configuration.