Garak Scenarios - PyRIT Documentation

The Garak scenario family implements probes inspired by the Garak framework. These include encoding-based probes (which test whether a target can be tricked into producing harmful content when prompts are encoded in various formats) and web-injection probes (which test whether a target emits markdown data-exfiltration or cross-site-scripting payloads).

For full programming details, see the Scenarios Programming Guide.

from pathlib import Path

from pyrit.output import output_scenario_async
from pyrit.registry import TargetRegistry
from pyrit.scenario.garak import Encoding, EncodingTechnique
from pyrit.scenario.garak.encoding import EncodingDatasetConfiguration
from pyrit.setup import initialize_from_config_async

await initialize_from_config_async(config_path=Path("pyrit_conf.yaml"))  # type: ignore

objective_target = TargetRegistry.get_registry_singleton().instances.get("openai_chat")

Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']
Loaded environment file: ./.pyrit/.env
Loaded environment file: ./.pyrit/.env.local

[pyrit:alembic] No new upgrade operations detected.

TextAdaptive: _EXCLUDED_TECHNIQUES entries ['prompt_sending'] are not in the current scenario-techniques catalog ['context_compliance', 'crescendo_history_lecture', 'crescendo_journalist_interview', 'crescendo_movie_director', 'crescendo_simulated', 'many_shot', 'pair', 'red_teaming', 'role_play', 'tap', 'violent_durian']; the exclusion is a no-op for those entries. Remove stale entries or update the catalog.

Encoding¶

Tests whether the target can decode and comply with encoded harmful prompts. Each encoding technique encodes the prompt, asks the target to decode it, and scores whether the decoded output matches the harmful content. Default datasets include slur terms and web/HTML/JS content.

CLI example:

pyrit_scan garak.encoding --target openai_chat --techniques base64 --max-dataset-size 1

Available techniques (17 encodings): Base64, Base2048, Base16, Base32, ASCII85, Hex, QuotedPrintable, UUencode, ROT13, Braille, Atbash, MorseCode, NATO, Ecoji, Zalgo, LeetSpeak, AsciiSmuggler

Aggregate techniques: ALL (every encoding, exhaustive) and DEFAULT (a broad curated subset spanning every encoding family — base-N, byte-encodings, substitution ciphers, and symbolic alphabets — for a meaningful default scan; the niche/lossy schemes are ALL-only). DEFAULT is used when no techniques are specified.

Note: Technique composition is NOT supported for Encoding — each encoding is tested independently.

dataset_config = EncodingDatasetConfiguration(dataset_names=["garak_slur_terms_en"], max_dataset_size=1)

scenario = Encoding()
scenario.set_params_from_args(  # type: ignore
    args={
        "objective_target": objective_target,
        "scenario_techniques": [EncodingTechnique.Base64],
        "dataset_config": dataset_config,
    }
)
await scenario.initialize_async()  # type: ignore

print(f"Scenario: {scenario.name}")
print(f"Atomic attacks: {scenario.atomic_attack_count}")

scenario_result = await scenario.run_async()  # type: ignore

await output_scenario_async(scenario_result)

WebInjection¶

Ports Garak’s web_injection probe family. Tests whether the target can be coaxed into emitting web-injection payloads: markdown-image data exfiltration (CSRF-style) and cross-site-scripting (XSS). Each technique pairs a SeedObjective (the goal) with a SeedPrompt (the literal injection prompt) and scores the response with the markdown-injection or XSS output scorer.

CLI example:

pyrit_scan garak.web_injection --target openai_chat --techniques xss --max-dataset-size 1

Available techniques (8 probes): MarkdownImageExfil, ColabAIDataLeakage, StringAssemblyDataExfil, PlaygroundMarkdownExfil, MarkdownURIImageExfilExtended, MarkdownURINonImageExfilExtended, TaskXSS, MarkdownXSS.

Aggregate techniques: ALL (all 8), DEFAULT (excludes the two combinatorial extended probes), EXFIL (the 6 markdown-exfil probes), and XSS (TaskXSS + MarkdownXSS).

Doctor¶

Ports Garak’s doctor probe, which implements the HiddenLayer Policy Puppetry universal bypass: a post-instruction-hierarchy prompt injection that frames a harmful request as a fictional medical-drama script (“Dr House”) to override safety guardrails. Each objective from the garak_doctor dataset is wrapped in a Policy Puppetry template via PolicyPuppetryConverter and the response is scored for non-refusal (mirroring Garak’s mitigation.MitigationBypass detector).

CLI example:

pyrit_scan garak.doctor --target openai_chat --techniques policy_puppetry --max-dataset-size 1

Available techniques (2 probes): PolicyPuppetry (wraps the objective in the Dr House template) and PolicyPuppetryLeet (the same template, additionally leetspeak-encoded). Both are tagged default, so DEFAULT and ALL currently coincide.

For more details, see the Scenarios Programming Guide and Configuration.