Loading Built-in Datasets

PyRIT includes many built-in datasets to help you get started with AI red teaming. While PyRIT aims to be unopinionated about what constitutes harmful content, it provides easy mechanisms to use datasets—whether built-in, community-contributed, or your own custom datasets.

Important Note: Datasets are best managed through PyRIT memory, where data is normalized and can be queried efficiently. However, this guide demonstrates how to load datasets directly as a starting point, and these can easily be imported into the database later.

The following command lists all built-in datasets available in PyRIT. Some datasets are stored locally, while others are fetched remotely from sources like HuggingFace.

Many of these datasets come from published research, including 0DIN Mozilla 0DIN, 2024, Aegis Ghosh et al., 2025, Agent Threat Rules Lin & ATR Community, 2026, ALERT Tedeschi et al., 2024, Anthropic Model-Written Evals Perez et al., 2023, BeaverTails Ji et al., 2023, CBT-Bench Zhang et al., 2024, CategoricalHarmfulQA (CatQA) Bhardwaj et al., 2024, CoCoNot Brahman et al., 2024, CyberSecEval Bhatt et al., 2023, DarkBench Kran et al., 2025, DecodingTrust Wang et al., 2023, Do Anything Now Shen et al., 2024, Do-Not-Answer Wang et al., 2023, EquityMedQA Pfohl et al., 2024, FigStep Gong et al., 2025, FORTRESS Knight et al., 2025, HarmBench Mazeika et al., 2024, HarmEval Banerjee et al., 2025, HarmfulQA Bhardwaj & Poria, 2023, JailbreakBench Chao et al., 2024, JailbreakV-28K Luo et al., 2024, LLM-LAT Sheshadri et al., 2024, MedSafetyBench Han et al., 2024, MM-SafetyBench Liu et al., 2024, Moral Integrity Corpus Ziems et al., 2022, MOSSBench Li et al., 2024, Multilingual Alignment Prism Aakanksha et al., 2024, Multilingual Vulnerabilities Tang et al., 2025, OR-Bench Cui et al., 2024, PKU-SafeRLHF Ji et al., 2024, SALAD-Bench Li et al., 2024, SemGuard Abughallous & Abufakher, 2026, SimpleSafetyTests Vidgen et al., 2023, SIUO Wang et al., 2025, SORRY-Bench Xie et al., 2024, SOSBench Jiang et al., 2025, StrongREJECT Souly et al., 2024, TDC23 Mazeika et al., 2023, ToxicChat Lin et al., 2023, VLSU Palaskar et al., 2025, VLGuard Zong et al., 2024, WildGuard Han et al., 2024, XL-SafetyBench Choi et al., 2026, XSTest Röttger et al., 2023, AILuminate Ghosh et al., 2025, Transphobia Awareness Scheuerman et al., 2025, Red Team Social Bias Taylor, 2024, and PromptIntel Roccia, 2024. Some datasets also originate from tools like garak Derczynski et al., 2024 and AdvBench Zou et al., 2023. The garak family includes per-language package-hallucination registries (garak_pypi_packages, garak_npm_packages, garak_crates_packages, garak_rubygems_packages, garak_dart_packages, garak_perl_packages, garak_raku_packages), system-prompt libraries (garak_drh_system_prompts, garak_tm_system_prompts), exploitation echo corpora (garak_exploitation_sql_injection, garak_exploitation_python_code_execution), PromptInject context and technique templates (prompt_inject_contexts, prompt_inject_techniques), API-key probe corpora (garak_api_key_services, garak_api_key_templates, garak_api_key_partial_keys, garak_api_key_safe_placeholders), the API-key service-to-pattern map (garak_api_key_service_patterns), ProPILE PII records and prompt templates (garak_propile_pii, garak_propile_templates), an audio jailbreak set (garak_audio_achilles_heel), and visual jailbreak sets (figstep, figstep_pro).

Loading Specific Datasets¶

You can retrieve all built-in datasets using SeedDatasetProvider.fetch_datasets_async(), or fetch specific ones by providing dataset names. This returns a list of SeedDataset objects containing the seeds.

Adding Datasets to Memory¶

While loading datasets directly is useful for quick exploration, storing them in PyRIT memory provides significant advantages for managing and querying your test data. Memory allows you to:

• Query seeds by harm category, data type, or custom metadata

• Track provenance and versions

• Share datasets across team members (when using Azure SQL)

• Avoid duplicate entries

The following example demonstrates adding datasets to memory. For comprehensive details on memory capabilities, see the memory documentation and seed database guide.

CyberSecEval: Separate Tactics from Harms¶

CyberSecEval’s ten MITRE categories describe cyberattack tactics, not ten distinct harm categories. The loader preserves them in metadata["mitre_category"] alongside the upstream technique ID/name. All prompts receive COORDINATION_HARM. C2, Evasion, Execution, Lateral Movement, Persistence and Privilege Escalation also receive MALWARE; Recon and Discovery are not automatically malware. Collection and Exfil receive additional malware/privacy labels only for curated, exact prompt matches. The pinned source has 39 PPI rows and 13 PROPRIETARY_INFO rows, with ten overlapping. Generic file transfer or collection does not by itself imply personal or confidential data. These are capability labels, not ground-truth judgments that every dual-use response is harmful.

Load the complete dataset into memory first. The following example independently samples three prompts from each category, rather than sampling thirty globally and accidentally omitting a category. Keep selection in the calling workflow, not in the dataset loader.

Keep these category groups when executing and reporting. In a custom scenario, construct one AtomicAttack per category with display_group set to that category; the existing scenario reports then show separate results for Recon, Exfil, Persistence, and the other tactics. Supply authored SeedObjective goals when adapting these literal SeedPrompt messages to an AttackSeedGroup. Do not blindly promote metadata["base_prompt"] to an objective: some upstream rows contain generation artifacts such as #ERROR!. This loader does not reproduce CyberSecEval’s expansion/judge protocol or decide which responses are harmful.