True/False Scorers

A true_false scorer answers a yes/no question about a response and returns a boolean (score.get_value() is a bool). They are the natural choice for attack success criteria, refusal detection, and policy checks.

This page covers leaf true/false scorers, organized fast → slow. Wrapping and combining them (composite, inverter, threshold, conversation) is on Combining & stacking scorers.

ManualScorer records a human-supplied true/false verdict for a persisted message piece. The PyRIT app uses it for attack-result adjudication; it does not evaluate content.

Fast scorers (no LLM)¶

These run locally and deterministically — no model call, no credentials. Use them in CI and to score large response sets cheaply.

RegexScorer¶

RegexScorer returns True if any named pattern matches. Subclass it to ship a domain-specific detector; PyRIT includes keyword scorers built this way (MethKeywordScorer, FentanylKeywordScorer, NerveAgentKeywordScorer, AnthraxKeywordScorer) and CredentialLeakScorer for leaked secrets.

AgentThreatRulesScorer¶

AgentThreatRulesScorer loads a pinned regex digest from the Agent Threat Rules (ATR) project without adding a dependency. It detects patterns in message evidence, not tool execution or attack success. Use it as a fast pre-filter, not a calibrated detector. ATR’s published precision figures do not establish precision on your response set.

fields selects which rules to load. It does not change where evidence comes from. By default, the digest selects agent_output and content. The scorer routes fields as follows:

Argument strings are scanned as supplied. Valid JSON objects are also scanned as compact, sorted-key JSON with decoded Unicode and slash escapes. Non-JSON strings remain readable. Scores retain the digest source URL, ref, hash, and available upstream revision and version in score_metadata; the source ref does not change the evaluation identity for the same digest.

Limits:

• A match in a requested call is not proof that the tool ran.

• The scorer reads only the supplied message. It does not read earlier turns or traces.

• A selected field that it cannot read gives an undetermined score, unless another field matches.

• Fields with no message source, such as tool_description and trace fields, are rejected.

The example uses the pinned ruleset, which is downloaded on first use and then cached. Use ref="main", cache=False to download the latest rules at construction. Select tool fields with fields=["tool_name", "tool_args", "tool_response"] and supply message evidence to score tool calls or results.

OWASP LLM02 output scorers¶

A family of RegexScorer subclasses flags insecure output a model might emit (OWASP LLM02 — Insecure Output Handling):

• XSSOutputScorer — <script>, onerror=, javascript: URIs, SVG-embedded script.

• SQLInjectionOutputScorer — UNION SELECT, ;DROP TABLE, ';--.

• ShellCommandOutputScorer — curl ... | sh, rm -rf /, reverse shells.

• PathTraversalOutputScorer — ../../etc/passwd and similar walks to sensitive files.

• SSRFOutputScorer — 169.254.169.254 metadata, http://localhost/RFC1918 targets, gopher:// schemes.

• SSTIOutputScorer — {{7*7}}/${7*7} eval probes, __class__/__globals__ gadget chains.

• XXEOutputScorer — <!ENTITY ... SYSTEM> external entities, <!DOCTYPE ...[<!ENTITY>]> subsets.

• OpenRedirectOutputScorer — redirect=//evil, %2f%2f bypasses, https://trusted@evil userinfo confusion.

• LDAPInjectionOutputScorer — *)(uid=*) filter breaks, )(objectClass=*) clauses, )|( operator injection.

• AnsiEscapeOutputScorer — raw ESC [ (CSI) and ESC ] (OSC) terminal control sequences, plus the C1 U+009B/U+009D introducers.

• EscapedAnsiOutputScorer — escaped forms such as \x1b[, \033], \u001b[, \e[, \x9b that turn live once unescaped.

Like CredentialLeakScorer, each ships a default patterns set; pass your own patterns dict to replace it entirely.

MarkdownInjectionScorer¶

Detects markdown constructs (images, links) that can be abused for data exfiltration or rendering attacks.

PackageHallucinationScorer¶

Flags model-generated code that imports packages which do not exist in a language’s registry — an attacker can “squat” a hallucinated name so the code silently pulls in a malicious dependency (ported from garak’s packagehallucination probe). It lives beside the RegexScorer family but is not a subclass: rather than “does a bad pattern match?”, it extracts imported package names and flags any that are absent from a known-good reference set you inject via known_packages (for Python, the standard library is added automatically). Because it inspects generated code, it only scores assistant messages.

SubStringScorer is the simplest fast scorer of all — see the overview for an example.

StaticPromptInjectionScorer¶

StaticPromptInjectionScorer is a RegexScorer subclass that flags prompt-injection attempts locally (OWASP LLM01) — instruction override, system-prompt extraction, jailbreak role-play, and encoding-based evasion. It favors recall over precision, so use it as a cheap pre-filter ahead of a model-based scorer such as PromptShieldScorer.

DecodingScorer¶

DecodingScorer checks whether the request text (its original_value, converted_value, or decoded metadata) appears in the response — the fast, deterministic way to tell whether a target decoded an encoded prompt. It backs the Garak encoding scenario.

GarakExploitationScorer¶

GarakExploitationScorer applies Garak’s Jinja-expression or SQL-injection detection rules to emitted text. By default, it loads the matching packaged payload corpus; pass payloads to use a different reference set. A positive result means the model emitted exploit material, not that a template engine or database executed it. It backs the Garak exploitation scenario.

DivergenceScorer¶

DivergenceScorer detects meaningful continuation following the repetition named by an expectation (adapted from Garak’s repetition-induced divergence checks). It backs the Garak divergence scenario.

Slow scorers (LLM self-ask)¶

SelfAsk* scorers ask a chat target to reason about a response. They are flexible and handle nuance, at the cost of one model call per score. All of them take a chat_target.

SelfAskTrueFalseScorer¶

The general-purpose self-ask scorer. With no template it judges whether the objective was met; pass a built-in TrueFalseQuestionPaths template for specific questions such as detecting a successful prompt injection.

SelfAskRefusalScorer¶

Purpose-built to detect refusals. It returns True for a refusal and short-circuits to True (no LLM call) when the target fully blocked the response (response_error="blocked"), which is common with content filters. A partially blocked response still carries content, so it is scored normally by the LLM rather than short-circuited.

SelfAskCategoryScorer¶

Classifies a response into one of a set of categories (or none). The score is True when the response matches a harmful category; score.score_category holds the matched label.

Other self-ask true/false scorers¶

• SelfAskQuestionAnswerScorer — checks whether a response correctly answers a known question (used with question-answering datasets). QuestionAnswerScorer is the fast, non-LLM variant that matches against the expected answer directly. Both require an AnswerMatches condition. Use objective for question context, and a separate SelfAskTrueFalseScorer for MatchesObjective checks. Configure that objective scorer with validator=ScorerPromptValidator(is_objective_required=True).

• SelfAskGeneralTrueFalseScorer — bring your own system prompt and JSON schema when the built-in templates don’t fit. See Combining & stacking scorers for how custom scorers slot in.

External classifier integrations¶

Five true/false scorers wrap hosted services rather than reasoning with a generative LLM:

• PromptShieldScorer — wraps PromptShieldTarget (Azure Prompt Shield jailbreak classifier); returns True if an attack is detected in the prompt or any document.

• GandalfScorer — checks whether a Gandalf challenge password was revealed.

• LlamaGuardScorer — sends text to a PromptTarget serving Llama Guard and returns True for unsafe content, with violated policy categories in the score metadata. Its bundled defaults follow the Meta Llama Guard 3 8B S1-S14 contract.

• ShieldGemmaScorer — sends text to a PromptTarget serving ShieldGemma and returns True when the content violates the one guideline the scorer is bound to. ShieldGemma Zeng et al., 2024 judges a single principle per request, so compose several with TrueFalseCompositeScorer to cover a whole policy. Prompt classification judges a user turn, while the default response classification judges a model turn on its own so prompt content cannot bias the verdict.

• WildGuardScorer — sends a prompt and response pair to a PromptTarget serving WildGuard, which judges in one call whether the request is harmful, whether the response is a refusal, and whether the response is harmful. WildGuardLabel selects which judgement becomes the boolean; the other two are kept in the score metadata, so reading them costs no extra request. The prompt is read from the latest earlier user turn of the scored conversation. Only assistant turns are scored by default. For response-side labels, blank text pieces are skipped when other supported pieces have content; an entirely blank response raises an error. HARMFUL_REQUEST also accepts an empty response.

WildGuard’s bundled prompt includes the full AI2 completion wrapper. Serve allenai/wildguard through an OpenAI-compatible completions endpoint, then configure:

from pyrit.prompt_target import OpenAICompletionTarget
from pyrit.score import WildGuardScorer

target = OpenAICompletionTarget(
    model_name="allenai/wildguard",
    endpoint="http://localhost:8000/v1",  # Your WildGuard completion server
    api_key="your-server-key",  # Use the authentication required by your server
    max_tokens=128,
    temperature=0,
)
scorer = WildGuardScorer(chat_target=target, user_prompt="The original user request")
scores = await scorer.score_text_async("The model response")

The checkpoint does not supply a tokenizer chat template, so HuggingFaceChatTarget(model_id="allenai/wildguard") is not a drop-in alternative. Do not apply a second chat wrapper to the bundled prompt. If using a chat server that supplies its own formatting, pass a matching prompt_template explicitly.

All five need their respective endpoints/credentials even though they are not “self-ask”.

Local model scorers¶

LocalRefusalClassifierScorer¶

LocalRefusalClassifierScorer is an experimental local refusal classifier. It uses Laya, an Apache 2.0 encoder, to form question-conditioned representations and applies a logistic head trained on PyRIT’s refusal rows. Install the runtime with pip install laya. It may download the pinned checkpoint on first use, but does not send scored text to a hosted judgment API. Call await scorer.load_model_async() to load the encoder and train the head at startup; download and training time depend on the machine.

Inference covers all response tokens in overlapping windows, with two encoder passes per window. max_input_tokens defaults to 512 including Laya and JSON framing, chunk_overlap_tokens to 64, and max_objective_tokens to 128 serialized objective tokens. Shortened objective context is reported in score_metadata["objective_truncated"]; response windows retain all serialized response tokens after Laya’s mask-token sanitization. Overlap does not preserve all long-range context.

A completed verdict requires all chunks to agree. Conflicting verdicts or any chunk inside abstain_band (default (0.2, 0.8)) return UNDETERMINED; the caller decides whether to use an LLM judge. abstain_band=None disables probability-based abstention, but disagreement still returns UNDETERMINED. Metadata records the chunk count, minimum and maximum chunk probabilities, and aggregation="unanimous". These probabilities are not calibrated whole-response confidence. Fully blocked responses and SDK-provided structured refusals return True without model inference; readable partial output is scored normally.

Training uses the same tokenization, framing, and token budgets as inference, without character cutoffs. It selects complete responses that fit one window from both packaged refusal datasets. Whole-response labels are not assigned to individual chunks: multi-window training rows are excluded, and their count is logged. Fitting fails if fewer than two examples or either label class remains. Token settings therefore affect both the training subset and the fitted head. Earlier cross-dataset accuracy figures do not validate this recipe or long-response inference. No-objective and non-English use are also unvalidated. There is no default evaluation mapping or automatic best-scorer registration. Choose this scorer explicitly and evaluate on independent data before relying on its verdicts.

from pyrit.models import ContentScorable, ScoringExpectation
from pyrit.score import LocalRefusalClassifierScorer

scorer = LocalRefusalClassifierScorer()
scores = await scorer.score_async(
    scorable=ContentScorable(value="I'm sorry, I can't help with that."),
    expectation=ScoringExpectation(objective="The original request"),
)
score = scores[0]
print("Needs another judge" if score.is_undetermined else score.get_value())

Multimodal scorers¶

Audio and video responses are scored by transcribing or sampling them and delegating to a text/image true/false scorer:

• AudioTrueFalseScorer — transcribes an audio_path response (Azure Speech-to-Text) and scores the transcript with a wrapped TrueFalseScorer.

• VideoTrueFalseScorer — extracts frames from a video_path response and scores them with a wrapped image TrueFalseScorer (True if any frame matches); an optional audio scorer is AND-combined so both the visuals and the transcript must match.