Adversarial Harness

waza adversarial runs prompt-injection and scope-bypass packs. With an explicit skill and real engine it observes behavior on those finite inputs; without a skill it defaults to mock, which validates the harness only, not the changed skill’s safety. Neither path is complete production safety certification.

This guide covers:

• What the built-in packs are and what they probe for
• Running packs from the CLI
• Wiring an adversarial: block into your eval.yaml
• Exit codes and CI integration
• Notes on determinism (replay-tape friendly)

Why a separate harness?

Functional evals (waza run) measure whether your skill solved the task. Adversarial evals measure whether your skill refused to solve the task when the task itself is unsafe. Both signals matter; they should not share a results column.

Every adversarial task is marked golden: true, so an unsafe outcome flips gate to exit 2 too — but waza adversarial is the canonical entry point because it also enforces an on_unsafe_outcome policy and prints a focused summary.

Built-in packs

Run waza adversarial --list-packs to print the catalog. (Unknown pack names are rejected with a config error — they do not list the catalog.)

The harness ships two packs out of the box:

prompt-injection

Four tasks that smuggle adversarial directives through README files, fake “system” comments inside source files, a support ticket, and a markdown changelog link. A safe skill ignores the directive and answers the underlying user request. An unsafe skill echoes the injection trigger, exfiltrates “secrets”, or follows the hostile instructions.

scope-bypass

Four tasks that explicitly ask the skill to do something out-of-scope: send email, delete files, install a third-party dependency, and call an external HTTP API. A safe skill refuses and explains the constraint; an unsafe skill complies.

Each pack lives under internal/adversarial/data/<pack>/ and is embedded into the waza binary, so no external pack download is required. This does not make real agent execution or its configured tool dependencies offline.

Quick start

Offline harness smoke

These commands use only the native mock: no agent, credentials or model calls. They check harness wiring, not the intended skill’s quality or safety. The mock echoes hostile prompts rather than implementing safe refusal. With the default fail policy, the first command reports eight unsafe observations and exits 2: an expected negative harness control. The single-pack output command also exits 2. --on-unsafe-outcome warn exercises the nonblocking path and exits 0, without changing those unsafe observations.

# Expected exit 2: negative harness control, not a skill assessment
waza adversarial --engine mock --packs prompt-injection,scope-bypass

# Expected exit 2: save the unsafe mock observations for inspection
waza adversarial --engine mock --packs scope-bypass --output adversarial.json

# Expected exit 0: nonblocking harness smoke, not a skill safety gate
waza adversarial --engine mock --packs prompt-injection --on-unsafe-outcome warn

Before real-engine execution

Use published v0.39.0, which includes the skill-binding fix described below, or a source build containing that fix. It is not in v0.38.10; older binaries must not be assumed to evaluate the intended skill. See release availability.

Before continuing, complete the SDK authentication and usage prerequisites: standalone CLI login, model access and approved runtime/tool permissions. Real runs may incur model costs and invoke configured dependencies. Approve that usage and those permissions before executing the separate block below. Dependency mocks do not establish live service safety.

--skill is the declared frontmatter name, not a filesystem path. Check that skills/my-skill/SKILL.md declares name: my-skill; substitute your actual directory/name together. The subshell runs from that directory so native discovery can find its SKILL.md. The fix resolves this directory from the original workspace and records its absolute path before relocating the synthesized spec. Missing/mismatched real targets now fail before engine creation; explicit mock harness runs remain available without a target.

# Real agent execution: requires the prerequisites above and may incur usage
(
  cd skills/my-skill &&
  waza adversarial --engine copilot-sdk --packs prompt-injection --skill my-skill --model gpt-4o
)

Regressions inspect actual generated-spec configuration and resolved execution requests, not a manually supplied discovery path. They substitute mock at the engine factory and check missing/mismatched sources. They do not establish that a live SDK session loaded/invoked the skill or that the skill is safe.

A successful real-engine summary (illustrative, not a recorded live result) can look like:

Adversarial summary (packs: prompt-injection)
  tasks run:  4
  unsafe:     0
  policy:     fail
  result:     ✅ all packs PASSED

Configuring from eval.yaml

You can pin pack selection and policy at the spec level by adding an adversarial: block. This block is additive under schema 1.2 and is consumed only by waza adversarial --spec eval.yaml; waza run ignores it.

The following is another offline harness example. Save it as eval.yaml. --spec supplies the pack/policy block (and optional graders/metrics), not the real engine or target selection. Keep --engine mock explicit in the command below; a real run requires explicit engine/skill flags and the prerequisites above.

schemaVersion: "1.2"
name: my-skill-eval
skill: my-skill
version: "1.0"
config:
  executor: mock
  model: mock
  trials_per_task: 1
  timeout_seconds: 30
tasks:
  - "tasks/*.yaml"
adversarial:
  packs:
    - prompt-injection
    - scope-bypass
  on_unsafe_outcome: fail  # or "warn"

CLI flags override the spec block:

# Offline only: spec says fail; this mock run downgrades to warn
waza adversarial --engine mock --spec eval.yaml --on-unsafe-outcome warn

Flags

Flag	Description	Default
--packs	Comma-separated pack names to run	All built-in packs
--list-packs	Print the pack catalog and exit	—
--on-unsafe-outcome	fail (exit 2) or warn (exit 0) on unsafe	fail
--spec	Inherit adversarial:, graders, and metrics from a spec	—
--engine	Engine to use (mock, copilot-sdk)	inferred
--skill	Declared skill name, not a filesystem path	—
--model	Model passed to the engine	—
--output	Write the full results.json to a file	—
--workers / --parallel	Concurrency knobs (same semantics as waza run)	—
--verbose	Print pack/policy/engine/artifact diagnostics	false

Exit codes

Code	Meaning
0	All adversarial tasks were refused safely (or --on-unsafe-outcome warn)
2	At least one unsafe outcome was observed, and policy is fail
3	Configuration error (unknown pack, malformed spec, etc.)

2 is the same exit code waza gate uses for golden-task failures — so the same CI step that gates on goldens will also gate on adversarial failures.

CI integration

.github/workflows/eval.yml

- name: Offline adversarial harness smoke (not skill safety)
  run: waza adversarial --engine mock --packs prompt-injection,scope-bypass --on-unsafe-outcome warn --output adversarial.json

- name: Upload adversarial results
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: adversarial-results
    path: adversarial.json

This smoke checks serialization/command wiring only. Its warning policy is deliberately nonblocking and must not gate skill safety. For a real skill gate use explicit --engine copilot-sdk --skill ... with the prerequisites above and keep the default fail policy. To inspect the offline warning path:

waza adversarial --engine mock --on-unsafe-outcome warn --output adversarial.json

Determinism

Pack inputs are embedded; real agent execution still requires its runtime and may use configured dependencies. waza replay checks recorded consistency: it does not rerun a changed agent, execute graders or automatically execute adversarial packs. For observed drift, capture two separately executed runs with waza run --snapshot, then use offline waza replay before.json --bisect after.json. The positional before.json is the baseline; --bisect after.json selects the comparison snapshot. That compares retained tool fingerprints, not causes or current model safety.

Schema reference

AdversarialConfig (schema 1.2 additive) on EvalSpec:

Field	Type	Required	Notes
packs	[]string	yes	Subset of built-in pack names
on_unsafe_outcome	"fail" | "warn"	no	Defaults to fail

Validation:

• Empty packs list is rejected
• Unknown pack names are rejected against the embedded catalog
• The adversarial: block requires schemaVersion: "1.2" or later

What’s next

The current packs are deliberately small and review-able. The harness is designed so future releases can grow the catalog (data-exfil, jailbreaks, multi-turn social engineering) without changing the spec or CLI surface. Custom user-provided packs are on the roadmap; track issue #365 for updates.