Snapshot & Replay

Snapshots preserve recorded observations, not a reconstructible workspace or proof of external state. Snapshot capture is local and opt-in; it adds no automatic export, upload, or fetching of paths named in metadata. Existing explicitly configured result storage still uploads results, including evidence metadata, but does not fetch or upload the separate snapshot files. See storage and portable evidence.

Available in v0.39.0: selected-file capture/regrade and ordinary evidence producers are included in the published standalone release. Capture remains opt-in, and selected-file private publication retains its Linux/macOS limits. See availability and platform limits. Replay never reruns a changed agent or the graders.

Operation capabilities

Operation	Inputs and checks	What it does not establish
Inspect evidence	Result metadata for every task trial; assessed manifest and full origin	Enforcement, complete execution history, external state, or preserved source bytes
Offline replay	Explicit snapshot; event sequence and stored score/pass consistency	Grader reruns, workspace reconstruction, or agent reproducibility
Bisect	Two explicit snapshots; ordered tool-name/argument fingerprints	A causal explanation for divergence
Regrade with workspace	Results and actual agent workspace via `--workspace`	That fixture hashes supply the workspace; prompt grading may call a paid judge
Regrade selected captured files	Explicit `--evidence-snapshot`, task, matching result origin, verified manifest and file content	Arbitrary graders, absent files, external receipts, or a complete workspace
Live replay	Unsupported; `--mode live` returns exit code `2`	No live replay behavior or delivery date is promised

Capture a mock run

From the repository root, this example uses the existing mock executor and makes no model calls. It records observations without selecting workspace contents:

waza run examples/code-explainer/eval.yaml \
  --context-dir examples/code-explainer/fixtures \
  --snapshot ./evidence-snapshots \
  --output ./evidence-results.json

Read and review the exact snapshot_path recorded in each task’s runs[] in evidence-results.json. With jq installed, select the first recorded path:

snapshot=$(jq -r '.tasks[0].runs[0].snapshot_path' ./evidence-results.json)
waza replay "$snapshot" --json
waza serve --results-dir .

The path is selected explicitly by the user, not inferred or fetched by a consumer. Ordinary snapshots retain legacy names/defaults. Selected-file capture uses randomized, non-overwriting filenames, private 0700 output directories, and 0600 files.

Immediate capture-to-write keeps this private route whenever any files were requested, even if every requested file is missing or rejected. This internal request marker is not serialized. Loaded or manually constructed zero-file snapshots retain historical metadata-only writer routing; existing nonempty file snapshots still use private publication. On unsupported platforms, selected publication returns an explicit error instead of falling back to ordinary output.

Frozen prompt judging is separate

Final prompt graders can opt into evidence-backed independent judging. The bounded runs[].judge_evidence bundle persists directly in results, without requiring --snapshot. A separately requested snapshot 1.2 references its already frozen digest as judgeEvidenceSHA256; post-grader snapshot workspace observations are not replacements for the files actually supplied to a judge. Ordinary snapshots keep version 1.1.

Evidence-only waza grade verifies that result bundle and makes new judge-model calls without the actor workspace. It does not follow snapshot paths or resume the actor. replay remains an offline consistency check, and the explicit file-only --evidence-snapshot operation does not perform model judging.

What’s recorded

Snapshot schema 1.1 carries a separately versioned evidence manifest 1.0. Missing-version defaults for legacy eval/results readers remain 1.4, while newly generated results use result schema 1.7. Task YAML has no independent schema-version field or default; legacy snapshots and results still load. Raw manifest JSON must satisfy the full embedded evidence schema, including required members and their types, before decoded metadata can be assessed. Missing fields or invalid null scalars are withheld without repairing zero values; schema-declared nullable fields remain valid. Evidence-aware consumers also reject unsupported manifest versions, unknown metadata, and duplicate JSON keys rather than interpreting them as assessed evidence.

Metadata-only API, interpretation reports and dashboard views do not fetch snapshot bytes. Requirement references in these views identify captured artifact metadata, not verified values inside it; artifact-relative value pointers are withheld and explicitly labeled unverified. Explicit snapshot-content verification checks digests and pointer existence when actual artifact bytes are supplied. Interpretation labels retain each checkpoint’s declared after_turn; observations from the same grader at different turns are not collapsed into the same label. --interpret lists captured-artifact IDs and source origins as metadata only, never verified value pointers. Present zero-valued source run/attempt fields denote unavailable attribution and are labeled unavailable, not reconstructed from the selected result. Known nonzero lineage mismatches remain withheld; omitted/null required fields remain invalid. Historical requirements with omitted, null, or empty check lists remain visible as unassessed in interpretation alongside populated sibling observations. Finite authored inputs use a separately selected manifest schema 1.1, not the native execution manifest 1.0. Both embedded schemas reject malformed required members without resealing or dropping their raw input; authored inputs cannot be bound to executed or cached runs.

Each manifest describes:

Full source origin: eval_id, task_id, run_number, attempt_count, and prior_attempts. References are observational and scoped to that entire origin. Earlier attempts are not inferred from the final attempt.
Requested engine/model, actual mock/live/unknown execution mode derived from the concrete MockEngine or CopilotEngine, requested tool policy, and verified enforcement. Dependency mode and verified enforcement remain unknown unless recorded.
waza, SDK, effective model, Go, and platform versions. Empty/unknown strings are unavailable, not inferred versions. Only SDK and effective-model versions may be null; waza, Go and platform metadata remain strings. Grader implementation, external rubric version, and lockfile provenance are explicitly unavailable.
runtime.no_skills (boolean|null) from the actual execution request and native_skill_control: unknown, sdk_default, requested_sdk_disable, or mock_not_applicable. This describes requested native skill control, not proof that the SDK enforced it. An empty directory is not a disablement proxy.
Prompt/follow-ups, ordered tool events, command invocations, grader outcomes, checkpoints, final output, and allowed environment values where recorded. The environment artifact fingerprints captured allow-list/configuration at /env; engine-config fingerprints captured configuration at /engine. Neither claims to capture the full host environment. Task/grader configuration, fixture, and instruction digests identify source material but do not preserve the source bytes.
Redaction rules that actually matched, match counts (including selected files), limitations, and fixed safe operational diagnostics.

Availability and completeness

Availability	Meaning
`captured`	Actual content digest plus a snapshot document/pointer; verify content for the requested operation
`digest_only`	Separate source digest, no preserved content and no content locator
`unavailable`	Reason recorded; no locator or digest that pretends to be content
`not_requested`	Capture was not requested for this artifact; reason, no locator

Most tapes have unknown completeness; an operational run error makes them partial. A selected workspace file can be individually complete, while the overall workspace remains partial or unavailable. External state is unavailable: a successful tool event is not an authoritative receipt.

Cached trials retain their source eval/task/run/attempt origin and are not new executions. A different eval ID is legitimate for cached source evidence; task, run, and attempt must still match. Every trial is exposed independently in the dashboard, including unassessed trials.

Capture and publication failures retain safely validated source eval/task identity without claiming successful content preservation. Missing or rejected identity produces an explicit origin_unavailable diagnostic; numeric lineage alone does not establish attribution. This leaves historical binding behavior and original execution result fields unchanged.

For legacy API compatibility, evidenceRuns is omitted when all task runs lack both evidence and requirement explanations; the dashboard’s fallback reports unassessed. Once any run has metadata, every trial is returned, including rows with missing historical metadata.

Historical absence is unassessed, not successful capture. Selected-file failures have fixed safe diagnostics and missing-file states. Capture or publication failure produces a fixed unavailable envelope and logs; rejected origin metadata is not copied into that envelope. Invalid metadata is withheld from the dashboard.

Select workspace files

Only explicitly reviewed agent-visible UTF-8 regular files can be selected:

waza run eval.yaml --snapshot ./evidence-snapshots \
  --snapshot-file report.txt \
  --snapshot-evaluator-only private-rubric.txt \
  --snapshot-max-file-bytes 1048576 \
  --snapshot-max-total-bytes 10485760 \
  --output ./evidence-results.json

--snapshot-file and --snapshot-evaluator-only are repeatable and require --snapshot. Exclusions additionally require --snapshot-file so workspace capture actually runs. Exclusions identify evaluator-only files; credentials and evaluator-only files must never be selected.
Explicit allow-listing is not authorization proof. The user must review agent visibility and exclude evaluator-only material; no automatic discovery grants capture permission.
Both byte limits are mandatory positive values when configured; defaults are 1 MiB per file and 10 MiB total.
Safe descriptor-based file reads are supported on Linux and macOS only; selected-file capture is unsupported elsewhere. Unsafe paths, nonregular files, invalid UTF-8, and over-limit files are not silently accepted.
Captured text has its own verified content digest. Source configuration and instruction/fixture digests cannot substitute for that text.

Filesystem admission and observation limits

On supported Linux/macOS hosts, evaluator-only exclusions are admitted before any selected file content is read. The metadata-only inventory is limited to 256 entry inspections and depth 32, including excluded directories and their descendants. Repeated/overlapping exclusions consume the same aggregate budget. This selected-content exclusion inventory is not the existing recursive fixture-digest inventory; it does not change fixture hashing. It does not read file contents, follow symlinks or traverse special entries. An inventory limit, inspection failure or descriptor/resource failure prevents selected-content capture; waza reports unavailable/partial evidence rather than pruning the inventory or raising process limits. The configured limit does not guarantee that the operating system has that many free descriptors.

Descriptor identities reject aliases of admitted excluded files/directories, including hard links to inspected descendants. Unrelated hard-linked public files remain eligible, and a missing exclusion does not reject neighboring files. Source capture roots and descendants use no-follow component admission. On macOS only, the literal system /var and /tmp aliases are accepted after checking their expected /private targets; arbitrary ancestor aliases are rejected.

Selected-state snapshot destination publication on Linux/macOS separately admits/creates each component without following arbitrary ancestor symlinks. Missing directories are created 0700; existing ancestors need not be private, but the final destination must be 0700. Non-creating identity checks precede evidence writes and surround publication; they never recreate a missing destination. A later failure may leave newly created empty directories. The existing cleaned destination spelling and ordinary metadata-only snapshot writer behavior are unchanged.

These checks observe admitted identities, not an immutable directory inventory: concurrent same-user renames, changes to excluded membership or later edits are not certified. Capture checks identity, size, modification time and change time before/after a bounded read, which is not an atomic or authentic snapshot. Materialization checks expected bytes and observed metadata immediately before publication, not continued integrity after the workspace is returned.

A late close error after verified private publication can leave a residual snapshot path. The result retains that path with unavailable capture evidence and an explicit failure diagnostic; it is not recorded as successful capture.

If private-file cleanup fails, waza can retain at most one independently verified complete residual location, preferring an eligible temporary name before the final name. The path may identify a hidden .json.tmp file; it is a failed-capture diagnostic, not a newly published snapshot or successful export. Additional possible residue is reported explicitly, but a single path cannot enumerate it. Partial/changed bytes and unverified moved-root locations are not represented as complete snapshots. Name, inode, byte and metadata checks are bounded observations, not a cleanup or moving-namespace guarantee.

Environment and redaction

Environment capture is default-deny. --snapshot-env-allow is repeatable and accepts explicit names or *-suffixed prefixes. Allowed values still undergo redaction; avoid broad prefixes containing credentials. Secret environment-key removals contribute to the redaction match count, as do actual selected-file rule matches.

For new captures, each artifact’s redacted flag describes changes to its own materialized content, not the capture-wide match count. Validation changes also mark the enclosing result; unrelated metadata, environment or workspace changes do not mark unchanged siblings. Core JSON representation-only normalization and no-op replacements do not mark an artifact changed, while removal of a materialized field does. Historical artifacts keep their recorded flags.

Typed JSON context, tool arguments/results, grader feedback/details, and structurally sensitive keys are sanitized before serialization, without modifying original results. Opaque bytes, unsafe keys or identifiers, malformed JSON, duplicate keys, trailing values, excessive nesting, and unsupported values are rejected. Numeric lexical tokens are preserved without float coercion.

Custom --redact policy.yaml rules replace defaults unless extend: true:

extend: true
rules:
  - name: internal-id
    pattern: 'INT-[0-9]{6}'

Pattern redaction is not a confidentiality proof. Review content before sharing. A file’s redacted: true can also mean JSON normalization, not a secret match. Summary counts describe actual rule matches, not normalization-only changes. Changed/redacted files are not eligible as original file-grading inputs.

Offline replay and bisect

waza replay ./reviewed-snapshot.json --json
waza replay ./reviewed-snapshot.json --bisect ./other-reviewed-snapshot.json --json

Replay checks that tool-event sequence is 1..N and, with --strict (default), that recorded grader score/pass values are consistent. It does not rerun graders. Exit codes: 0 consistent, 1 divergent, 2 snapshot load/parse/I/O, JSON output, unsupported live mode, or unknown mode error. Bisect compares ordered tool-name/argument fingerprints, ignoring durations and raw output text. It identifies divergence, not its cause.

Regrade selected captured files

waza grade eval.yaml --results ./evidence-results.json --task report-task \
  --evidence-snapshot ./reviewed-snapshot.json --output ./regraded-results.json

This explicit Linux/macOS mode requires exactly one assessed matching run for the selected task. It verifies the original manifest SHA-256, full source origin, and complete, unredacted file contents, including each text digest. It never follows snapshot_path implicitly and cannot be combined with an explicit --workspace. Verification also binds the snapshot header to the full origin, checks that each declared workspace pointer names the actual file index, and rejects raw supplied workspace JSON with unknown/unhashed members or duplicate keys.

Only file graders with explicit required must_exist / content_patterns paths are supported. Only those required files are materialized in a private workspace. Wildcard checks, absence checks, file graders without explicit required checks, all non-file graders, and output-based expected.output_contains, expected.output_not_contains, and expected.output_contains_any checks are rejected before materialization. Checks cover only the materialized captured subset, not absence from the original workspace. A must_not_match pattern checks the complete verified text of its selected file. This is not general workspace reconstruction. Ordinary grade --workspace grading and this snapshot mode both validate the manifest digest and recorded eval/task/run/attempt attribution of every selected evidence-bearing run before grading. Cached results may retain their source eval ID, but never a foreign task, run or attempt. Historical runs without evidence remain supported. Duplicate task IDs in the results are rejected before grading, even if one duplicate has no runs or evidence; an overwritten row cannot bypass preflight. Ordinary prompt graders may contact a paid judge; evidence inspection adds no automatic calls.

The --snapshot-max-file-bytes and --snapshot-max-total-bytes limits govern capture, not admission of an independently supplied snapshot. Workspace verification/materialization checks identity and required content but does not reapply those capture budgets. Review supplied artifact sizes before selecting them; a valid digest is not a resource-limit or trust guarantee.

Regrading retains the original capture SHA-256 as source_manifest_sha256 but invalidates original validation, result, grader-configuration, and checkpoint links. Original checkpoints are never observations of newly declared checks.

Scoped requirement observations

Declared requirement checks resolve named graders only when the name is unique in the declared scope. Missing names and flattened-name collisions are unresolved. See the task authoring example for the requirements YAML shape and scope selection. passed / failed describe existing grader observations, not requirement enforcement, a new requirement score, or changed pass/fail defaults.

Prompt checks are qualitative; code/program operational status remains typed unavailable where it cannot be certified. File/diff inputs are uncertified unless verified for the requested operation. Overall operational errors remain distinct from grader outcomes. Categories and references do not infer agent intent, causal classification, or hypotheses.

Optional interpretation output

waza run ... --interpret prints a summary with per-trial scoped requirement check observations and evidence metadata when present. Existing no-metadata summary text remains unchanged. Malformed evidence and its assessment references are withheld, not treated as assessed observations. The API and report also withhold requirement observations when a declared task differs from the selected task or a reference fails manifest-origin, artifact-ID or pointer validation. Reference-free historical observations remain displayable. Interpretation does not add enforcement, causal diagnosis, or automatic model calls.

Snapshot compatibility

The snapshot wire format is independent 1.1; historical 1.0 snapshots remain readable. Eval and result formats retain their own independent versions.

MAJOR mismatch is rejected at load (exit 2).
Same-major minor additions remain readable; typed snapshot decoding ignores unknown fields rather than preserving them verbatim.

Portable hash contract

json-v1 content and manifest digests use sorted-map Go JSON canonicalization: UTF-8 object keys sorted via map encoding, exact numeric lexical tokens, and array order preserved. Go encoding/json escaping applies, including HTML escaping of <, >, and &, and escaping U+2028/U+2029 as \u2028/\u2029. The manifest hash omits the sha256 member entirely (not an empty string) and includes all recognized metadata. Timings participate in hashes; no cross-run hash stability is promised. json-v1 identifies canonical JSON digests; utf8 identifies captured text bytes; source-bytes identifies source bytes only, not captured content.

These independently tested vectors are JSON content digests, not full manifest hashes. The integer token is preserved without rounding:

Canonical JSON input	SHA-256
`{"a/b":["ok"],"count":9007199254740993}`	`ae787d745a6a9350aff0908f9bfb76ccdaf889b874927d3e5067392db310c003`
`{"a":9007199254740993,"z":"ok"}`	`7b80b3c931fd14341f4f635196e2a8004ddd3b98f1d9b5cb2788a78353e4c90e`

The vectors are checked in internal/evidence/manifest_test.go.

Ask Waza