Skip to content
waza

Ask Waza

Ask about Waza commands, writing evals, or troubleshooting an error. Paste YAML below for a schema check.

AI answers can be wrong. Do not share secrets or sensitive information. History may be retained.

CLI reference | Privacy & limits

    Check pasted YAML (optional)

    External Assertion Interoperability

    This source-build preview, targeted for v0.40.0, converts one explicitly selected AgentV case-sensitive contains assertion to a native text grader. It does not import AgentEvals, AgentV, Harbor or ASSERT tasks or whole suites. Issue #789 remains open.

    Import a selected assertion

    Section titled “Import a selected assertion”

    From the repository root, with a binary built from this source:

    Terminal window
    # From the repository root, using a binary built from this source.
    adapter=agentv-68df6d2206cea61f721d0445b405efcb5e94aff0-contains-v1
    waza interop import --adapter "$adapter" --test contains-check --assertion 1 \
      < internal/interop/testdata/agentv/suite.yaml > one-native-grader.yaml

    The command saves stdout to one-native-grader.yaml for the export below. Stdout is one native grader mapping, not a runnable eval. Place it in a native eval’s graders list. Stderr reports ASSERTION-ONLY PROJECTION: task inputs, providers, fixtures, sibling assertions, thresholds, suite metadata and execution settings are not converted or validated. Selected assertions with unsupported fields fail rather than losing semantics. Excluded scripts or hooks are never executed. Source digest comments are advisory and unverified.

    The exact commit pin is mandatory. Only type: contains, a nonempty string value and optional nonblank string metric are supported. Case-sensitive substring matching maps to native contains_cs, not case-insensitive contains. Scores are binary 0/1, higher is better. No regex, negation, required, weights, thresholds, provider settings, evidence or whole-task translation is claimed.

    Export one native grader

    Section titled “Export one native grader”
    Terminal window
    waza interop export --adapter "$adapter" < one-native-grader.yaml

    Input must have exactly name, type: text, and config: {contains_cs: [one-nonempty-string]}. Output is a single AgentV assertion for an upstream assert list, not a suite. Multiple native checks are rejected because their fractional scores would change the upstream assertion. A missing upstream metric imports as agentv-contains; export makes that metric explicit. Round trips preserve supported meaning, not original bytes.

    Both commands use stdin/stdout only. They reject unknown adapter versions, duplicate/merge keys, aliases, anchors, multiple documents, custom tags, non-UTF-8 input and input exceeding 4 MiB. No model/tool calls, ambient config loading or reference reads occur. A value resembling a path remains literal text. Selected strings containing ${, {{, }}, {% or {# are rejected in both directions rather than interpreting environment interpolation or Nunjucks expressions, statements or comments. Validation/conversion errors before output delivery return exit 2 with no converted stdout. Delivery errors propagate, but failed or short writes can expose partial output that cannot be rolled back. A broken pipe may terminate the process instead of returning exit 2. Shell redirection can create or truncate a file before failure; discard output from failed commands.

    Coverage and evidence

    Section titled “Coverage and evidence”
    Source Current coverage
    AgentV 68df6d22, MIT Selected literal contains assertion import/export only
    AgentEvals specification 61ec10dc, MIT No direct adapter; specification dialect differs from AgentV
    Harbor f6cf2e2e, Apache-2.0 No adapter; real task success depends on container and executable verifier semantics
    ASSERT 45da4c56, MIT No adapter; interchangeable definition schema and offline scoring oracle not established

    The retained real upstream YAML suite, executable TypeScript assertion oracle, and MIT license are checksum-pinned. Production Waza CLI conformance compares supported positive and negative decisions and scores against that oracle without inference. This is not whole-suite or agent-quality evidence.

    Terminal window
    go build -o /tmp/waza-interop ./cmd/waza
    bash examples/interop/run.sh /tmp/waza-interop

    The example reports and retains scratch files on success or failure, rather than recursively deleting a pathname that could have been replaced. Use a trusted temporary parent and inspect the reported location before manual removal; an external rename or replacement can make that path stale.

    The example uses a mock executor and checks one passing and one failing control. It requires Bash, jq, and the repository fixtures. The Go conformance tests additionally require Node 22.18+ or 24+ for TypeScript type stripping.

    See the verified compatibility matrix, source pins, licensing and remaining acceptance gaps and the native eval authoring guide. Full fixture/model/tool/task semantics, combined suite decisions, and full evidence/provenance round trips remain unsupported. No Lab/result schema changes are introduced.