Skip to content
Vally

Vally 0.8.0

Released on 2026-07-08.

Features

  • New vally oracle command — verifies an eval harness against a stimulus’s golden_patch without running an agent. It materializes the starting environment, applies the patch, and runs the graders; --no-golden-patch grades the unpatched baseline as a sanity check (which should fail). (#615)
  • New vally experiment merge <shard-dir>... -o <merged-dir> — reconciles the per-shard outputs of a sharded vally experiment run into one authoritative result (concatenated per-variant results.jsonl, regenerated run-summary.jsonl, merged plan-snapshot.json, a single report.md, and a promoted experiment-manifest.json), with a specific error per validation failure mode. Completes end-to-end experiment sharding. (#626)
  • New Backend plugin type — separates the compute environment (where a trial runs) from the executor (the agent that runs it). vally eval and vally experiment run accept --backend <name-or-package> and repeatable --backend-args <key=value>; the built-in local backend is the default. Backends register via registerBackends(registry) and implement prepare/runTrial/shutdown. (#584)
  • New @microsoft/vally-executor-cca executor plugin — runs the Copilot Coding Agent (CCA) V3 binary as a local executor, converting its events JSONL stream into a Vally trajectory. Single-turn in v1. (#642, #650)
  • New custom-metrics grader — asserts on arbitrary numeric/boolean/string signals a run writes to a JSON file (default <workspace>/custom_metrics.json), beyond Vally’s built-in trajectory metrics. (#487)
  • New transcript-contains and transcript-matches graders (plus -not- variants) — assert on any assistant message produced during a run, not just the final output. (#627)
  • Trajectory graders accept an optional scope (parent, subagent, or { agent: "<name>" }) to assert on a specific agent’s behavior rather than the whole run; composes with the existing turn field. (#647)
  • tool-calls grader gains a parallel predicate — passes only when a set of matching calls all occur within a single model-response batch (one agent step). (#636)
  • panel grader: added unanimous aggregation strategy (mean score with strict all-pass verdict). (#632)
  • panel grader: added structured per-criterion aggregation via criteria + overall_threshold, supporting per-criterion pass_threshold, weight, and advisory criteria (required: false). (#632)
  • Tool-call simulation — a stimulus can declare canned responses for first-party tools (simulation.tool_overrides) so the agent runs against a deterministic, mocked toolset, enabling side-effect-free behavioral evals. (#648)
  • A stimulus can declare a supported_executors allow-list; in a mixed run, stimuli incompatible with the active executor are skipped (not failed), with an executor-skip-all diagnostic when a filter removes everything. (#489)
  • vally experiment run now preserves per-trial executor session logs, matching vally eval. (#621)

Breaking

  • vally eval and vally experiment run flatten preserved executor session logs: the executor-session-logs/ level is removed and each trial collapses into a single directory named by its 0-based trial number, nested as <eval>/<stimulus>/<model>/<trial>/ (experiment runs prepend a <variant>/ segment). Retries append __attempt-<m> (first attempt unsuffixed); sanitization collisions get a numeric -2/-3 suffix. Tooling that located logs under executor-session-logs/…/trial-N/ must be updated.
  • vally eval / vally experiment run now reject an eval whose sanitized directory name (case-insensitively) collides with a reserved run-output artifact name (results.jsonl, eval-results.md, eval-results.junit.xml, otel-spans, otel-spans.jsonl, diffs, run-summary.jsonl, artifacts), since flattened session-log directories are now siblings of those artifacts and would otherwise overwrite them.
  • Removed RunPlan.dispose(). Callers that execute planned items are responsible for shutting down backends, executors, reporters, and session logs directly.
  • TrialPhase collapsed the two workspace-lifecycle phases acquire-workspace and prepare-workspace into a single preparing phase.
  • Removed the remote-executor workspace-sync API: Executor.finalizeWorkspace(), the exported FinalizeWorkspaceContext and WorkspaceStatus types, and Trajectory.workspaceStatus. Executors must write to the provided local workDir, and remote compute environments should be implemented as a Backend.
  • panel grader: judge failures are now strict rather than failure-isolated. Any judge that errors (crash, timeout, malformed response) forces the panel to passed: false, even when the surviving judges’ consensus would otherwise pass. Previously a failed judge was dropped and the panel could still pass on the survivors. The surviving judges still produce the reported score (and metadata.failed_judges still lists what failed), but the verdict is forced to fail. Existing evals with panels that occasionally lose a judge to rate limits/timeouts may now see those trials fail where they previously passed. (#632)