Skip to content
Vally

Vally 0.18.0

Released on 2026-10-05.

Breaking

  • The unused filter field has been removed from experiment files; experiments that set it are now rejected. Suite tag filters and --eval-filter are unchanged. (#1242)
  • Public API (@microsoft/vally): the pre-stable Executor/Backend cleanup contract changed. cleanup() now accepts { preserveWorkspace?: boolean } and is no longer skipped after failed egress, so custom backends must honor preserveWorkspace to avoid deleting recovery files. (#1233)
  • The minimum Node.js version is now 22.14.0. For @microsoft/vally-server, the raw-connection API is replaced by openDatabase(), which returns { dataSource, ingestDirectory, close }. (#1263)

Features

  • New built-in Docker backend runs trials in isolated Docker Compose environments: vally eval --backend docker. (#1246)
  • New vally simulate record command generates a simulation.tool_overrides block from a recorded run so evals can replay tool calls deterministically. (#1227)
  • Machine-readable outputs now carry a stable top-level runId on trial-result and run-summary records. vally eval, grade, oracle, and experiment run accept --run-id, experiment merge validates and preserves it, and reporter plugins receive it at run start. (#1234)
  • vally eval gains --workspace <path> --in-place to run directly in an existing directory, avoiding expensive workspace copies and worktree creation in large repositories. Supports whole-run diff grading in clean Git checkouts. (#1268)
  • Executors can register per-attempt cleanup callbacks via ExecutorOptions.registerCleanup, which run after grading instead of at the end of execute(). (#1233)
  • better-sqlite3 is now an optional peer dependency; vally serve and vally ingest fall back to Node’s built-in SQLite when it is not installed. (#1263)

Fixed

  • Copilot SDK operations now honor the trial execution deadline during startup and session creation, and SDK teardown is bounded to prevent timed-out work from hanging trials or leaking resources. (#1253)
  • Cumulative diff capture failures are now preserved in saved results, so offline re-grading reports the real error instead of a generic “No workspace diff is available”.
  • vally oracle no longer reports golden_custom_metrics as agent changes in the workspace diff, fixing diff graders failing on such stimuli. Graded runs now supply the metrics to the custom-metrics grader through the artifact directory instead of the workspace, so other graders reading custom_metrics.json from the workspace no longer see it. (#1264)
  • Output and transcript contains graders now report the effective substring, case_sensitive, and negation settings in result metadata, including defaults and negation implied by the grader name. (#1244)