Skip to content
Vally

Vally 0.4.0

Released on 2026-05-04.

Breaking

  • Node.js minimum version is now >=22.
  • constraints.max_duration_ms renamed to constraints.max_duration. Accepts duration strings (e.g. "2m").
  • Duration fields (config.timeout, grader timeout, MCP timeout, constraints.max_duration, CLI --timeout) now require a unit suffix (e.g. 5m, 30s, 500ms). Bare numbers are no longer accepted.

Features

  • Grade Copilot CLI output produced outside of vally (#212)
  • Preserve executor session logs (#203)
  • Add captured stdout and stderr to the result json files (#263)
  • Add metric-threshold graders: token-budget, tool-call-count, turn-count, error-count, wall-time — check trajectory metrics against configurable max thresholds in eval.yaml.
  • Human-friendly duration support: timeout fields now accept duration strings like 5m, 30s, 500ms, 2h with a required unit suffix. Applies to eval config timeout, grader timeout (run-command, program), MCP server timeout, constraints.max_duration, and CLI --timeout. New parseDuration, isValidDuration, formatDuration utilities exported from @microsoft/vally (#264)

Changed

  • @github/copilot-sdk is now a required runtime dependency for core/CLI Copilot execution and LLM grading paths. (#258)
  • Clarify why –executor-plugin is repeatable (#261)
  • Harbor: emit oracles for completed and exit-success graders (#208)
  • Harbor: make dir names valid Docker image refs (#238)
  • Update the install instructions now that we’re on public npm, and also make sure we test the doc site builds in CI (#252)