Skip to content
Vally

Vally 0.11.0

Released on 2026-07-22.

Breaking

  • The exported Backend interface (@microsoft/vally/backend) now requires a gradeTrial(options) method; custom backends must implement it. (#649)

Features

  • New vally init command scaffolds a ready-to-run starter eval.yaml. (#742)
  • Vally now reports Copilot AI Credit usage (totalNanoAiu) per stimulus through eval and experiment results. (#740)
  • vally grade gains --diff-file and --artifact-dir for grading pre-hydrated output fully offline. (#758)
  • New agent-side reasoning_effort option, threaded end-to-end to executors like model. (#747)
  • New optional attachments field on Stimulus attaches files to the agent’s initial prompt. (#710)
  • The Harbor exporter now exports environment.mcpServers instead of dropping it. (#706)
  • The custom-metrics grader now reads its metrics JSON from trajectory.artifactDir before falling back to the workspace. (#746)
  • Executors now get a durable per-trial artifacts directory so their output survives in the report dir. (#748)
  • The ATIF adapter now accepts ATIF v1.7 trajectory shapes while keeping v1.6 support. (#726)
  • --grader-plugin is now repeatable, so multiple external grader packages can load in one run. (#721)
  • New --reporter-plugin seam lets external packages or local paths register custom reporters without forking the CLI. (#612)
  • Tool-call events gain an optional turnId so the tool-calls grader batches parallel checks by turn identity. (#693)

Fixed

  • CopilotAdapter no longer mislabels tool results when parent and subagent calls reuse the same toolCallId. (#769)
  • vally eval / vally experiment now cap workers at the number of trial items. (#762)
  • Suite globs now reach dot-directories and sibling trees. (#635)
  • vally lint now correctly emits unknown-environment-ref for unresolved environment references. (#738)
  • The valid-refs grader now validates reference-style Markdown link/image definitions. (#750)
  • The valid-refs grader now uses a CommonMark AST, eliminating false positives from paths inside code. (#731)
  • The Docker build no longer fails on stale npm registry metadata. (#777)