Skip to content
Vally

Vally 0.20.0

Released on 2026-10-22.

Breaking

  • Custom executors and other error producers must set retryable: true on the error they throw to authorize retries. Retry consumers no longer infer retryability from error messages, nested causes, or AggregateError members. (#1241)
  • Public API (@microsoft/vally/graders): withRolledUpGraderError() has been removed. Aggregate graders must propagate blocking errors to their top-level status; nested advisory errors no longer invalidate an otherwise valid aggregate verdict, but still cause CLI commands to exit 1. Use hasGraderError() to detect errors recursively. (#1222)

Features

  • Terminal reporting now uses compact result cards, clearer eval/model/experiment summaries, aggregate trial counts, and explicit flakiness and partial-run labels. Failure diagnostics and verbose evidence are easier to inspect. (#1295)
  • The Docker backend builds and caches its default task/grade image locally as vally-task:<content-hash> instead of pulling a versioned image from Azure Container Registry. The image contains the installed core package; uncached builds require network access, and explicit image overrides remain supported. (#1337)
  • The Copilot SDK executor supports HydraFusion V2 routing with model: auto and executor.config.autoTier: hydrafusion or hydrafusion-max, including experiment variants. (#1347)
  • Reasoning effort settings accept model-defined levels, including none, minimal, and max, across evals, experiments, graders, and CLI overrides, with validation against model capabilities when available. (#1318)
  • Prompt and panel judges support defaults.judge_timeout and vally eval --judge-timeout, allowing per-turn timeouts longer than the existing two-minute default while remaining bounded by scoring.timeout. (#1324)
  • Experiment runs and merges emit OpenTelemetry traces, saving local trace artifacts by default or exporting to a collector with --otlp-endpoint. (#1297)
  • The Copilot SDK executor handles MCP sampling requests through the Copilot runtime, allowing tools that depend on sampling to complete. (#1336)

Fixed

  • Grading failures are distinguished from valid failing verdicts across terminal, JSONL, markdown, and JUnit reports. Missing grading results are treated as errors, and experiment cleanup failures produce exit code 1; valid failing verdicts still require --require-pass to fail the command. (#1222)
  • Evals whose stimuli execute successfully without graders pass without an overall score, even with a threshold configured. Successful --skip-grade runs remain passing in both single- and multi-trial execution. (#1222, #1295)
  • Copilot executor and judge calls reject unavailable models and model-catalog lookup failures before creating a session. BYOK providers remain exempt from Copilot catalog validation. (#1352, #1358)
  • Copilot clients recover after model-catalog timeouts instead of reusing a stalled client; judge client startup is bounded so later calls can retry with a fresh client. (#1358)
  • Retry classification is owned by error producers, and comparison diagnostics preserve the error class, retryability, attempt count, and message instead of treating all exhausted judge retries as rate limits. (#1241)
  • Copilot SDK tool results preserve structured error payloads for downstream graders and trajectory consumers. (#1340)
  • Docker grading honors the eval’s configured grading timeout instead of imposing a fixed deadline. (#1343)
  • In-place workspace diff capture preserves sparse indexes instead of expanding them. (#1345)
  • vally oracle --no-golden-input runs turn- and agent-scoped output and trajectory graders against the empty baseline instead of rejecting them; absence-only graders report n/a for the negative control. (#1323, #1342)
  • Harbor exports preserve agent environment variables and avoid reinstalling Node when the base image already provides a compatible version. (#1339, #1338)