Skip to content
Vally

Vally 0.10.0

Released on 2026-07-16.

Breaking

  • LLM prompt/panel graders no longer automatically receive the resolved golden patch under vally grade and vally oracle. Configurations that need reference-solution context must now add golden_patch to the grader’s config.evidence (e.g. evidence: [trajectory, golden_patch]). Normal vally eval never resolved golden patches, and configurations without evidence continue to receive trajectory evidence. (#700)
  • vally lint now rejects an environment.files[] entry that is missing dest (previously only caught at load time), so specs that passed lint with a missing dest will now fail. The lint diagnostic code for a missing src was also renamed missing-required-field → environment-file-missing-field (both src and dest share it); tooling that keys off the old code must be updated. (#702)
  • The exported TrajectoryEvent union gained a new system member (SystemEvent), and the ATIF adapter now emits typed system events for context-management steps (e.g. compaction) instead of the previous generic custom events. TypeScript consumers with exhaustive switch statements over TrajectoryEvent must handle the new variant, and custom graders that matched those steps as type: "custom" must match type: "system" instead. (#669)
  • Public API (@microsoft/vally): the exported assertOracleGraders(graderConfigs) helper now takes a second options argument ({ hasGoldenTrajectory?, goldenTrajectoryHasTurns? }) so oracle grader validation can account for a supplied golden trajectory. Callers relying on the previous single-argument signature should pass the new options. (#690)

Features

  • prompt and panel LLM graders gain an explicit config.evidence selector (trajectory, diff, golden_patch) that controls what a judge sees. diff injects the agent’s cumulative workspace changes as ## Agent Diff; golden_patch (under vally grade/vally oracle) injects the reference solution. Omitted evidence defaults to trajectory evidence; when present it is a strict, complete selector. Agent Diff and Golden Patch sections are capped at 20,000 characters and nonce-fenced as untrusted evidence, and diff cannot be combined with turn/scope. (#700)
  • New environment.env field sets environment variables on the agent process, threaded into the copilot-sdk, claude-cli, and cca executors. Executors advertise supportsEnvVars; the pipeline warns rather than silently dropping vars for executors that don’t. Includes Windows-safe case-insensitive env merging, env-var name validation, and best-effort secret redaction in executor error output. (#711)
  • vally oracle gains two optional stimulus fields — golden_trajectory (an ATIF trajectory, inline or path, graded instead of the empty synthesized one, which lifts the rejection for behavior/metric/transcript graders including turn-scoped ones) and golden_custom_metrics (a metrics JSON object written into the workspace wherever a custom-metrics grader reads). Both are withheld by --no-golden-patch and combine with golden_patch. (#690)
  • The claude-cli executor now parses Claude Code stream-json output into full Vally trajectories — normalizing Claude’s tools (Bash, Edit, Read, …) to the shared canonical vocabulary, using real event timestamps, and taking token usage, cost, and model from Claude’s authoritative result event — so Claude runs are gradeable with the same graders as Copilot. (#696)
  • The ATIF → Trajectory adapter now emits reasoning events (from each step’s reasoning_content, in both main and subagent paths) and a new typed system event (type: "system") carrying compaction/context_management semantics, instead of collapsing non-skill system steps into a generic custom event. Both render in the LLM trajectory formatter rather than as [UNKNOWN]. (#669)
  • New Trajectory.raw escape hatch carries the original, unconverted source document (e.g. the raw ATIF) so custom (program) graders can reach fields Vally’s normalized flat-event model drops. It is populated by fromAtif, in-process only, and stripped before JSONL serialization on both the vally eval reporter and vally grade --output jsonl paths. (#662)

Fixed

  • Temp-dir cleanup in the vally-executor-claude-cli and vally-executor-cca plugins now routes through the retry-hardened removeDir() helper, closing the last gaps in Windows EBUSY/EPERM/ENOTEMPTY cleanup hardening. (#703)

Changed

  • The live eval progress line now reads 0/1 completed (workers: 5) instead of the misleading Running 0/1 (workers: 5) — the number is completed trials, not in-flight ones. (#714)
  • Documentation: added a discoverable schema/reference map (AGENTS.md, README golden path), multi-turn guide/reference cross-links, and a runnable samples/multi-turn-memory/ example so humans and agents can find the right docs without grepping source. (#699)
  • Renamed the private root workspace package from evaluate to vally. The root package is "private": true and never published, so this is a cosmetic change with no effect on @microsoft/vally*. (#704)