Skip to content
Vally

Vally 0.15.0

Released on 2026-08-31.

Breaking

  • vally oracle gains --skip-grade to produce golden trajectories without running graders. (#1031)
  • vally eval and vally experiment now distinguish failing scores from grader execution errors end-to-end, matching the vally grade exit-code semantics introduced in 0.14.0. (#1032)
  • Redundant behavioral-constraint grader aliases have been removed; use the canonical constraint names instead. (#1085)
  • Deprecated top-level eval.yaml configuration support has been removed; move settings under their documented nested keys. (#1100)

Features

  • LLM prompt graders can opt into workspace evidence delivery. (#730)
  • Executors can discover Copilot agents natively from the workspace instead of requiring explicit configuration. (#797)
  • The internal GER migration bridge plugin gains broader task.yaml translation coverage, including setup.copilot_settings and solution.commands → goldenPatch. (#942, #1020, #1030)
  • Runs can capture the full workspace diff as a workspace.patch artifact. (#975)
  • New system event grader validates agent lifecycle/system events. (#987)
  • The tool-calls grader gains write_only and read_only flags. (#1013)
  • New step-count grader checks the number of agent turns/steps taken. (#1016)
  • vally eval agent is now available as an alias for the existing eval command. (#1035)
  • Harbor exports support verifierSetupScript for Windows tasks, Git environments in Windows Harbor tasks, and now use the Vally oracle to produce golden solutions. (#1036, #1051, #1054)
  • Multi-turn evals can be exported as Harbor multi-step tasks. (#1037)
  • Git worktree/clone setup timeout is now configurable. (#1040)
  • Stimuli selection supports a --tag key!=value exclusion filter. (#1047)
  • Graders and Harbor exports can use outside-workspace assets and artifacts directories. (#1077)
  • New grading_environment block keeps grader-only files out of the agent-visible workspace, recorded diff, and exported task while staging them into a disposable grading workspace. The agent-visible block is also renamed to agent_environment, with environment kept as a deprecated alias for compatibility. (#1106)

Fixed

  • vally compare retries transient judge failures and reports a partial/inconclusive state instead of a hard failure. (#838)
  • The linter only warns about scoring-defaults-applied when multiple graders are present, reducing noisy single-grader warnings. (#1048)
  • vally oracle --no-golden-input is now a real negative control for trajectory graders. (#1049)
  • Golden-patch warning scope, report ordering, and CI eval verdict computation were corrected. (#1050)
  • An escaped $${ sequence is no longer silently dropped when it’s the only ${…} in an eval. (#1062)
  • Harbor no longer applies default task resource limits, avoiding unintended resource caps on exported tasks. (#1067)
  • Fixed an incorrect eval runner tag-filter assertion. (#1071)
  • eval-authoring setup now works cross-platform instead of requiring a POSIX shell. (#1101)
  • Pinned @github/copilot CLI to a known-good version to fix intermittent LLM-judge failures caused by an upstream package export removal. (#1108)