Skip to content
Vally

Vally 0.14.0

Released on 2026-08-18.

Breaking

  • vally grade now reserves exit code 1 for input, configuration, and grader execution errors; a valid failing verdict exits 0 unless the new --require-pass flag is set. Grader results can now report status: error, and automation that treated every failed score as a command error should opt into --require-pass. (#985)
  • vally lint now rejects invalid or effectively empty tools regex filters for tool-call-count, max-repeat, and loop-outcome, surfacing configurations that previously failed only at evaluation time. (#928)
  • Eval loading and linting now reject blank, null, and otherwise invalid defaults.executor values instead of accepting them as an absent executor. (#969)

Features

  • LLM prompt and panel graders gain evidence: [repo], which supplies a bounded snapshot of the final workspace while excluding noise directories, binary contents, and symlink targets. (#922)
  • New defaults.judge_provider configuration points LLM judge graders at a custom OpenAI-, Azure-, or Anthropic-compatible endpoint, with support for environment-referenced credentials and Copilot environment discovery. (#944)
  • Experiment variants and matrix axes can now vary reasoning_effort and judge_reasoning_effort. (#936)
  • New eval-provider plugin support lets vally eval discover non-native eval formats through --eval-plugin and --eval-spec; the internal GER migration bridge can run existing task.yaml suites directly, audit compatibility, or convert them to native eval.yaml. (#942)
  • The Harbor exporter gains Windows-container output through targetOs=windows, including Windows verifier/oracle scripts and optional installation of the Visual C++ runtime, Node.js, and Vally. (#937, #1008)
  • Harbor exports can customize generated task.toml sections with addTaskToml, including environment resources, networking, variables, MCP servers, and verifier images. (#913)
  • Harbor task metadata now declares when an exported eval requires an LLM judge, allowing Harbor to identify the task’s grading requirements. (#949)
  • Harbor’s verifierFiles option keeps private grading assets out of the agent image and overlays them only when verification starts. (#924)
  • Harbor verifier setup scripts can provide a cumulative DIFF_FILE, enabling unscoped diff graders and LLM judges using diff evidence in exported tasks. (#964)
  • The CCA executor is now distributed through the internal package feed instead of returning a 404 when installed. (#911)

Fixed

  • Grader execution failures are now represented as errors rather than ordinary failing scores, including nested grader failures; console output reports ERROR separately from FAILED. (#985)
  • Harbor verifiers write rewards only after grading succeeds and a score is extracted, preventing infrastructure or grading errors from appearing as a valid zero reward. (#915)
  • Harbor exports no longer emit the unsupported agent.max_turns field, and no longer emit placeholder category and tags values when the corresponding eval metadata is absent. Configure agent turn limits through Harbor instead. (#1014, #1012)
  • An npm engine restriction (>=11.11.1) accidentally introduced in Vally packages for 0.12.0 and 0.13.0 was removed. (#950)