Skip to content
Vally

Vally 0.13.0

Released on 2026-08-07.

Breaking

  • The tool-calls grader now validates its config semantics at vally lint time, so specs with cross-list or per-entry config mistakes that previously only failed when an eval ran are now caught at lint. (#582)
  • Graders now derive descriptive names from their config, so grader display names change in the console, markdown, and stored results, and the analytics DB schema bumps to v7 (delete eval-history.db and re-ingest); consumers keyed on the old bare type names should switch to the new graderType field. (#883)

Features

  • defaults.executor can now be an { name, config } object; the copilot-sdk executor gains a provider block to point the agent under test at a custom OpenAI-compatible endpoint (BYOK), with secrets referenced by env-var name. (#835)
  • New assistant-contains-tool-output grader captures a value from a tool result and asserts a later assistant message echoes it. (#912)
  • Graders now derive a descriptive name from their config, so multiple graders of the same type are distinguishable in reports instead of collapsing into one line. (#883)
  • The tool-call-count grader gains an optional tools filter, so budgets can target specific tools (e.g. web_search) instead of only the total tool-call count. (#900)
  • vally experiment run gains an opt-in --compare flag that runs vally compare against the baseline after the run. (#851)
  • vally experiment run gains a --workspace <path> flag that preserves each variant’s agent workspaces for inspection, matching vally eval. (#917)
  • The Harbor exporter now materializes worktree and clone environment.git environments as self-contained detached checkouts baked into the task image. (#834)
  • The Harbor exporter gains a verifierSetupScript option to inject container-side setup (e.g. acquiring a short-lived token for an LLM judge) into the generated verifier. (#846)
  • Remote/relocating backends now materialize trajectory.artifactDir to a host-readable path, so grading pre-hydrated output offline works across the backend boundary. (#892)

Fixed

  • Large tool-output spillover is now redirected to a readable absolute path, so the agent no longer wastes turns hunting for a session-relative spill file and polluting resource-usage metrics. (#880)
  • Preserving a worktree environment.git run to --workspace now repairs the worktree metadata, so git status in the preserved directory no longer fails with fatal: not a git repository. (#853)
  • The copilot-sdk executor no longer orphans its temporary home directory when it exits early on a fatal precondition. (#852)
  • vally lint now warns when a prompt/panel grader sets config.prompt but defines no rubric, and validates rubric shape, surfacing problems that previously failed silently or crashed mid-grading. (#888)