Vally 0.19.0

Released on 2026-10-13.

Experiment files no longer declare plugin paths; move reusable grader, executor, reporter, eval-provider, and backend plugin configuration to .vally.yaml. (#1058)

.vally.yaml can now register project-wide plugins and select default backends and eval providers, giving vally eval and vally experiment run consistent, reproducible runtime configuration. (#1058)
Evals can set scoring.timeout to enforce one wall-clock grading budget per stimulus and trial across vally eval, vally grade, and vally oracle. Completed grader results are preserved when the deadline expires. (#1179)
Harbor exports support taskCustomizer=<path> modules that can add, replace, or suppress generated task files before they are written. (#1304)
Harbor exports can use joinTurnsWith=<delimiter> to combine a multi-turn stimulus into one instruction while preserving the existing multi-step behavior by default. (#1306)

Absence-only graders, such as tool-calls configured only with disallowed, now grade empty agent scopes correctly; oracle baselines report these scoped checks as n/a. (#1325)
vally experiment run --compare now inherits each eval’s effective judge model, reasoning effort, and provider settings. The command also accepts --judge-model and --judge-reasoning-effort overrides for comparison without changing trial grading. (#1276)
Suite and experiment discovery now accepts inherited project directories reached with different casing on case-insensitive filesystems while keeping casing checks for paths written in eval patterns. (#1286)
Harbor verifier and oracle specs now preserve literal ${...} values when reloaded instead of treating them as unresolved parameter placeholders. (#1299)
Grader execution errors can no longer pass threshold scoring when threshold is 0. (#1179)
LLM graders can again authenticate with Copilot credentials saved in the OS keychain while retaining workspace-judge restrictions. (#1298)