Skip to content
Vally

Vally 0.5.0

Released on 2026-05-21.

Breaking

  • CLI: The --eval command line switch for lint has been removed, in favor of lint --eval-spec or lint -e. (#277)
  • Program-grader now uses the workspace folder as its default working directory (#274)
  • CLI: Removed the inline prompt positional argument from vally eval. Use --eval-spec, --suite, or --tag to drive evaluations.

Features

  • EvalRunOptions.onDiagnostic callback for structured runtime warnings (falls back to console.warn when not set).
  • New built-in mock executor for testing graders without a live agent. Returns a pass-through trajectory (stimulus prompt as output) or loads fixture responses from <workDir>/.mock-executor/<stimulus.name>.json. Registered alongside copilot-sdk in the global executor registry.
  • Program and run-command graders: added env config field for passing custom environment variables to grader subprocesses. Reserved variables (EVALUATE_WORKSPACE, EVALUATE_GRADER_INPUT) cannot be overridden by program grader config.
  • Allow program and run-command graders to run from a configured subdirectory under the preserved workspace, accepting either slash style (#278)
  • Multi-judge panel grader — runs the same rubric across N judge models in parallel and combines their verdicts with a configurable aggregation strategy (majority / mean / median / min). Failure isolation keeps surviving judges scoring when one judge fails; per-judge breakdowns appear in details; the raw spread of judge scores (max − min) is surfaced via metadata.disagreement. (#189)
    • The panel grader intentionally does not expose a per-invocation panel_concurrency knob — judges fan out together via Promise.allSettled. LLM rate-limit / throughput control belongs at a global layer (e.g. on the LlmClient or runner) so it applies uniformly across every LLM-using grader rather than only within a single panel call. Tracked in #363.
    • max is intentionally not included in the aggregation strategies — it was only there for symmetry with min and has no concrete use case (“any single judge can pass the panel” is rarely what you actually want). Can be added back if a real workload calls for it.
    • The panel grader does not flag disagreement against a threshold or affect passed based on it. The raw spread is recorded in metadata.disagreement so downstream tooling (reporters, dashboards, alerts) can apply its own policy. Wiring disagreement into diagnostics / reporter footnotes / soft-fail semantics is tracked in #365.

Fixed

  • panel grader: validateEvalSpec/vally lint now rejects an empty models list at lint time instead of letting the empty config slip through and only fail at runtime. Backed by a new generic minItems schema constraint for array-shaped grader config fields.
  • panel grader: validateEvalSpec/vally lint now also rejects empty-string entries in models (e.g., models: ["gpt-5", ""]), which would previously pass lint and be silently dropped at runtime. Backed by a new generic itemMinLength schema constraint for string-array fields. The constraint trims before measuring length, so whitespace-only entries like [" "] are also rejected.
  • panel grader: parseConfig() now trims model entries so whitespace-only model names (" ") are dropped rather than passed through to the LLM client.
  • panel grader: PanelGraderConfig.models is now typed as required (string[]) to match the validator schema and the eval.yaml shape, giving programmatic callers compile-time enforcement consistent with vally lint.
  • panel grader: per-judge metadata key failed_judges is now snake_case in both the partial-failure and all-judges-failed result paths (the all-failed path previously emitted failedJudges), consistent with the other emitted metadata keys (token_usage, latency_ms, raw_overall_score).
  • panel grader: per-judge information (score, raw score, latency, token usage, reasoning) is now only emitted in details[*]; the redundant metadata.judges array (a denormalized copy of the same data) is gone. Consumers should walk details to inspect per-judge results.
  • Eval-level tags are now merged onto each stimulus during resolution (child-wins), so custom executors receive the effective tags. (#337)
  • CLI: standardized -e as the short alias for --eval-spec across eval, lint, grade, compare, and export. (#277)
  • Use the proper workdir for the program grader. (#274)
  • Eval runs now abort when MCP servers fail to load instead of silently continuing. Each failed MCP server load will show up as an error event in the trajectory. (#288)
  • tool-calls grader: the name field of a ToolMatch is now matched as an unanchored regex, matching its documented behavior. Use ^name$ for an exact match; the inline-flag convention ((?i), (?m), (?s)) is supported. Empty name patterns are now rejected with a clear error (an empty regex would otherwise match every tool call). Behavior change: configs whose tool names contain any regex metacharacter will now be interpreted as regex — escape each metacharacter with \ to preserve a literal match (and optionally wrap with ^...$ to also require an exact, non-substring match).

Changed

  • CLI: vally eval now always writes result files to a directory. The default is ./vally-results/<timestamp>/ (a summary at the end of the run gives the exact path). Use --output-dir to specify a different directory.
  • CLI: The --junit requires --output-dir and --keep-executor-session-logs requires --output-dir guards have been removed since output is now always written to disk.