Skip to content
Vally

Vally 0.6.0

Released on 2026-06-11.

Breaking

  • Removed LlmClient.chat(), LlmChatOptions, LlmResponse, chatAndExtractJson, and extractJson from the public API — all LLM judge calls now use LlmClient.judge() exclusively.
  • PromptGrader and PairwiseGrader constructors no longer accept RetryOptions (unused — no caller or config surface ever set it).
  • Removed withRetry from the public API (no longer used by either LLM grader after the tool-call refactor; the function remains internal-only).
  • Reporter API rebuilt around the new PlanReporter interface. All four built-in reporters (EvalConsoleReporter, EvalJsonlReporter, EvalMarkdownReporter, EvalJunitReporter) and CompositeEvalReporter were rewritten in place; the legacy BufferedEvalReporter, StringWritable, and EvalReporterRegistry types were removed. The following public exports were also removed: EvalReporter, EvalRunStartInfo, EvalCompleteInfo, EvalReporterRegistration, RegisteredReporter, ReporterMutex, and createEvalReporterRegistry.
  • Eval pipeline rebuilt around concurrent trials: runMatchedEval and runMultiTrial were removed, as were the legacy orchestrator (packages/core/src/pipeline/orchestrate.ts) and the multi-trial module (packages/core/src/pipeline/multi-trial.ts), including their exported computeMultiTrialVerdict and decomposeMultiTrialOutcomes helpers. The associated public types RunEvalOptions, RunEvalResult, StimuliExecutionResult, StimulusExecutionContext, MultiTrialOptions, and TrialProgressEvent were also removed.
  • JSONL output: TrialResultRecord and the per-eval entries of the run-summary record now include a required variant: string field. Plain vally eval runs use a default variant name; experiment runs carry the configured variant name. Downstream consumers that parse the JSONL stream should expect this field on every line.
  • vally-server: the outcomes table gained experiment_name, experiment_run_id, variant, eval_file, eval_hash, and config_hash columns plus two indexes (idx_outcomes_experiment, idx_outcomes_variant). SCHEMA_VERSION bumped to 2. There is no migration path — pre-existing SQLite databases must be deleted and re-ingested.
  • CLI: Removed vally eval --keep-executor-session-logs; executor session logs are now preserved by default under the run output directory.
  • Removed the unused onDiagnostic callback from EvalRunOptions (public API). runEval no longer emitted diagnostics through it; callers that set it can drop the option.
  • Skills are no longer auto-discovered from the working directory. Specify skills for a run either in an eval file via environment.skills (paths relative to the eval file), or for a whole vally eval run via the opt-in --skill-dir <path> flag. When environment.skills is set it fully replaces the --skill-dir base (replace, not merge), and vally experiment run never auto-discovers skills, so baseline-vs-skilled A/B comparisons stay hermetic. Paths that are missing, not a directory, lack a SKILL.md, or contain an unparseable SKILL.md now fail the run fast instead of being silently skipped.

Features

  • New vally experiment run CLI command for declarative multi-variant evaluation. Reads an *.experiment.yaml file that declares the eval set, the varying dimensions (typically /environment/skills), and the named variants; runs every (eval × variant) combination through a single shared worker pool; emits per-variant results.jsonl + run-summary.jsonl plus an experiment-level markdown report.
  • vally experiment run --dry-run resolves the experiment and prints the plan (one row per (variant × eval)) without executing trials.
  • Console and markdown reporters now surface the variant name in headings and the progress table when a plan spans more than one variant. Single-variant runs (every plain vally eval invocation) are unchanged.
  • CLI: vally eval now accepts --shutdown-timeout <duration> (e.g. 5s, 500ms; default 5000ms) to bound how long cleanup waits for executors to shut down, including on Ctrl+C. Previously this window was hardcoded to 5 seconds. (#112)
  • tool-calls grader: added a sequence field that asserts tools were called in a given relative order (an ordered subsequence). Other tool calls may be interleaved between the sequence entries. Entries use the same ToolMatch | string form as required/disallowed, and can be combined with them. (#452)
  • tool-calls grader: added at_step / before_step positional constraints to required entries, asserting that a tool was invoked in a specific agent step (at_step: N) or within the first N steps (before_step: N). Steps are counted by turn_start events; trajectories without turn boundaries collapse to a single step 0.
  • tool-calls grader: extended ToolMatch with min_count (require at least N matching calls on a required entry), args (regex matching against arbitrary call arguments, not just command/path), result (regex matching against the tool’s output/observation, on required/disallowed entries), and final (assert the matching call is the final tool call). (#479)

Fixed

  • CLI: vally eval now handles SIGINT/SIGTERM (Ctrl+C) gracefully — it shuts down executors, flushes partial JSONL output, and finalizes partial markdown/JUnit reports before exiting (code 130 on SIGINT) instead of being torn down by Node’s default action mid-run. A second Ctrl+C forces an immediate exit. (#112)

  • LLM judges (prompt, pairwise) now deliver verdicts via a submit_grade / submit_pairwise_grade tool call instead of parsing JSON out of model prose, eliminating intermittent “No valid JSON object found in LLM response” failures. (#345)

  • Grader validation error messages now include the grader’s name when set or its type, if it has no name.

  • Eval loader now rejects blank or whitespace-only grader name values with a clear error instead of accepting them.

  • tool-calls grader: a dangling tool_call (no matching tool_result, e.g. a truncated run) now only errors when a sequence is configured — required/disallowed-only configs grade such partial runs normally instead of throwing.

  • CLI: vally eval now utilizes all available workers to run trials.

  • vally experiment run: the experiment-relative eval path recorded in results.jsonl / run-summary.jsonl (experiment.evalFile) is now computed against the canonicalized experiment directory, so symlinked or temp-dir checkouts no longer produce ../private/var/…-style paths.

Changed

  • Server ingest now emits the VALLY_LEGACY_JSONL deprecation warning for legacy JSONL input.

  • defaults.timeout in an eval spec is now honored as the per-stimulus timeout when --timeout is not passed on the command line. Previously the field was accepted and validated but never applied, so the eval always fell back to the built-in default. The CLI --timeout flag still takes precedence over the spec default.

  • run-command graders now continue evaluating configured stdout/stderr assertions against partial output when a command times out or exceeds maxBuffer.

  • copilot-sdk executor now aborts in-flight backend work before disconnecting when a run fails or times out, instead of only disconnecting (which left the backend request running). Both the abort and disconnect are best-effort and individually time-bounded so a wedged connection can’t block teardown. (#112)

  • vally grade can now consume vally eval --skip-grade --output jsonl output again. EvalJsonlReporter now always emits explicit gradeResult and trajectory keys (as null when absent) on every trial-result line, so ungraded success records are valid EvalOutcome shapes and round-trip through the documented “grade later” workflow instead of failing with stdin did not match a recognized shape. (#492)

  • Harbor exporter: generated Dockerfiles now install Node.js and the vally CLI by default, requiring a Debian/Ubuntu-family base image (apt-get). Pass installRuntime=false to skip the install block when using a base image that already provides the runtime.