Skip to content
Vally

Vally 0.7.0

Released on 2026-06-30.

Features

  • vally eval collects OpenTelemetry traces from the copilot-sdk executor and writes them to otel-spans.jsonl in the output directory. Specify --otlp-endpoint <url> to instead export traces to an OTLP/HTTP collector; the endpoint must be an http(s) URL without embedded credentials.
  • CLI: vally export now accepts repeatable -e, --eval-spec flags to export multiple eval specs in one invocation. When more than one spec is given, each spec’s output is written into its own subdirectory of the output directory, named after the spec’s name field.
  • CLI: vally eval now copies per-stimulus configured workspace files into the run output directory whenever artifacts is present on that stimulus. Include/exclude patterns are configured via artifacts.include and artifacts.exclude; when a path matches both, exclude wins. Copied files are placed in an artifacts/ subdirectory of that trial’s session-log directory (<eval>/<stimulus>/<model>/<trial>/artifacts/), alongside its events.jsonl/metadata.json.
  • CLI: vally experiment run now also copies each stimulus’s configured workspace files whenever artifacts is present, co-located under that trial’s per-variant session-log directory (<variant>/<eval>/<stimulus>/<model>/<trial>/artifacts/), matching vally eval behavior.
  • run-command grader: stdout_contains, stderr_contains, stdout_matches, and stdout_not_matches now accept either a single non-empty string or a non-empty array of non-empty strings. For positive fields all listed entries must match; for stdout_not_matches none of the listed regexes may match. Each array entry produces its own GraderResult detail (named stdout-contains[0], stdout-contains[1], …) so failure evidence still points at the specific check that failed.
  • vally experiment run now also writes two run-level artifacts — plan-snapshot.json and shard-manifest.json — alongside the per-variant directories. The snapshot captures the planned eval/stimulus metadata and grader configs that the per-trial JSONL omits; the manifest records run identity (a machine-independent planDigest, variant names, varied axes) and shard identity (an unsharded run is recorded as shard 1/1). These are the artifact contract that the forthcoming vally experiment merge will consume to reconcile sharded runs. New public helpers computeShardKey, computePlanDigest, assertUniqueShardKeys, assemblePlanSnapshot, assembleShardManifest, and singleShardIdentity are exported from @microsoft/vally.
  • vally experiment run is now shardable across N runners via --shard i/n, --run-id <id> (required with --shard), and --shard-strategy <specifier> (default round-robin). A shard runs only its slice of the planned trials and writes outputs under <output-dir>/<run-id>/shard-NN-of-NN/; the experiment-level markdown report is skipped in shard mode (a per-shard report is incomplete by definition). The experiment manifest block and planDigest stay byte-identical across every shard of a run, and shard-manifest.json now records the real shard block (index/total, chosen strategy, selected/completed shard keys). Non-sharded runs keep today’s timestamped layout.
  • New public ShardStrategy interface in @microsoft/vally plus two built-in strategies — round-robin (spreads trials evenly) and by-stimulus (keeps every trial of a stimulus on one shard). Strategies receive inert ShardableItem[] descriptors (never live trial closures) and return the shard keys to run. Custom strategies load from a module specifier (--shard-strategy ./my-strategy.js or a package name) via the shared plugin resolver and must export createShardStrategy(): ShardStrategy (trusted code, no sandbox). Exports: ShardStrategy, ShardableItem, ShardSelectContext, roundRobinStrategy, byStimulusStrategy, resolveShardStrategy, validateSelectedShardKeys, BUILT_IN_SHARD_STRATEGIES, DEFAULT_SHARD_STRATEGY.
  • Each trial-result JSONL line now carries an optional shardKey field (the stable per-trial identity from computeShardKey), making the merge step’s results→shardKey mapping direct rather than re-derived.
  • Multi-turn stimuli via a new turns: [...] field — each prompt is sent sequentially to the same agent session so the agent retains full conversation context. Any grader can be scoped to a specific turn with turn: <N> (0-based, matching the gh-eval YAML convention) on its config; the pipeline slices the trajectory to that turn before grading, so scoping works for every built-in and plugin grader. Executors opt in via supportsMultiTurn; unsupported executors fail fast with a clear error. New exports: stimulusPrompts, eventTurn, sliceTrajectoryToTurn, stampConversationTurns. (#481)

Breaking

  • vally-server: the outcomes table gained a nullable shard_key column (populated from the new optional shardKey field on each trial-result JSONL line). SCHEMA_VERSION bumped to 3. There is no migration path — pre-existing SQLite databases must be deleted and re-ingested. Note: raw per-shard outputs (under <run-id>/shard-NN-of-NN/) are not meant for direct ingestion; ingest the merged output instead.
  • scoring.weights now requires a normalized distribution: all weight values must sum to 1.0 (±0.01 tolerance for two-decimal-place rounding). Previously, scoring.weights was accepted and validated but silently ignored — all graders contributed equally regardless. Now the aggregate score is a weighted sum keyed by grader type — Σ(weight_t × avg_score_t): when multiple graders share the same type, their scores are averaged first and the type weight is applied once. Graders absent from the map receive weight 0 and are excluded from scoring. Comparative graders (e.g. pairwise) are always excluded from the weighted score. The linter (vally lint) reports a scoring-weight-sum error when weights do not sum to approximately 1.0. (#416)
  • When scoring.threshold is configured, per-trial pass — and therefore pass@k / pass^k, the per-trial glyphs/counts, the markdown and JUnit verdicts, and the vally grade exit code — is now determined by score >= threshold rather than the binary all-graders-passed flag. This applies to any threshold eval, including equal-weight ones: existing threshold evals where some graders pass and others fail will see their pass@k/pass^k numbers and verdicts shift to reflect the aggregate score. (#416)
  • ScoringConfig.threshold and ScoringConfig.weights are now typed as optional (threshold: number → number?, weights: Record<string, number> → Record<string, number>?). This is a type-level change that aligns the interface with existing runtime behavior — validation already accepted a scoring block with either field absent, and an omitted threshold already defaulted to the binary all-graders-pass verdict. No spec-authoring behavior changes; but TypeScript consumers that read config.threshold / config.weights must now handle undefined. (#416)
  • Removed the exported resolveThreshold() helper. It returned an implicit default threshold of 1.0 when none was configured, which contradicted the new verdict semantics where an omitted threshold falls back to the binary all-graders-pass flag (see resolveGradePass). It was unused internally; consumers should read scoring.threshold directly (treating undefined as “no threshold”). (#416)
  • run-command grader: empty values for stdout_contains, stderr_contains, stdout_matches, and stdout_not_matches are now rejected at validation time (empty scalar strings, empty arrays, and empty-string array entries). Previously an empty scalar was silently treated as a no-op, so a check could vanish without warning. Remove or fill in any such entries before upgrading.
  • vally serve: the analytics server now validates the Host header to defend against DNS-rebinding attacks (which bypass CORS) — a malicious page can rebind its hostname to 127.0.0.1 and read eval data otherwise. Requests are rejected with 403 forbidden_host unless the Host is a loopback name (localhost/127.0.0.1/::1), the bound --host, an IP literal (IP-literal Hosts are inherently safe from rebinding), or a name passed via the new repeatable --allowed-host <host> flag. /api/health is exempt so liveness probes can address the server by a Service DNS name. Upgrade note: if you front vally serve with a reverse proxy or ingress that forwards a custom Host (e.g. nginx proxy_set_header Host $host;), add that hostname with --allowed-host or requests will be rejected. The effective allow-list is logged at startup.
  • vally-server: the outcomes table gained a stimulus_turns column (JSON-serialized Stimulus.turns for multi-turn stimuli). SCHEMA_VERSION bumped to 3. There is no migration path — pre-existing SQLite databases must be deleted and re-ingested.
  • vally-server schema bump 3 → 5 (breaking): the outcomes table gained an end_reason TEXT column (trajectory end reason: completed or agent_timeout). Action required: delete your analytics database and re-ingest — the server will refuse to open a pre-version-5 database.
  • Removed the orphan-files skill linter and its exported OrphanFilesGrader. vally lint no longer reports orphaned references/ files — it had a high false-positive rate and no rigorous definition. (#599)
  • Removed the grader portability tier: the PortabilityTier type, the GraderMetadata.portability field, and GraderRegistry.getByPortability() are gone. Custom graders must drop the portability property.
  • max_duration is now enforced (behavior change): previously max_duration on a stimulus was parsed but not reliably applied as a hard abort. It now is. Any eval with a max_duration shorter than the agent typically needs will be aborted and fail where before it ran to the default 120 s timeout. Review all max_duration values before upgrading.
  • Removed the pairwise grader type; comparison is now a mode of the prompt judge, run via vally compare. Eval specs that declare type: pairwise fail validation with a migration error.
  • Public API (@microsoft/vally): the pairwise grading surface was removed and renamed. Removed: PairwiseGrader, GraderExecution, GraderBehavior.execution, and GraderInput.trajectoryB. Renamed: PAIRWISE_MAGNITUDE_SCORES → COMPARISON_MAGNITUDE_SCORES, PairwiseJudgeResponseSchema → ComparisonJudgeResponseSchema, PairwiseJudgeResponse → ComparisonJudgeResponse, PairwiseRubricResult → ComparisonRubricResult, PairwiseMagnitude → JudgeMagnitude. Also removed PairwiseGraderConfig (the comparison judge reuses PromptGraderConfig). New comparison contract types: GraderComparison, GraderComparisonInput, GraderComparisonResult, ComparisonEntry, ComparisonWinner, ComparisonMagnitude. A judge advertises comparison support by implementing the optional Grader.compare() method (replacing the old behavior.execution === "comparative" flag).
  • vally compare: --run-a / --run-b are now --baseline / --treatment, plus an experiment-directory form (vally compare <experiment-dir>). --eval-spec is no longer required.
  • vally-server: the outcomes table gained a baseline column (the experiment’s baseline variant, recorded per trial-result record). Part of the schema bump above; delete the analytics database and re-ingest.

Fixed

  • vally eval --workspace placement failures now report where the original workspace was preserved (or that a backup directory was orphaned) instead of failing silently.