Skip to content

Vally 0.3.0

Released on 2026-04-29.

Features

  • Add --threshold option to vally eval for overriding eval scoring thresholds during a command invocation.
  • Add optional name field to grader config for human-readable display names in reports.
  • Add vally eval --keep-executor-session-logs to preserve native executor events.jsonl files under --output-dir for session visualizers such as agentviz.
  • Remote executor workspace sync contract — finalizeWorkspace() hook on Executor interface enables remote executors to materialize workspace files locally before grading. workspaceStatus on Trajectory enables fail-closed safety for file-based graders. requiresWorkspace metadata on GraderBehavior enables dynamic workspace-dependency detection.
  • Add --known-domains option to vally lint for URL reference scanning (#157)
  • Add --allowed-external-deps option to vally lint for structural dependency scanning (#157)
  • Multi-model eval support (#140)
  • Grader plugin system — ship graders in separate packages (#141)
  • Grader improvements — regex matching, cross-platform normalization, better diagnostics (#137)
  • Eval-authoring meta-eval and document directory support in environment.files (#135)
  • Workspace caching for multi-trial runs (#132)
  • JUnit XML reporter for eval pipeline (#128)
  • Export command and exporter plugin system (#113)

Fixed

  • Inject environment.skills into executor and honor .vally.yaml paths.skills (#139, #138)
  • Flatten multi-trial JUnit XML for CI parser compatibility (#131)
  • Add actions:read permission for dorny/test-reporter (#130)
  • Fix bug when path to node contains spaces (#127)

Documentation

  • Clarify GitHub Packages installation guide (#136)
  • Document directory support in environment.files (#135)
  • Docs for program, skill-invocation and tool-call graders (#124)
  • Fix install instructions to be less destructive (#122)
  • Document export command and architecture (#120)
  • Common template for environment fields (vally-yaml and eval-spec) (#118)
  • Document git and mcpServers in JSON schema (#116)

CI

  • Fix fork-pr-status credentials (#126, #123)