CLI: export

Usage

vally export --eval-spec <path> [--eval-spec <path> ...] --exporter <specifier> [options] [-- key=value ...]

Description

Convert one or more eval specs to a format consumed by another evaluation tool. The export command validates each eval spec, resolves environments and stimuli, loads an exporter plugin, and invokes it to generate output files.

--eval-spec is repeatable. When a single spec is given, output is written directly to the output directory. When multiple specs are given, each spec’s output is written into its own subdirectory of the output directory, named after that spec’s (sanitized) name field.

For Harbor exports, both type: worktree and type: clone are materialized as a self-contained, pinned checkout baked into the task image, with the source remote and reflogs stripped. Harbor uses host Git configuration for authentication, including credential helpers such as Git Credential Manager, while overriding line-ending settings for deterministic LF checkouts. HTTP(S) clone URLs must not contain userinfo, query/fragment parameters, or backslashes, and SSH URLs must not contain passwords; SSH agents and GIT_ASKPASS are supported. Non-LFS custom filter declarations are rejected because their commands cannot be reproduced in the image. Worktree exports retain history reachable from the selected commit; clone exports honor their configured shallow depth. Large worktree histories therefore increase the exported task size. The checkout replaces the base image’s entire /app directory, so custom base images must keep required runtime files elsewhere. The Docker build may still require network access for the base image, runtime installation, or environment commands. Normal vally eval remains the source of truth for environment.git; Git submodules and Git LFS remain warnings because their contents are not materialized.

Options

Flag	Type	Required	Description
--eval-spec <path>	string	Yes	Path to an eval.yaml file. Repeatable.
--exporter <specifier>	string	Yes	Exporter plugin to load (npm package name or local path)
--grader-plugin <specifier>	string	No	Grader plugin to load (npm package name or local path). Prevents unknown-grader-type errors for plugin graders during validation. Repeatable.
--output-dir <dir>	string	No	Directory for output files (default: ./<exporter-name>/)
--tag <key=values>	string	No	Filter stimuli by tag (repeatable, e.g., --tag cost=free,low)
--param <key=value>	string	No	Set a param value (repeatable, e.g. --param MODEL=gpt-4o). Overrides eval param files and .vally.yaml.
--verbose	boolean	No	Show resolved eval schema and plugin details

Exporter-specific options

Options after -- are forwarded to the exporter plugin as key=value pairs.

Exporter resolution

The --exporter specifier is resolved differently depending on its shape:

Specifier type	Examples	Resolved from
Local path	./my-exporter, ../plugins/exp, /abs/path	Current working directory
Bare package name	@contoso/vally-exporter-foo, my-exporter	Eval project directory (dirname(eval.yaml))

When multiple --eval-spec values are given, a bare exporter specifier is resolved from the first spec’s project directory, so a bare exporter is expected to be a shared dependency across the specs.

Exit codes

Code	Meaning
0	Export completed successfully
1	Validation error, plugin load failure, or export error

Examples

Basic export

vally export --eval-spec eval.yaml --exporter @contoso/vally-exporter-foo

With output directory and tag filter

vally export --eval-spec ./evals/my-eval/eval.yaml \
  --exporter @contoso/vally-exporter-foo \
  --output-dir ./exported \
  --tag cost=free,low

Pass exporter-specific options

vally export --eval-spec eval.yaml \
  --exporter @contoso/exporter \
  -- baseImage=ubuntu:22.04 format=compact

Bulk export (multiple specs)

vally export \
  --eval-spec ./evals/login/eval.yaml \
  --eval-spec ./evals/checkout/eval.yaml \
  --exporter @contoso/vally-exporter-foo \
  --output-dir ./exported

Each spec is written into its own subdirectory of ./exported, named after the spec’s name field (sanitized) — for example ./exported/login/ and ./exported/checkout/. If two specs resolve to the same subdirectory name (compared case-insensitively, for cross-platform safety), the command reports an error and exits without writing anything.