Docker backend

Use the Docker backend when you want each eval trial to run in a fresh container instead of on the host, with grading separated from the agent run. Docker adds image and daemon setup overhead; use the default local backend when you don’t need container isolation.

The agent runs in a task project with its own workspace. Once that project stops, Vally captures workspace evidence and, if grading is requested, runs graders in a separate grade project. Graders receive a copy of the task workspace only when needed. The agent does not receive hidden grader inputs or judge credentials, and grading cannot change the workspace exported after the trial. Graders within a trial share an environment, so they are not isolated from one another; a grader that executes agent-authored files must treat them as untrusted.

Before you start

• Install Docker with Compose and make sure your user can access a running Docker daemon.
• Have an eval.yaml.
• For custom images, see Image requirements. The task image must include the executor you select; the grade image must include the graders you select.

Run an eval

vally eval --backend docker --eval-spec eval.yaml

By default, Vally builds one shared task/grade appliance locally and caches it as vally-task:<content-hash>. The grade image defaults to the effective task image. An uncached build needs network access.

To use separate task and grade images:

vally eval --backend docker \
  --backend-args task-image=my-task-image@sha256:... \
  --backend-args grade-image=my-grade-image@sha256:... \
  --eval-spec eval.yaml

Digest references are recommended. If you specify only task-image, Vally also uses it for grading. Task and grading containers remain separate even when they share an image. Pass --backend-args again for each additional setting, such as --backend-args task-env=MY_TOKEN to forward a named host variable to the task role. Forward grade credentials with grade-env, not task-env.

Image requirements

Each role image must:

• target Linux and declare a non-root USER;
• provide the exact host version of @microsoft/vally, including vally-docker-runner;
• provide a POSIX sh;
• not declare a nested VOLUME below a backend-managed mount (for example, /workspace/cache). Declaring the mount path itself is allowed.

The task image supplies its executor and task tools. The grade image supplies its graders and evidence tools. It must provide GNU coreutils, and needs Git for live trials that capture workspace evidence. The task image also needs Git when the eval uses a Git source.

Backend arguments

Pass arguments with repeatable --backend-args key=value options.

Key	Default	Description
task-image	locally built default image	Linux image for setup, task execution, and executor plugins. The default contains the installed @microsoft/vally package.
grade-image	effective task-image	Linux image for evidence, grading, and grader plugins.
task-network	default	Docker network mode or existing network name for the task container.
grade-network	default	Docker network mode or existing network name for the grading container.
task-env	-	Repeatable exact host environment variable name forwarded to the task container.
grade-env	-	Repeatable exact host environment variable name forwarded to the grading container.
compose-file	-	Optional Compose file for support services.

Plugins

Use --executor-plugin and --grader-plugin with either:

• a bare package installed on the host, such as @acme/vally-grader-policy;
• an explicit package directory, such as ./plugins/my-executor.

The package needs a name, version, and compatible, non-optional @microsoft/vally peer dependency. Package subpaths, individual files, and file: URLs are not supported. A bare package must be resolvable through a CommonJS-compatible entry; for import-only conditional exports, pass its directory instead. Vally installs each plugin inside the corresponding image, so that image needs npm. Plugins and their install scripts are trusted; npm installation does not inherit your host npm configuration, authentication, or proxy variables. Keep host executor-plugin packages available until the first trial has prepared the task role; grader-plugin packages are needed through initial backend preparation.

Export and cleanup

Vally exports the task workspace or selected artifacts before releasing trial resources. If export or cleanup fails, check the warning for a preserved host path or a docker-volume://<volume-name> recovery location. An abrupt host or daemon failure may also leave resources needing manual cleanup.