CLI: init

Usage

vally init [dir] [options]

Description

vally init scaffolds a starter eval.yaml so you don’t have to write your first spec from scratch. The generated eval asks the agent to write a hello.json file and grades it with deterministic graders (file-exists and file-matches), so grading needs no judge model or extra runtime dependencies; only vally eval itself needs an executor (the built-in one uses the Copilot SDK). It also includes commented-out “level-up” graders (running a command to parse the JSON, and an LLM judge with a rubric), an environment block for loading a skill or MCP server, constraints, and a pointer to experiments.

By default it writes to evals/hello/eval.yaml. Pass a directory to write somewhere else.

Arguments

Argument	Required	Default	Description
dir	No	evals/hello	Directory to create the sample eval in.

Options

Flag	Type	Default	Description
--force	boolean	false	Overwrite an existing eval.yaml in the target directory.

Behavior

• Creates the target directory (and any missing parents) if needed.
• Refuses to overwrite an existing eval.yaml unless --force is passed, so you never lose work by accident.
• Prints the next commands to run after scaffolding.

Exit codes

Code	Meaning
0	The sample eval was written.
1	The target eval.yaml already exists (without --force) or an error occurred.

Example

# Scaffold evals/hello/eval.yaml
vally init

# Then validate and run it
vally lint -e evals/hello/eval.yaml
vally eval -e evals/hello/eval.yaml

Tip

Open the scaffolded spec and uncomment its OPTIONAL sections (LLM judge, rubric, constraints, experiment) to see more of the eval.yaml format.