Run Your First Eval

This is the third and final step to run your first eval.

Run these commands from the same shell where you configured the runtime. You’re ready to scaffold a spec, execute an agent, capture a trajectory, and grade the results.

You need Node.js >=22.14.0, the Vally CLI, and a GitHub account with an active Copilot subscription.

Scaffold a starter eval

vally init writes a starter spec so you don’t have to start from an empty file:
Terminal window
```
vally init
```
This creates evals/hello/eval.yaml.

Run the eval

vally eval \
  --eval-spec evals/hello/eval.yaml \
  --output-dir ./results \
  --verbose

vally eval validates the spec before running the agent.

A successful run ends with a passing summary and the saved artifact paths:

────────────────────────────────────────────────────────

Summary
  hello  ✓ 1/1 (score 100% ≥ 100%)

Saved  ./results/<timestamp>/

What `vally init` created

The generated spec defines one stimulus: a prompt asking the agent to write a hello.json file. Deterministic graders check the result without needing a judge model.

name: hello
description: A minimal example eval — the agent writes a small hello.json and we check it.
type: capability

defaults:
  timeout: 2m

stimuli:
  - name: greeting-json
    prompt: |
      Create a file named hello.json that describes a greeting. It must be
      valid JSON with two fields:

        - "message": a short, friendly greeting
        - "language": the two-letter code "en"
    graders:
      - type: file-exists
        config:
          path: hello.json
      - type: file-matches
        config:
          path: hello.json
          pattern: '"language"\s*:\s*"en"'
      - type: file-matches
        config:
          path: hello.json
          pattern: '"message"\s*:\s*".+"'

scoring:
  weights:
    file-exists: 0.34
    file-matches: 0.66
  threshold: 1.0

Understanding the output

As your eval runs, the terminal shows which workers are active and updates the summary as each stimulus finishes all its repetitions. Failures include grader evidence to help you understand what went wrong; use --verbose to see passing result cards and more failure detail.

At completion, unweighted evals show PASS, FAIL, or FLAKY. Weighted evals instead show how many stimuli passed and whether the score met the threshold. If evidence is truncated in the terminal, the saved results and session logs contain the full details.

Metrics

Every eval run captures a trajectory — a record of everything the agent did:

Metric	What it means
Tokens	Total input + output tokens across all LLM calls
Turns	Number of agent conversation turns
Tool calls	How many tools the agent invoked
Wall time	Real clock time for the run
Skills used	How many skills were activated by the agent

Grader results

Each grader produces a pass/fail with evidence explaining why:

✔ file-exists Files matching 'hello.json' found: hello.json — the grader checked, file exists, passed.
✘ file-matches Pattern /"language"\s*:\s*"en"/ not found in any file matching 'hello.json' — the grader checked, pattern missing, failed.

Scores

By default the verdict is all-or-nothing: the stimulus passes only if every grader passed. A score is also reported, but it is the mean of the grader scores, so a partial failure shows as a decimal rather than 0. See Scoring for how verdicts and scores differ, and how weights and threshold change them.

Optional: Re-grade saved trajectories

Each run’s results.jsonl contains one trial-result record per trial, with the full trajectory embedded inline. Pipe it back through vally grade to re-score with different graders without re-running the agent:

cat ./results/<timestamp>/results.jsonl | vally grade --eval-spec evals/hello/eval.yaml

What to try next

If you want to…	Start here
Add or improve behavior checks	Choose a grader
Test a multi-turn conversation	Write a multi-turn eval
Compare a model or configuration	Run your first experiment
Automate an eval quality gate	Add Vally to CI
Configure weights and thresholds	Understand scoring
Diagnose a validation or run failure	Debug an eval
Explore results across runs	Start the dashboard