Skip to content
Vally

Run Your First Eval

This quickstart runs a full evaluation: scaffold a spec, execute an agent, capture a trajectory, and grade the results.

What you’ll do

Section titled “What you’ll do”
  1. Scaffold an eval.yaml with vally init
  2. Run the eval and inspect the trajectory
  3. Understand the scores

  1. Scaffold a starter eval

    vally init writes a starter spec so you don’t have to start from an empty file:

    Terminal window
    vally init

    This creates evals/hello/eval.yaml. It defines one stimulus (a prompt asking the agent to write a hello.json file) and grades the result with deterministic graders, which don’t need a judge model.

    evals/hello/eval.yaml
    name: hello
    description: A minimal example eval — the agent writes a small hello.json and we check it.
    type: capability
    

    defaults:
      timeout: 2m
    

    stimuli:
      - name: greeting-json
        prompt: |
          Create a file named hello.json that describes a greeting. It must be
          valid JSON with two fields:
    

            - "message": a short, friendly greeting
            - "language": the two-letter code "en"
        graders:
          - type: file-exists
            config:
              path: hello.json
          - type: file-matches
            config:
              path: hello.json
              pattern: '"language"\s*:\s*"en"'
          - type: file-matches
            config:
              path: hello.json
              pattern: '"message"\s*:\s*".+"'
    

    scoring:
      weights:
        file-exists: 0.34
        file-matches: 0.66
      threshold: 1.0

  2. Validate it

    Before running an agent, a fast static check catches config errors:

    Terminal window
    vally lint --eval-spec evals/hello/eval.yaml
    ✔ evals/hello/eval.yaml is valid

  3. Run the eval

    Terminal window
    vally eval \
      --eval-spec evals/hello/eval.yaml \
      --output-dir ./results \
      --verbose

    You’ll see output like:

    ━━━ hello · greeting-json ━━━
      Metrics
      ─────────────────────────────────────────
      Tokens        1,204
      Turns         2
      Tool calls    1
      Wall time     4.1s
      Errors        0
      Skills used   0
      Model         gpt-5.5
    

      Graders (3/3)
      ─────────────────────────────────────────
      ✔ file-exists  Files matching 'hello.json' found: hello.json
      ✔ file-matches #1  Pattern /"language"\s*:\s*"en"/ matched in hello.json
      ✔ file-matches #2  Pattern /"message"\s*:\s*".+"/ matched in hello.json
    

      All graders passed.
    

      Agent Output
      ─────────────────────────────────────────
      Created `hello.json` with a friendly greeting message and `"language": "en"`.
    

    Summary
    ───────────────────────────────────────────
    ✔ hello  score: 100.0% (threshold: 100.0%)
    

    Saved artifacts
    JSONL → ./results/2025-01-15T10-30-00/results.jsonl
    Markdown → ./results/2025-01-15T10-30-00/eval-results.md
    Session logs → ./results/2025-01-15T10-30-00
    OpenTelemetry traces → ./results/2025-01-15T10-30-00/otel-spans.jsonl

  4. Re-grade saved trajectories

    Each run’s results.jsonl contains one trial-result record per trial, with the full trajectory embedded inline. You can pipe it back through vally grade to re-score with different graders without re-running the (expensive) agent execution:

    Terminal window
    cat ./results/2025-01-15T10-30-00/results.jsonl | vally grade --eval-spec evals/hello/eval.yaml

Understanding the output

Section titled “Understanding the output”

Metrics

Section titled “Metrics”

Every eval run captures a trajectory — a record of everything the agent did:

Metric What it means
Tokens Total input + output tokens across all LLM calls
Turns Number of agent conversation turns
Tool calls How many tools the agent invoked
Wall time Real clock time for the run
Skills used How many skills were activated by the agent

Grader results

Section titled “Grader results”

Each grader produces a pass/fail with evidence explaining why:

  • ✔ file-exists Files matching 'hello.json' found: hello.json — the grader checked, file exists, passed.
  • ✘ file-matches Pattern /"language"\s*:\s*"en"/ not found in any file matching 'hello.json' — the grader checked, pattern missing, failed.

Scores

Section titled “Scores”

The final score is a weighted combination of grader results against a threshold. See Scoring for the math.

Next steps

Section titled “Next steps”