Skip to content
Vally

Debugging Evals

Evals fail. That’s their job — to tell you something isn’t right. The failure can happen before a run starts, while the agent or graders are running, or after a completed run produces an unexpected verdict. This guide narrows those possibilities in that order.

Debugging workflow

Section titled “Debugging workflow”

Start with Step 1 and follow the transition at the end of each step. Stop once the evidence explains the failure, then use Step 4 to test the correction.

Step 1: Make the eval valid

Section titled “Step 1: Make the eval valid”

Start by running the eval. Before starting an agent, vally eval automatically validates the spec:

Terminal window
vally eval --eval-spec eval.yaml

If validation fails, use the diagnostic code to find the check in the validation diagnostic reference. Correct the reported field or path, then rerun the eval.

Once validation succeeds, eval starts the agent. If execution does not complete, continue to Step 2.

Step 2: Get the eval to complete

Section titled “Step 2: Get the eval to complete”

If the command stops before producing a useful result, match its leading message in the table below. Messages can come from an executor or model provider, so the remaining text may vary.

What you see Next action
Session was not created with authentication… Configure the executor using Runtime Setup, then rerun the command from the same shell.
Trial execution error: … timed out … Rerun with --verbose and inspect any available session logs to see whether the agent was progressing. Raise the cap only when needed; see eval execution options.

If the leading text is not listed, use --verbose, preserve the full message, and follow the executor or provider details it contains.

If the command completes but the result is unexpected, continue to Step 3.

Step 3: Find why the result was unexpected

Section titled “Step 3: Find why the result was unexpected”

A completed run contains the evidence needed to distinguish an agent failure from a grader or scoring problem. Inspect progressively more detail until the result makes sense.

1. Read the grader evidence. Every result explains why it passed or failed:

✘ file-exists add.test.js          No files matching 'add.test.js' found
✔ output-contains 'test'           'test' found in output
✔ output-matches /ActivitySource/  Output matches pattern /ActivitySource/

Use the displayed grader name to find its configuration in the eval. Set an explicit name when several graders are difficult to distinguish. If the evidence explains the result, continue to Step 4.

2. Reveal the agent output. If the evidence is not enough, rerun with --verbose:

Terminal window
vally eval --eval-spec eval.yaml --verbose

This adds per-trial grader details. On multi-trial runs, agent output is shown only for failing trials; inspect results.jsonl for passing-trial output. If that explains the result, continue to Step 4.

3. Inspect the saved records. If verbose output is still insufficient, open results.jsonl and the per-trial session logs in the timestamped output directory:

Useful fields to inspect in results.jsonl:

Symptom What to check in trajectory
Grader says file doesn’t exist Check tool_call events — did the agent actually write the file?
Unexpected output Check assistant_message events — what did the agent actually say?
Timeout Check wallTimeMs in metrics — is the timeout too short?
No tool calls Check toolCallCount — maybe the agent solved it conversationally
Errors Filter for error events — network issues, permission problems, etc.

See eval trajectory output for the complete file layout and session-log semantics. Once the evidence explains the result, continue to Step 4.

Step 4: Verify the fix

Section titled “Step 4: Verify the fix”

Avoid another agent run when the trajectory already contains the evidence your revised graders need. Re-run only when the fix changes agent inputs, execution settings, or workspace behavior.

If the correction changes only graders or scoring, use grade to test it against the saved trajectory without another agent run:

Terminal window
# Re-grade saved outcomes
vally grade --eval-spec eval.yaml < path/to/results.jsonl

This is much faster than re-running the agent. Use it to:

  • Adjust grader config (change the substring, loosen the pattern)
  • Add or remove graders
  • Tweak scoring weights and thresholds

If the correction depends on generated files or workspace behavior, rerun with --workspace to keep the agent’s working directory:

Terminal window
vally eval \
  --eval-spec eval.yaml \
  --workspace ./debug-workspace

After the run, assign the workspace path Vally prints to WORKSPACE, then inspect it:

Terminal window
ls -la "$WORKSPACE"
cat "$WORKSPACE/add.test.js"

After making the correction, repeat the eval or grade command that exposed the failure.

Troubleshoot by symptom

Section titled “Troubleshoot by symptom”

Find the symptom in the first column. If none matches, return to Step 3 and inspect the saved evidence before changing the eval.

Symptom Check and next action
A file grader says a file is missing Compare the path in tool_call events with the grader’s glob and preserved workspace. Correct the path or glob; see file-exists.
An output grader fails on text that is present Check case_sensitive and the selected turn. Correct the casing or turn; see output-contains.
Trials alternate between pass and fail Compare per-trial results and pass@k / pass^k. Increase runs to measure reliability; see scoring functions.
Every grader passes but the aggregate score fails Check grader scores, scoring.weights, and scoring.threshold; recalculate using the scoring rules.
A prompt grader gives unexpectedly low scores Rerun with --verbose, inspect per-criterion reasoning, then adjust the rubric, scale, or model one at a time; see prompt.
LLM judge failed: … or LLM panel judge failed: … Check judge authentication and model availability; see prompt or panel.
A comparison becomes a tie after position swap Compare the reasoning from both response orders. Treat disagreement as unresolved; see comparison behavior.
Judge retries are exhausted Inspect the comparison failure diagnostics before choosing a remedy; see LLM judge failures.
Local and CI results differ Compare explicit model, environment, skills, MCP, and EVALUATE_USE_HOST_COPILOT_HOME settings; see Runtime Setup.

When a correction changes only graders or scoring, use the Step 4 re-grading path instead of running the agent again.

LLM judge failures

Section titled “LLM judge failures”

Inspect the error in the grader’s evidence before choosing a remedy. Ordinary prompt grading reports a failed LLM call without retrying it. For comparisons, the output JSONL records failure.errorClass, failure.retryable, failure.attempts, and failure.message. An error class such as "timeout" does not authorize retries: only explicitly retryable failures are retried. The built-in Copilot LLM client marks recognized SDK timeout, rate-limit, and connection failures retryable; the comparison runner consumes that decision. Retry exhaustion alone does not identify the cause.

If the error indicates API rate limiting:

  • Reduce --workers to lower concurrent API calls
  • Use --judge-model to switch to a model with available capacity
  • Separate grading from execution: eval --skip-grade, then grade at a lower concurrency

Next steps

Section titled “Next steps”