Results API

The vally serve command exposes a REST API for querying eval results. The API is designed to be consumed by agents, scripts, and CI pipelines as well as the built-in dashboard.

Start a server:

vally serve ./vally-results

Conventions

Errors return { error: { code, message } } where code is a stable machine-readable string like not_found or invalid_params.

Pagination uses { page: { nextCursor, limit, hasMore, total } }. Pass ?cursor=<nextCursor>&limit=100 to page through results. Maximum limit is 1000.

Outcome status is a constrained enum: success, error, or skipped.

Metric definitions are available at GET /api/metrics. Each metric includes its id, unit (ratio, ms, count, tokens), direction (higher_is_better, lower_is_better, or neutral), and aggregation type (rate, mean, sum, p50, p95). Agents can use these to interpret values without external documentation.

Endpoints

Runs

`GET /api/runs`

List all runs. Returns { items: RunSummary[], page }.

Each RunSummary includes id, source, models, stimulusCount, outcomeCount, and passRate.

`GET /api/runs/:id`

Run detail. Adds stimuli (list of stimulus names) and config (eval configuration) to the run summary.

`GET /api/runs/:id/outcomes`

Outcomes scoped to this run. Same response shape as GET /api/outcomes with an implicit runId filter.

`GET /api/runs/:id/matrix?metric=pass_rate`

Score matrix for a run. Returns { rows, columns, cells, metric } where rows are stimulus names, columns are model names, and each cell has a value, outcomeCount, and optional flags.

The metric query parameter selects which metric to compute. Defaults to pass_rate. Use GET /api/metrics to discover available metrics.

`GET /api/runs/:id/ranking`

Model leaderboard. Returns { models: ModelRank[] } sorted by pass rate (descending), then duration (ascending). Each entry includes rank, passRate, meanDurationMs, meanTurns, meanToolCalls, token totals, and outcomeCount.

`GET /api/runs/:id/graders`

Grader verdict and error analysis. Returns { graders: GraderStats[] } sorted by pass rate (ascending, worst first). Each entry includes totalCount, passCount, failCount, errorCount, passRate, meanScore, and up to three strings in each of sampleFailureEvidence and sampleErrorEvidence. failCount contains valid verdicts that failed; errorCount contains invocations that produced no valid verdict. passRate is calculated over all invocations, so grader errors count as not passed. meanScore uses valid verdicts only because grader-error scores are not valid measurements; it is null when no invocation produced a valid score.

`GET /api/runs/:id/compare?modelA=...&modelB=...`

Within-run model comparison. Returns per-stimulus pass/fail and scores for two models, plus a summary with agreementRate and divergentCount.

`GET /api/runs/:id/tools`

Tool usage analysis. Returns { tools: ToolStats[] } sorted by call count. Each entry includes callCount, successRate, outcomePassRate (pass rate of outcomes that used this tool), and models (which models used it).

Outcomes

`GET /api/outcomes`

List outcomes with filters. Supports query parameters:

Parameter	Description
`runId`	Filter by run
`model`	Filter by model name
`stimulus`	Filter by stimulus name
`status`	Filter by status (success/error/skipped)
`passed`	Filter by pass/fail (true/false)
`grader`	Filter by grader type or name (see below)
`limit`	Page size (max 1000)
`cursor`	Pagination cursor

Returns { items: OutcomeSummary[], page }. Each summary includes the outcome id, stimulusName, model, trialIndex, status, passed, score, and all metric values (duration, turns, tool calls, errors, tokens). It also includes graderSummary, an array of { name, graderType?, status, passed, score } for each grader. Grader status is success when the grader produced a valid verdict and error when it did not; use passed only when status is success.

Server outcome summaries always carry an explicit grader status. Built-in ingestion normalizes results from older JSONL that omitted this field to success. Custom DataSource implementations must perform the same normalization before returning an OutcomeSummary.

name is the grader’s display name. When the eval author set one explicitly it is a stable identifier — a slug, unique across the eval file — and safe to group or trend on. Otherwise it is derived from the grader’s config (output-matches /Cosmos/), so it changes when that config is edited. graderType is the stable registered type (output-matches); group on it when you want every instance of a type together, or when the graders you are trending are unnamed. It is absent for outcomes ingested from JSONL written before graders reported their type.

Two caveats when grouping on name. graderSummary cannot tell you whether a name was authored or derived — only the full outcome detail below carries that distinction. And name uniqueness is enforced per eval file, so a run spanning several eval files can contain two unrelated graders with the same name.

grader= matches either field, so a filter on a bare type still selects every instance of it and older rows stay reachable by name. One sharp edge follows from the union: a grader name equal to some other grader’s type matches both. vally lint rejects that name only when the colliding type belongs to a different grader in the same eval file, so the edge does not disappear.

`GET /api/outcomes/:id`

Full outcome detail. Extends OutcomeSummary with graderResults (full GraderResult objects with evidence strings, each also carrying the optional graderType and, for graders the eval author named, configuredName) and error (if the outcome errored). configuredName is the authored stable identifier; its presence is how you tell an authored name from a config-derived one.

`GET /api/outcomes/:id/trajectory`

Raw trajectory for an outcome. Returns the full Trajectory object with all events (tool calls, tool results, messages, errors, token usage).

`GET /api/outcomes/:id/tools`

Tool call events for an outcome. Returns { items: ToolEvent[], total } with per-call details: toolName, success, argsSizeBytes, resultSizeBytes.

argsSizeBytes is the UTF-8 byte length of the argument value’s JSON serialization. Explicit null is measured as null (4 bytes), while an omitted arguments field uses the legacy empty-object representation {} (2 bytes). Older Vally versions normalized explicit null to {}, so argument-size values for those calls are not directly comparable across ingests made before and after non-object argument support.

Cross-run comparison

`GET /api/compare?runs=id1,id2,id3`

Compare outcomes across multiple runs. The comparison unit is a subject: a (run, model) pair. A single multi-model run produces multiple subjects.

Returns:

subjects with subjectId, runId, model, label, and outcomeCount
coverage showing how many stimuli overlap between subjects (intersectionCount, unionCount, and per-subject matched/total)
rows, one per stimulus, each containing a cells map from subjectId to a cell with status (ok/failed/error/missing), sampleSize, and aggregated metrics (passRate, meanDurationMs, meanTurns, meanToolCalls, errorCount, token averages)

Agent usage patterns

The API is structured as a funnel. Start broad, narrow to the problem:

GET /api/runs                            → pick the latest run
GET /api/runs/:id/ranking                → identify the worst model
GET /api/outcomes?model=...&passed=false  → find its failures
GET /api/outcomes/:id                    → read the grader evidence

For regression detection:

GET /api/compare?runs=<new>,<old>        → compare two runs
→ look for cells where status changed from "ok" to "failed"
→ coverage.intersectionCount tells you how comparable the runs are

For tool debugging:

GET /api/runs/:id/tools                  → find tools with low successRate
GET /api/outcomes?runId=...&limit=100    → check outcomes that used that tool