CLI: serve

Usage

vally serve [directory] [options]

Description

Start an HTTP server that provides a REST API and built-in web dashboard for exploring eval results.

There are three ways to use it:

# Serve from a directory (in-memory, no persistence)
vally serve ./results

# Serve from an existing database (no directory needed)
vally serve --store eval-history.db

# Ingest a directory into a database and serve
vally serve ./results --store eval-history.db

For programmatic use, @microsoft/vally-server exposes one database handle that owns the selected connection, ingestion, and query store:

import { openDatabase } from "@microsoft/vally-server";

const database = openDatabase("./eval-history.db");
try {
  await database.ingestDirectory("./results");
  const runs = await database.dataSource.listRuns();
  console.log(runs.items);
} finally {
  database.close();
}

Arguments

Argument	Description
[directory]	Path to eval output directory. Optional when using --store

Options

Flag	Type	Default	Description
--port <port>	number	3200	Port to listen on
--host <host>	string	127.0.0.1	Host to bind to. Use 0.0.0.0 to expose externally.
--cors	boolean	false	Enable CORS for cross-origin browser requests
--allowed-host <host>	string	—	Additional Host header value to accept (repeatable). See Security.
--store <path>	string	—	Persist data to a SQLite file (enables historical mode)

Exit codes

Code	Meaning
0	Server started successfully
1	Error (no runs found, bad port)

REST API

The server exposes a JSON API at /api/. Key endpoints:

Endpoint	Description
GET /api/runs	List all runs
GET /api/runs/:id	Run detail (stimuli, models, config)
GET /api/runs/:id/outcomes	Outcomes scoped to a run
GET /api/runs/:id/matrix	Score matrix (any metric)
GET /api/runs/:id/ranking	Model leaderboard
GET /api/runs/:id/graders	Grader failure analysis
GET /api/runs/:id/tools	Per-tool usage stats
GET /api/outcomes	List/filter outcomes across runs
GET /api/outcomes/:id	Full outcome with grader details
GET /api/outcomes/:id/trajectory	Raw trajectory events
GET /api/compare?runs=id1,id2	Cross-run comparison
GET /api/metrics	Self-describing metric definitions

All responses use standardized error envelopes ({ error: { code, message } }) and explicit pagination ({ page: { nextCursor, limit, hasMore, total } }).

Examples

# Basic usage — view results from an eval run
vally serve ./vally-results/

# Custom port
vally serve ./vally-results/ --port 8080

# Persistent database — keeps history across sessions
vally serve ./vally-results/ --store eval-history.db

# Query the API with curl
curl http://127.0.0.1:3200/api/runs
curl http://127.0.0.1:3200/api/runs/<run-id>/ranking

SQLite troubleshooting

No additional SQLite installation or configuration is required for serve or ingest. Both commands use Node’s built-in SQLite unless the optional better-sqlite3 package is installed where the CLI can resolve it.

SQLite experimental warning

On Node 22, node:sqlite may emit an ExperimentalWarning. This is not a command failure and requires no action.

Errors loading better-sqlite3

If an installed better-sqlite3 fails to load, Vally reports the error rather than switching backends. Repair the addon installation for your Node version and platform, or remove it if you do not need it so Vally can use built-in SQLite:

# From the local project where the addon was installed:
npm uninstall better-sqlite3

# For an addon installed in the CLI's global npm prefix:
npm uninstall -g better-sqlite3

Installing or removing the addon in a different project or npm prefix does not affect the CLI’s backend selection. Database-open errors, such as file permissions or an incompatible schema, must be resolved separately; changing backends does not fix them.

Security

• Binds to 127.0.0.1 by default

• CORS is disabled by default

• Internal errors return a generic message (no paths or SQL fragments are exposed)

• Host-header validation (DNS-rebinding protection). The server rejects any request whose Host header is missing or not allowed with 403 forbidden_host. This blocks DNS-rebinding attacks, where a malicious web page repoints its hostname to 127.0.0.1 to bypass CORS and read your eval data. A Host is allowed when it is:

  • a loopback name (localhost, 127.0.0.1, ::1),
  • the value passed to --host,
  • any IP literal (IPv4 or IPv6) — accessing the server directly by IP, including a LAN IP or a 0.0.0.0 bind, is inherently safe because rebinding requires a hostname the attacker can flip in DNS, or
  • a name allowlisted via --allowed-host <host> (repeatable), e.g. when fronting the server with a custom domain.

  # Allow a custom hostname (e.g. behind a reverse proxy)
  vally serve ./results --allowed-host eval.internal.example.com

  The /api/health endpoint is exempt so liveness probes can address the server by a Service DNS name; it returns no eval data.