Writing Custom Reporters

Vally ships built-in reporters that render console progress, markdown, JUnit XML, and JSONL. Custom reporters let you observe the same run lifecycle from your own code — to stream results to a dashboard, archive trajectories, push metrics, or build any other run-aware integration.

Reporters are the fifth plugin seam, alongside executors, graders, eval providers, and lint reporters. They follow the same loader, registry, and --<kind>-plugin conventions.

Interface

A reporter implements one or more lifecycle methods. Every method except onRunComplete and onTrialResult is optional, so a minimal reporter can implement just the events it cares about:

interface PlanReporter {
  onRunStart?(ctx: PlanRunStartContext): Promise<void>;
  onTrialStart?(item: TrialWorkItem): Promise<void>;
  onTrialResult(event: { item: TrialWorkItem; result: TrialResult }): Promise<void>;
  onTrialPhase?(itemId: string, phase: TrialPhase, detail?: TrialPhaseDetail): Promise<void>;
  onDiagnostics?(diagnostics: ScopedDiagnostic[]): Promise<void>;
  onEvalComplete?(summary: EvalSummary): Promise<void>;
  onRunComplete(
    summary: RunSummary,
    artifacts: RunArtifacts,
    status?: RunCompletionStatus,
  ): Promise<void>;
}

Method	When it fires
onRunStart	Once, before any trial. Carries run-level metadata (see below).
onTrialStart	(Optional) When a trial is admitted to the worker pool.
onTrialResult	At most once per planned trial (exactly once on a completed run), with its terminal result.
onTrialPhase	(Optional) Progress sub-events within a trial (execute, grade, …).
onDiagnostics	(Optional) Plan-time / run-level / per-trial warnings outside the result stream.
onEvalComplete	(Optional) When one (eval × model × variant) finishes.
onRunComplete	Once, after the last trial, with the run summary, artifact paths, and status.

Implementation

Here’s a complete, runnable reporter that collects results in memory and prints a one-line summary at the end:

src/collecting-reporter.ts

import type {
  PlanReporter,
  PlanRunStartContext,
  RunArtifacts,
  RunCompletionStatus,
  RunSummary,
  TrialResult,
  TrialWorkItem,
} from "@microsoft/vally";

export class CollectingReporter implements PlanReporter {
  private readonly trials: Array<{ id: string; passed: boolean }> = [];

  async onRunStart(ctx: PlanRunStartContext): Promise<void> {
    const evalNames = ctx.evals.map((e) => e.evalName).join(", ");
    console.error(`[collecting] run started: ${evalNames} (${ctx.totalItems} trials)`);
  }

  async onTrialResult(event: { item: TrialWorkItem; result: TrialResult }): Promise<void> {
    // Use a stable id for idempotent delivery; copy what you need now —
    // the trajectory may be reclaimed after the run.
    const id = event.item.id;
    const passed = event.result.status === "success" && (event.result.grade?.passed ?? true);
    this.trials.push({ id, passed });
  }

  async onRunComplete(
    summary: RunSummary,
    _artifacts: RunArtifacts,
    status: RunCompletionStatus = "completed",
  ): Promise<void> {
    const passed = this.trials.filter((t) => t.passed).length;
    console.error(`[collecting] run ${status}: ${passed}/${this.trials.length} trials passed`);
  }
}

This is enough to receive completed trial results and the terminal run event. Next, package and load the reporter; the production contracts later on this page are only needed for durable or distributed integrations.

Shipping as a plugin package

A reporter plugin exports a registerReporters function that receives the reporter registry and adds one or more reporters. Reporters are additive — registering several is normal, and there is no name-conflict check.

1. Create the package

   src/index.ts

   import type { ReporterRegistry } from "@microsoft/vally";
   import { CollectingReporter } from "./collecting-reporter.js";
   
   export function registerReporters(registry: ReporterRegistry): void {
     registry.register(new CollectingReporter());
   }

   Assuming your build writes src/index.ts to dist/index.js, configure the package so Vally can load that file by package name. type=module tells Node to treat emitted .js files as ES modules; main and exports identify the compiled file. Then install @microsoft/vally as a peer dependency:

   npm pkg set type=module main=dist/index.js exports=./dist/index.js
   npm install --save-peer @microsoft/vally

2. Use it from the CLI

   The --reporter-plugin flag is repeatable and accepts an npm package name or a local path:

   # npm package
   vally eval --reporter-plugin @myorg/vally-reporter-custom --eval-spec eval.yaml
   
   # local path (useful during development)
   vally eval --reporter-plugin ./dist/index.js --eval-spec eval.yaml
   
   # multiple reporters fan out in the order listed
   vally eval --reporter-plugin @myorg/a --reporter-plugin @myorg/b -e eval.yaml
   
   # works for experiment runs too
   vally experiment run experiment.yaml --reporter-plugin @myorg/vally-reporter-custom

The loader validates each reporter before the run starts. Missing onTrialResult or onRunComplete methods fail fast with an error that identifies the plugin. Repeating the same --reporter-plugin specifier registers it once and emits a warning.

How plugin specifiers resolve

• npm package specifiers (e.g. @myorg/vally-reporter-custom) resolve via normal Node module resolution.
• Local / relative path specifiers (e.g. ./dist/index.js) resolve relative to the directory of the input file, not your current working directory:
  • vally eval — relative to the --eval-spec file’s directory. When no --eval-spec is given, the base is the skill/working directory (--skill-dir, else --work-dir, else the current working directory).
  • vally experiment run — relative to the experiment file’s directory.

To avoid ambiguity, prefer an absolute path for local plugins.

Production considerations

The reporter above is complete. Read the remaining sections when your integration persists events, handles interrupted runs, or forwards data to another system.

Lifecycle and ordering

For a reporter loaded with --reporter-plugin, the CLI guarantees this order:

1. onRunStart fires first. Exactly once, strictly before any onTrialResult.

2. onTrialResult fires at most once per planned trial. Each callback contains a complete terminal result; any trajectory or grade it carries is immutable. Concurrent trials arrive in completion order, not plan order. A completed run delivers every planned trial; a failed or cancelled run may stop before some trials produce a result.

3. onRunComplete fires last. It fires exactly once after the final delivered trial, including failed and cancelled runs. In those cases, status describes the abort and summary may be partial.

Callbacks are serialized within one reporter, so your implementation does not need an internal lock to preserve this order. Optional callbacks may interleave with trial results, but always occur after onRunStart and before onRunComplete.

Trajectory memory is reclaimed after the run

Treat TrialResult (including result.trajectory) as immutable, and copy anything you need to retain during your onTrialResult call. After the run summarizes, the CLI may null out trajectory references to free memory; later access is undefined.

Run completion status

onRunComplete receives an optional trailing status: RunCompletionStatus describing the run’s terminal lifecycle state, independent of the pass/fail verdict and validity signals:

• summary.passed is the aggregate pass/fail verdict.
• summary.hadExecutionErrors indicates that one or more trials failed to execute.
• summary.hadGradingErrors indicates that one or more graders failed to produce a valid result.

status	Meaning
"completed"	Every planned trial produced a terminal result. Read the verdict from summary.
"failed"	The run aborted on an unrecoverable error. summary may be partial.
"cancelled"	The user interrupted the run (Ctrl+C / SIGTERM). summary may be partial.

The parameter is optional and trailing, so reporters that only implement onRunComplete(summary, artifacts) stay valid — treat an absent status as "completed".

Run-level metadata

onRunStart receives a PlanRunStartContext rich enough to open a run-level container with full provenance, not just counts:

interface PlanRunStartContext {
  runId?: string; // built-in orchestration supplies a stable ID; legacy lifecycle drivers may omit it
  evals: EvalPlanMetadata[]; // eval names, models, executor names, planned stimulus names + counts, runs
  totalItems: number; // total planned trials
  workers: number; // max concurrent in-flight trials
  source?: { name: string; version: string }; // producer identity (tool + version)
  graderNames?: string[]; // sorted union of grader names in this run
  tagFilter?: Record<string, string[]>; // active include tag filter, if any
  tagExclude?: Record<string, string[]>; // active exclude tag filter, if any
  metadata?: Record<string, ConfigMetadataValue>; // effective root + suite metadata
  suite?: {
    name: string;
    metadata?: Record<string, ConfigMetadataValue>;
  }; // selected named suite and its passive metadata
}

metadata contains the effective passive integration metadata for every run. Root-level .vally.yaml metadata supplies the defaults; when a named suite is selected, its metadata is shallow-merged over the root and replaces matching top-level keys.

suite is present when the run was started with a named suite. Its name identifies the selected suite, and its metadata contains the same effective merged metadata. The entire suite field is omitted for runs that were not started with a named suite.

These two fields report the active tag filter, and each behaves differently:

• tagFilter is present whenever any tag filter is active — including an exclusion-only run (--tag key!=value with no include), where it is an empty map {}. So don’t read a present tagFilter as “there are include conditions”; check Object.keys(tagFilter).length > 0 before rendering include criteria.
• tagExclude is present only when at least one exclusion is active, so an include-only run omits it.

Both fields are omitted entirely when no tag (or suite) filter is applied.

Per-trial payload

onTrialResult carries the full normalized trajectory plus the grade:

interface TrialResult {
  itemId: string; // matches TrialWorkItem.id
  durationMs: number;
  status: "success" | "error" | "skipped";
  trajectory: Trajectory | null; // null for a skipped trial or an executor error before output
  grade: StimulusGradeResult | null; // null when skipped, --skip-grade, or no graders
  error?: string; // error message when status is error
  skipReason?: string; // reason when status is skipped
  workspacePath?: string; // absolute path when the trial workspace was preserved
}

trajectory.events is the complete normalized event stream (user / assistant messages, tool calls and results, token usage, reasoning, skill activations, errors, …). grade extends GraderResult, exposing per-grader name, kind, passed, score, evidence, optional label, nested details, and structured metadata. The item carries stimulus, model, variant, and trial-bucketing fields.

Stable identifiers

Use event.item.id as the per-trial idempotency key. It is always present and unique within a run across (eval × model × variant × stimulus × trial).

event.result.trajectory?.id and event.result.trajectory?.metadata.sessionID are executor-owned identifiers for correlation, not trial idempotency. A resumed session may span several trials, and trajectory is null for skipped trials or executor errors before output.

Event ordering within a single trajectory (trajectory.events) is deterministic — events appear in the order the executor emitted them.

Error isolation

If a reporter callback throws or rejects, the CLI logs the error and continues the run and sibling reporters. At the end of the run, one warning reports the total callback errors. Reporter failures do not change the run’s exit code.

Next steps

• CLI reference: eval — --reporter-plugin flag
• CLI reference: experiment — --reporter-plugin flag
• Writing custom executors — the executor plugin system follows the same pattern
• Trajectory format — the data structure delivered to onTrialResult