Writing Custom Reporters

Vally ships built-in reporters that render console progress, markdown, JUnit XML, and JSONL. Custom reporters let you observe the same run lifecycle from your own code — to stream results to a dashboard, archive trajectories, push metrics, or build any other run-aware integration.

Reporters are the fifth plugin seam, alongside executors, graders, eval providers, and lint reporters. They follow the same loader, registry, and --<kind>-plugin conventions.

The PlanReporter interface

A reporter implements one or more lifecycle methods. Every method except onRunComplete and onTrialResult is optional, so a minimal reporter can implement just the events it cares about:

interface PlanReporter {
  onRunStart?(ctx: PlanRunStartContext): Promise<void>;
  onTrialStart?(item: TrialWorkItem): Promise<void>;
  onTrialResult(event: { item: TrialWorkItem; result: TrialResult }): Promise<void>;
  onTrialPhase?(itemId: string, phase: TrialPhase, detail?: TrialPhaseDetail): Promise<void>;
  onDiagnostics?(diagnostics: ScopedDiagnostic[]): Promise<void>;
  onEvalComplete?(summary: EvalSummary): Promise<void>;
  onRunComplete(
    summary: RunSummary,
    artifacts: RunArtifacts,
    status?: RunCompletionStatus,
  ): Promise<void>;
}

Method	When it fires
onRunStart	Once, before any trial. Carries run-level metadata (see below).
onTrialStart	(Optional) When a trial is admitted to the worker pool.
onTrialResult	At most once per planned trial (exactly once on a completed run), with the complete trajectory + grade.
onTrialPhase	(Optional) Progress sub-events within a trial (execute, grade, …).
onDiagnostics	(Optional) Plan-time / run-level / per-trial warnings outside the result stream.
onEvalComplete	(Optional) When one (eval × model × variant) finishes.
onRunComplete	Once, after the last trial, with the run summary, artifact paths, and status.

Lifecycle, ordering, and timing contract

The CLI guarantees the following ordering for every reporter — built-in or plugin-loaded — except for the terminal-delivery clause in step 3, which is scoped to plugin-loaded reporters:

1. onRunStart fires first. Exactly once, strictly before any onTrialResult.

2. onTrialResult fires at most once per planned trial. Ordering across concurrent trials is not guaranteed — trials complete in pool-finish order, not plan order (the pool runs up to --workers trials at a time). The stronger guarantee — exactly once per planned trial — holds only when the run completes (status === "completed"); on a failed or cancelled run, trials that never reached a terminal result before shutdown are simply never delivered. However, each individual onTrialResult delivers a complete, immutable trajectory and grade for that one trial; you never observe a partially-populated trial.

3. onRunComplete fires last. Strictly after the last onTrialResult. On a completed run it fires exactly once for every reporter. On a failed or cancelled run, only reporters loaded via --reporter-plugin are guaranteed a terminal onRunComplete (carrying the abort status and a possibly-partial summary). The built-in file reporters (markdown, and JUnit when --junit is set) are also finalized with a direct terminal onRunComplete on abort so their report files are still written, then immediately sealed from further events. The built-in console reporter is the sole exception: it is intentionally skipped on abort, to avoid rendering a partial summary.

Within a single reporter, the composite serializes callbacks (your reporter advances down its own queue), so you observe your own events in the order above without needing an internal lock. The optional callbacks interleave with onTrialResult and always fire after onRunStart (they never precede it). Every reporter that receives a terminal onRunComplete — every reporter on a completed run, and on a failed/cancelled run both the plugin-loaded reporters and the finalized built-in file reporters (see point 3) — sees those optional callbacks strictly before it, closing the onRunStart … onRunComplete envelope: the file reporters are sealed the instant they are finalized, so the late onDiagnostics the CLI may emit afterward (e.g. an OTel trace-location notice) never reaches them. The only reporter that observes such late onDiagnostics with no closing onRunComplete is the console reporter, which is not finalized on abort.

Trajectory memory is reclaimed after the run

Treat TrialResult (including result.trajectory) as immutable, and copy anything you need to retain during your onTrialResult call. After the run summarizes, the CLI may null out trajectory references to free memory; later access is undefined.

Run completion status

onRunComplete receives an optional trailing status: RunCompletionStatus describing the run’s terminal lifecycle state, independent of the pass/fail verdict (which lives in summary.passed / summary.hadExecutionErrors):

status	Meaning
"completed"	Every planned trial ran to its natural end. Read the verdict from summary.
"failed"	The run aborted on an unrecoverable error. summary may be partial.
"cancelled"	The user interrupted the run (Ctrl+C / SIGTERM). summary may be partial.

The parameter is optional and trailing, so reporters that only implement onRunComplete(summary, artifacts) stay valid — treat an absent status as "completed".

Run-level metadata

onRunStart receives a PlanRunStartContext rich enough to open a run-level container with full provenance, not just counts:

interface PlanRunStartContext {
  evals: EvalPlanMetadata[]; // eval names, models, executor names, planned stimulus names + counts, runs
  totalItems: number; // total planned trials
  workers: number; // max concurrent in-flight trials
  source?: { name: string; version: string }; // producer identity (tool + version)
  graderNames?: string[]; // sorted union of grader names in this run
  tagFilter?: Record<string, string[]>; // active tag filter, if any
}

Per-trial payload

onTrialResult carries the full normalized trajectory plus the grade:

interface TrialResult {
  itemId: string; // matches TrialWorkItem.id
  durationMs: number;
  status: "success" | "error";
  trajectory: Trajectory | null; // full event stream; null only if the executor errored first
  grade: StimulusGradeResult | null; // per-grader breakdown; null with --skip-grade or no graders
  error?: string;
}

trajectory.events is the complete normalized event stream (user / assistant messages, tool calls and results, token usage, reasoning, skill activations, errors, …). grade extends GraderResult, exposing per-grader name, kind, passed, score, evidence, optional label, nested details, and structured metadata. The item carries stimulus, model, variant, and trial-bucketing fields.

Stable identifiers (idempotency / at-least-once delivery)

Retry-safe consumers need a stable idempotency key per trial. Each trial exposes two:

• event.item.id — the TrialWorkItem id, globally unique within a run across (eval × model × variant × stimulus × trial). Always present.
• event.result.trajectory?.id and event.result.trajectory?.metadata.sessionID — the executor-level trajectory / session id. trajectory is null only when the executor errored before producing one (result.status === "error"); use item.id as the fallback key.

Event ordering within a single trajectory (trajectory.events) is deterministic — events appear in the order the executor emitted them.

Error isolation

Every reporter runs inside a composite that isolates failures: a reporter that throws or returns a rejected promise is logged to stderr and skipped for that one event. A failing reporter never fails the run and never blocks sibling reporters. All lifecycle methods may be async.

Because those runtime errors are swallowed to preserve isolation, the CLI keeps them visible: each throw is logged with the reporter’s stable name (for plugin reporters, the --reporter-plugin specifier — not the unhelpful Object), and at the end of the run a single summary warning reports how many reporter callback errors occurred. The run’s exit code is unaffected.

A malformed plugin is different from a runtime throw. A reporter that does not implement the required onTrialResult and onRunComplete methods fails fast at load with a clear error naming the specifier and the missing method, rather than throwing on every callback during the run. The optional lifecycle methods (onRunStart, onTrialStart, onTrialPhase, onDiagnostics, onEvalComplete) may be omitted freely.

Passing the same --reporter-plugin specifier more than once registers it only once: the duplicate is dropped with a warning so a reporter’s side effects are not silently doubled.

Implementing a reporter

Here’s a complete, runnable reporter that collects results in memory and prints a one-line summary at the end:

src/collecting-reporter.ts

import type {
  PlanReporter,
  PlanRunStartContext,
  RunArtifacts,
  RunCompletionStatus,
  RunSummary,
  TrialResult,
  TrialWorkItem,
} from "@microsoft/vally";

export class CollectingReporter implements PlanReporter {
  private readonly trials: Array<{ id: string; passed: boolean }> = [];

  async onRunStart(ctx: PlanRunStartContext): Promise<void> {
    const evalNames = ctx.evals.map((e) => e.evalName).join(", ");
    console.error(`[collecting] run started: ${evalNames} (${ctx.totalItems} trials)`);
  }

  async onTrialResult(event: { item: TrialWorkItem; result: TrialResult }): Promise<void> {
    // Use a stable id for idempotent delivery; copy what you need now —
    // the trajectory may be reclaimed after the run.
    const id = event.result.trajectory?.metadata.sessionID ?? event.item.id;
    this.trials.push({ id, passed: event.result.grade?.passed ?? false });
  }

  async onRunComplete(
    summary: RunSummary,
    _artifacts: RunArtifacts,
    status: RunCompletionStatus = "completed",
  ): Promise<void> {
    const passed = this.trials.filter((t) => t.passed).length;
    console.error(`[collecting] run ${status}: ${passed}/${this.trials.length} trials passed`);
  }
}

Shipping as a plugin package

A reporter plugin exports a registerReporters function that receives the reporter registry and adds one or more reporters. Reporters are additive — registering several is normal, and there is no name-conflict check.

1. Create the package

   src/index.ts

   import type { ReporterRegistry } from "@microsoft/vally";
   import { CollectingReporter } from "./collecting-reporter.js";
   
   export function registerReporters(registry: ReporterRegistry): void {
     registry.register(new CollectingReporter());
   }

   Set @microsoft/vally as a peer dependency:

   package.json

   {
     "name": "@myorg/vally-reporter-custom",
     "main": "dist/index.js",
     "peerDependencies": {
       "@microsoft/vally": "^0.6.0"
     }
   }

2. Use it from the CLI

   The --reporter-plugin flag is repeatable and accepts an npm package name or a local path:

   # npm package
   vally eval --reporter-plugin @myorg/vally-reporter-custom --eval-spec eval.yaml
   
   # local path (useful during development)
   vally eval --reporter-plugin ./dist/index.js --eval-spec eval.yaml
   
   # multiple reporters fan out in the order listed
   vally eval --reporter-plugin @myorg/a --reporter-plugin @myorg/b -e eval.yaml
   
   # works for experiment runs too
   vally experiment run experiment.yaml --reporter-plugin @myorg/vally-reporter-custom

How plugin specifiers resolve

• npm package specifiers (e.g. @myorg/vally-reporter-custom) resolve via normal Node module resolution.
• Local / relative path specifiers (e.g. ./dist/index.js) resolve relative to the directory of the input file, not your current working directory:
  • vally eval — relative to the --eval-spec file’s directory. When no --eval-spec is given, the base is the skill/working directory (--skill-dir, else --work-dir, else the current working directory).
  • vally experiment run — relative to the experiment file’s directory.

To avoid ambiguity, prefer an absolute path for local plugins.

Tip

Plugin reporters are layered onto the same composite as the built-in reporters, so they inherit its error isolation automatically — a bug in your reporter can’t fail the run.

Next steps

• CLI reference: eval — --reporter-plugin flag
• CLI reference: experiment — --reporter-plugin flag
• Writing custom executors — the executor plugin system follows the same pattern
• Trajectory format — the data structure delivered to onTrialResult