Writing Custom Executors

By default, Vally runs evals locally using the built-in copilot-sdk executor. The framework also ships a built-in mock executor that returns a pass-through trajectory for testing graders without a live agent — use executor: mock in your eval config for grader-only testing. Custom executors let you run evals on different backends — remote sandboxes, alternative AI SDKs, or specialized testing harnesses.

Interface

Only name, execute(), and shutdown() are required. The remaining fields advertise optional capabilities or validate executor-specific configuration:

interface Executor {
  name: string;

  // Optional — when true, the pipeline may prepare a workspace once and copy it per trial
  supportsPreparedWorkspace?: boolean;

  // Optional — when true, the executor supports multi-turn stimuli (Stimulus.turns)
  // Must be set to opt in; runEval() rejects multi-turn stimuli on executors without this flag
  supportsMultiTurn?: boolean;

  // Required with a multi-turn turn-scoped static diff grader. The executor must
  // await options.onTurnComplete() after each started configured turn on an
  // execution path that returns a trajectory.
  supportsTurnCompletion?: boolean;

  // Optional — when true, the executor supports tool-call simulation
  supportsSimulation?: boolean;

  // Optional — when true, the executor can simulate configured MCP tool calls;
  // requires supportsSimulation because MCP overrides live under Stimulus.simulation
  supportsMcpSimulation?: boolean;

  // Optional — when true, the executor supports prompt attachments
  supportsAttachments?: boolean;

  // Optional — when true, the executor applies ExecutorOptions.env (agent-process
  // env vars from agent_environment.env). Without this flag, runEval() warns that any
  // agent_environment.env variables will have no effect rather than silently dropping them.
  supportsEnvVars?: boolean;

  // Optional — validate executor-specific config from `defaults.executor.config`
  // (opaque to core). Called once at plan time. Throw with an actionable message
  // on invalid input. If a spec supplies `config` for an executor that does NOT
  // implement this hook, planning fails closed for that eval.
  validateConfig?(config: unknown): void;

  execute(stimulus: Stimulus, options: ExecutorOptions): Promise<Trajectory>;
  shutdown(): Promise<void>;
}

Method	Purpose
name	Unique identifier used in defaults.executor in eval specs
execute()	Run a single stimulus and return a trajectory with events, metrics, and output
shutdown()	Clean up resources (connections, processes, sandboxes)
supportsPreparedWorkspace	(Optional) When true, the pipeline prepares the workspace once and copies it per trial
supportsMultiTurn	(Optional) When true, the executor handles Stimulus.turns — each prompt is sent sequentially on the same session. Without this flag, multi-turn stimuli fail fast with a clear error.
supportsTurnCompletion	(Optional) When true, the executor awaits onTurnComplete after every started configured turn. Required for multi-turn diff-contains, diff-not-contains, or diff-empty graders with turn:.
supportsSimulation	(Optional) When true, the executor handles Stimulus.simulation; otherwise those stimuli are rejected.
supportsMcpSimulation	(Optional) With supportsSimulation, enables simulation of configured MCP tool calls without invoking the real tool.
supportsAttachments	(Optional) When true, the executor handles Stimulus.attachments; otherwise those stimuli are rejected.
supportsEnvVars	(Optional) When true, the executor applies ExecutorOptions.env (agent-process env vars from agent_environment.env). Without this flag, the pipeline warns that any agent_environment.env variables will have no effect.
validateConfig()	(Optional) Validate defaults.executor.config (see Configuration). Called once at plan time; throw on invalid input. Executors without this hook reject any supplied config (fail-closed).

Implementation

This minimal executor wraps a hypothetical AI SDK and returns a Vally trajectory:

src/my-executor.ts

import type { Executor, ExecutorOptions, Stimulus, Trajectory } from "@microsoft/vally";
import { computeMetrics } from "@microsoft/vally";

export class MyExecutor implements Executor {
  name = "my-executor";

  async execute(stimulus: Stimulus, options: ExecutorOptions): Promise<Trajectory> {
    if (options.maxAgentDurationMs !== undefined) {
      throw new Error("my-executor does not support constraints.max_agent_duration");
    }
    if (options.sessionID !== undefined) {
      throw new Error("my-executor does not support session resumption");
    }

    const startedAt = new Date();

    // Call your AI backend
    const response = await myAiSdk.complete({
      prompt: stimulus.prompt,
      model: options.model,
      reasoningEffort: options.reasoningEffort,
      timeout: options.timeout,
      workDir: options.workDir,
    });

    const completedAt = new Date();
    const events = convertToTrajectoryEvents(response);
    const metrics = computeMetrics(events);

    return {
      id: crypto.randomUUID(),
      stimulus,
      events,
      output: response.text,
      workDir: options.workDir,
      metadata: {
        startedAt,
        completedAt,
        model: options.model ?? "unknown",
        executor: this.name,
        skillsLoaded: [],
        sessionID: crypto.randomUUID(),
      },
      metrics: {
        ...metrics,
        wallTimeMs: completedAt.getTime() - startedAt.getTime(),
      },
    };
  }

  async shutdown(): Promise<void> {
    // Clean up connections, pools, etc.
  }
}

ExecutorOptions

The pipeline passes these options to every execute() call:

interface ExecutorOptions {
  skills?: Skill[]; // Skills to load for this run
  timeout: number; // Timeout in milliseconds
  maxAgentDurationMs?: number; // Agent working-time limit in milliseconds
  workDir: string; // Working directory for the agent
  model?: string; // Model override
  reasoningEffort?: string; // Non-empty model-supported value; unset = model default
  executorConfig?: unknown; // Executor-specific config from defaults.executor.config (opaque; you cast/validate it)
  sessionID?: string; // Resume an existing session
  mcpServers?: Record<string, McpServerConfig>; // MCP servers to attach
  env?: Record<string, string>; // Agent-process env vars (from agent_environment.env; merged over process.env)
  onRawEvent?: (event: unknown) => void; // Callback for each raw SDK event during execution
  registerCleanup?: (cleanup: () => void | Promise<void>) => void; // Attempt-local post-grading cleanup
  onTurnComplete?: (completion: {
    turn: number;
    status: "completed" | "interrupted" | "unavailable";
    final?: boolean;
  }) => Promise<void>;
  sessionLog?: ExecutorSessionLogOptions; // Per-run scratch directory for authoritative session logs
  traceContext?: ExecutorTraceContext; // W3C trace context to propagate to traced runtimes
}

interface ExecutorTraceContext {
  traceparent?: string;
  tracestate?: string;
}

interface ExecutorSessionLogOptions {
  rootDir: string; // Local scratch directory for authoritative executor session logs
  sessionID?: string; // Optional stable session ID for executors that support caller-provided IDs
  // Optional durable dir (under the persisted report dir) for full per-trial
  // artifacts (e.g. OUTPUT_DIR/process-*.log); retained in the viewer.
  // Executors are responsible for creating it. When unset, behavior is unchanged.
  executorArtifactsDir?: string;
}

maxAgentDurationMs is the agent’s working-time budget. Supporting it means stopping and settling backend work at expiry, then returning the partial trajectory with endReason: "agent_timeout". onTurnComplete is host work and counts against neither this budget nor the hard execution deadline. Propagate traceContext when launching a traced runtime.

When provided, await onTurnComplete once per started turn before returning a trajectory. Use completed after normal completion, interrupted after a recoverable stop settles, and unavailable when no stable workspace boundary exists. Set final: true on the last started turn. Do not call it for turns that never started; a hard failure may reject without a final callback.

Configuration

Use the object form when your executor needs custom configuration:

defaults:
  executor:
    name: my-executor
    config: { anything: your-executor-defines }

• Vally passes config unchanged as ExecutorOptions.executorConfig.
• Implement validateConfig(config) to parse it at plan time and throw an actionable Error when invalid. Supplying config without this method fails planning instead of being ignored.
• vally lint validates config for built-in executors. Plugin executor config is validated by vally eval, because lint does not load executor plugins.

Shipping as a plugin package

Custom executors can be shipped in a separate npm package and loaded at runtime via --executor-plugin.

1. Create the package

   Your package exports a registerExecutors function that receives the executor registry:

   src/index.ts

   import type { ExecutorRegistry } from "@microsoft/vally";
   import { MyExecutor } from "./my-executor.js";
   
   export function registerExecutors(registry: ExecutorRegistry): void {
     registry.register(new MyExecutor());
   }

   Assuming your build writes src/index.ts to dist/index.js, configure the package so Vally can load that file by package name. type=module tells Node to treat emitted .js files as ES modules; main and exports identify the compiled file. Then install @microsoft/vally as a peer dependency:

   npm pkg set type=module main=dist/index.js exports=./dist/index.js
   npm install --save-peer @microsoft/vally

2. Use it from the CLI

   # npm package
   vally eval --executor-plugin @myorg/vally-executor-custom --eval-spec eval.yaml
   
   # local path (useful during development)
   vally eval --executor-plugin ./my-executor --eval-spec eval.yaml
   
   # multiple executor plugins — useful when a single invocation runs several
   # eval specs (e.g., via -e A.yaml -e B.yaml or a suite) that select
   # different executors via their `defaults.executor` field
   vally eval --executor-plugin @myorg/exec-a --executor-plugin @myorg/exec-b -e A.yaml -e B.yaml

3. Reference it in eval.yaml

   Use the executor’s name property in the eval spec:

   eval.yaml

   defaults:
     executor: my-executor
     model: gpt-5.5

How plugin specifiers resolve

• npm package names resolve from the project using Node module resolution.
• Local paths resolve from the first --eval-spec file’s directory, or from the skill/working directory when no eval spec is supplied.

Prefer an absolute path when the resolution base is not obvious.

Tip

Executor plugins are loaded before the run resolves executors, so vally eval will catch unknown executor names with a clear error message listing all available executors.

Lifecycle and resource management

• execute() may be called many times concurrently (controlled by --workers)
• The CLI calls shutdown() once for every executor loaded from a plugin and every executor used by the run during normal, failed, and interrupted cleanup, bounded by --shutdown-timeout. A plugin executor may be shut down without being selected or having execute() called.
• If your executor manages a pool of resources (connections, sandboxes), initialize lazily in execute() and clean up in shutdown()

Retryable failures

The executor decides whether a failed attempt is safe to retry. Mark retryable failures before throwing:

throw Object.assign(new Error("Agent service temporarily unavailable", { cause }), {
  retryable: true,
});

Vally reads only the retryable boolean on the error you throw.

Per-attempt resources needed by graders

Use options.registerCleanup for ephemeral resources that must survive execute(), such as an isolated agent home read by validators. Register immediately after allocation, before any operation that might fail:

if (!options.registerCleanup) {
  throw new Error("This executor requires a host with post-grading cleanup support");
}
const home = await mkdtemp(join(tmpdir(), "agent-home-"));
options.registerCleanup(() => rm(home, { recursive: true, force: true }));

Import mkdtemp and rm from node:fs/promises, and join / tmpdir from node:path / node:os, respectively.

The local backend awaits callbacks during per-trial cleanup after grading and workspace egress, including grading errors and skipped or absent grading. On execution failure, callbacks run before workspace teardown and before a retry. Each attempt has its own registry, including concurrent calls to the same executor. Callbacks run sequentially in reverse registration order, once per registration; one failure does not prevent later callbacks. Each callback has a 30-second deadline; on timeout the host warns and continues with remaining cleanup and workspace teardown. A timed-out callback is not canceled and may still finish later, so callbacks must tolerate resources already being removed. Callbacks must not block the event loop: an in-process timer cannot interrupt synchronous code. Failures and timeouts emit executor-cleanup-failed warnings without replacing execution or grading results. Registration after execute() settles is rejected.

Resources remain available through workspace export. Cleanup runs before workspace disposal, even when --workspace preserves the workspace or an export fails. Callbacks must not depend on the original workspace path surviving a successful export, since export may move it. Register only ephemeral resources: durable session logs and trajectory.artifactDir must remain available for export and offline re-grading. shutdown() is still appropriate for shared connections, not per-attempt state. Forced process termination cannot guarantee callbacks run.

The hook is optional because direct execute() callers and older hosts may not provide it. Require a supporting host (as above), or explicitly implement a run-wide shutdown() fallback; do not silently skip resource cleanup.

Callers using runEval() directly must await result.cleanup() in a finally after grading. That releases executor resources before temporary workspace data; on execution failure runEval() does this automatically. Repeated/concurrent cleanup calls share the same executor-resource cleanup completion. Pass EvalRunOptions.onDiagnostic to receive cleanup warnings; delivery has a 5-second deadline. Without a sink, or if it throws or times out, warnings go to console.warn so diagnostic delivery cannot hold up remaining cleanup.

Relocating backends must own registration and disposal on the worker where the executor and graders run, for example by using runEval() or LocalBackend there. Do not serialize callback functions in a trajectory, trial outcome, or remote request. Forward cleanup diagnostics to the host separately.

Backend results implement cleanup(options?: { preserveWorkspace?: boolean }): Promise<void>. The host calls cleanup({ preserveWorkspace: true }) after failed egress that requires retaining workspace or artifact recovery sources. Backends must honor this option, releasing only non-workspace resources in that case. Normal completion and retries call cleanup() without options. Repeated calls should share one completion and the first call’s preservation decision.

Backend migration: cleanup is no longer skipped after failed egress. Existing backends must handle preserveWorkspace before upgrading; ignoring it could delete recovery files. There is no capability flag or separate pre-egress phase. Direct runEval() callers can also use cleanup({ preserveWorkspace: true }) to release executor resources and managed scratch data while retaining the workspace.

Workspace location

Executors must write the agent’s files to the options.workDir the pipeline provides; that local directory is what file-based graders (file-exists, file-contains, run-command, etc.) read.

Running the agent on a different machine (a remote sandbox, container, or VM) is a Backend concern, not an executor one: implement a Backend that owns the remote workspace and transfers files back via the Backend egress SPI, rather than handling remoteness inside the executor.

Next steps

• Eval spec reference — defaults.executor field
• CLI reference: eval — --executor-plugin flag
• Writing custom graders — the grader plugin system follows the same pattern
• Trajectory format — the data structure your executor must return