Writing Custom Executors

By default, Vally runs evals locally using the built-in copilot-sdk executor. The framework also ships a built-in mock executor that returns a pass-through trajectory for testing graders without a live agent — use executor: mock in your eval config for grader-only testing. Custom executors let you run evals on different backends — remote sandboxes, alternative AI SDKs, or specialized testing harnesses.

The Executor interface

Every executor implements this interface:

interface Executor {
  name: string;

  // Optional — when true, the pipeline may prepare a workspace once and copy it per trial
  supportsPreparedWorkspace?: boolean;

  // Optional — when true, the executor supports multi-turn stimuli (Stimulus.turns)
  // Must be set to opt in; runEval() rejects multi-turn stimuli on executors without this flag
  supportsMultiTurn?: boolean;

  // Required with a multi-turn turn-scoped static diff grader. The executor must
  // await options.onTurnComplete() after each started configured turn on an
  // execution path that returns a trajectory.
  supportsTurnCompletion?: boolean;

  // Optional — when true, the executor applies ExecutorOptions.env (agent-process
  // env vars from environment.env). Without this flag, runEval() warns that any
  // environment.env variables will have no effect rather than silently dropping them.
  supportsEnvVars?: boolean;

  // Optional — validate executor-specific config from `defaults.executor.config`
  // (opaque to core). Called once at plan time. Throw with an actionable message
  // on invalid input. If a spec supplies `config` for an executor that does NOT
  // implement this hook, planning fails closed for that eval.
  validateConfig?(config: unknown): void;

  execute(stimulus: Stimulus, options: ExecutorOptions): Promise<Trajectory>;
  shutdown(): Promise<void>;
}

Method	Purpose
name	Unique identifier used in defaults.executor in eval specs
execute()	Run a single stimulus and return a trajectory with events, metrics, and output
shutdown()	Clean up resources (connections, processes, sandboxes)
supportsPreparedWorkspace	(Optional) When true, the pipeline prepares the workspace once and copies it per trial
supportsMultiTurn	(Optional) When true, the executor handles Stimulus.turns — each prompt is sent sequentially on the same session. Without this flag, multi-turn stimuli fail fast with a clear error.
supportsTurnCompletion	(Optional) When true, the executor awaits onTurnComplete after every started configured turn. Required for multi-turn diff-contains, diff-not-contains, or diff-empty graders with turn:.
supportsEnvVars	(Optional) When true, the executor applies ExecutorOptions.env (agent-process env vars from environment.env). Without this flag, the pipeline warns that any environment.env variables will have no effect.
validateConfig()	(Optional) Validate defaults.executor.config (see Executor config). Called once at plan time; throw on invalid input. Executors without this hook reject any supplied config (fail-closed).

Executor config

An eval selects an executor with either a bare name or an object that adds executor-specific config:

defaults:
  executor:
    name: my-executor
    config: { anything: your-executor-defines }

The config payload is opaque to vally core — it travels unchanged to your executor as ExecutorOptions.executorConfig and is interpreted only by you. Implement validateConfig(config) to parse and validate it at plan time (throw an Error with an actionable message on bad input). Vally fails closed: if a spec supplies config but your executor doesn’t implement validateConfig, planning records a per-eval failure rather than silently ignoring it. vally lint runs the same check for the built-in executors (copilot-sdk, mock), so those authoring mistakes surface before a run; lint has no --executor-plugin flag today, so a spec naming a plugin executor is validated at plan time (vally eval) rather than at lint time.

ExecutorOptions

The pipeline passes these options to every execute() call:

interface ExecutorOptions {
  skills?: Skill[]; // Skills to load for this run
  timeout: number; // Timeout in milliseconds
  workDir: string; // Working directory for the agent
  model?: string; // Model override
  reasoningEffort?: "low" | "medium" | "high" | "xhigh"; // Agent reasoning effort; unset = model default
  executorConfig?: unknown; // Executor-specific config from defaults.executor.config (opaque; you cast/validate it)
  sessionID?: string; // Resume an existing session
  mcpServers?: Record<string, McpServerConfig>; // MCP servers to attach
  env?: Record<string, string>; // Agent-process env vars (from environment.env; merged over process.env)
  onRawEvent?: (event: unknown) => void; // Callback for each raw SDK event during execution
  onTurnComplete?: (completion: {
    turn: number;
    status: "completed" | "interrupted" | "unavailable";
    final?: boolean;
  }) => Promise<void>;
  sessionLog?: ExecutorSessionLogOptions; // Per-run directory for native session files (e.g. events.jsonl)
}

interface ExecutorSessionLogOptions {
  rootDir: string; // Local root directory for executor-native session state for this run
  sessionID?: string; // Optional stable session ID for executors that support caller-provided IDs
  // Optional durable dir (under the persisted report dir) for full per-trial
  // artifacts (e.g. OUTPUT_DIR/process-*.log); retained in the viewer.
  // Executors are responsible for creating it. When unset, behavior is unchanged.
  executorArtifactsDir?: string;
}

When onTurnComplete is provided, call and await it exactly once after each started configured turn on a path that returns a trajectory. Use completed after normal completion, interrupted only after the agent work has settled following a recoverable stop, and unavailable when no stable workspace boundary can be established. Set final: true when no later configured turn will start. Do not call it for turns that never started. A hard execution failure may reject without a final callback because it produces no trajectory to grade.

The callback performs host-side grading work (such as capturing a workspace diff), not agent work. Exclude its elapsed time from both your agent working-time budget and hard execution deadline.

Implementing an executor

Here’s a minimal executor that wraps a hypothetical AI SDK:

src/my-executor.ts

import type { Executor, ExecutorOptions, Stimulus, Trajectory } from "@microsoft/vally";
import { computeMetrics } from "@microsoft/vally";

export class MyExecutor implements Executor {
  name = "my-executor";

  async execute(stimulus: Stimulus, options: ExecutorOptions): Promise<Trajectory> {
    const startedAt = new Date();

    // Call your AI backend
    const response = await myAiSdk.complete({
      prompt: stimulus.prompt,
      model: options.model,
      timeout: options.timeout,
    });

    const completedAt = new Date();
    const events = convertToTrajectoryEvents(response);
    const metrics = computeMetrics(events);

    return {
      id: crypto.randomUUID(),
      stimulus,
      events,
      output: response.text,
      workDir: options.workDir,
      metadata: {
        startedAt,
        completedAt,
        model: options.model ?? "unknown",
        executor: this.name,
        skillsLoaded: [],
        sessionID: "",
      },
      metrics: {
        ...metrics,
        wallTimeMs: completedAt.getTime() - startedAt.getTime(),
      },
    };
  }

  async shutdown(): Promise<void> {
    // Clean up connections, pools, etc.
  }
}

Shipping as a plugin package

Custom executors can be shipped in a separate npm package and loaded at runtime via --executor-plugin.

1. Create the package

   Your package exports a registerExecutors function that receives the executor registry:

   src/index.ts

   import type { ExecutorRegistry } from "@microsoft/vally";
   import { MyExecutor } from "./my-executor.js";
   
   export function registerExecutors(registry: ExecutorRegistry): void {
     registry.register(new MyExecutor());
   }

   Set @microsoft/vally as a peer dependency:

   package.json

   {
     "name": "@myorg/vally-executor-custom",
     "main": "dist/index.js",
     "peerDependencies": {
       "@microsoft/vally": "^0.3.0"
     }
   }

2. Use it from the CLI

   # npm package
   vally eval --executor-plugin @myorg/vally-executor-custom --eval-spec eval.yaml
   
   # local path (useful during development)
   vally eval --executor-plugin ./my-executor --eval-spec eval.yaml
   
   # multiple executor plugins — useful when a single invocation runs several
   # eval specs (e.g., via -e A.yaml -e B.yaml or a suite) that select
   # different executors via their `defaults.executor` field
   vally eval --executor-plugin @myorg/exec-a --executor-plugin @myorg/exec-b -e A.yaml -e B.yaml

3. Reference it in eval.yaml

   Use the executor’s name property in the eval spec:

   eval.yaml

   defaults:
     executor: my-executor
     model: gpt-5.5

Tip

Executor plugins are loaded before the run resolves executors, so vally eval will catch unknown executor names with a clear error message listing all available executors.

Lifecycle and resource management

• execute() may be called many times concurrently (controlled by --workers)
• shutdown() is called once when the eval run completes, even if errors occurred
• If your executor manages a pool of resources (connections, sandboxes), initialize lazily in execute() and clean up in shutdown()
• Register a signal handler for SIGINT/SIGTERM if your resources are expensive to leak

Workspace location

Executors must write the agent’s files to the options.workDir the pipeline provides; that local directory is what file-based graders (file-exists, file-contains, run-command, etc.) read.

Running the agent on a different machine (a remote sandbox, container, or VM) is a Backend concern, not an executor one: implement a Backend that owns the remote workspace and transfers files back via the Backend egress SPI, rather than handling remoteness inside the executor.

Next steps

• Eval spec reference — defaults.executor field
• CLI reference: eval — --executor-plugin flag
• Writing custom graders — the grader plugin system follows the same pattern
• Trajectory format — the data structure your executor must return