Writing Custom Graders

Vally ships with built-in graders, but real-world evals often need domain-specific checks. This guide walks through building a custom grader from scratch.

The Grader interface

Every grader implements this interface:

interface Grader {
  metadata: GraderMetadata;
  grade(input: GraderInput): Promise<GraderResult>;
  // Optional: derive a display name from this instance's config (see below).
  defaultName?(config: Record<string, unknown>): string;
  // Optional: implement to support head-to-head comparison (vally compare).
  compare?(input: GraderComparisonInput): Promise<GraderComparisonResult>;
  // Optional: declare that this instance's config asserts the ABSENCE of something
  // (see the oracle baseline / negative-control note below).
  assertsAbsence?(config: Record<string, unknown>): boolean;
}

interface GraderMetadata {
  name: string;
  description: string;
  behavior: GraderBehavior;
  determinism: "static" | "complex-static" | "slm" | "llm";
  reference: "reference-free" | "reference-based";
  temporalScope: "point-in-time" | "trajectory-level" | "cross-trajectory";
  costProfile: "free" | "low" | "medium" | "high";
}

interface GraderBehavior {
  requiresLlmClient?: boolean;
  requiresWorkspace?: boolean;
}

The grade method receives a GraderInput containing:

trajectory — the full event log from the agent run
stimulus — the prompt and config that produced it
config — grader-specific config from the eval spec

Reading files from the run

trajectory exposes three on-disk locations, kept distinct:

trajectory.workDir — the graded workspace the agent mutated.
trajectory.artifactDir — run outputs (present only when the run recorded one). Read outputs from here first, falling back to workDir; honor trajectory.artifactDirStrict, which requires the referenced path to exist under the artifact dir rather than silently falling back.
trajectory.assetsDir — setup inputs staged outside the workspace via dest_root: assets (present only when the run staged assets). These are agent-visible and writable — the agent receives the path as EVALUATE_ASSETS — so treat their contents as untrusted at grade time and never store grader logic there.

Both artifactDir and assetsDir are host paths that may be absent when a stored trajectory is re-graded on a different host, so guard against undefined and a missing directory. The shared helper resolveGraderReadRoot (@microsoft/vally/graders) implements the artifact-dir-first, workspace-fallback, strict-aware read used by the built-in file graders.

Instance names (optional)

A stimulus can configure several instances of your grader with different config. By default they all report under the bare type name, leaving reports to tell them apart by position. Implement defaultName to derive a name from an instance’s config:

import { describeGraderValue } from "@microsoft/vally/graders";

defaultName(config: Record<string, unknown>): string {
  const target = describeGraderValue(config.target); // sanitizes + truncates
  return target ? `${this.metadata.name} ${target}` : this.metadata.name;
}

These names reach console output, markdown, JUnit XML, and the analytics store, so:

Return a pure function of config. A name that varies between calls (a timestamp, a random id) makes the same grader report under a different name in every trial, fragmenting aggregation, JUnit property keys, and stored results.
Never derive it from config that can carry secrets. Prefer a discriminating key — a command, a path, a pattern — over one that may hold credentials, like an env block.

An eval author’s explicit name always wins, and two instances that still resolve to the same name get a #1/#2 suffix.

What the runtime does with your return value:

Return	Result
A non-empty string	Sanitized, truncated to at most 60 grapheme clusters (the ellipsis counts toward that), used as the name
A string that normalizes to empty	Treated as “no default” — the grader falls back to its own result name
A non-string (including a `Promise` from an `async` implementation)	Rejected as “no default”, with a `process.emitWarning`
A throw	Caught and warned; never fails the run

Control characters become spaces and whitespace runs collapse, since names render on one line. To opt out for a particular config, return this.metadata.name rather than undefined — the declared return type is string.

Comparison capability (optional)

A grader can support head-to-head comparison by implementing the optional compare() method — implementing it is what marks the grader as comparison-capable. It receives a GraderComparisonInput (a baseline and a treatment trajectory for the same stimulus), returns a signed, treatment-relative GraderComparisonResult, and is invoked by vally compare. The built-in prompt grader implements it.

Absence assertions (optional)

vally oracle --no-golden-input is a negative control: it withholds the golden solution and expects every grader to fail, proving the grader actually depends on the solution. A grader that still passes is flagged as trivially passing.

Some checks, though, legitimately pass with nothing to withhold — a tool-calls grader that only lists disallowed tools, or an output-not-contains check. Against the empty baseline they pass because there was genuinely nothing to do, not because they are broken. Implement assertsAbsence to tell the oracle that a given config is one of these: when it returns true and the grader passes on the baseline, that grader is reported N/A instead of counted as a false failure.

// tool-calls: only forbidding calls (a non-empty `disallowed` with no
// `required`/`sequence`/`parallel`) is an absence assertion — an empty
// trajectory satisfies it.
assertsAbsence(config: Record<string, unknown>): boolean {
  const cfg = config as {
    required?: unknown[];
    sequence?: unknown[];
    parallel?: unknown[];
    disallowed?: unknown[];
  };
  const isEmptyList = (v: unknown[] | undefined): boolean => v == null || v.length === 0;
  return (
    !isEmptyList(cfg.disallowed) &&
    isEmptyList(cfg.required) &&
    isEmptyList(cfg.sequence) &&
    isEmptyList(cfg.parallel)
  );
}

Return a pure function of config (like defaultName). Only implement it when a passing result on empty input is genuinely correct; graders whose passing on the baseline would be a real problem (e.g. file-not-exists, diff-not-contains) should omit it so the negative control still catches them.

Example: a “no-errors” grader

Let’s build a grader that checks whether the agent produced any errors during its run.

Define the grader class

import type { Grader, GraderMetadata, GraderInput, GraderResult } from "@microsoft/vally";

export class NoErrorsGrader implements Grader {
  metadata: GraderMetadata = {
    name: "no-errors",
    description: "Checks that the agent produced no error events",
    behavior: {},
    determinism: "static",
    costProfile: "free",
    reference: "reference-free",
    temporalScope: "trajectory-level",
  };

  async grade(input: GraderInput): Promise<GraderResult> {
    if (!input.trajectory) {
      throw new Error("Missing trajectory");
    }

    const errors = input.trajectory.events.filter((e) => e.type === "error");
    const passed = errors.length === 0;

    return {
      name: this.metadata.name,
      kind: "code",
      passed,
      score: passed ? 1 : 0,
      evidence: passed
        ? "No error events in trajectory"
        : `${errors.length} error(s): ${errors.map((e) => e.data.message).join(", ")}`,
      label: passed ? "correct" : "incorrect",
    };
  }
}

Register it

import { createGraderRegistry } from "@microsoft/vally";
import { NoErrorsGrader } from "./no-errors-grader.js";

const registry = createGraderRegistry();
registry.register(new NoErrorsGrader());

Use it in eval.yaml

stimuli:
  - name: test-case
    prompt: "Do something"
    graders:
      - type: no-errors

Example: a tool-count grader

A grader that checks the agent used a reasonable number of tool calls:

import type { Grader, GraderMetadata, GraderInput, GraderResult } from "@microsoft/vally";

interface Config {
  min?: number;
  max?: number;
}

export class ToolCountGrader implements Grader {
  metadata: GraderMetadata = {
    name: "tool-count",
    description: "Checks that tool call count is within expected range",
    behavior: {},
    determinism: "static",
    costProfile: "free",
    reference: "reference-free",
    temporalScope: "trajectory-level",
  };

  async grade(input: GraderInput): Promise<GraderResult> {
    if (!input.trajectory) throw new Error("Missing trajectory");

    const config = (input.config ?? {}) as Config;
    const count = input.trajectory.metrics.toolCallCount;
    const min = config.min ?? 0;
    const max = config.max ?? Infinity;
    const passed = count >= min && count <= max;

    return {
      name: this.metadata.name,
      kind: "code",
      passed,
      score: passed ? 1 : 0,
      evidence: `${count} tool calls (expected ${min}–${max === Infinity ? "∞" : max})`,
      label: passed ? "correct" : "incorrect",
    };
  }
}

Use in eval.yaml:

graders:
  - type: tool-count
    config:
      min: 1
      max: 10

Taxonomy guidelines

Choose taxonomy values honestly — they’re surfaced in reports and help eval authors decide whether to include your grader in fast inner-loop runs or reserve it for outer-loop evaluation:

If your grader…	Set determinism to…	Set cost to…
Does string/file operations only	`static`	`free` or `low`
Runs a subprocess or does I/O	`complex-static`	`low`
Calls an embedding/small model	`slm`	`medium`
Calls GPT-5.5 or similar	`llm`	`high`

Testing your grader

Write tests that exercise both passing and failing cases:

import { describe, it, expect } from "vitest";
import { NoErrorsGrader } from "./no-errors-grader.js";

describe("NoErrorsGrader", () => {
  const grader = new NoErrorsGrader();

  it("passes when no errors", async () => {
    const result = await grader.grade({
      trajectory: {
        id: "test-1",
        events: [
          {
            type: "tool_call",
            timestamp: new Date(),
            data: { toolName: "read_file", toolCallId: "1" },
          },
        ],
        metrics: { errorCount: 0 },
        output: "done",
        workDir: "/tmp",
      },
    });
    expect(result.passed).toBe(true);
    expect(result.score).toBe(1);
  });

  it("fails when errors exist", async () => {
    const result = await grader.grade({
      trajectory: {
        id: "test-2",
        events: [{ type: "error", timestamp: new Date(), data: { message: "timeout" } }],
        metrics: { errorCount: 1 },
        output: "",
        workDir: "/tmp",
      },
    });
    expect(result.passed).toBe(false);
    expect(result.evidence).toContain("timeout");
  });
});

Shipping as a plugin package

Custom graders can be shipped in a separate npm package and loaded at runtime via --grader-plugin. This lets teams share graders across repos without forking vally.

Create the package

Your package exports a registerGraders function that receives the grader registry:

import type { GraderRegistry } from "@microsoft/vally";
import { NoErrorsGrader } from "./no-errors-grader.js";

export function registerGraders(registry: GraderRegistry): void {
  registry.register(new NoErrorsGrader());
}

Set @microsoft/vally as a peer dependency in your package.json:

{
  "name": "@myorg/vally-grader-quality",
  "main": "dist/index.js",
  "peerDependencies": {
    "@microsoft/vally": "^0.2.0"
  }
}

Use it from the CLI

Pass the package name or path to any vally command:

# npm package
vally eval --grader-plugin @myorg/vally-grader-quality --eval-spec eval.yaml

# local path
vally eval --grader-plugin ./my-graders --eval-spec eval.yaml

# repeatable — load several grader plugins at once
vally eval --grader-plugin @myorg/vally-grader-quality --grader-plugin ./my-graders --eval-spec eval.yaml

# works with lint, grade, oracle, and export too
vally lint --eval-spec eval.yaml --grader-plugin ./my-graders

Reference plugin graders in eval.yaml

Plugin graders are referenced by name, just like built-ins:

graders:
  - type: no-errors
  - type: output-contains
    config:
      substring: "hello"

Next steps

Grader taxonomy — deep dive on taxonomy dimensions
Grader catalog — built-in grader examples
How it works — where graders fit in the pipeline