Writing Custom Graders

Vally ships with built-in graders, but real-world evals often need domain-specific checks. This guide walks through implementing, testing, and loading a custom grader. Only metadata and grade() are required; the other methods add specialized capabilities.

Interface

Every grader implements this interface:

interface Grader {
  metadata: GraderMetadata;
  grade(input: GraderInput): Promise<GraderResult>;
  // Optional: derive a display name from this instance's config (see below).
  defaultName?(config: Record<string, unknown>): string;
  // Optional: implement to support head-to-head comparison.
  compare?(input: GraderComparisonInput): Promise<GraderComparisonResult>;
  // Optional: declare that this instance's config asserts the ABSENCE of something
  // (see the oracle baseline / negative-control note below).
  assertsAbsence?(config: Record<string, unknown>): boolean;
}

interface GraderMetadata {
  name: string;
  description: string;
  behavior: GraderBehavior;
  determinism: "static" | "complex-static" | "slm" | "llm";
  reference: "reference-free" | "reference-based";
  temporalScope: "point-in-time" | "trajectory-level" | "cross-trajectory";
  costProfile: "free" | "low" | "medium" | "high";
}

interface GraderBehavior {
  requiresLlmClient?: boolean;
  requiresWorkspace?: boolean;
}

Implementation

Let’s build a grader that checks whether the agent produced any errors during its run.

1. Define the grader class

   no-errors-grader.ts

   import type { Grader, GraderMetadata, GraderInput, GraderResult } from "@microsoft/vally";
   
   interface NoErrorsConfig {
     ignoredTypes: string[];
   }
   
   function parseConfig(config: Record<string, unknown>): NoErrorsConfig {
     const ignoredTypes = config.ignoredTypes ?? [];
     if (!Array.isArray(ignoredTypes) || ignoredTypes.some((value) => typeof value !== "string")) {
       throw new Error("no-errors config.ignoredTypes must be an array of strings");
     }
     return { ignoredTypes };
   }
   
   export class NoErrorsGrader implements Grader {
     metadata: GraderMetadata = {
       name: "no-errors",
       description: "Checks that the agent produced no error events",
       behavior: {},
       determinism: "static",
       costProfile: "free",
       reference: "reference-free",
       temporalScope: "trajectory-level",
     };
   
     async grade(input: GraderInput): Promise<GraderResult> {
       if (!input.trajectory) {
         throw new Error("Missing trajectory");
       }
   
       const { ignoredTypes } = parseConfig(input.config ?? {});
       const errors = input.trajectory.events
         .filter((event) => event.type === "error")
         .filter((event) => !ignoredTypes.includes(event.data.type ?? ""));
       const passed = errors.length === 0;
   
       return {
         name: this.metadata.name,
         kind: "code",
         passed,
         score: passed ? 1 : 0,
         evidence: passed
           ? "No error events in trajectory"
           : `${errors.length} error(s): ${errors.map((e) => e.data.message).join(", ")}`,
         label: passed ? "correct" : "incorrect",
       };
     }
   
     assertsAbsence(): boolean {
       return true;
     }
   }

2. Configure it in eval.yaml

   After packaging and loading the grader plugin as shown below, reference its metadata name:

   eval.yaml

   name: no-errors-demo
   
   stimuli:
     - name: test-case
       prompt: "Do something"
       graders:
         - type: no-errors
           config:
             ignoredTypes: [TransientWarning]

GraderInput and run files

grade() receives a GraderInput. Every field is optional in the public contract, so narrow the values your grader requires and return an actionable error when one is absent:

interface GraderInput {
  trajectory?: Trajectory;
  stimulus?: Stimulus;
  config?: Record<string, unknown>;
  diffBaseDir?: string;
  goldenPatch?: string;
  signal?: AbortSignal;
  timeoutMs?: number;
  judgeTimeoutMs?: number;
}

• trajectory is the normalized agent event log.
• stimulus is the prompt and configuration that produced the trajectory.
• config is this grader instance’s config object from the eval spec. Validate its values before use, as NoErrorsGrader does with ignoredTypes above.
• diffBaseDir resolves a relative trajectory.diffPath sidecar.
• goldenPatch contains the reference-solution diff when golden-patch grading provides one.
• signal fires when the stimulus’s scoring.timeout budget expires. Enforcement is cooperative: forward it to subprocesses and fetch/SDK calls, or your grader will keep running past the budget.
• timeoutMs is the remaining total grading budget when the grader starts, when scoring.timeout is configured.
• judgeTimeoutMs is the configured per-turn budget in milliseconds for LLM judge calls, from defaults.judge_timeout or vally eval --judge-timeout. Unlike timeoutMs, it does not bound the entire grader. Custom LLM graders should apply it to each judge turn while honoring signal and the remaining timeoutMs budget. Built-in LLM graders default to 2m when it is absent.

When reading files, keep these trajectory locations distinct:

• trajectory.workDir — the graded workspace the agent mutated.
• trajectory.artifactDir — run outputs. Read outputs from here first, falling back to workDir; honor trajectory.artifactDirStrict, which forbids that fallback.
• trajectory.assetsDir — setup inputs staged outside the workspace via dest_root: assets. These are agent-visible and writable, so treat them as untrusted and never store grader logic there.

artifactDir and assetsDir are host paths and may be absent during offline re-grading. The resolveGraderReadRoot helper from @microsoft/vally/graders provides artifact-first, workspace-fallback, strict-aware lookup for output-reading graders. The built-in custom-metrics grader uses it; workspace file graders read workDir directly.

Optional capabilities

The runtime detects each capability by the presence of its method. Do not implement methods your grader does not need.

Instance names (optional)

Implement defaultName() when authors may configure several instances of your grader with different config. The runtime calls it automatically unless the author supplied an explicit name:

import { describeGraderList } from "@microsoft/vally/graders";

defaultName(config: Record<string, unknown>): string {
  const ignoredTypes = describeGraderList(config.ignoredTypes);
  return ignoredTypes ? `${this.metadata.name} except ${ignoredTypes}` : this.metadata.name;
}

Return a pure function of config, and never derive a name from values that may contain secrets. Names reach console output, markdown, JUnit XML, and the analytics store. Explicit author names win, and duplicate resolved names receive #1/#2 suffixes.

Return	Runtime behavior
A non-empty string	Sanitized, truncated to at most 60 grapheme clusters (including the ellipsis), and used as the name
A string that normalizes to empty	Ignored; the grader falls back to its result name
A non-string (including a Promise from an async implementation)	Ignored with a process.emitWarning
A throw	Caught and warned; the run continues

Control characters become spaces and whitespace runs collapse. To opt out for one config, return this.metadata.name; the declared return type does not allow undefined.

Comparison capability (optional)

Implement compare() only when a future or programmatic caller needs head-to-head trajectory judgment. Its presence marks the grader as comparison-capable; there is no metadata flag. It receives GraderComparisonInput and returns a signed, treatment-relative GraderComparisonResult. GraderComparisonInput.judgeTimeoutMs supplies the per-turn budget for LLM judge calls, just as it does in GraderInput.

The current vally compare command constructs the built-in prompt grader directly. It does not provide a plugin-driven way to select a custom comparison grader.

Absence assertions (optional)

Implement assertsAbsence() only when passing on an empty baseline is correct. NoErrorsGrader returns true because an empty trajectory contains no errors; another example is a tool-calls grader configured only with disallowed. vally oracle --no-golden-input otherwise expects each grader to fail when the golden solution is withheld. When assertsAbsence() returns true, a passing baseline result is reported as N/A instead of trivially passing.

assertsAbsence(config: Record<string, unknown>): boolean {
  const cfg = config as {
    required?: unknown[];
    sequence?: unknown[];
    parallel?: unknown[];
    disallowed?: unknown[];
  };
  const isEmptyList = (value: unknown[] | undefined): boolean =>
    value == null || value.length === 0;
  return (
    !isEmptyList(cfg.disallowed) &&
    isEmptyList(cfg.required) &&
    isEmptyList(cfg.sequence) &&
    isEmptyList(cfg.parallel)
  );
}

Like defaultName(), this must be a pure function of config. Omit it when a baseline pass would be a real problem so the negative control still catches the grader.

Taxonomy guidelines

Choose taxonomy values honestly — they’re surfaced in reports and help eval authors decide whether to include your grader in fast inner-loop runs or reserve it for outer-loop evaluation:

If your grader…	Set determinism to…	Set cost to…
Does string/file operations only	static	free or low
Runs a subprocess or does I/O	complex-static	low
Calls an embedding/small model	slm	medium
Calls GPT-5.5 or similar	llm	high

Testing your grader

Write tests that exercise both passing and failing cases:

no-errors-grader.test.ts

import { describe, it, expect } from "vitest";
import type { Trajectory } from "@microsoft/vally";
import { NoErrorsGrader } from "./no-errors-grader.js";

function trajectory(events: Trajectory["events"]): Trajectory {
  return {
    id: "test-1",
    stimulus: { name: "test-case", prompt: "Do something" },
    events,
    metrics: {
      tokenUsage: {
        inputTokens: 0,
        outputTokens: 0,
        totalTokens: 0,
        cacheReadTokens: 0,
        cacheWriteTokens: 0,
        callCount: 0,
        byModel: {},
      },
      toolCallCount: 0,
      toolCallBreakdown: {},
      simulatedToolCallCount: 0,
      skillActivationCount: 0,
      skillActivationBreakdown: {},
      turnCount: 0,
      wallTimeMs: 0,
      errorCount: events.filter((event) => event.type === "error").length,
    },
    output: "done",
    workDir: "/tmp",
    metadata: { model: "test", skillsLoaded: [], executor: "test", sessionID: "test-1" },
  };
}

describe("NoErrorsGrader", () => {
  const grader = new NoErrorsGrader();

  it("passes when no errors", async () => {
    const result = await grader.grade({
      trajectory: trajectory([]),
    });
    expect(result.passed).toBe(true);
    expect(result.score).toBe(1);
  });

  it("fails when errors exist", async () => {
    const result = await grader.grade({
      trajectory: trajectory([{ type: "error", data: { message: "timeout" } }]),
    });
    expect(result.passed).toBe(false);
    expect(result.evidence).toContain("timeout");
  });

  it("ignores configured error types", async () => {
    const result = await grader.grade({
      trajectory: trajectory([
        { type: "error", data: { message: "retrying", type: "TransientWarning" } },
      ]),
      config: { ignoredTypes: ["TransientWarning"] },
    });
    expect(result.passed).toBe(true);
  });

  it("rejects invalid config", async () => {
    await expect(
      grader.grade({
        trajectory: trajectory([]),
        config: { ignoredTypes: "TransientWarning" },
      }),
    ).rejects.toThrow("no-errors config.ignoredTypes must be an array of strings");
  });
});

Shipping as a plugin package

Custom graders can be shipped in a separate npm package and loaded at runtime via --grader-plugin. This lets teams share graders across repos without forking vally.

1. Create the package

   Your package exports a registerGraders function that receives the grader registry:

   src/index.ts

   import type { GraderRegistry } from "@microsoft/vally";
   import { NoErrorsGrader } from "./no-errors-grader.js";
   
   export function registerGraders(registry: GraderRegistry): void {
     registry.register(new NoErrorsGrader());
   }

   Assuming your build writes src/index.ts to dist/index.js, configure the package so Vally can load that file by package name. type=module tells Node to treat emitted .js files as ES modules; main and exports identify the compiled file. Then install @microsoft/vally as a peer dependency:

   npm pkg set type=module main=dist/index.js exports=./dist/index.js
   npm install --save-peer @microsoft/vally

2. Use it from the CLI

   Pass the package name or path to a command that supports --grader-plugin:

   # npm package
   vally eval --grader-plugin @myorg/vally-grader-quality --eval-spec eval.yaml
   
   # local path
   vally eval --grader-plugin ./my-graders --eval-spec eval.yaml
   
   # repeatable — load several grader plugins at once
   vally eval --grader-plugin @myorg/vally-grader-quality --grader-plugin ./my-graders --eval-spec eval.yaml
   
   # works with lint, grade, oracle, and export too
   vally lint --eval-spec eval.yaml --grader-plugin ./my-graders

3. Reference plugin graders in eval.yaml

   Plugin graders are referenced by name, just like built-ins:

   eval.yaml

   graders:
     - type: no-errors
     - type: output-contains
       config:
         substring: "hello"

How plugin specifiers resolve

• npm package names resolve from the project using Node module resolution.
• For vally eval, local paths resolve from the first --eval-spec file’s directory, or from the skill/working directory when no eval spec is supplied.
• For vally grade, vally oracle, and vally export, local paths resolve from the eval spec’s directory.
• For vally lint, local paths resolve from the directory containing the resolved --eval-spec argument.

Prefer an absolute path when the command’s resolution base is not obvious.

Tip

The plugin is loaded before validation, so vally lint will accept your custom grader types without errors.

Next steps

• Grader taxonomy — deep dive on taxonomy dimensions
• Grader catalog — built-in grader examples
• How it works — where graders fit in the pipeline