Evaluating Custom Agents

What is a custom agent?

VS Code custom agents are specialized AI personas defined in .agent.md files. They extend VS Code’s Copilot with custom instructions, tool constraints, and behavioral guidelines. Unlike general skills, agents can specify which tools they’re allowed to use, define model preferences, and coordinate with other agents through handoffs.

For details on creating custom agents in VS Code, see the VS Code custom agents documentation.

Key differences from SKILL.md

Waza treats .agent.md files as first-class evaluation targets, just like SKILL.md. The key differences:

Aspect	SKILL.md	.agent.md
Purpose	Define reusable Copilot skills	Define custom VS Code agent personas
Frontmatter fields	name, description, triggers	name, description, tools, model, handoffs, mcp-servers, agents
Tool usage	No built-in constraint	Declares allowed tools in tools: field
Auto constraints	Manual tool_constraint grader needed	Implicit tool_constraint auto-injected if tools: field present
When both exist	SKILL.md takes priority in same directory	.agent.md evaluated only if SKILL.md absent

Quick start

1. Run the example

Waza includes a complete security-reviewer agent example:

waza run examples/custom-agent/eval.yaml -v

This evaluates a custom security-reviewer agent with tasks covering code review, vulnerability detection, and compliance checking.

2. Point waza at your agent

Create an eval.yaml:

name: my-agent-eval
description: Evaluating my custom agent
skill: my-agent  # Points to my-agent.agent.md in the same directory
version: "1.0"

config:
  model: claude-sonnet-4.6
  timeout_seconds: 300

graders:
  - type: text
    name: identifies_issues
    config:
      regex_match:
        - "(?i)(bug|issue|security|error)"

tasks:
  - "tasks/*.yaml"

3. Run it

waza run eval.yaml -v

Waza discovers your my-agent.agent.md, parses its frontmatter, and auto-injects a tool_constraint grader if your agent declares a tools: field.

Anatomy of a .agent.md file

Here’s a complete annotated custom agent:

---
name: security-reviewer
description: |
  A specialized agent that reviews code for security vulnerabilities,
  compliance issues, and best practices. Flags suspicious patterns
  and suggests hardening strategies.

model: claude-opus-4.6
tools:
  - codeSearch
  - fileRead
  - fileWrite
  - runCommand

mcp-servers:
  - url: "sse://internal-tools.example.com"
    name: "compliance-checker"

handoffs:
  - name: legal-review
    when: "sensitive compliance issues found"
  - name: performance-analyst
    when: "security patch impacts performance"

agents:
  - name: code-analyzer
    scope: "async dependency analysis"
---

# Security Review Agent

You are an expert security reviewer specializing in code analysis, vulnerability detection, and compliance validation.

## Your role

Analyze code submissions for:
- Common security vulnerabilities (injection, XSS, CSRF, etc.)
- Cryptographic issues
- Data handling compliance (PII, sensitive data)
- Third-party dependency risks
- Infrastructure-as-code misconfigurations

## Tool usage guidelines

- Use `codeSearch` to identify similar patterns across the codebase
- Use `fileRead` to examine context and dependencies
- Use `runCommand` to check for known vulnerability patterns (SAST integration)
- Use `fileWrite` only to create reports, never modify source directly

## Response format

Provide:
1. **Risk Level:** Critical / High / Medium / Low
2. **Issue Description:** What was found and why it matters
3. **Affected Code:** File path and line numbers
4. **Remediation:** Concrete fix or best practice
5. **References:** CWE/CVE links if applicable

Frontmatter fields explained

Field	Type	Required	Waza behavior
name	string	Yes	Agent identifier; must match filename (security-reviewer for security-reviewer.agent.md)
description	string	Yes	Summary of agent purpose; used in discovery and trigger tests
model	string	No	Preferred model; waza uses config.model from eval.yaml (field is informational)
tools	array	No	Allowed tool names; waza auto-injects tool_constraint grader to validate only these tools are used
mcp-servers	array	No	Model Context Protocol servers; parsed but not yet evaluated (P2 roadmap)
handoffs	array	No	Handoff definitions; parsed but not yet evaluated (P2 roadmap)
agents	array	No	Coordinating agents; parsed but not yet evaluated (P2 roadmap)

Body content

The markdown body becomes the agent’s system message. Waza injects it into the execution context to guide the agent’s behavior during evaluation tasks, prefixed with Base directory for this skill: <absolute path> so relative links in the body resolve from the agent file’s directory.

How waza evaluates agents

Discovery

Waza searches for .agent.md files in the same locations as SKILL.md:

• Current directory
• ./agents/ subdirectory
• ./skills/ subdirectory
• Paths specified in eval.yaml

# Discovers any .agent.md or SKILL.md in the current directory
waza run eval.yaml

# Explicitly target an agent in a subdirectory
waza run my-evals/eval.yaml --context-dir ./agents

Auto-injected tool constraint

When your .agent.md declares a tools: field, waza automatically adds a tool_constraint grader to validate that only those tools were called during task execution.

Auto-injection is transparent

You won’t see the auto-injected grader in your eval.yaml, but it appears in the results and can be overridden.

Before (without agent’s tools field):

graders:
  - type: text
    name: code_quality
    config:
      regex_match: ["(?i)(refactored|improved)"]

After (if agent has tools: [codeSearch, fileRead, runCommand]):

graders:
  - type: tool_constraint  # 🔄 Auto-injected
    name: allowed_tools
    config:
      allow_only:
        - tool: codeSearch
        - tool: fileRead
        - tool: runCommand
  - type: text
    name: code_quality
    config:
      regex_match: ["(?i)(refactored|improved)"]

allow_only is an upper bound: the agent may use any subset of these tools (or none at all), but any tool call outside the list fails the grader. tool matches by exact case-insensitive name — bash will not accidentally match bashful. Declaring tools: [] in the frontmatter injects an empty allow-list, denying all tool use.

Override auto-injection

To use a different tool constraint or opt out entirely, declare your own in eval.yaml:

graders:
  - type: tool_constraint
    name: my_tool_policy
    config:
      allow_only:                       # Stricter than agent declares
        - tool: codeSearch
        - tool: fileRead
      reject_tools:
        - tool: fileDelete
        - tool: runCommand

  - type: text
    name: code_quality
    config:
      regex_match: ["(?i)(refactored|improved)"]

When you declare a tool_constraint grader, waza skips auto-injection for that agent.

Runtime tool policy enforcement (copilot-sdk executor)

The tool_constraint grader above is a post-run check: it grades the transcript after the agent has already finished. When the executor is copilot-sdk, waza additionally enforces the same tools: declaration at runtime, before any tool call executes:

tools: frontmatter	Runtime behavior
Omitted	Unrestricted — tools provided by the Copilot runtime may be used, unchanged from prior behavior.
tools: []	Every model-initiated tool attempt (built-in, MCP, subagent) is denied before it executes.
tools: [name, ...]	Only the declared tools are permitted; every other tool is denied before it executes.

The SDK session receives a native tool filter, a pre-tool-use hook denies undeclared calls (including tools that do not request permission), and a fail-closed permission handler checks permission requests. Unrecognized requests are denied. These checks apply to initial and resumed sessions, including static follow-ups and responder-driven turns. A denied attempt fails the run even if the final answer is otherwise correct.

The effective policy and every denial appear in results.json under each run’s session_digest (tool_policy_mode, tool_policy_denials), in the run_complete event when --session-log is enabled, and in the dashboard’s trajectory session digest. A selected agent with omitted tools: explicitly records unrestricted.

Runtime policy and the implicit allow_only grader resolve the selected agent by name, using the task’s effective skill_paths (task overrides take precedence), direct or one-level nested discovery, and SKILL.md precedence. Fully disabled skills receive neither policy nor implicit grader. Malformed agent definitions fail resolution rather than silently granting unrestricted access. An explicit tool_constraint grader replaces the implicit grader, not the runtime policy.

Matching is exact, case-insensitive, and uses the same fixed aliases as allow_only; patterns and wildcards do not grant additional tools:

Equivalent names	Native built-in filter
read, readFile, fileRead, view	builtin:view
write, writeFile, fileWrite, edit	builtin:edit
bash, runCommand	builtin:bash
fetch, web_fetch	builtin:web_fetch

Bare names refer to built-ins. For MCP tools, declare mcp:<server-key>-<tool-name> (for example mcp:github-list_issues); for caller-provided tools, use custom:<tool-name> (for example custom:CodeSearch). Permission matching preserves these namespaces: CodeSearch does not authorize custom:CodeSearch, and github-list_issues does not authorize mcp:github-list_issues. Declaring one MCP tool never authorizes another server or its other tools. Custom/MCP identifiers retain their original case and source qualifier in the SDK filter and do not use built-in aliases.

Hooks and transcripts may omit source metadata. For those observations, declared custom/MCP names match the literal unqualified name; source isolation then relies on the SDK filter. Explicit source information is never discarded when present.

A launch must pass its tool-name check (typically task for subagent launches) and any associated permission check. For SDK workflow permission requests, waza matches the workflow’s Name exactly and case-insensitively against the declared tools, using the fixed aliases above. An empty or omitted name falls back to task; permitting task alone does not authorize a named workflow. Permitting a launcher is not a separate sandbox for the subagent.

The bundled Copilot CLI 1.0.85 still uses the legacy factory permission kind, which the current SDK compatibility path preserves as a raw request. Waza accepts this specific legacy kind only when its JSON parses into the compatible workflow fields and the payload’s kind matches the factory discriminator. Valid legacy requests use the same name matching and unnamed task fallback as workflow requests. Malformed legacy payloads and all other unrecognized kinds remain denied. Allowed requests are forwarded unchanged to the existing permission handler, preserving its decisions and managed-approval metadata.

The opt-in live SDK regression exercises an allowed file read and a current-information request in both an initial and resumed session:

ENABLE_COPILOT_TESTS=true go test ./internal/execution -run '^TestToolPolicyReadOnly_Live$' -count=1 -v

Not a full sandbox

Runtime tool policy enforcement restricts which tools the model may invoke. It does not restrict what those tools can access once permitted (host files, credentials, network) — that is a separate sandboxing concern.

Writing your eval.yaml for an agent

Target an agent by name or file path:

name: security-agent-eval
description: Evaluate the security-reviewer agent
skill: security-reviewer  # Resolves to security-reviewer.agent.md

version: "1.0"

config:
  model: claude-sonnet-4.6
  timeout_seconds: 300
  trials_per_task: 1

# Graders: define what success looks like
graders:
  - type: text
    name: identifies_vulnerability
    config:
      regex_match:
        - "(?i)(vulnerability|bug|issue)"
        - "(?i)(risk level|severity)"

  - type: code
    name: suggests_fix
    config:
      assertions:
        - "output_length > 200"

tasks:
  - "tasks/*.yaml"

Key points:

• Use the skill: field with the agent name to target an agent
• All other eval.yaml structure is identical
• Discovery and grading work the same way
• The agent’s tools: field auto-injects a constraint (unless you override it)

When SKILL.md and .agent.md both exist

If both files are present in the same directory, SKILL.md takes priority. Waza evaluates the skill and ignores the agent.

Workaround: If you want to evaluate the agent instead:

• Rename or move the SKILL.md file temporarily, or
• Place the .agent.md in a different directory and reference it explicitly in eval.yaml

This behavior ensures backward compatibility—existing SKILL.md-based projects work unchanged.

Coverage reporting

The waza coverage command now reports both SKILL.md and .agent.md files:

waza coverage

# Output:
# Skills found: 3
#   ✓ code-explainer (SKILL.md)
#   ✓ documentation-writer (SKILL.md)
#   ✓ security-reviewer (.agent.md)
#
# Evaluations found: 3
#   ✓ evals/code-explainer.yaml
#   ✓ evals/documentation-writer.yaml
#   ✓ evals/security-reviewer.yaml

Each agent is counted alongside skills. Use this to audit your evaluation coverage.

Example: security-reviewer agent

The examples/custom-agent/ directory contains a complete, runnable security-reviewer agent with evaluation suite.

File structure

examples/custom-agent/
├── security-reviewer.agent.md    # Agent definition with tools
├── eval.yaml                      # Evaluation spec
├── tasks/
│   ├── 01-injection-detection.yaml
│   ├── 02-crypto-review.yaml
│   ├── 03-data-handling.yaml
│   └── 04-infrastructure-scan.yaml
└── fixtures/
    ├── vulnerable-sql.py
    ├── weak-crypto.js
    └── pii-exposure.tf

Running the example

cd examples/custom-agent
waza run eval.yaml -v

# Output:
# Running: security-reviewer eval
# Task: injection-detection ... ✓ PASS
# Task: crypto-review ... ✓ PASS
# Task: data-handling ... ⚠ PARTIAL (4/5 validators)
# Task: infrastructure-scan ... ✓ PASS
#
# Results saved to results.json

What each task tests

Task	Purpose	Validates
injection-detection	SQL/command injection vulnerabilities	Tool constraint (codeSearch, fileRead only)
crypto-review	Cryptographic weaknesses	Identifies weak algorithms and recommends alternatives
data-handling	PII and sensitive data exposure	Detects hardcoded secrets, unencrypted fields
infrastructure-scan	Terraform/IaC misconfigurations	Checks for open security groups, missing logging

The auto-injected tool_constraint grader validates that only [codeSearch, fileRead, runCommand] are called—the tools declared in the agent’s frontmatter.

Sample task file

name: injection-detection
description: Agent identifies SQL injection vulnerabilities

triggers:
  - "Review this Python code for security issues"

fixtures:
  - vulnerable-sql.py

expected_outcome: |
  - Identifies the SQL injection vulnerability on line 8
  - Explains the risk
  - Suggests parameterized queries as mitigation

Limitations & roadmap

• handoffs field — Parsed but not yet evaluated. P2 feature to test agent coordination.
• mcp-servers field — Parsed but not yet evaluated. P2 feature to validate MCP server integration.
• Multiple .agent.md files — Runtime selection matches the eval’s skill: name against agent frontmatter names (falling back to the filename without .agent.md). Use unique names.
• No handoff testing — Coming in a future release to validate agent coordination.

See GitHub issue #225 for status.

See also

• Writing Eval Specs — Full eval.yaml reference
• Validators & Graders — All available graders, including tool_constraint
• CLI Commands — waza run, coverage, and check commands
• VS Code Custom Agents — Creating agents in VS Code