CLI: simulate

Usage

vally simulate record <input> [--output <path>] [--mcp-server <name>]...

Description

simulate record captures the tool calls from a recorded run once and turns them into a paste-ready YAML simulation.tool_overrides block, so an eval can replay recorded tool responses instead of calling the real tools.

It extracts completed tool calls from a Vally trajectory, an ATIF trajectory, or a prior eval run, and writes the overrides to stdout or a file. The generated block is a starting point: review outputs for sensitive or environment-specific data before adding it to an eval spec.

The generated overrides match on exact recorded arguments and have no default, so replay still runs the agent, and any tool call whose arguments differ from what was recorded falls through to the real tool. Treat the output as a recorded starting point, not a fully offline sandbox — add a default or broaden the patterns if you need every call intercepted.

Arguments

Argument	Description
<input>	A Vally trajectory JSON file, an ATIF JSON file, a Vally results.jsonl file, a directory containing results.jsonl, or - for stdin.

Options

Flag	Type	Default	Description
--output, -o <path>	string	stdout	Write the YAML to a file. Without it, the YAML goes to stdout and warnings to stderr.
--mcp-server <name>	string	—	MCP server name for inputs that omit server configuration. Repeatable. Needed for ATIF and other inputs that don’t record MCP servers so successful MCP calls can be told apart from first-party tools.

Exit codes

Code	Meaning
0	Overrides generated
1	Invalid input or write failure

Generated overrides

The command preserves each runtime tool name and creates anchored argument patterns. Recorded result text becomes output, MCP structuredContent becomes structured_output, and unsuccessful results include is_error: true. Identical tool/arguments/result tuples are emitted once. Shell tools (bash, shell, powershell) are aliases at replay time, so their patterns are merged under a single shell key.

Successful non-shell calls are only emitted when the tool matches a configured MCP server name (from the recorded trajectory or --mcp-server); other successful non-shell calls are omitted with a warning because the simulation schema can only replay failures for first-party tools. Failed calls are always retained.

The generated file contains a top-level simulation block. In an eval spec, simulation lives inside a stimulus, so indent the generated block under the target stimulus rather than pasting it at the document root:

stimuli:
  - name: my-stimulus
    prompt: ...
    # ← paste the generated `simulation:` block here, indented to this level
    simulation:
      tool_overrides:
        work-search:
          patterns:
            - pattern: ^\{"query":"project atlas"\}$
              output: Found one result
              structured_output:
                count: 1

If the same tool and arguments returned different results, the generated simulation keeps the first result and prints a warning. The current override schema cannot represent response sequences; edit the generated block when later calls should behave differently.

MCP tools with output schemas

structured_output is only emitted when the recorded result carried structured content. If an MCP tool declares an output schema, its simulated result must include structured_output, or the eval run fails at replay time with an error like has an output schema but the override provided no structured_output. Some inputs (notably ATIF trajectories) do not carry structured content, so add the matching structured_output by hand for those tools before running the eval.

Examples

# Generate overrides from a trajectory and print them to stdout
vally simulate record trajectory.json

# Write overrides to a file. `vally eval` writes results.jsonl into a
# timestamped run directory under --output-dir, so point at that run directory
# (or pass the results.jsonl file directly).
vally simulate record ./vally-results/2025-01-01T00-00-00-000Z/ --output simulation.yaml

# Read from stdin and name the MCP servers an ATIF export omits
cat run.atif.json | vally simulate record - --mcp-server work --mcp-server docs