Trajectory Format

A trajectory is the complete record of a single agent run. Saved as JSON via --output-dir, trajectories can be re-graded, compared, and analyzed.

ATIF input

When importing an Agent Trajectory Interchange Format document, Vally requires steps to contain at least one entry. Represent a run with no agent activity using a synthetic non-agent placeholder step rather than an empty array.

Top-level structure

interface Trajectory {
  id: string; // Unique run identifier
  stimulus: Stimulus; // The prompt + config that produced this
  events: TrajectoryEvent[]; // Flat array of typed events
  metrics: TrajectoryMetrics; // Computed aggregates
  output: string; // Final agent output text
  workDir: string; // Workspace path during run
  artifactDir?: string; // Artifact dir outside the workspace; some graders (e.g. custom-metrics) read it before workDir
  artifactDirStrict?: boolean; // When true, artifactDir is authoritative: artifact-reading graders do NOT fall back to workDir
  assetsDir?: string; // Setup-input dir staged outside the workspace (the EVALUATE_ASSETS dir); host path, absent when re-graded elsewhere
  diff?: string; // Cumulative workspace diff
  diffPath?: string; // Cumulative diff sidecar path
  turnDiffs?: Array<
    { turn: number } & ({ diff: string } | { diffPath: string } | { error: string })
  >;
  workspacePatchSource?: "fast-path" | "snapshot" | "reused-diff"; // Which strategy produced workspace.patch
  workspacePatchCaptureError?: string; // Set when workspace.patch capture was attempted but failed
  metadata: TrajectoryMetadata; // Execution context
  raw?: unknown; // Escape hatch: unconverted source doc (e.g. raw ATIF). In-process only — stripped from JSONL output, never persisted.
}

workspace.patch — per-trial workspace diff artifact

Written to <session-dir>/workspace.patch for every trial with a workspace, regardless of whether diff-based graders are configured. An empty file means the workspace was unchanged.

The content is stripped from results.jsonl (it’s already on disk). If capture fails — including when the diff exceeds the 64 MiB cap — workspacePatchCaptureError appears in the JSONL record and a workspace-patch-capture-failed warning is emitted; the trial still succeeds. workspacePatchSource records which strategy produced the patch (fast-path, snapshot, or reused-diff).

Sensitive content

workspace.patch may contain secrets, credentials, or proprietary code. Treat it with the same care as diff / diffPath.

Workspace diff artifacts

diff / diffPath represent the cumulative workspace change from the start of a run to its final state. Multi-turn runs that use a turn-scoped static diff grader also persist turnDiffs: one 0-based record per turn Vally attempted to bound, containing exactly one of an inline diff, a relative diffPath sidecar, or an error when Vally could not safely capture that boundary. These diffs may contain raw workspace content and should be treated as sensitive.

An absent, malformed, or error turn record fails a scoped diff grader; Vally never substitutes the cumulative diff except for turn 0 when both the caller-supplied and recorded stimuli are explicitly single-turn, where the two are equivalent by construction. JSONL sidecar paths resolve relative to the run directory supplied to vally grade with --run-dir.

Library users calling sliceTrajectoryToTurn(trajectory, turn) receive an event/output/metrics view that preserves the trajectory’s other fields, including cumulative diff evidence. The grading pipeline uses a separate internal projection for turn-scoped static diff graders, which strips cumulative diff sources before injecting only the selected turnDiffs record.

raw — the escape hatch

raw carries the original, unconverted source document a trajectory was built from (e.g. the raw ATIF object) for custom graders that need fields the normalized events array doesn’t capture. It is populated only by offline adapters like fromAtif — live executor runs leave it undefined — and is stripped from JSONL output, so it reaches graders during a run but is never persisted. Because it isn’t persisted, a grader that reads raw works only on the live pass: re-grading from JSONL (vally grade <file.jsonl>) will see undefined.

Prefer the normalized events

Reaching into raw opts out of Vally’s portability contract: it is untyped and its shape is dictated by the producer, so a grader keyed on it couples to a foreign schema and breaks when the input comes from a different executor/format. Only use it when the flat event model genuinely can’t express what you need.

artifactDir / artifactDirStrict — where artifact graders read

artifactDir is a directory of run artifacts written outside the graded workDir (e.g. a benchmark harness’s output dir). Artifact-reading graders such as custom-metrics look up their files there first, then fall back to workDir when the file is absent from the artifact dir. A live executor run typically sets artifactDir and keeps this lenient fallback; the exception is a relocating backend whose artifact dir the runner could not materialize to a host-readable path (see Remote backends below), which makes it strict.

artifactDirStrict makes artifactDir authoritative: when it is true, those graders do not fall back to workDir, so a file missing from the artifact dir fails the grader instead of silently resolving against a (possibly stale) workspace copy. It is set in two cases: by grade --artifact-dir for offline re-grade, where the workspace is usually absent or stale and a silent fallback would report a misleading result; and by the runner when a relocating backend’s artifact dir could not be materialized to a host-readable path (see below). Unlike raw, it is a normal additive field that round-trips through JSONL.

Remote backends

When a trial runs on a relocating backend (Docker/remote) the executor writes artifactDir on the far side, so the raw path is not readable from the grading host. The runner mirrors what it already does for workDir: it materializes the artifact dir back to a host-readable location under the run’s report directory and rewrites artifactDir to that placed path, so on success the value recorded in results.jsonl points at a location the host can read (including for offline re-grade). Materialization is opt-in: a backend requests it by listing "artifact-dir" in DisposableTrialResult.supportedExportKinds. A backend that omits it — including the local backend, whose artifact dir is already host-readable — is left untouched (materialization is driven by the declared capability, never inferred from a stat of the recorded path, since a relocating worker can report a far-side path that coincidentally exists on the host). When materialization cannot deliver readable artifacts, artifactDir follows the fallback rules below instead.

If materialization can’t deliver readable artifacts to the host, the runner emits an artifact-dir-materialize-skipped / artifact-dir-materialize-failed diagnostic and never leaves a stale far-side path behind. When the artifacts survive at a host-readable recovery location, artifactDir points there. Otherwise, when a host destination was resolved, artifactDir is set to that destination with artifactDirStrict = true so a later offline re-grade fails clearly against the missing artifact location instead of silently falling back to a stale workspace copy; only when no host destination exists at all is artifactDir dropped.

assetsDir — where setup inputs were staged

assetsDir is the managed per-trial directory holding files staged with environment.files[].dest_root: "assets". It sits outside the graded workspace, so its contents never appear in diff / turnDiffs or to tree-scanning graders, and it is removed when the trial ends.

It holds run inputs — recorded session logs, fixtures a grader reads — which is what distinguishes it from artifactDir, the run outputs directory. The two are never conflated.

Graders read it as trajectory.assetsDir; the program grader also receives it as the EVALUATE_ASSETS environment variable, the same variable the agent sees.

Because it is a host path allocated fresh per trial, it is normally absent when re-grading a stored trajectory — on another host, or after the trial’s cleanup removed it. Treat an unset or non-existent assetsDir as the normal case. vally grade --assets-dir <path> stamps it explicitly for an offline re-grade, and exporters such as Harbor set it to the fixed non-workspace image path where they baked the same files.

Assets are agent-visible and writable — the agent receives the path as EVALUATE_ASSETS — so a grader must not treat their contents as trusted or assume they survived the run unmodified.

Events

Events are a discriminated union on the type field. Every event has a timestamp.

tool_call

Agent invoked a tool.

{
  "type": "tool_call",
  "timestamp": "2025-01-15T10:30:00.000Z",
  "data": {
    "toolName": "write_file",
    "toolCallId": "call_abc123",
    "turnId": "0",
    "arguments": { "path": "add.test.js", "content": "..." }
  }
}

turnId (optional) identifies the model-response turn that issued the call, matching the surrounding turn_start/turn_end events when present (subagent-inlined calls carry a turn id but no surrounding boundaries). Graders use it to group parallel tool-call batches by turn identity (see the tool-calls grader); it is absent for executors that don’t carry a turn id.

launchedAgentIds (optional) identifies subagents linked to the call by the source trajectory. Offline adapters use it to preserve launch boundaries when events are scoped to a named subagent.

When tool-call simulation intercepts a call, the event carries "simulated": true so graders and reports can distinguish it from a real invocation.

tool_result

Tool returned a result.

{
  "type": "tool_result",
  "timestamp": "2025-01-15T10:30:01.000Z",
  "data": {
    "toolName": "write_file",
    "toolCallId": "call_abc123",
    "success": true,
    "result": "File written successfully"
  }
}

token_usage

LLM token counts from a single API call.

{
  "type": "token_usage",
  "timestamp": "2025-01-15T10:30:00.500Z",
  "data": {
    "inputTokens": 1500,
    "outputTokens": 350,
    "model": "gpt-5.5",
    "cacheReadTokens": 200,
    "cacheWriteTokens": 0,
    "cost": {
      "provider": "github-copilot",
      "unit": "nano-aiu",
      "amount": 1250000000
    }
  }
}

Copilot SDK sources can include an optional provider-labelled cost. Live SDK events report a request cost; saved Copilot logs are reconstructed from per-model session totals. Other executors and older Copilot logs may omit cost.

cost_unavailable

The source could not provide a complete usage summary for a provider and unit. This event prevents tokenUsage.cost from being reported as a misleading subtotal while preserving token metrics from other valid events.

{
  "type": "cost_unavailable",
  "timestamp": "2025-01-15T10:30:00.500Z",
  "data": {
    "provider": "github-copilot",
    "unit": "nano-aiu",
    "model": "gpt-5.5",
    "reason": "malformed_usage"
  }
}

turn_start / turn_end

Conversation turn boundaries.

{ "type": "turn_start", "timestamp": "...", "data": { "turnId": "turn-1" } }
{ "type": "turn_end",   "timestamp": "...", "data": { "turnId": "turn-1" } }

assistant_message / user_message

Text messages in the conversation.

{ "type": "assistant_message", "timestamp": "...", "data": { "content": "I'll write tests for..." } }
{ "type": "user_message",      "timestamp": "...", "data": { "content": "Write tests for add()" } }

skill_activation

A skill was loaded by the agent.

{
  "type": "skill_activation",
  "timestamp": "...",
  "data": {
    "name": "test-writer",
    "path": "/skills/test-writer/SKILL.md",
    "pluginName": "copilot",
    "allowedTools": ["write_file", "read_file"]
  }
}

reasoning

The agent’s reasoning (thinking) for a step, when the source captures it. Emitted before the step’s assistant_message.

{
  "type": "reasoning",
  "timestamp": "...",
  "data": { "content": "First I'll inspect the failing test, then trace the null deref." }
}

system

A typed system-level event, such as context compaction. eventType carries the semantic kind (e.g. "compaction") so graders can match on it rather than parsing free text. message and observation are optional and omitted when the source provides no text. details carries structured, source-defined fields for the eventType (e.g. tokensRemoved); field names are only stable within a given eventType, not across all system events.

A system event is emitted only when the source marks the step with a typed kind. Other system steps remain custom events, so consumers should not expect every system step to appear as a system event. eventType values are not a fixed platform enum — new ones can appear as executors add support for more agent lifecycle events. Known values include:

• compaction — the conversation was successfully compacted (summarized) to free up context
• truncation — the conversation was hard-truncated to fit the context window
• snapshot_rewind — the session was rewound to an earlier point, discarding later events
• mode_changed — the agent’s operating mode changed (e.g. interactive, plan, autopilot)
• model_set — the session selected its initial model
• model_change — the session transitioned from one model to another
• task_complete — the agent’s request to complete the task was accepted
• handoff — the session was handed off from another session
• session_limits_changed — the session’s usage/rate limits were set or cleared

{
  "type": "system",
  "timestamp": "...",
  "data": {
    "eventType": "compaction",
    "message": "Compacted 40 messages",
    "observation": "Reduced to 12 messages, removed 8000 tokens",
    "details": { "success": true, "messagesRemoved": 28, "tokensRemoved": 8000 }
  }
}

error

Something went wrong during the run.

{
  "type": "error",
  "timestamp": "...",
  "data": {
    "message": "Request timed out",
    "type": "TimeoutError",
    "code": 408
  }
}

Metrics

Computed from events after the run completes:

interface TrajectoryMetrics {
  tokenUsage: {
    inputTokens: number;
    outputTokens: number;
    totalTokens: number;
    cacheReadTokens: number;
    cacheWriteTokens: number;
    callCount: number;
    cost?: ProviderCost;
    byModel: Record<
      string,
      {
        inputTokens: number;
        outputTokens: number;
        callCount: number;
      }
    >;
  };
  toolCallCount: number;
  toolCallBreakdown: Record<string, number>; // { "write_file": 3, "read_file": 1 }
  simulatedToolCallCount: number; // tool calls that were simulated, not run for real
  skillActivationCount: number;
  skillActivationBreakdown: Record<string, number>; // { "test-writer": 1 }
  turnCount: number;
  wallTimeMs: number;
  errorCount: number;
}

type ProviderCost = {
  provider: "github-copilot";
  unit: "nano-aiu";
  amount: number;
};

tokenUsage.cost is the provider-labelled usage cost for the evaluated agent across the trajectory’s LLM calls. It is present only when every token_usage event reports the same provider and unit with a safe, non-negative integer amount; an explicit 0 is preserved, while an omitted field means the complete total is unknown. It does not include LLM grader or judge usage, and Vally does not convert nano-AI units to displayed AI Credits.

Metadata

interface TrajectoryMetadata {
  model: string; // Model used for execution
  skillsLoaded: string[]; // Names of skills that were loaded
  startedAt: Date;
  completedAt: Date;
  executor: string; // Which executor ran this
  sessionID: string;
}

Working with trajectories

# Save trajectories (one trial-result record per trial, in results.jsonl)
vally eval --eval-spec eval.yaml --output-dir ./vally-results

# Re-grade trajectories from a previous run
cat ./vally-results/*/results.jsonl | vally grade --eval-spec eval.yaml

# Inspect trial trajectories with jq. `results.jsonl` interleaves
# `trial-result` records (with a nested `trajectory` field) with a final
# `run-summary` line, so filter on `.type == "trial-result"` first.
cat ./vally-results/*/results.jsonl | jq 'select(.type == "trial-result") | .trajectory.metrics'
cat ./vally-results/*/results.jsonl | jq 'select(.type == "trial-result") | .trajectory.events[] | select(.type == "tool_call") | .data.toolName'
cat ./vally-results/*/results.jsonl | jq 'select(.type == "trial-result") | .trajectory.events | length'