Trajectory Format

A trajectory is the complete record of a single agent run. Saved as JSON via --output-dir, trajectories can be re-graded, compared, and analyzed.

Top-level structure

interface Trajectory {
  id: string; // Unique run identifier
  stimulus: Stimulus; // The prompt + config that produced this
  events: TrajectoryEvent[]; // Flat array of typed events
  metrics: TrajectoryMetrics; // Computed aggregates
  output: string; // Final agent output text
  workDir: string; // Workspace path during run
  artifactDir?: string; // Artifact dir outside the workspace; some graders (e.g. custom-metrics) read it before workDir
  artifactDirStrict?: boolean; // When true, artifactDir is authoritative: artifact-reading graders do NOT fall back to workDir
  diff?: string; // Cumulative workspace diff
  diffPath?: string; // Cumulative diff sidecar path
  turnDiffs?: Array<
    { turn: number } & ({ diff: string } | { diffPath: string } | { error: string })
  >;
  metadata: TrajectoryMetadata; // Execution context
  raw?: unknown; // Escape hatch: unconverted source doc (e.g. raw ATIF). In-process only — stripped from JSONL output, never persisted.
}

Workspace diff artifacts

diff / diffPath represent the cumulative workspace change from the start of a run to its final state. Multi-turn runs that use a turn-scoped static diff grader also persist turnDiffs: one 0-based record per turn Vally attempted to bound, containing exactly one of an inline diff, a relative diffPath sidecar, or an error when Vally could not safely capture that boundary. These diffs may contain raw workspace content and should be treated as sensitive.

An absent, malformed, or error turn record fails a scoped diff grader; Vally never substitutes the cumulative diff except for turn 0 when both the caller-supplied and recorded stimuli are explicitly single-turn, where the two are equivalent by construction. JSONL sidecar paths resolve relative to the run directory supplied to vally grade with --run-dir.

Library users calling sliceTrajectoryToTurn(trajectory, turn) receive an event/output/metrics view that preserves the trajectory’s other fields, including cumulative diff evidence. The grading pipeline uses a separate internal projection for turn-scoped static diff graders, which strips cumulative diff sources before injecting only the selected turnDiffs record.

raw — the escape hatch

raw carries the original, unconverted source document a trajectory was built from (e.g. the raw ATIF object) for custom graders that need fields the normalized events array doesn’t capture. It is populated only by offline adapters like fromAtif — live executor runs leave it undefined — and is stripped from JSONL output, so it reaches graders during a run but is never persisted. Because it isn’t persisted, a grader that reads raw works only on the live pass: re-grading from JSONL (vally grade <file.jsonl>) will see undefined.

Prefer the normalized events

Reaching into raw opts out of Vally’s portability contract: it is untyped and its shape is dictated by the producer, so a grader keyed on it couples to a foreign schema and breaks when the input comes from a different executor/format. Only use it when the flat event model genuinely can’t express what you need.

artifactDir / artifactDirStrict — where artifact graders read

artifactDir is a directory of run artifacts written outside the graded workDir (e.g. a benchmark harness’s output dir). Artifact-reading graders such as custom-metrics look up their files there first, then fall back to workDir when the file is absent from the artifact dir. A live executor run typically sets artifactDir and keeps this lenient fallback; the exception is a relocating backend whose artifact dir the runner could not materialize to a host-readable path (see Remote backends below), which makes it strict.

artifactDirStrict makes artifactDir authoritative: when it is true, those graders do not fall back to workDir, so a file missing from the artifact dir fails the grader instead of silently resolving against a (possibly stale) workspace copy. It is set in two cases: by grade --artifact-dir for offline re-grade, where the workspace is usually absent or stale and a silent fallback would report a misleading result; and by the runner when a relocating backend’s artifact dir could not be materialized to a host-readable path (see below). Unlike raw, it is a normal additive field that round-trips through JSONL.

Remote backends

When a trial runs on a relocating backend (Docker/remote) the executor writes artifactDir on the far side, so the raw path is not readable from the grading host. The runner mirrors what it already does for workDir: it materializes the artifact dir back to a host-readable location under the run’s report directory and rewrites artifactDir to that placed path, so on success the value recorded in results.jsonl points at a location the host can read (including for offline re-grade). Materialization is opt-in: a backend requests it by listing "artifact-dir" in DisposableTrialResult.supportedExportKinds. A backend that omits it — including the local backend, whose artifact dir is already host-readable — is left untouched (materialization is driven by the declared capability, never inferred from a stat of the recorded path, since a relocating worker can report a far-side path that coincidentally exists on the host). When materialization cannot deliver readable artifacts, artifactDir follows the fallback rules below instead.

If materialization can’t deliver readable artifacts to the host, the runner emits an artifact-dir-materialize-skipped / artifact-dir-materialize-failed diagnostic and never leaves a stale far-side path behind. When the artifacts survive at a host-readable recovery location, artifactDir points there. Otherwise, when a host destination was resolved, artifactDir is set to that destination with artifactDirStrict = true so a later offline re-grade fails clearly against the missing artifact location instead of silently falling back to a stale workspace copy; only when no host destination exists at all is artifactDir dropped.

Events

Events are a discriminated union on the type field. Every event has a timestamp.

tool_call

Agent invoked a tool.

{
  "type": "tool_call",
  "timestamp": "2025-01-15T10:30:00.000Z",
  "data": {
    "toolName": "write_file",
    "toolCallId": "call_abc123",
    "turnId": "0",
    "arguments": { "path": "add.test.js", "content": "..." }
  }
}

turnId (optional) identifies the model-response turn that issued the call, matching the surrounding turn_start/turn_end events when present (subagent-inlined calls carry a turn id but no surrounding boundaries). Graders use it to group parallel tool-call batches by turn identity (see the tool-calls grader); it is absent for executors that don’t carry a turn id.

When tool-call simulation intercepts a call, the event carries "simulated": true so graders and reports can distinguish it from a real invocation.

tool_result

Tool returned a result.

{
  "type": "tool_result",
  "timestamp": "2025-01-15T10:30:01.000Z",
  "data": {
    "toolName": "write_file",
    "toolCallId": "call_abc123",
    "success": true,
    "result": "File written successfully"
  }
}

token_usage

LLM token counts from a single API call.

{
  "type": "token_usage",
  "timestamp": "2025-01-15T10:30:00.500Z",
  "data": {
    "inputTokens": 1500,
    "outputTokens": 350,
    "model": "gpt-5.5",
    "cacheReadTokens": 200,
    "cacheWriteTokens": 0,
    "cost": {
      "provider": "github-copilot",
      "unit": "nano-aiu",
      "amount": 1250000000
    }
  }
}

Copilot SDK sources can include an optional provider-labelled cost. Live SDK events report a request cost; saved Copilot logs are reconstructed from per-model session totals. Other executors and older Copilot logs may omit cost.

cost_unavailable

The source could not provide a complete usage summary for a provider and unit. This event prevents tokenUsage.cost from being reported as a misleading subtotal while preserving token metrics from other valid events.

{
  "type": "cost_unavailable",
  "timestamp": "2025-01-15T10:30:00.500Z",
  "data": {
    "provider": "github-copilot",
    "unit": "nano-aiu",
    "model": "gpt-5.5",
    "reason": "malformed_usage"
  }
}

turn_start / turn_end

Conversation turn boundaries.

{ "type": "turn_start", "timestamp": "...", "data": { "turnId": "turn-1" } }
{ "type": "turn_end",   "timestamp": "...", "data": { "turnId": "turn-1" } }

assistant_message / user_message

Text messages in the conversation.

{ "type": "assistant_message", "timestamp": "...", "data": { "content": "I'll write tests for..." } }
{ "type": "user_message",      "timestamp": "...", "data": { "content": "Write tests for add()" } }

skill_activation

A skill was loaded by the agent.

{
  "type": "skill_activation",
  "timestamp": "...",
  "data": {
    "name": "test-writer",
    "path": "/skills/test-writer/SKILL.md",
    "pluginName": "copilot",
    "allowedTools": ["write_file", "read_file"]
  }
}

reasoning

The agent’s reasoning (thinking) for a step, when the source captures it. Emitted before the step’s assistant_message.

{
  "type": "reasoning",
  "timestamp": "...",
  "data": { "content": "First I'll inspect the failing test, then trace the null deref." }
}

system

A typed system-level event, such as context compaction. eventType carries the semantic kind (e.g. "compaction") so graders can match on it rather than parsing free text. message and observation are optional and omitted when the source provides no text.

A system event is emitted only when the source marks the step with a typed kind (from ATIF extra.context_management.type). Other system steps remain custom events, so consumers should not expect every system step to appear as a system event.

{
  "type": "system",
  "timestamp": "...",
  "data": {
    "eventType": "compaction",
    "message": "Compacting 40 messages",
    "observation": "Reduced to 12 messages, removed 8000 tokens"
  }
}

error

Something went wrong during the run.

{
  "type": "error",
  "timestamp": "...",
  "data": {
    "message": "Request timed out",
    "type": "TimeoutError",
    "code": 408
  }
}

Metrics

Computed from events after the run completes:

interface TrajectoryMetrics {
  tokenUsage: {
    inputTokens: number;
    outputTokens: number;
    totalTokens: number;
    cacheReadTokens: number;
    cacheWriteTokens: number;
    callCount: number;
    cost?: ProviderCost;
    byModel: Record<
      string,
      {
        inputTokens: number;
        outputTokens: number;
        callCount: number;
      }
    >;
  };
  toolCallCount: number;
  toolCallBreakdown: Record<string, number>; // { "write_file": 3, "read_file": 1 }
  simulatedToolCallCount: number; // tool calls that were simulated, not run for real
  skillActivationCount: number;
  skillActivationBreakdown: Record<string, number>; // { "test-writer": 1 }
  turnCount: number;
  wallTimeMs: number;
  errorCount: number;
}

type ProviderCost = {
  provider: "github-copilot";
  unit: "nano-aiu";
  amount: number;
};

tokenUsage.cost is the provider-labelled usage cost for the evaluated agent across the trajectory’s LLM calls. It is present only when every token_usage event reports the same provider and unit with a safe, non-negative integer amount; an explicit 0 is preserved, while an omitted field means the complete total is unknown. It does not include LLM grader or judge usage, and Vally does not convert nano-AI units to displayed AI Credits.

Metadata

interface TrajectoryMetadata {
  model: string; // Model used for execution
  skillsLoaded: string[]; // Names of skills that were loaded
  startedAt: Date;
  completedAt: Date;
  executor: string; // Which executor ran this
  sessionID: string;
}

Working with trajectories

# Save trajectories (one trial-result record per trial, in results.jsonl)
vally eval --eval-spec eval.yaml --output-dir ./vally-results

# Re-grade trajectories from a previous run
cat ./vally-results/*/results.jsonl | vally grade --eval-spec eval.yaml

# Inspect trial trajectories with jq. `results.jsonl` interleaves
# `trial-result` records (with a nested `trajectory` field) with a final
# `run-summary` line, so filter on `.type == "trial-result"` first.
cat ./vally-results/*/results.jsonl | jq 'select(.type == "trial-result") | .trajectory.metrics'
cat ./vally-results/*/results.jsonl | jq 'select(.type == "trial-result") | .trajectory.events[] | select(.type == "tool_call") | .data.toolName'
cat ./vally-results/*/results.jsonl | jq 'select(.type == "trial-result") | .trajectory.events | length'