Telemetry

The Vally CLI collects a small amount of pseudonymous usage telemetry so we can see which commands people run, whether they succeed, and where they’re slow. It’s on by default, and you can turn it off at any time.

We designed this to tell us how the tool is used, never what you use it on. Vally emits at most one record per command you run, and that record describes how the command went, not the data it ran against. Purely informational displays like --help and --version emit nothing at all.

What Vally collects

Each time you run a vally command, Vally records:

The command name (for example eval, grade, or experiment run). Just the command, not the arguments or flags you passed it.
The Vally version you’re running.
The outcome: whether the command succeeded, failed, or was interrupted, plus the exit code and a coarse health category (for example a user error versus an internal error). When something fails, we record a fixed error code from a small predefined list, never the actual error message.
How long the command took, in milliseconds.
Aggregate counts and coarse run facts: how many evals, trials, and grader invocations ran, which built-in grader types and executor were used, and whether a custom grader, executor, or exporter was involved. Counts and built-in names only. Anything custom (a plugin grader, executor, or backend) is recorded as custom, never by its real name.
A persistent device identifier, when one is available. This is a randomly generated identifier shared across Microsoft developer tools, which is what makes it pseudonymous rather than anonymous: it lets us count unique developer environments and correlate across tools, but it carries no name, email, account, or hostname. It’s stored per user account, so two accounts on the same computer count separately.
Coarse OS and runtime info: the operating system family and its major version, the CPU architecture, and the major version of Node.js. Major versions only, so 24 rather than 24.3.1.
Whether you’re running in CI or interactively.

A few commands add one or two more details of the same kind (for example whether vally serve finished starting up, and how long that took), and each record carries internal bookkeeping like a schema version so we can interpret it later.

The Vally-defined fields are filtered through a fixed allow-list in the code, so a field that isn’t on that list doesn’t get sent.

What Vally does not collect

This is the important part. Nothing Vally sends to Microsoft includes any of the following:

Your prompts. Nothing you write to an agent is collected.
Your datasets or eval content. The evals you run, their inputs, and their expected outputs stay on your machine.
File paths. No paths to your files, projects, or working directory.
Command arguments or flag values. We record that you ran eval, not what you pointed it at.
Model names, scores, or grader results. We record that graders ran and how many, never the model you used, the scores they produced, or the content behind them.
Anything that identifies you personally. No name, no email, no account, no hostname.

Eval tracing is a separate stream

Usage telemetry is not the only thing Vally can emit. When you run vally eval, Vally also produces detailed OpenTelemetry traces of the run itself, and those are much richer: they can include eval file paths, model names, and grader scores.

By default this stream stays on your machine: Vally writes it to an otel-spans.jsonl file alongside your other run artifacts, and none of it goes to Microsoft. (Vally stages the spans in an otel-spans/ directory while the run is in flight, then merges them into that file at the end, so you’ll only find the directory left behind if a run doesn’t shut down cleanly.) It only leaves your machine if you pass --otlp-endpoint, and then it goes only to the collector you name. The opt-out variables below turn off usage telemetry; eval tracing is governed by where you point it.

Turning it off

Set either of these environment variables before running Vally:

# Vally-specific opt-out
export VALLY_TELEMETRY_OPTOUT=1

# Or the cross-tool community convention
export DO_NOT_TRACK=1

On Windows PowerShell:

$env:VALLY_TELEMETRY_OPTOUT = "1"
# or
$env:DO_NOT_TRACK = "1"

VALLY_TELEMETRY_OPTOUT accepts 1, true, or yes. DO_NOT_TRACK accepts 1 or true. With either one set, Vally is fully opted out: it doesn’t initialize telemetry, collect anything, or make any network call. Opt-out always wins over every other setting.

To make it permanent, add the variable to your shell profile (for example ~/.bashrc or ~/.zshrc) or set it in your CI environment.

How long data is kept

If you haven’t opted out, the raw telemetry is kept for 90 days. Aggregated data derived from it is kept for up to about 18 months.

Those windows apply to Vally’s default destination. You can send usage telemetry somewhere else instead by setting OTEL_EXPORTER_OTLP_ENDPOINT, VALLY_TELEMETRY_ENDPOINT, or APPLICATIONINSIGHTS_CONNECTION_STRING. Any of those replaces the default destination, so the records go only where you point them and retention becomes whatever that collector does.

The first-run notice

The first time telemetry is active for your user profile, Vally prints a one-time notice to stderr and then writes a small marker file at ~/.vally/usage-telemetry-notice so it doesn’t show again. The marker lives in your home directory, so another account on the same computer sees the notice on its own first run. If you saw it fly by in your terminal, here it is in full:

Vally collects pseudonymous usage telemetry (command name, version, outcome, duration, persistent device identifier (when available), and coarse OS/runtime info) to improve the product. No prompts, datasets, file paths, or arguments are collected.
Opt out any time by setting VALLY_TELEMETRY_OPTOUT=1 or DO_NOT_TRACK=1.
Learn more: https://aka.ms/vally/telemetry

If you want to see the notice again, delete that marker file and run any Vally command.