Runtime Setup

This is the second of three steps to run your first eval.

Previous step

1. Install Vally

Prerequisites

• Node.js >=22.14.0 and the Vally CLI.
• A GitHub account with an active Copilot subscription.

Vally uses the built-in copilot-sdk executor by default. The quickest local setup is to reuse a Copilot CLI login.

Sign in with Copilot CLI

1. Install Copilot CLI if you don’t already have it:

   npm install -g @github/copilot

2. Launch Copilot CLI:

   copilot

   Enter /login and follow the prompts if you are not already signed in.

3. In the shell where you will run Vally, allow the executor to use that login:

   macOS / Linux

   export EVALUATE_USE_HOST_COPILOT_HOME=1

   PowerShell

   $env:EVALUATE_USE_HOST_COPILOT_HOME = "1"

Caution

Host-home mode exposes persisted Copilot settings to the run. Prefer token authentication when evals need a reproducible runtime configuration.

Tip

Next: Run your first eval from the same shell. That walkthrough verifies authentication while it scaffolds, validates, and runs your first evaluation.

Alternative: Authenticate with a token

For a reproducible local setup that does not use persisted Copilot settings, create a fine-grained GitHub personal access token with Account permissions > Copilot Requests. Expose it to Vally with an environment variable:

macOS / Linux

export COPILOT_GITHUB_TOKEN="<your-token>"

PowerShell

$env:COPILOT_GITHUB_TOKEN = "<your-token>"

Do not put the token in eval.yaml or commit it to source control.

This token authenticates both the default executor and LLM judge features such as prompt, panel, vally compare, and experiment --compare.

For automation credentials and a complete workflow, continue later with Add to CI.