Get Started

Vally is an extensible evaluation platform for AI agents — run agents against test cases, grade their behavior, and track quality over time.

If you’re new to Vally, How It Works explains the evaluation pipeline.

Prerequisites

• Node.js >=22.14.0, which includes npm.
• A GitHub account with an active Copilot subscription for Vally’s default runtime.

Complete these steps in order to run your first eval:

1. Install Vally

   npm install -g @microsoft/vally-cli

   See Installation for verification and library usage.

2. Set up the default runtime

   macOS / Linux

   # Install Copilot CLI
   npm install -g @github/copilot
   
   # Launch it, enter /login, then exit after signing in
   copilot
   
   # Allow Vally to use that login in this shell
   export EVALUATE_USE_HOST_COPILOT_HOME=1

   PowerShell

   # Install Copilot CLI
   npm install -g @github/copilot
   
   # Launch it, enter /login, then exit after signing in
   copilot
   
   # Allow Vally to use that login in this shell
   $env:EVALUATE_USE_HOST_COPILOT_HOME = "1"

   See Runtime Setup for token authentication.

3. Run your first eval

   Scaffold a spec, run an agent, and grade the results end-to-end.