Add to CI

This quickstart adds linting to every PR and behavioral evaluation to trusted changes.

vally lint is a fast, free static check — no agent needed. It validates your SKILL.md files (skill definitions that describe your agent’s capabilities) and your eval.yaml specs for config errors. Run it on every PR as a first gate, then use vally eval for full behavioral checks on trusted code.

1. Add the lint step (every PR)

   Create .github/workflows/skill-lint.yml:

   .github/workflows/skill-lint.yml

   name: Skill Lint
   on:
     pull_request:
       paths:
         - "**/SKILL.md"
         - "**/*.yaml"
   
   jobs:
     lint:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v4
   
         - uses: actions/setup-node@v4
           with:
             node-version: "22"
             check-latest: true
   
         - run: npm install -g @microsoft/vally-cli
   
         - name: Lint skills
           run: vally lint .
   
         - name: Validate eval specs
           run: vally lint --eval-spec evals/hello/eval.yaml --strict

   This lint pass catches:

   • Invalid SKILL.md format (name, description, frontmatter)
   • Broken file references

2. Add behavioral evals

   Run agent-level evals before merging branches from the same repository and after merging fork contributions. First, configure one of these authentication options:

   Recommended: GitHub Actions token

   1. For an organization-owned repository, ask an organization owner to enable Allow use of Copilot CLI billed to the organization.
   2. Grant the workflow copilot-requests: write permission.
   3. Pass ${{ github.token }} to Vally as GITHUB_TOKEN, as shown in the workflow below.

   The built-in token is short-lived, authenticates both the default copilot-sdk executor and LLM judges, and does not require a repository secret. See GitHub’s Copilot CLI in GitHub Actions setup.

   .github/workflows/skill-eval.yml

   name: Skill Eval
   on:
     pull_request:
       paths:
         - "**/SKILL.md"
         - "**/eval.yaml"
     push:
       branches: [main]
       paths:
         - "**/SKILL.md"
         - "**/eval.yaml"
     workflow_dispatch:
   
   permissions:
     contents: read
     copilot-requests: write
   
   jobs:
     eval:
       # Fork PRs cannot receive Copilot write permission or repository secrets.
       if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v4
   
         - uses: actions/setup-node@v4
           with:
             node-version: "22"
             check-latest: true
   
         - run: npm install -g @microsoft/vally-cli
   
         - name: Run evals
           run: |
             vally eval \
               --eval-spec evals/hello/eval.yaml \
               --skill-dir . \
               --require-pass \
               --output-dir ./results
           env:
             GITHUB_TOKEN: ${{ github.token }}
   
         - name: Upload trajectories
           if: always()
           uses: actions/upload-artifact@v4
           with:
             name: eval-results
             path: ./results/

   Fork pull requests

   Fork workflows cannot access copilot-requests: write or repository secrets, even after approval. This workflow skips their pre-merge eval and runs it automatically after merge on main. Never use pull_request_target to execute untrusted fork code with write permissions or secrets.

   Tip

   Always upload trajectory artifacts — they let you re-grade offline with vally grade without re-running the agent.

   Fallback: Fine-grained personal access token

   If your organization cannot enable the Copilot CLI policy:

   1. Create a fine-grained personal access token with Account permissions > Copilot Requests.
   2. Save it as a GitHub Actions repository secret named COPILOT_CLI_TOKEN.
   3. Replace the env block in the Run evals step with:

   env:
     GH_TOKEN: ${{ secrets.COPILOT_CLI_TOKEN }}
     GITHUB_COPILOT_API_TOKEN: ${{ secrets.COPILOT_CLI_TOKEN }}

   The same secret is mapped twice because the default executor reads GH_TOKEN, while LLM judges read GITHUB_COPILOT_API_TOKEN.

   This fallback also skips fork PRs because repository secrets are not provided to them; the push trigger runs their eval after merge.

3. Use exit codes for gating

   Both lint and eval set process.exitCode = 1 on errors, so GitHub Actions will automatically mark the step as failed.

   • vally lint → exits 1 if any skill fails validation
   • vally eval → exits 1 for grader, execution, configuration, or tooling errors. Add --require-pass to also exit 1 when a valid eval verdict fails.

Running Specific Suites

Define suites in .vally.yaml to run different eval subsets in different CI contexts:

.vally.yaml

suites:
  ci-gate:
    filter: { priority: [p0, p1] }
  nightly:
    filter: { priority: [p0, p1, p2] }

# .github/workflows/ci.yml — fast suite on every PR
- name: Run CI gate evals
  run: vally eval --suite ci-gate --require-pass

# .github/workflows/nightly.yml — comprehensive nightly run
- name: Run nightly evals
  run: vally eval --suite nightly

The PR suite uses --require-pass to gate on valid failing verdicts. The nightly suite omits it so valid failures are reported without failing the scheduled workflow; operational errors still fail both commands.

Next steps

• Runtime setup — executor and credential options for local runs and CI
• Grader taxonomy — pick graders by determinism and cost
• Writing eval specs — design better stimuli