Defining evaluation criteria

Evaluation criteria describe what makes a run successful. The judge evaluates the agent's output against each criterion and produces the run's report.

Criteria are first-class, reusable entities: each criterion has a stable ID and a prompt describing what to look for, and a request references criteria by ID.

The criteria graph

Every criterion lives in a directed acyclic graph (DAG). Dependencies between criteria are declared with dependsOn. A criterion is evaluated only when all of its parents pass; if a parent fails, its descendants are skipped and reported as such.

A criterion with no dependsOn is a root — it has no prerequisites and is always evaluated. A graph that is all roots is still a DAG; it just has no edges. There is no separate "flat checklist" mode.

Writing a single criterion

A criterion is a single, observable statement. The judge reads the agent's output (files, logs, command output) and decides whether the statement holds.

Good criteria are:

• Observable — a human reading the same output would reach the same conclusion.
• Atomic — one thing per criterion. Split compound statements.
• Outcome-focused — describe the result, not the path. "The server returns JSON on /health" is better than "The agent uses Express's built-in JSON middleware."

Examples:

• ✅ Has a package.json with express as a dependency.
• ✅ GET /health returns HTTP 200 with a JSON body.
• ✅ No tests fail when running npm test.
• ❌ Code is high-quality. (not observable consistently)
• ❌ Has Express routes and a database connection. (not atomic)

Building the graph

Use dependsOn when:

• A later check is meaningless without an earlier one. (Don't bother checking GET / returns hello-world if there's no Express server.)
• You want to express alternative paths. (Pass if SQLite is set up correctly OR Postgres is set up correctly.)
• A task has natural stages and you want a clear report of how far the agent got.

Example:

- id: has_package_json
  prompt: Has a package.json with express as a dependency
- id: has_express_server
  prompt: Has a main entry file that creates an Express server
  dependsOn: [has_package_json]
- id: has_root_route
  prompt: GET / returns a hello world response
  dependsOn: [has_express_server]
- id: configurable_port
  prompt: The server listens on a configurable port
  dependsOn: [has_express_server]

The judge evaluates has_package_json first, then has_express_server only if it passed, then has_root_route and configurable_port in parallel only if has_express_server passed.

If you don't need ordering or conditional evaluation, just don't set dependsOn — every criterion becomes a root and they're all evaluated independently. Cycles in dependsOn are rejected.

For the full schema (field names, validation rules), see the Criteria schema reference.

Auto-generating evaluation prompts

When you create a criterion, Scope can generate the underlying evaluation prompt for you using GitHub Models. The prompt describes how the judge should look for evidence in the run's output. You can edit the generated prompt before saving.

This is exposed in the Portal's criterion editor as Generate prompt and through the API as a related endpoint. See the Swagger reference for the exact route.

Creating criteria

Portal

1. Open Criteria in the navigation.
2. Click New criterion, give it an ID and a prompt.
3. To make it depend on others, set dependsOn to the IDs of its parents.
4. Save.

REST API

Criteria are created one at a time:

curl

curl --request POST \
  --url https://your-scope.example.com/api/v1/criteria \
  --header 'Content-Type: application/json' \
  --data '{
  "id": "has_package_json",
  "prompt": "Has a package.json with express as a dependency"
}'

JS (fetch)

const url = 'https://your-scope.example.com/api/v1/criteria';
const options = {
  method: 'POST',
  headers: {'Content-Type': 'application/json'},
  body: '{"id":"has_package_json","prompt":"Has a package.json with express as a dependency"}'
};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

Python

import requests

url = "https://your-scope.example.com/api/v1/criteria"

payload = {
    "id": "has_package_json",
    "prompt": "Has a package.json with express as a dependency"
}
headers = {"Content-Type": "application/json"}

response = requests.post(url, json=payload, headers=headers)

print(response.json())

Go

package main

import (
  "fmt"
  "strings"
  "net/http"
  "io"
)

func main() {

  url := "https://your-scope.example.com/api/v1/criteria"

  payload := strings.NewReader("{\n  \"id\": \"has_package_json\",\n  \"prompt\": \"Has a package.json with express as a dependency\"\n}")

  req, _ := http.NewRequest("POST", url, payload)

  req.Header.Add("Content-Type", "application/json")

  res, _ := http.DefaultClient.Do(req)

  defer res.Body.Close()
  body, _ := io.ReadAll(res.Body)

  fmt.Println(res)
  fmt.Println(string(body))

}

Java

OkHttpClient client = new OkHttpClient();

MediaType mediaType = MediaType.parse("application/json");
RequestBody body = RequestBody.create(mediaType, "{\n  \"id\": \"has_package_json\",\n  \"prompt\": \"Has a package.json with express as a dependency\"\n}");
Request request = new Request.Builder()
  .url("https://your-scope.example.com/api/v1/criteria")
  .post(body)
  .addHeader("Content-Type", "application/json")
  .build();

Response response = client.newCall(request).execute();

C#

using System.Net.Http.Headers;
var client = new HttpClient();
var request = new HttpRequestMessage
{
    Method = HttpMethod.Post,
    RequestUri = new Uri("https://your-scope.example.com/api/v1/criteria"),
    Content = new StringContent("{\n  \"id\": \"has_package_json\",\n  \"prompt\": \"Has a package.json with express as a dependency\"\n}")
    {
        Headers =
        {
            ContentType = new MediaTypeHeaderValue("application/json")
        }
    }
};
using (var response = await client.SendAsync(request))
{
    response.EnsureSuccessStatusCode();
    var body = await response.Content.ReadAsStringAsync();
    Console.WriteLine(body);
}

With a dependency:

curl

curl --request POST \
  --url https://your-scope.example.com/api/v1/criteria \
  --header 'Content-Type: application/json' \
  --data '{
  "id": "has_express_server",
  "prompt": "Has a main entry file that creates an Express server",
  "dependsOn": ["has_package_json"]
}'

JS (fetch)

const url = 'https://your-scope.example.com/api/v1/criteria';
const options = {
  method: 'POST',
  headers: {'Content-Type': 'application/json'},
  body: '{"id":"has_express_server","prompt":"Has a main entry file that creates an Express server","dependsOn":["has_package_json"]}'
};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

Python

import requests

url = "https://your-scope.example.com/api/v1/criteria"

payload = {
    "id": "has_express_server",
    "prompt": "Has a main entry file that creates an Express server",
    "dependsOn": ["has_package_json"]
}
headers = {"Content-Type": "application/json"}

response = requests.post(url, json=payload, headers=headers)

print(response.json())

Go

package main

import (
  "fmt"
  "strings"
  "net/http"
  "io"
)

func main() {

  url := "https://your-scope.example.com/api/v1/criteria"

  payload := strings.NewReader("{\n  \"id\": \"has_express_server\",\n  \"prompt\": \"Has a main entry file that creates an Express server\",\n  \"dependsOn\": [\"has_package_json\"]\n}")

  req, _ := http.NewRequest("POST", url, payload)

  req.Header.Add("Content-Type", "application/json")

  res, _ := http.DefaultClient.Do(req)

  defer res.Body.Close()
  body, _ := io.ReadAll(res.Body)

  fmt.Println(res)
  fmt.Println(string(body))

}

Java

OkHttpClient client = new OkHttpClient();

MediaType mediaType = MediaType.parse("application/json");
RequestBody body = RequestBody.create(mediaType, "{\n  \"id\": \"has_express_server\",\n  \"prompt\": \"Has a main entry file that creates an Express server\",\n  \"dependsOn\": [\"has_package_json\"]\n}");
Request request = new Request.Builder()
  .url("https://your-scope.example.com/api/v1/criteria")
  .post(body)
  .addHeader("Content-Type", "application/json")
  .build();

Response response = client.newCall(request).execute();

C#

using System.Net.Http.Headers;
var client = new HttpClient();
var request = new HttpRequestMessage
{
    Method = HttpMethod.Post,
    RequestUri = new Uri("https://your-scope.example.com/api/v1/criteria"),
    Content = new StringContent("{\n  \"id\": \"has_express_server\",\n  \"prompt\": \"Has a main entry file that creates an Express server\",\n  \"dependsOn\": [\"has_package_json\"]\n}")
    {
        Headers =
        {
            ContentType = new MediaTypeHeaderValue("application/json")
        }
    }
};
using (var response = await client.SendAsync(request))
{
    response.EnsureSuccessStatusCode();
    var body = await response.Content.ReadAsStringAsync();
    Console.WriteLine(body);
}

A request references criteria by ID in scenario.criteria — see Submitting requests (REST API).

Editing criteria

Criteria are mutable. Editing replaces the stored definition. Past run reports are computed from the criteria as they were at submission time, so edits don't change historical results.

Tips

• Add edges only when they buy you something. A graph with no dependsOn edges is fine — don't invent dependencies just to make the graph "look richer."
• Keep IDs stable. Criterion IDs appear in reports and analyses; rename sparingly.
• Don't bake the answer in. Criteria should describe the outcome, not the implementation choice. Otherwise you're benchmarking the agent's ability to guess your style, not solve the task.
• One responsibility per criterion. Compound criteria make pass/fail rationale ambiguous.

See also

• Criteria schema reference
• Submitting requests (Portal)