Working with prompt features

A prompt feature is a boolean characteristic detected on a task prompt — for example asks_for_api, asks_for_testing, or asks_for_docker. Features let you slice and group runs across heterogeneous tasks: "how does profile X do on prompts that ask for testing?"

Features are managed in a shared catalog. Each feature has an ID, a description, and a detection prompt the LLM uses to decide whether the feature applies to a given task prompt.

Extracting features from a task prompt

Feature extraction is on-demand, not automatic on run submission. Trigger it from the Portal or the API.

Portal

Open a task prompt's detail page and click Extract features. The page shows which catalog features were detected, plus any new features the LLM suggests adding based on the prompt.

REST API

curl

curl --request POST \
  --url https://your-scope.example.com/api/v1/task-prompts/%7Bid%7D/extract-features

JS (fetch)

const url = 'https://your-scope.example.com/api/v1/task-prompts/%7Bid%7D/extract-features';
const options = {method: 'POST'};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

Python

import requests

url = "https://your-scope.example.com/api/v1/task-prompts/%7Bid%7D/extract-features"

response = requests.post(url)

print(response.json())

Go

package main

import (
  "fmt"
  "net/http"
  "io"
)

func main() {

  url := "https://your-scope.example.com/api/v1/task-prompts/%7Bid%7D/extract-features"

  req, _ := http.NewRequest("POST", url, nil)

  res, _ := http.DefaultClient.Do(req)

  defer res.Body.Close()
  body, _ := io.ReadAll(res.Body)

  fmt.Println(res)
  fmt.Println(string(body))

}

Java

OkHttpClient client = new OkHttpClient();

Request request = new Request.Builder()
  .url("https://your-scope.example.com/api/v1/task-prompts/%7Bid%7D/extract-features")
  .post(null)
  .build();

Response response = client.newCall(request).execute();

C#

using System.Net.Http.Headers;
var client = new HttpClient();
var request = new HttpRequestMessage
{
    Method = HttpMethod.Post,
    RequestUri = new Uri("https://your-scope.example.com/api/v1/task-prompts/%7Bid%7D/extract-features"),
};
using (var response = await client.SendAsync(request))
{
    response.EnsureSuccessStatusCode();
    var body = await response.Content.ReadAsStringAsync();
    Console.WriteLine(body);
}

Add ?force=true to bypass the cache and re-run extraction. The response returns the list of detected features, the extraction timestamp, and any suggested new features.

Extraction is cached: re-extracting the same task prompt returns the cached result unless you pass force=true.

Browsing the feature catalog

The Portal's Prompt features page lists every defined feature, its ID, and its detection prompt. Use it to:

• See what the LLM is looking for under each label.
• Search and filter features by ID or text.
• Open a feature's detail page to read its full detection prompt.

Manually overriding a detection

Sometimes the LLM gets it wrong, or you want to mark a feature as applying for reasons the prompt doesn't make obvious.

Portal

On the task prompt detail page, toggle the feature's Detected switch. The change is recorded as a manual evaluation and persists across re-extractions.

REST API

curl

curl --request PATCH \
  --url https://your-scope.example.com/api/v1/task-prompts/%7Bid%7D/features/%7BfeatureId%7D \
  --header 'Content-Type: application/json' \
  --data '{ "detected": true }'

JS (fetch)

const url = 'https://your-scope.example.com/api/v1/task-prompts/%7Bid%7D/features/%7BfeatureId%7D';
const options = {
  method: 'PATCH',
  headers: {'Content-Type': 'application/json'},
  body: '{"detected":true}'
};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

Python

import requests

url = "https://your-scope.example.com/api/v1/task-prompts/%7Bid%7D/features/%7BfeatureId%7D"

payload = { "detected": True }
headers = {"Content-Type": "application/json"}

response = requests.patch(url, json=payload, headers=headers)

print(response.json())

Go

package main

import (
  "fmt"
  "strings"
  "net/http"
  "io"
)

func main() {

  url := "https://your-scope.example.com/api/v1/task-prompts/%7Bid%7D/features/%7BfeatureId%7D"

  payload := strings.NewReader("{ \"detected\": true }")

  req, _ := http.NewRequest("PATCH", url, payload)

  req.Header.Add("Content-Type", "application/json")

  res, _ := http.DefaultClient.Do(req)

  defer res.Body.Close()
  body, _ := io.ReadAll(res.Body)

  fmt.Println(res)
  fmt.Println(string(body))

}

Java

OkHttpClient client = new OkHttpClient();

MediaType mediaType = MediaType.parse("application/json");
RequestBody body = RequestBody.create(mediaType, "{ \"detected\": true }");
Request request = new Request.Builder()
  .url("https://your-scope.example.com/api/v1/task-prompts/%7Bid%7D/features/%7BfeatureId%7D")
  .patch(body)
  .addHeader("Content-Type", "application/json")
  .build();

Response response = client.newCall(request).execute();

C#

using System.Net.Http.Headers;
var client = new HttpClient();
var request = new HttpRequestMessage
{
    Method = HttpMethod.Patch,
    RequestUri = new Uri("https://your-scope.example.com/api/v1/task-prompts/%7Bid%7D/features/%7BfeatureId%7D"),
    Content = new StringContent("{ \"detected\": true }")
    {
        Headers =
        {
            ContentType = new MediaTypeHeaderValue("application/json")
        }
    }
};
using (var response = await client.SendAsync(request))
{
    response.EnsureSuccessStatusCode();
    var body = await response.Content.ReadAsStringAsync();
    Console.WriteLine(body);
}

Adding a new feature

You can add features yourself when an existing one doesn't capture what you want.

AI-assisted creation (recommended)

In the Portal, the Create feature wizard accepts a plain-English description of the behavior you want to detect ("does the prompt ask for a database?") and uses an LLM to:

• Suggest a snake_case ID.
• Draft a detection prompt.
• Optionally suggest parent/child relationships with existing features.

You can edit any of the generated fields before saving.

The same is exposed via the API:

curl

curl --request POST \
  --url https://your-scope.example.com/api/v1/prompt-features/generate-prompt \
  --header 'Content-Type: application/json' \
  --data '{ "behavior": "Asks the agent to write a Dockerfile or container setup" }'

JS (fetch)

const url = 'https://your-scope.example.com/api/v1/prompt-features/generate-prompt';
const options = {
  method: 'POST',
  headers: {'Content-Type': 'application/json'},
  body: '{"behavior":"Asks the agent to write a Dockerfile or container setup"}'
};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

Python

import requests

url = "https://your-scope.example.com/api/v1/prompt-features/generate-prompt"

payload = { "behavior": "Asks the agent to write a Dockerfile or container setup" }
headers = {"Content-Type": "application/json"}

response = requests.post(url, json=payload, headers=headers)

print(response.json())

Go

package main

import (
  "fmt"
  "strings"
  "net/http"
  "io"
)

func main() {

  url := "https://your-scope.example.com/api/v1/prompt-features/generate-prompt"

  payload := strings.NewReader("{ \"behavior\": \"Asks the agent to write a Dockerfile or container setup\" }")

  req, _ := http.NewRequest("POST", url, payload)

  req.Header.Add("Content-Type", "application/json")

  res, _ := http.DefaultClient.Do(req)

  defer res.Body.Close()
  body, _ := io.ReadAll(res.Body)

  fmt.Println(res)
  fmt.Println(string(body))

}

Java

OkHttpClient client = new OkHttpClient();

MediaType mediaType = MediaType.parse("application/json");
RequestBody body = RequestBody.create(mediaType, "{ \"behavior\": \"Asks the agent to write a Dockerfile or container setup\" }");
Request request = new Request.Builder()
  .url("https://your-scope.example.com/api/v1/prompt-features/generate-prompt")
  .post(body)
  .addHeader("Content-Type", "application/json")
  .build();

Response response = client.newCall(request).execute();

C#

using System.Net.Http.Headers;
var client = new HttpClient();
var request = new HttpRequestMessage
{
    Method = HttpMethod.Post,
    RequestUri = new Uri("https://your-scope.example.com/api/v1/prompt-features/generate-prompt"),
    Content = new StringContent("{ \"behavior\": \"Asks the agent to write a Dockerfile or container setup\" }")
    {
        Headers =
        {
            ContentType = new MediaTypeHeaderValue("application/json")
        }
    }
};
using (var response = await client.SendAsync(request))
{
    response.EnsureSuccessStatusCode();
    var body = await response.Content.ReadAsStringAsync();
    Console.WriteLine(body);
}

From a suggestion

When you extract features on a task prompt, the response (and the Portal page) often shows suggested features the LLM thinks would be useful additions to the catalog. Click Promote in the Portal, or take the suggestion through the wizard, to add it.

Tips

• Use features to find patterns. They're most useful when you filter or group runs by them in dashboards — not when read one at a time.
• Promote only stable features. Each catalog feature is a commitment to keep its detection prompt healthy. Don't promote every suggestion.
• Edit detection prompts when results drift. If a feature is consistently misfiring, edit its detection prompt — that's the knob.

Limitations

• Feature extraction and generation use an LLM and can return 503 when the model service is unavailable or rate-limited. Retry, or re-trigger extraction from the Portal.
• Detection is probabilistic. For high-stakes filtering, spot-check a sample and use manual overrides where needed.

See also

• Prompt feature schema reference
• Managing task prompts