Quality checklist

Use this checklist to self-review before running an evaluation or asking someone else to review its quality.

Scenario quality

• The prompt is something a developer would actually type
• Contains no evaluation metadata, scoring hints, or instructions about how to be evaluated
• Be mindful of vocabulary: developers may not know the internal product names or specific terms your team uses. Don't assume they'll use the "right" words

Criteria quality: accuracy

• You have a prerequisite gate: one or at most two unconditional checks. If they don't pass, further evaluation makes no sense. Example: you ask the agent to build a database solution assuming Cosmos DB (without naming it), and the agent writes MongoDB code. Evaluating the correctness of that MongoDB solution is pointless because the basic technology choice is wrong.
• Each criterion tests an aspect of your technology that matters for correct usage
• Pass conditions describe what correct usage looks like with specific packages, versions, or APIs named
• Fail conditions describe specific incorrect patterns (not just "does not use the technology correctly")
• Skip conditions cover cases where the criterion doesn't apply
• A perfect implementation would pass all criteria
• A clearly broken implementation would fail at least one criterion

Criteria quality: calibration

• Criteria are aligned with reasonable expectations for a developer using this technology, not with what your extension provides
• No criterion is so loose that everything passes
• No criterion is so strict that only one exact implementation passes
• Each criterion tests one thing (not multiple concerns in one criterion)
• Criteria test correctness, not style
• Criteria don't test things the agent can't control

Criteria quality: consistency

• Each criterion has explicit pass/fail/skip conditions that leave nothing to interpretation
• Criteria use technology-specific vocabulary the judge can verify (package names, API patterns, config keys) rather than vague quality terms

Criteria calibration

• Each criterion passes a known-good artifact and fails a known-bad artifact
• Repeated judgments of the same artifacts produce consistent verdicts
• Someone who knows the product has confirmed technical accuracy
• Repeated scoring has confirmed the wording produces consistent verdicts
• Calibration runs are kept separate from production results

Criteria maintenance

• The locked criteria set has a version, and production results record that version
• Material changes to the product, scenario, or starting workspace trigger a criteria review
• Criteria for older product versions are retained only while those versions remain supported
• Defects found after production are documented and corrected in a new criteria-set version
• Judge changes preserve backward compatibility or establish a clearly labeled measurement boundary

Profile selection

• Bare baseline is included (unless you're only comparing extension variations)
• If there's an extension to compare, at least one extension profile is included
• If there's no extension yet, running just the baseline is fine. It tells you whether gaps exist and helps you decide how to address them.
• Each profile tests a specific question

Before running

• At least 5 runs per scenario-profile combination
• Use environment and agent where the majority of your audience is to ensure you're optimizing for the maximum impact first

Ongoing measurement

• Rerun cadence reflects changes to the coding agents your audience uses and the team's capacity to respond
• Recurring evaluations use the same locked definition so that results remain comparable
• Product, scenario, or criteria changes start a new version rather than continuing the old trend

Evaluation system readiness

• Reproduces the operating system, harness, model, and extensions used by the largest audience cohort
• Configures scenarios, starting workspaces, criteria, and profiles independently
• Resets every run to the same starting workspace and runs each scenario-profile combination repeatedly
• Supports mechanical gates and structured checks for verifiable outcomes
• Supports context-aware criteria with explicit pass, fail, and skip conditions for semantic decisions
• Preserves generated artifacts, verdict evidence, execution results, task costs, and complete trajectories per run
• Compares extension profiles against a bare baseline, including quality and task cost
• Keeps run-level evidence available when results are aggregated