Code Review
This category evaluates an agent’s ability to review a Business Central (AL) pull request. Given a diff, the agent produces structured review comments, which are scored against an expected (gold) set of findings.
Unlike the pass/fail categories, code review is scored with Precision / Recall / F1 over the matched comments. Every same-file expected/generated pair is sent to an LLM judge before one-to-one assignment; only pairs confirmed to describe the same underlying issue are eligible. There is no line-distance threshold or top-k pruning: a nearby unrelated finding must not hide a farther semantic match. Matched comments are additionally scored on how closely the agent’s severity classification tracks the expected severity.
A gold entry may also declare ignored_comments — legitimate-but-optional observations (out-of-scope nitpicks, maintainer-judgment calls) that should be neither required nor penalized. All same-file ignored/generated pairs are judged in the same single pass as the expected candidates, using the fixed judge model rather than the experiment model. The final joint assignment first maximizes expected matches, then ignored matches, then minimizes total line distance. Each gold and generated comment can be used at most once; equal-value duplicate comments remain distinct. An assigned ignored match is dropped from scoring entirely: it earns no recall and does not count against precision. Thus expected credit takes precedence, but equally creditable assignments preserve as much ignored neutralization as possible.
The judge receives sum((expected_in_file + ignored_in_file) * generated_in_file) candidates. This deliberate increase in judge input avoids irreversible location-only pruning; entries without candidates need no judge call. Judge failures propagate rather than falling back to structural matches. Existing published results are not rescored by this change, and historical recall improvements must be measured rather than assumed.
CodeReviewResult.create requires explicit expected and ignored match lists from the caller. It only computes metrics from those established matches; it never infers matches from comment locations or invokes the judge itself.
Category and runners
code-review is the evaluation contract: it owns the dataset, structured review.json output, scorer, result schema, and leaderboard schema. A runner is the system under test. The same entries can be evaluated through the generic GitHub Copilot CLI and Claude Code runners, allowing direct cross-system comparisons under one scorer.
BC PR Review is a separate agent harness fixed to the code-review category. It runs the production BC-ALAgents review engine with BCQuality, while generic Copilot and Claude runners continue to use their own prompts and configuration:
bcbench evaluate copilot <entry> --category code-review
bcbench evaluate claude <entry> --category code-review
bcbench evaluate pr-review <entry>
BC-ALAgents is the PR Review harness boundary. Its repo and default commit are pinned with the other harnesses in .github/actions/install-agent-harnesses, and that engine commit owns the BCQuality version through its own configuration. Changing the default pin still requires a new BC-Bench version and must record the BC-ALAgents commit SHA in the release notes.
For a durable experiment, push the pipeline and/or BCQuality changes through a BC-ALAgents branch, update the action pin to that immutable commit, then run BC-Bench from a branch with a draft PR describing the experiment (see EXPERIMENT.md). This keeps the reproducible dependency chain BC-Bench -> BC-ALAgents -> BCQuality, and records why the revision was evaluated.
The pr-review workflow also accepts an engine-sha input — a full 40-character BC-ALAgents commit SHA — as a convenience for a quick look at a revision without branching or re-pinning. It does not replace the process above: nothing records the intent behind the run, so an override is scored but never published to Braintrust/Kusto or the leaderboard. Blank keeps the default pin, requeued repeats retain the override, and the SHA that ran is recorded as agent_version on every result — read it from the job summary and run artifacts.
Either way, hold everything else fixed: the benchmark version, the model, the Copilot CLI version the engine uses internally, and the configured minimum severity. Locally, bcbench evaluate pr-review --engine-path <checkout> requires a clean engine checkout and uses the configured severity; use bcbench run pr-review for dirty-checkout smoke tests or --min-severity overrides.
BC PR Review records wall-clock duration, prompt/completion/total tokens, and exact AI credits. Usage values come from the engine’s strictly validated schema-v1 _run-metrics.json, never from console transcripts. API-call details, knowledge-filter counts, token subcategories, completeness diagnostics, and producer metadata remain in that raw artifact rather than being promoted into BC-Bench result and leaderboard schemas.
Unavailable AI credits remain null in bceval exports; observed zero remains zero. The pinned bc-eval 0.3.14 consumer requires numeric prompt/completion tokens, so its existing zero fallbacks for missing tokens remain unchanged. Use the original per-entry result metrics, not bceval token fields, to distinguish unknown usage from measured zero.
Baseline Leaderboard
| Agent | Model | Micro F1 (95% CI) | Precision | Recall | Valid Output | Avg Time | Ver |
|---|---|---|---|---|---|---|---|
| BC PR Review | gemini-3-7-flash | 38.0% (37.3-39.3%) | 35.8% | 40.6% | 100.0% | 539.6s | 0.10.0 |
| BC PR Review | gpt-5-3-codex | 32.6% (30.2-36.0%) | 29.9% | 35.9% | 100.0% | 337.2s | 0.10.0 |
| GitHub Copilot | claude-opus-4-6 | 32.2% (30.6-33.3%) | 28.7% | 36.6% | 100.0% | 67.1s | 0.6.0 |
| GitHub Copilot | claude-opus-4-6 | 31.8% (29.2-33.0%) | 26.4% | 40.0% | 100.0% | 51.0s | 0.7.0 |
| GitHub Copilot | claude-opus-4-8 | 31.4% (29.3-32.9%) | 26.6% | 38.2% | 100.0% | 60.7s | 0.6.0 |
| BC PR Review | mai-code-1-1-flash | 31.2% (30.5-31.9%) | 35.6% | 27.8% | 100.0% | 183.8s | 0.10.0 |
| GitHub Copilot | claude-opus-4-8 | 30.6% (29.3-32.6%) | 24.4% | 41.0% | 100.0% | 62.5s | 0.7.1 |
| Claude Code | claude-opus-4-8 | 29.3% (28.2-30.7%) | 22.6% | 41.8% | 100.0% | 57.2s | 0.6.0 |
| GitHub Copilot | claude-opus-4-8 | 28.6% (27.8-30.1%) | 22.5% | 39.4% | 100.0% | 62.4s | 0.7.0 |
| BC PR Review | gpt-5-6-sol | 28.2% (28.0-28.5%) | 19.6% | 50.0% | 100.0% | 372.4s | 0.10.0 |
| BC PR Review | gpt-5-6-luna | 27.9% (25.9-31.3%) | 21.1% | 41.2% | 100.0% | 291.8s | 0.10.0 |
| GitHub Copilot | claude-sonnet-4-6 | 27.3% (25.8-28.6%) | 20.9% | 39.4% | 100.0% | 83.2s | 0.6.0 |
| BC PR Review | gpt-5-6-terra | 25.4% (24.5-26.8%) | 18.7% | 39.3% | 100.0% | 342.4s | 0.10.0 |
| BC PR Review | claude-opus-5 | 24.4% (23.2-26.4%) | 15.8% | 53.7% | 100.0% | 801.7s | 0.10.0 |
| BC PR Review | claude-sonnet-5 | 24.1% (23.1-25.9%) | 17.0% | 40.7% | 100.0% | 369.8s | 0.10.0 |
| Claude Code | claude-sonnet-4-6 | 23.9% (23.2-24.5%) | 16.8% | 41.3% | 100.0% | 75.7s | 0.6.0 |
| GitHub Copilot | claude-opus-4-7 | 22.1% (21.6-23.4%) | 15.5% | 38.3% | 100.0% | 49.5s | 0.6.0 |
| GitHub Copilot | gpt-5-6-luna | 20.0% (18.8-20.9%) | 19.5% | 20.6% | 100.0% | 70.0s | 0.11.0 |
| GitHub Copilot | gpt-5-5 | 19.7% (18.9-21.0%) | 22.8% | 17.3% | 100.0% | 50.0s | 0.6.0 |
| GitHub Copilot | claude-haiku-4-5 | 19.7% (18.0-21.3%) | 18.8% | 20.6% | 100.0% | 51.5s | 0.6.0 |
| GitHub Copilot | gpt-5-6-terra | 16.7% (15.6-18.8%) | 16.0% | 17.6% | 100.0% | 64.9s | 0.11.0 |
Performance Leaderboard
| Agent | Model | Avg Time | Avg Prompt Tokens | Avg Completion Tokens | Avg Total Tokens | Avg AI Credits | Ver |
|---|---|---|---|---|---|---|---|
| GitHub Copilot | gpt-5-6-luna | 48.7s | — | — | — | 2.2104 | 0.11.0 |
| GitHub Copilot | claude-opus-4-7 | 49.5s | 120680 | 2571 | — | — | 0.6.0 |
| GitHub Copilot | gpt-5-5 | 50.0s | 162263 | 2257 | — | — | 0.6.0 |
| GitHub Copilot | claude-opus-4-6 | 51.0s | 91834 | 1411 | — | — | 0.7.0 |
| GitHub Copilot | claude-haiku-4-5 | 51.5s | 125101 | 1748 | — | — | 0.6.0 |
| Claude Code | claude-opus-4-8 | 57.2s | 87985 | 3837 | — | — | 0.6.0 |
| GitHub Copilot | claude-opus-4-8 | 60.7s | 153333 | 3350 | — | — | 0.6.0 |
| GitHub Copilot | claude-opus-4-8 | 62.4s | 154633 | 3510 | — | — | 0.7.0 |
| GitHub Copilot | claude-opus-4-8 | 62.5s | 156808 | 3464 | — | — | 0.7.1 |
| GitHub Copilot | gpt-5-6-terra | 64.9s | — | — | — | 18.2619 | 0.11.0 |
| GitHub Copilot | claude-opus-4-6 | 67.1s | 107224 | 2025 | — | — | 0.6.0 |
| GitHub Copilot | gpt-5-6-luna | 70.0s | — | — | — | 4.5454 | 0.11.0 |
| GitHub Copilot | gpt-5-5 | 72.6s | 365586 | 3538 | — | — | 0.6.1 |
| Claude Code | claude-sonnet-4-6 | 75.7s | 78632 | 4581 | — | — | 0.6.0 |
| GitHub Copilot | claude-sonnet-4-6 | 83.2s | 105088 | 3623 | — | — | 0.6.0 |
| Claude Code | claude-opus-4-8 | 104.1s | 318077 | 7423 | — | — | 0.6.1 |
| GitHub Copilot | claude-opus-4-6 | 119.5s | 310331 | 4176 | — | — | 0.6.1 |
| GitHub Copilot | claude-opus-4-8 | 138.0s | 622184 | 9463 | — | — | 0.6.1 |
| GitHub Copilot | claude-opus-4-8 | 138.8s | 598062 | 9017 | — | — | 0.7.1 |
| BC PR Review | mai-code-1-1-flash | 183.8s | 355684 | 10238 | 365923 | 4.322 | 0.10.0 |
| BC PR Review | gpt-5-6-luna | 291.8s | — | — | — | — | 0.10.0 |
| BC PR Review | gpt-5-3-codex | 337.2s | — | — | — | — | 0.10.0 |
| BC PR Review | gpt-5-6-terra | 342.4s | — | — | — | — | 0.10.0 |
| BC PR Review | claude-sonnet-5 | 369.8s | 939078 | 18410 | 957488 | 49.682 | 0.10.0 |
| BC PR Review | gpt-5-6-sol | 372.4s | — | — | — | — | 0.10.0 |
| BC PR Review | gemini-3-7-flash | 539.6s | 1094217 | 7655 | 1101872 | 30.4883 | 0.10.0 |
| BC PR Review | claude-opus-5 | 801.7s | — | — | — | — | 0.10.0 |
Experiment Leaderboard
Compares review-knowledge configurations for the same model (see the Baseline Leaderboard above for the plain agent):
- Inline knowledge (pre-#8700) — the review checklists BCApps shipped inline before adopting BCQuality, injected as custom instructions.
| Variant | Agent | Model | Micro F1 (95% CI) | Macro F1 (95% CI) | Precision | Recall | Valid Output | Avg Time | Ver |
|---|---|---|---|---|---|---|---|---|---|
| Inline knowledge (pre-#8700) | GitHub Copilot | gpt-5-5 | 53.2% (49.7-54.8%) | 64.2% (60.8-67.4%) | 52.0% | 54.5% | 100.0% | 72.6s | 0.6.1 |
| Inline knowledge (pre-#8700) | GitHub Copilot | claude-opus-4-8 | 53.1% (49.2-55.8%) | 58.6% (55.4-61.6%) | 42.9% | 69.9% | 100.0% | 138.8s | 0.7.1 |
| Inline knowledge (pre-#8700) | Claude Code | claude-opus-4-8 | 52.6% (51.5-53.6%) | 62.5% (59.6-65.4%) | 44.0% | 65.5% | 100.0% | 104.1s | 0.6.1 |
| Inline knowledge (pre-#8700) | GitHub Copilot | claude-opus-4-8 | 50.4% (48.9-52.8%) | 59.9% (56.8-62.8%) | 41.9% | 63.2% | 100.0% | 138.0s | 0.6.1 |
| Inline knowledge (pre-#8700) | GitHub Copilot | claude-opus-4-6 | 44.1% (42.6-46.1%) | 50.1% (46.7-53.4%) | 38.2% | 52.1% | 100.0% | 119.5s | 0.6.1 |
| Other | GitHub Copilot | gpt-5-6-luna | 34.9% (33.6-36.3%) | 41.9% (38.9-45.0%) | 30.6% | 40.8% | 100.0% | 48.7s | 0.11.0 |
How metrics are computed
- Precision — of the scorable comments the agent generated (generated minus ignored), the fraction that matched an expected finding. Penalizes noisy reviews.
- Recall — of the expected findings, the fraction the agent caught. Penalizes missed issues.
- F1 — harmonic mean of precision and recall; balances both equally (the β=1 case of Fβ).
- Fβ (β=0.5) — precision-leaning F-score; use when false positives are costly (noisy reviews waste reviewer time).
- Fβ (β=2) — recall-leaning F-score; weights catching issues more than avoiding noise.
- Severity MAE — mean absolute error between the agent’s and the expected severity levels, over matched comments only. Lower is better;
0means every matched comment got the severity exactly right. - Ignored — generated comments that matched an entry’s
ignored_commentsset. These are excluded from precision (they are neither correct nor incorrect); the count is surfaced for transparency only. - Valid output rate — fraction of tasks whose output parsed into a structured review. Failures score zero on every other metric. (Reported per run.)
- Micro vs. Macro — Micro sums matched, scorable generated (generated minus ignored), and expected across all tasks (tasks with many comments dominate); Macro averages per-task scores (every task counts equally).
- 95% CI — confidence interval bootstrapped over the per-task F1 scores, so the leaderboard reports sampling uncertainty even for a single run. The micro
F1CI resamples runs; theMacro F1CI resamples tasks.