Code Review
This category evaluates an agent’s ability to review a Business Central (AL) pull request. Given a diff, the agent produces structured review comments, which are scored against an expected (gold) set of findings.
Unlike the pass/fail categories, code review is scored with Precision / Recall / F1 over the matched comments. Expected and generated comments are paired by a globally optimal (one-to-one) assignment on file and line proximity (within a configured line tolerance), and a pair is only counted as matched when an LLM judge confirms the two describe the same underlying issue. Matched comments are additionally scored on how closely the agent’s severity classification tracks the expected severity.
A gold entry may also declare ignored_comments — legitimate-but-optional observations (out-of-scope nitpicks, maintainer-judgment calls) that should be neither required nor penalized. Ignored comments are structurally paired against the generated comments and validated by the same LLM judge, in a single judge pass alongside the expected comments. Any generated comment the judge confirms as an ignored match is dropped from scoring entirely: it earns no recall and does not count against precision. Expected always takes precedence, so a comment that could match both is credited as a real find; a comment that does not hold up as an expected match can still be neutralized as ignored rather than counting as a false positive. Most entries leave ignored_comments empty, which scores identically to before.
Category and runners
code-review is the evaluation contract: it owns the dataset, structured review.json output, scorer, result schema, and leaderboard schema. A runner is the system under test. The same entries can be evaluated through the generic GitHub Copilot CLI and Claude Code runners, allowing direct cross-system comparisons under one scorer.
BC PR Review is a separate agent harness fixed to the code-review category. It runs the production BC-ALAgents review engine with BCQuality, while generic Copilot and Claude runners continue to use their own prompts and configuration:
bcbench evaluate copilot <entry> --category code-review
bcbench evaluate claude <entry> --category code-review
bcbench evaluate pr-review <entry>
BC-ALAgents is the PR Review harness boundary. Its repo and commit are pinned with the other harnesses in .github/actions/install-agent-harnesses, and that engine commit owns the BCQuality version through its own configuration. Engine updates require a new BC-Bench version and must record the BC-ALAgents commit SHA in the release notes.
For an experiment, push the pipeline and/or BCQuality changes through a BC-ALAgents branch, update the action pin to that immutable commit, then run BC-Bench from the corresponding BC-Bench commit. This keeps the reproducible dependency chain BC-Bench -> BC-ALAgents -> BCQuality. A local BC-ALAgents checkout can be supplied with --engine-path for smoke testing.
BC PR Review records wall-clock duration, prompt/completion/total tokens, and exact AI credits. Usage values come from the engine’s strictly validated schema-v1 _run-metrics.json, never from console transcripts. API-call details, knowledge-filter counts, token subcategories, completeness diagnostics, and producer metadata remain in that raw artifact rather than being promoted into BC-Bench result and leaderboard schemas.
Baseline Leaderboard
| Agent | Model | Micro F1 (95% CI) | Precision | Recall | Valid Output | Avg Time | Ver |
|---|---|---|---|---|---|---|---|
| GitHub Copilot | claude-opus-4-6 | 32.2% (30.6-33.3%) | 28.7% | 36.6% | — | 67.1s | 0.6.0 |
| GitHub Copilot | claude-opus-4-6 | 31.8% (29.2-33.0%) | 26.4% | 40.0% | — | 51.0s | 0.7.0 |
| GitHub Copilot | claude-opus-4-8 | 31.4% (29.3-32.9%) | 26.6% | 38.2% | — | 60.7s | 0.6.0 |
| GitHub Copilot | claude-opus-4-8 | 30.6% (29.3-32.6%) | 24.4% | 41.0% | — | 62.5s | 0.7.1 |
| Claude Code | claude-opus-4-8 | 29.3% (28.2-30.7%) | 22.6% | 41.8% | — | 57.2s | 0.6.0 |
| GitHub Copilot | claude-opus-4-8 | 28.6% (27.8-30.1%) | 22.5% | 39.4% | — | 62.4s | 0.7.0 |
| GitHub Copilot | claude-sonnet-4-6 | 27.3% (25.8-28.6%) | 20.9% | 39.4% | — | 83.2s | 0.6.0 |
| Claude Code | claude-sonnet-4-6 | 23.9% (23.2-24.5%) | 16.8% | 41.3% | — | 75.7s | 0.6.0 |
| GitHub Copilot | claude-opus-4-7 | 22.1% (21.6-23.4%) | 15.5% | 38.3% | — | 49.5s | 0.6.0 |
| GitHub Copilot | gpt-5-5 | 19.7% (18.9-21.0%) | 22.8% | 17.3% | — | 50.0s | 0.6.0 |
| GitHub Copilot | claude-haiku-4-5 | 19.7% (18.0-21.3%) | 18.8% | 20.6% | — | 51.5s | 0.6.0 |
Performance Leaderboard
| Agent | Model | Avg Time | Avg Prompt Tokens | Avg Completion Tokens | Avg Total Tokens | Avg AI Credits | Ver |
|---|---|---|---|---|---|---|---|
| GitHub Copilot | claude-opus-4-7 | 49.5s | — | — | — | — | 0.6.0 |
| GitHub Copilot | gpt-5-5 | 50.0s | — | — | — | — | 0.6.0 |
| GitHub Copilot | claude-opus-4-6 | 51.0s | — | — | — | — | 0.7.0 |
| GitHub Copilot | claude-haiku-4-5 | 51.5s | — | — | — | — | 0.6.0 |
| Claude Code | claude-opus-4-8 | 57.2s | — | — | — | — | 0.6.0 |
| GitHub Copilot | claude-opus-4-8 | 60.7s | — | — | — | — | 0.6.0 |
| GitHub Copilot | claude-opus-4-8 | 62.4s | — | — | — | — | 0.7.0 |
| GitHub Copilot | claude-opus-4-8 | 62.5s | — | — | — | — | 0.7.1 |
| GitHub Copilot | claude-opus-4-6 | 67.1s | — | — | — | — | 0.6.0 |
| GitHub Copilot | gpt-5-5 | 72.6s | — | — | — | — | 0.6.1 |
| Claude Code | claude-sonnet-4-6 | 75.7s | — | — | — | — | 0.6.0 |
| GitHub Copilot | claude-sonnet-4-6 | 83.2s | — | — | — | — | 0.6.0 |
| Claude Code | claude-opus-4-8 | 104.1s | — | — | — | — | 0.6.1 |
| GitHub Copilot | claude-opus-4-6 | 119.5s | — | — | — | — | 0.6.1 |
| GitHub Copilot | claude-opus-4-8 | 138.0s | — | — | — | — | 0.6.1 |
| GitHub Copilot | claude-opus-4-8 | 138.8s | — | — | — | — | 0.7.1 |
Experiment Leaderboard
Compares review-knowledge configurations for the same model (see the Baseline Leaderboard above for the plain agent):
- Inline knowledge (pre-#8700) — the review checklists BCApps shipped inline before adopting BCQuality, injected as custom instructions.
| Variant | Agent | Model | Micro F1 (95% CI) | Macro F1 (95% CI) | Precision | Recall | Valid Output | Avg Time | Ver |
|---|---|---|---|---|---|---|---|---|---|
| Inline knowledge (pre-#8700) | GitHub Copilot | gpt-5-5 | 53.2% (49.7-54.8%) | 64.2% (60.8-67.4%) | 52.0% | 54.5% | — | 72.6s | 0.6.1 |
| Inline knowledge (pre-#8700) | GitHub Copilot | claude-opus-4-8 | 53.1% (49.2-55.8%) | 58.6% (55.4-61.6%) | 42.9% | 69.9% | — | 138.8s | 0.7.1 |
| Inline knowledge (pre-#8700) | Claude Code | claude-opus-4-8 | 52.6% (51.5-53.6%) | 62.5% (59.6-65.4%) | 44.0% | 65.5% | — | 104.1s | 0.6.1 |
| Inline knowledge (pre-#8700) | GitHub Copilot | claude-opus-4-8 | 50.4% (48.9-52.8%) | 59.9% (56.8-62.8%) | 41.9% | 63.2% | — | 138.0s | 0.6.1 |
| Inline knowledge (pre-#8700) | GitHub Copilot | claude-opus-4-6 | 44.1% (42.6-46.1%) | 50.1% (46.7-53.4%) | 38.2% | 52.1% | — | 119.5s | 0.6.1 |
How metrics are computed
- Precision — of the scorable comments the agent generated (generated minus ignored), the fraction that matched an expected finding. Penalizes noisy reviews.
- Recall — of the expected findings, the fraction the agent caught. Penalizes missed issues.
- F1 — harmonic mean of precision and recall; balances both equally (the β=1 case of Fβ).
- Fβ (β=0.5) — precision-leaning F-score; use when false positives are costly (noisy reviews waste reviewer time).
- Fβ (β=2) — recall-leaning F-score; weights catching issues more than avoiding noise.
- Severity MAE — mean absolute error between the agent’s and the expected severity levels, over matched comments only. Lower is better;
0means every matched comment got the severity exactly right. - Ignored — generated comments that matched an entry’s
ignored_commentsset. These are excluded from precision (they are neither correct nor incorrect); the count is surfaced for transparency only. - Valid output rate — fraction of tasks whose output parsed into a structured review. Failures score zero on every other metric. (Reported per run.)
- Micro vs. Macro — Micro sums matched, scorable generated (generated minus ignored), and expected across all tasks (tasks with many comments dominate); Macro averages per-task scores (every task counts equally).
- 95% CI — confidence interval bootstrapped over the per-task F1 scores, so the leaderboard reports sampling uncertainty even for a single run. The micro
F1CI resamples runs; theMacro F1CI resamples tasks.