Overview
Unblocked Code Review in Deep mode, scored with the published pipeline of Martian’s open-source Code Review Bench and compared with the published results of other AI code reviewers on Martian’s leaderboard, including its top three. Every score, how the scores are calculated, and the result for each of the 158 known bugs, with a link to the comment that caught it.
How these numbers were produced. Martian built Code Review Bench and publishes the other reviewers’ results: each reviewer’s best-scoring version, run by Martian on earlier versions of those products. This page shows a selection of them, including the top three by F2. We ran Unblocked through Martian’s published pipeline, with the same issue extraction, de-duplication, judge and answer key. Unblocked is not yet listed on Martian’s leaderboard, which additionally requires several hundred public reviews and Martian running the pipeline itself.
Results
Unblocked and the reviewers compared here, scored the same way and sorted by F2, the score Martian’s dashboard shows by default. F2 counts recall twice as heavily as precision.
| # | Reviewer | F2 | Recall | Precision | Bugs caught, of 158 | Issues graded |
|---|---|---|---|---|---|---|
| 1 |  Unblocked | 66.2 | 68.4% | 58.7% | 108 | 184 |
| 2 |  Qodo Deep | 65.1 | 64.6% | 67.1% | 102 | 152 |
| 3 |  Cubic | 64.9 | 65.8% | 61.5% | 104 | 169 |
| 4 |  Augment | 64.0 | 65.2% | 59.5% | 103 | 173 |
| 5 |  GitHub Copilot | 58.0 | 66.5% | 38.5% | 105 | 273 |
| 6 |  Greptile | 51.2 | 51.3% | 50.9% | 81 | 159 |
| 7 |  CodeRabbit | 51.1 | 59.5% | 32.6% | 94 | 288 |
| 8 |  Cursor Bugbot | 48.6 | 46.8% | 56.9% | 74 | 130 |
| 9 |  Claude Code | 47.7 | 48.1% | 46.3% | 76 | 164 |
| 10 |  Devin | 44.8 | 41.1% | 69.1% | 65 | 94 |
F-scores are computed from unrounded precision and recall. Other reviewers’ figures come from the results Martian publishes for the offline leaderboard; where a reviewer has several versions there, the best-scoring version is shown: Qodo Deep (Qodo extended v2), Cubic v2, GitHub Copilot v2 and Greptile v4.1.
How scoring works
Each pull request comes with an answer key of known bugs. The benchmark splits every comment a reviewer posted into the individual issues it raises, and an AI judge decides which answer-key bug, if any, each issue describes. Three numbers follow from that.
- Recall (bugs caught)
bugs matched ÷ 158- The share of answer-key bugs the reviewer flagged in at least one comment. Unblocked: 108 of 158, or 68.4%.
- Precision (against the answer key)
issues matched ÷ issues graded- The share of the issues in the reviewer’s comments that match an answer-key bug.
- F-score (Fβ)
(1 + β²) · P · R ÷ (β² · P + R)- One number that counts recall β times as heavily as precision. Martian’s default is β = 2. Unblocked’s F2: 66.2.
Precision here measures agreement with the answer key, not whether a comment is correct. The key only lists the bugs people wrote down, so a real defect missing from it counts against every reviewer.
By severity
Every answer-key bug carries a severity. Among the reviewers compared here, Unblocked catches the most high-severity bugs, ties Cubic for the most critical bugs, and catches the most critical and high-severity bugs combined: 55 of 66, six more than the next best.
| Reviewer | Critical (12) | High (54) | Medium (59) | Low (33) | Critical + high (66) |
|---|---|---|---|---|---|
| Unblocked | 1083% | 4583% | 4271% | 1133% | 5583% |
| Qodo Deep | 975% | 4074% | 3763% | 1648% | 4974% |
| Cubic | 1083% | 3870% | 3763% | 1958% | 4873% |
| Augment | 867% | 4176% | 3966% | 1545% | 4974% |
| GitHub Copilot | 975% | 3870% | 4271% | 1648% | 4771% |
| Greptile | 867% | 3056% | 3051% | 1339% | 3858% |
| CodeRabbit | 867% | 3463% | 3966% | 1339% | 4264% |
| Cursor Bugbot | 758% | 3361% | 2542% | 927% | 4061% |
| Claude Code | 758% | 2852% | 3051% | 1133% | 3553% |
| Devin | 650% | 2750% | 2542% | 721% | 3350% |
By project
How well each reviewer holds up across five codebases in five languages. Unblocked catches at least 64% of the bugs in every project; every other reviewer drops below 60% on at least one.
| Reviewer | Cal.com (37) | Discourse (38) | Grafana (25) | Keycloak (25) | Sentry (33) |
|---|---|---|---|---|---|
| Unblocked | 2876% | 2668% | 1664% | 1664% | 2267% |
| Qodo Deep | 2568% | 2258% | 1768% | 1768% | 2164% |
| Cubic | 1951% | 2258% | 2496% | 1456% | 2576% |
| Augment | 2978% | 2258% | 1768% | 1352% | 2267% |
| GitHub Copilot | 2978% | 2771% | 1560% | 1664% | 1855% |
| Greptile | 2054% | 2053% | 1664% | 1040% | 1545% |
| CodeRabbit | 2978% | 2258% | 1352% | 1560% | 1545% |
| Cursor Bugbot | 1951% | 1950% | 1144% | 1248% | 1339% |
| Claude Code | 1951% | 2053% | 1560% | 624% | 1648% |
| Devin | 1643% | 1539% | 1560% | 728% | 1236% |
All 158 bugs
Every answer-key bug and whether each reviewer caught it. Click a bug to read it in full. A caught mark links to the review comment that raised the bug, and a missed mark to that reviewer’s review of the pull request. Unblocked’s reviews are in its public benchmark repositories; the other reviewers’ are where Martian has published them.
Loading all 158 bugs…
Martian’s pipeline merges a reviewer’s comments before splitting them into issues, so it does not record which comment a match came from. We link each caught bug to the comment whose text overlaps the judged issue most: 903 of 912 caught marks link to a comment, including all 108 of Unblocked’s, and the 9 ambiguous ones link to the pull request. The link helps you find the comment; the match itself is the judge’s.
Methodology
- Benchmark
- The offline benchmark from Martian’s Code Review Bench: 50 pull requests from five open-source projects, each a change that introduced bugs later found and fixed.
- Answer key
- 173 known issues. The leaderboard’s default view counts the 158 that are bugs, security, concurrency, data, API, performance, test-gap or documentation defects, and leaves out 15 style and speculative issues. Severity labels come from the answer key.
- Judge
- The judge Martian’s pipeline uses. Every reviewer’s comments go through the same extraction, de-duplication and judging steps.
- Unblocked
- Unblocked Code Review in Deep mode, running as the Unblocked GitHub app. The 50 pull requests were opened in public copies of the five projects under the AI-Code-Review-Eval organization, where every review can be read.
- Other reviewers
- Martian’s published results for a selection of the reviewers on its leaderboard, including the top three by F2, each at its best-scoring version, run by Martian on earlier versions of those products. Newer versions may score differently.
- Training data
- All 50 changes come from public repositories, dated 2013 to 2025, along with their fixes, so any model may have seen them in training. That applies to every reviewer on this page, and we did not test for it.