# Unblocked catches the most bugs

URL: https://getunblocked.com/benchmarks/code-review/

On Martian's Code Review Bench, Unblocked in Deep mode caught 108 of 158 known bugs in 50 real pull requests, more than any reviewer on Martian's leaderboard.

---

Unblocked ranks #1 on Martian's Code Review Bench

- **Highest recall**: 108 of 158 known bugs caught, more than any reviewer on Martian's leaderboard (Unblocked 68.4%, GitHub Copilot 66.5%, Cubic 65.8%)
- **Most severe bugs**: 55 of 66 critical and high-severity bugs caught, six more than any other reviewer compared here (Qodo Deep 49, Augment 49)

## Overview

Unblocked Code Review in Deep mode, scored with the published pipeline of Martian's open-source Code Review Bench and compared with the published results of other AI code reviewers on Martian's leaderboard, including its top three. Every score, how the scores are calculated, and the result for each of the 158 known bugs, with a link to the comment that caught it.

**How these numbers were produced.** Martian built Code Review Bench and publishes the other reviewers' results: each reviewer's best-scoring version, run by Martian on earlier versions of those products. This page shows a selection of them, including the top three by F2. We ran Unblocked through Martian's published pipeline, with the same issue extraction, de-duplication, judge and answer key. Unblocked is not yet listed on [Martian's leaderboard](https://codereview.withmartian.com/?mode=offline), which additionally requires several hundred public reviews and Martian running the pipeline itself.

## Results

Unblocked and the reviewers compared here, scored the same way and sorted by F2, the score Martian's dashboard shows by default. F2 counts recall twice as heavily as precision.

| # | Reviewer | F2 | Recall | Precision | Bugs caught, of 158 | Issues graded |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | Unblocked | 66.2 | 68.4% | 58.7% | 108 | 184 |
| 2 | Qodo Deep | 65.1 | 64.6% | 67.1% | 102 | 152 |
| 3 | Cubic | 64.9 | 65.8% | 61.5% | 104 | 169 |
| 4 | Augment | 64.0 | 65.2% | 59.5% | 103 | 173 |
| 5 | GitHub Copilot | 58.0 | 66.5% | 38.5% | 105 | 273 |
| 6 | Greptile | 51.2 | 51.3% | 50.9% | 81 | 159 |
| 7 | CodeRabbit | 51.1 | 59.5% | 32.6% | 94 | 288 |
| 8 | Cursor Bugbot | 48.6 | 46.8% | 56.9% | 74 | 130 |
| 9 | Claude Code | 47.7 | 48.1% | 46.3% | 76 | 164 |
| 10 | Devin | 44.8 | 41.1% | 69.1% | 65 | 94 |

F-scores are computed from unrounded precision and recall. Other reviewers' figures come from the results Martian publishes for the [offline leaderboard](https://codereview.withmartian.com/?mode=offline); where a reviewer has several versions there, the best-scoring version is shown: Qodo Deep (Qodo extended v2), Cubic v2, GitHub Copilot v2 and Greptile v4.1.

## How scoring works

Each pull request comes with an answer key of known bugs. The benchmark splits every comment a reviewer posted into the individual issues it raises, and an AI judge decides which answer-key bug, if any, each issue describes. Three numbers follow from that.

- **Recall (bugs caught)**, `bugs matched ÷ 158`: The share of answer-key bugs the reviewer flagged in at least one comment. Unblocked: 108 of 158, or 68.4%.
- **Precision (against the answer key)**, `issues matched ÷ issues graded`: The share of the issues in the reviewer's comments that match an answer-key bug.
- **F-score (Fβ)**, `(1 + β²) · P · R ÷ (β² · P + R)`: One number that counts recall β times as heavily as precision. Martian's default is β = 2. Unblocked's F2: 66.2.

Precision here measures agreement with the answer key, not whether a comment is correct. The key only lists the bugs people wrote down, so a real defect missing from it counts against every reviewer.

## By severity

Every answer-key bug carries a severity. Among the reviewers compared here, Unblocked catches the most high-severity bugs, ties Cubic for the most critical bugs, and catches the most critical and high-severity bugs combined: 55 of 66, six more than the next best.

| Reviewer | Critical (12) | High (54) | Medium (59) | Low (33) | Critical + high (66) |
| --- | --- | --- | --- | --- | --- |
| Unblocked | 10 (83%) | 45 (83%) | 42 (71%) | 11 (33%) | 55 (83%) |
| Qodo Deep | 9 (75%) | 40 (74%) | 37 (63%) | 16 (48%) | 49 (74%) |
| Cubic | 10 (83%) | 38 (70%) | 37 (63%) | 19 (58%) | 48 (73%) |
| Augment | 8 (67%) | 41 (76%) | 39 (66%) | 15 (45%) | 49 (74%) |
| GitHub Copilot | 9 (75%) | 38 (70%) | 42 (71%) | 16 (48%) | 47 (71%) |
| Greptile | 8 (67%) | 30 (56%) | 30 (51%) | 13 (39%) | 38 (58%) |
| CodeRabbit | 8 (67%) | 34 (63%) | 39 (66%) | 13 (39%) | 42 (64%) |
| Cursor Bugbot | 7 (58%) | 33 (61%) | 25 (42%) | 9 (27%) | 40 (61%) |
| Claude Code | 7 (58%) | 28 (52%) | 30 (51%) | 11 (33%) | 35 (53%) |
| Devin | 6 (50%) | 27 (50%) | 25 (42%) | 7 (21%) | 33 (50%) |

## By project

How well each reviewer holds up across five codebases in five languages. Unblocked catches at least 64% of the bugs in every project; every other reviewer drops below 60% on at least one.

| Reviewer | Cal.com (37) | Discourse (38) | Grafana (25) | Keycloak (25) | Sentry (33) |
| --- | --- | --- | --- | --- | --- |
| Unblocked | 28 (76%) | 26 (68%) | 16 (64%) | 16 (64%) | 22 (67%) |
| Qodo Deep | 25 (68%) | 22 (58%) | 17 (68%) | 17 (68%) | 21 (64%) |
| Cubic | 19 (51%) | 22 (58%) | 24 (96%) | 14 (56%) | 25 (76%) |
| Augment | 29 (78%) | 22 (58%) | 17 (68%) | 13 (52%) | 22 (67%) |
| GitHub Copilot | 29 (78%) | 27 (71%) | 15 (60%) | 16 (64%) | 18 (55%) |
| Greptile | 20 (54%) | 20 (53%) | 16 (64%) | 10 (40%) | 15 (45%) |
| CodeRabbit | 29 (78%) | 22 (58%) | 13 (52%) | 15 (60%) | 15 (45%) |
| Cursor Bugbot | 19 (51%) | 19 (50%) | 11 (44%) | 12 (48%) | 13 (39%) |
| Claude Code | 19 (51%) | 20 (53%) | 15 (60%) | 6 (24%) | 16 (48%) |
| Devin | 16 (43%) | 15 (39%) | 15 (60%) | 7 (28%) | 12 (36%) |

## All 158 bugs

Every answer-key bug and whether each reviewer caught it. Click a bug to read it in full. A caught mark links to the review comment that raised the bug, and a missed mark to that reviewer's review of the pull request. Unblocked's reviews are in its public benchmark repositories; the other reviewers' are where Martian has published them.

Martian's pipeline merges a reviewer's comments before splitting them into issues, so it does not record which comment a match came from. We link each caught bug to the comment whose text overlaps the judged issue most: 903 of 912 caught marks link to a comment, including all 108 of Unblocked's, and the 9 ambiguous ones link to the pull request. The link helps you find the comment; the match itself is the judge's.

## Methodology

| Project | Language | Pull requests | Bugs counted |
| --- | --- | --- | --- |
| [Cal.com](https://github.com/AI-Code-Review-Eval/cal-com-unblocked) | TypeScript | 10 | 37 |
| [Discourse](https://github.com/AI-Code-Review-Eval/discourse-unblocked) | Ruby | 10 | 38 |
| [Grafana](https://github.com/AI-Code-Review-Eval/grafana-unblocked) | Go | 10 | 25 |
| [Keycloak](https://github.com/AI-Code-Review-Eval/keycloak-unblocked) | Java | 10 | 25 |
| [Sentry](https://github.com/AI-Code-Review-Eval/sentry-unblocked) | Python | 10 | 33 |
| **Total** | | **50** | **158** |

- **Benchmark.** The offline benchmark from Martian's [Code Review Bench](https://github.com/withmartian/code-review-benchmark): 50 pull requests from five open-source projects, each a change that introduced bugs later found and fixed.
- **Answer key.** 173 known issues. The leaderboard's default view counts the 158 that are bugs, security, concurrency, data, API, performance, test-gap or documentation defects, and leaves out 15 style and speculative issues. Severity labels come from the answer key.
- **Judge.** The judge Martian's pipeline uses. Every reviewer's comments go through the same extraction, de-duplication and judging steps.
- **Unblocked.** Unblocked Code Review in Deep mode, running as the Unblocked GitHub app. The 50 pull requests were opened in public copies of the five projects under the [AI-Code-Review-Eval](https://github.com/AI-Code-Review-Eval) organization, where every review can be read.
- **Other reviewers.** Martian's published results for a selection of the reviewers on its leaderboard, including the top three by F2, each at its best-scoring version, run by Martian on earlier versions of those products. Newer versions may score differently.
- **Training data.** All 50 changes come from public repositories, dated 2013 to 2025, along with their fixes, so any model may have seen them in training. That applies to every reviewer on this page, and we did not test for it.

## Try Unblocked Code Review

- Book a Demo: <https://getunblocked.com/demo/>
- Get Started for Free: <https://getunblocked.com/dashboard/get-started>