Part 1 of 2 · Next: Code Review Memory: less noise, same recall
Recall first: how we built Unblocked's Deep code review
What precision and recall mean for the way your team ships, which side of the trade-off fits your workflow, and the architecture behind an AI review agent built to miss as few bugs as possible.
Coding agents write a growing share of pull requests. On GitHub, more than 40% now start from an agent's draft, up from 17% six months earlier (GitHub Octoverse). Review hasn't kept pace: at one company studied by Carnegie Mellon and Stanford researchers, pull requests tripled while human reviewers grew by half, and the share that got any human review fell from 89% to 68% (He et al.). Review is now the last place a person looks at the code before it ships, and an overloaded human reviewer starts to skim.An AI review agent can help, but it can fail in two ways. It can miss a real bug, or it can flag something that isn't one. Every design decision in a review agent is a choice about which of those two failures to risk more. This post explains how to think about that choice for your own team, and how we built Deep Review mode, the recall-first option in Unblocked Code Review.
Two ways to get review wrong#
Benchmarks measure the two failures separately. Recall asks: of the real bugs in a pull request, how many did the agent catch? Precision asks: of the comments it posted, how many pointed at a real problem?
Noise is rarely the agent being wrong. When we had AI raters score Deep Review's comments from the pull request author's point of view, only 5 to 7% were wrong. Most of what people call noise is true but unwanted: a style the team doesn't enforce, a convention written down nowhere, a risk the team has already accepted. A stricter agent can't fix that. One that learns your team can, which is the subject of part 2.
Most AI code review tools report the share of their comments that got fixed. That number only counts comments the tool actually posted. It treats a correct comment the author chose to ignore the same as noise, and it can never show a missed bug, because a missed bug leaves no comment to count.
An agent that flags everything gets high recall and low precision. One that only speaks when it is certain gets the opposite. Neither number is useful alone. What matters is where your team sets the balance.
The trade-off is a dial#
Picture two review agents on the same 100 pull requests, which hold 40 real bugs between them.
| Quiet agent | Thorough agent | |
|---|---|---|
| Comments posted | 30 | 60 |
| Real bugs caught, of 40 | 24 | 32 |
| Bugs the review missed | 16 | 8 |
| Recall | 60% | 80% |
| Comments nobody needed | 6 | 28 |
| Precision | 80% | 53% |
The thorough agent trades 22 extra comments for 8 fewer missed bugs. Whether that is a good trade depends on what each one costs your team.
Where the trade lands in your pipeline#
What each failure costs depends on where review sits in your delivery flow. Two questions decide it. Who acts on review comments? And how much stands between review and your users?
| Your workflow | What costs more | Lean toward |
|---|---|---|
| An agent writes the code and an agent works through the review comments | A missed bug. An extra comment costs the agent seconds. | Recall |
| An agent writes the code and a person skims the diff | A missed bug, because nobody read every line closely | Recall, with memory to cut repeat comments |
| A person writes the code and a teammate reviews it closely | Reading time. The teammate is a second net for bugs. | Precision |
| Changes to authentication, payments or data migrations | A missed bug, by a wide margin | Recall |
Most of those rows lean toward recall, and for the same reason: the two failures are paid for at different points in the pipeline, and only one of them grows.
That growth is measurable: in a NIST-commissioned study, a bug fixed after release took three to five times the hours of one fixed during coding. Agent-written code also makes the bill more common: merged pull requests opened by coding agents needed a follow-up fix at 1.62 times the odds of those written by people (Takerngsaksiri et al.).
How Deep Review is built#
Unblocked Code Review has two modes. Standard, the default, puts precision first and posts fewer comments. Deep mode puts recall first. Recall doesn't come from asking a model to try harder (”make no mistakes” or “look carefully”). It comes from how the review is structured.
- Understand the change. Before looking for bugs, the agents work out what the pull request is trying to do and what depends on the code it touches. Many bugs live outside the changed lines: a caller that wasn't updated, or a migration that wasn't written. They draw on Unblocked's context engine for this: the tickets the pull request links to in Jira, Linear and other trackers, the Slack and Teams threads that mention it, and the author's other recent pull requests. A bug is often a mismatch between the code and what the ticket asked for.
- Review in parallel. Several agents review the same change, each from a different angle, one of them dedicated to security. They work inside a sandbox with the repository checked out, and can open any file, search the codebase and run commands, not just read the diff.
- Account for every changed file. Each changed file has to be closed with a line of code that justifies the verdict. Files left open are sent back for another pass.
- Verify every finding. A separate judge agent checks each candidate against the code. It can post it, correct an overstated claim, fold it into another finding with the same root cause, or drop it.
- Post what survives. Findings that share a root cause become one comment that lists every place it occurs.
What we learned building it#
Agents talk themselves out of real bugs#
Reading the reasoning behind missed bugs, we kept finding the same pattern. The agent worked out the defect, then dismissed it as "unlikely", "minor" or "out of scope" and never wrote it down. In one 50-PR benchmark run, 22 of 32 such misses left no trace at all, so nothing downstream could recover them. That is why every changed file now has to be closed with evidence, and why a hedge doesn't count as evidence. A doubt gets written down so the verification step can rule on it.
Rules in a prompt are advice; checks in the system are guarantees#
When we asked for a behavior in the prompt, the model often satisfied it in spirit and skipped it in practice. So the steps that matter are enforced by the system around the model. Coverage is checked by code, not by asking. In a paired test, removing that check cost about 15 points of recall, and a single correction turn won them back with precision unchanged.
The finder shouldn't grade its own work#
The review agents are told to report anything they can't rule out. A separate judge agent, reading the same code, decides what is worth posting, and a claim is never evidence for itself. Dropping a finding needs an explicit reason, so a verification step that fails can't silently empty a review.
Filters after the fact remove real bugs#
We tried cutting noise by filtering finished findings: by confidence, by category, by how local the change was. Every filter removed real bugs along with the comments people didn't want. Precision comes from verifying findings against the code, and from learning what your team doesn't want to hear, which is the subject of part 2.
When to choose recall first#
We ran Martian's open Code Review Bench, a public set of 50 real pull requests with human-labeled bugs, on Unblocked Deep Review ourselves. It caught 108 of the 158 known bugs, more than any tool in Martian's published results, while raising about 3.7 issues per pull request. Our benchmark post has the full results and links every review.
Deep mode fits best when agents write most of your code. A coding agent can work through review comments in seconds, so an extra comment costs little and a missed bug costs a lot. Standard fits teams where people read and act on every comment themselves, or that prefer quieter reviews. Neither is better. It's a preference, and teams can switch at any time.
Recall first does mean more comments, and some will be true but not useful to your team. In part 2, Code Review Memory, we show how Unblocked learns your team's preferences from the conversations on your pull requests, cutting the comments you don't want while keeping the bugs.
