# Code Review Memory: less noise, same recall


URL: https://getunblocked.com/blog/code-review-memory/
Published: 2026-11-09T17:00:00.000Z
Author: Kaihao Zhao
Categories: AI Code Review
Series: Part 2 of 2 · Previous: [Recall first: how we built Unblocked's Deep code review](https://getunblocked.com/blog/recall-first-deep-code-review/)

A recall-first review agent raises more comments. Most of the unwanted ones are true; they cover things your team has already decided. Here is how Unblocked learns those decisions from the conversations on your pull requests.

---
## Most noise is true

As [part 1](/blog/recall-first-deep-code-review/) noted, when we had AI raters score Deep Review's comments from the pull request author's point of view, only 5 to 7% were wrong. About a quarter were true but minor, and that quarter is what people usually mean by noise. The replies to those comments tend to look like this:

- **Taste:** "Literal values are fine in tests. Keep named constants for app code."
- **Convention:** "In our Go services, never start a bare goroutine. Use an errgroup."
- **Accepted risk:** "Yes, this endpoint is unbounded. It's internal and we know."
- **Missing context:** "We only support Postgres, so MySQL compatibility doesn't matter."

The agent is right in every one of these. It doesn't know what the team has already decided, because those decisions aren't written down anywhere.

## Precision means two things

On a benchmark, precision is measured against a set of golden answers: does the comment match a known bug? On your team, precision is measured by whether anyone wanted the comment. Those are different numbers, and they fail for different reasons.

The usual flawed way to tune precision is using a confidence threshold. Every finding comes out of the agent with a confidence score, and a filter posts only the findings above a cutoff: a comment scored 0.9 is posted, one scored 0.5 is dropped. Raise the cutoff and fewer comments appear; lower it and more do.

A **wrong comment** is the agent's own mistake, and the reviewer catches most of those before posting using a judge that must verify every finding. A **true comment your team doesn't want** is a fact about your team. No threshold can know it, and as part 1 showed, filtering findings removes real bugs along with the comments people don’t want. The missing information lives with the team, so **the agent has to learn it from the team**.

## Why not just a rules file?

Files such as `AGENTS.md` help, and Unblocked can read those too. But in the repositories we work with, they often fall out of date as the code moves, or stay too general to act on in a review. "Write clean, well-tested code" doesn't tell an agent whether a literal value in a test is fine. And they only hold what someone thought to write down ahead of time.

Memory comes from the other end. Each rule is learned at the moment a comment missed the mark, from the person who knows that code, checked against the code before it is saved, and scoped to where it applies. Rules nobody uses lose their place on their own. The two work together: the file says what the team planned, and memory records what the team decided in review.

## How it works

There is nothing to configure and no rule language to learn. The conversation on the pull request is the interface.

[From a reply to a rule. Unblocked checks every reply against the code before it saves anything, and asks the expertise graph only when the reply conflicts with an existing rule. A reply the code contradicts gets a pushback instead, and a thanks or a one-off isn't saved.](https://cdn.sanity.io/files/31mw1ch6/production/6ece8a63d8332283fa693e95b44efcf10ec9940f.webm)


### You teach it by replying

When someone replies to an Unblocked review comment, Unblocked decides whether the reply holds something worth remembering and saves it as a short plain-language rule. If it answers the reply, it says future reviews will take it into account; if it doesn't answer, it still learns, silently. "Fixed, thanks" or "we'll accept the risk on this one" isn't saved.

### Rules have a scope, and work in both directions

Each rule applies to the whole repository, a folder, a module or a file type. A team can say "don't ask for tests on small UI changes", and also "always ask for tests on payment code". The second kind makes the agent stricter where the team wants it, which raises recall where it matters most to that team.

### It checks claims against the code

A reply is a claim, and some claims are wrong. Before saving anything, Unblocked opens the code, and it pushes back if the code disagrees. A missing rule costs one repeated comment. A false rule can hide a real bug on every pull request after it.

![A pull request thread where Unblocked answers a wrong claim with the code that contradicts it](https://cdn.sanity.io/images/31mw1ch6/production/8e287bd7cc102a699c7b9fbeb89a2b21432b844a-2010x1138.png?w=760&auto=format)
_**A wrong claim is not saved.** The reply is answered with the code that contradicts it._


**No rule can silence a security or crash finding.**

### When people disagree

A correction updates the existing rule instead of piling a contradicting one next to it. That's easy when the team agrees, and harder when two people don't.

Unblocked's expertise graph, built from your team's commit history, knows who has actually worked on each part of the code, and who has left. When a reply conflicts with a rule, Unblocked compares the two sides' experience in that part of the code. If the person replying knows it better, the rule changes. If the people who set the rule know it better, the rule stays, and Unblocked asks them once, on the thread, to weigh in. It won't make a quiet exception for one pull request, and it never posts anyone's expertise score on the thread.

![A pull request thread where Unblocked keeps a billing rule and asks its owner, maria, to weigh in](https://cdn.sanity.io/images/31mw1ch6/production/a3c90f552856ebe592b262f2267f9c7dd2399087-2010x1138.png?w=760&auto=format)
_**A conflict goes to the people who know the code.** The rule changes only when they agree, or when the person replying knows that code better._


### Every rule is always in view

Each rule is stored as a structured record. Every review sees only a one-line summary of each rule, and the agent opens the full record when a rule looks relevant to the change.

![The rule index every review reads, one line per rule with its kind and scope, beside the full record of the goroutine rule with where it applies, its kind, the rule, code that does not and does follow it, and where it was learned](https://cdn.sanity.io/images/31mw1ch6/production/157f3ce12ad1cb50d11342eeac831ae20fe9604b-2010x924.png?w=760&auto=format)
_**How a rule is stored.** Every review reads the one-line index. When a change starts a goroutine, the agent opens that rule's full record, with its scope, examples and where it was learned._


We don't look rules up by search. "Never start a bare goroutine" shares almost no words with `go sendReceipt(order)`, so a search for relevant rules would miss it. Reading every summary, the agent doesn't.

![Search by shared words misses the goroutine rule; reading every rule summary finds it](https://cdn.sanity.io/images/31mw1ch6/production/7739dc909056a05545a8e364f6680d82739655fd-2010x843.png?w=760&auto=format)
_**Why rules aren't found by search.** With 20 to 1,020 rules in one repository, the one-line index found 80 to 90% of the rules that applied to a change but shared few words with it. Embedding search found 14%._


### Rules earn their place

The summaries stay short, and so does the list. Each repository has a cap on how many rules can be active at once; the rest are dormant, kept but out of the agent's view. Each rule has a score from its recent reviews: a point each time it shaped a posted comment or made the judge agent drop one, a point for each positive reaction or reply to those comments, minus two for each negative one, and a point for every 50 points of expertise its teacher has in that code.

When a new rule arrives and the pool is full, the lowest-scoring rule that nobody has taught recently goes dormant. A recently taught rule is never put to sleep, and teaching a dormant rule again brings it back. The prompt never fills up with rules nobody uses.

## Does it work?

For a recall-first agent, the question is whether it gets quieter without losing bugs. We test that by teaching a repository a few preferences, then running the same pull requests with memory on and off. Each pull request holds a pattern the team asked it to skip and a real bug. Turning memory on cut comments by a quarter. The comments the team had taught it to skip mostly stopped, the check the team had asked for appeared every time instead of occasionally, and every real bug was still found.

A larger run over 100 pull requests of teaching and follow-up held up the same way. No rule suppressed a planted bug or security issue, no rule applied outside its scope, and replies that made false claims about the code or tried to slip in instructions were never saved.

## Getting the most out of it

Memory gets better the more the team talks to it. A bare "no" tells Unblocked something was off. A reply that says why tells it what to do next time, on every pull request after. A few seconds spent explaining a pushback pays off on every review that follows.

![A pull request thread where a reply about internal helpers becomes a rule](https://cdn.sanity.io/images/31mw1ch6/production/c36d6ad943640b820b0af40910ba34183613761a-2010x1014.png?w=760&auto=format)
_**Worth learning.** The reply says how future reviews should work, so it becomes a rule for internal helpers._


![A pull request thread where a fixed bug gets a thumbs-up and nothing is saved](https://cdn.sanity.io/images/31mw1ch6/production/0067a64d21be647e0478dfce070bd2d129f59c26-2010x898.png?w=760&auto=format)
_**Not worth learning.** The comment was right and the bug is fixed. That settles this pull request and teaches nothing new for future reviews, so Unblocked reacts with a thumbs-up and saves nothing. Promises about a follow-up aren't saved either._


Put together with Part 1, this is the full shape of the trade. Deep Review raises recall. Memory raises precision as your team defines it, one reply at a time, without turning recall back down.

## What we're still watching

It's early. We're watching for rules that pull the agent's attention away from nearby bugs, checking how scopes behave in very large monorepos, and working on retiring rules once the code they describe is gone.

**Reply to any Unblocked review comment to start teaching it.**