We put Jev in production against a cross-encoder. Here are the numbers.

At Unblocked, we have many use cases for a System One model like Jev. This post shows how it compares to our existing production solution on the memory selection problem.

Decisions are expensive#

Answering one question at Unblocked is mostly a chain of small decisions. Is this message even a question? Which repositories and documents bear on it? Which of the things we remember about this organization are relevant right now? Should we answer at all, or stay out of the thread?

The paragraph we generate at the end is the cheap part. The decisions in front of it set the quality, the latency and most of the bill, and we make nearly all of them with text models. We pay generation prices and generation latency for what is just a yes or a no.

So Jev arrived with an unusually specific fit. You hand it a state and a set of questions, and for each question you get a typed judgment back:

The state is ingested once and every question runs against it in parallel, in a single round trip, up to 64k tokens across the whole request. Input costs $0.042 per million tokens. Output is free.

The open question about this class of model is whether the calibration holds up outside a demo, on a messy production workload, against a system that has been tuned for the job. We decided to find out properly and run it as an evaluation rather than a vibe check.

The decision we picked: which memories the agent sees#

Our agent answers questions about an organization's code and institutional knowledge, and it writes notes as it goes: that a service owns retries, that a user wants short answers, that a deploy script moved. When the next question comes in, it has to decide which notes help and which hurt.

The failure is asymmetric. A missed note costs a slightly worse answer. A wrong note injected into the prompt costs a confident, specific, wrong answer. That is much harder for a reader to catch, and much more expensive to Unblocked's credibility. So the solution has two jobs: rank memories by relevance, and stay silent when it has nothing to offer.

Our current solution is a two-step algorithm:

1. Pick the top 100 memories by cosine similarity between the query embedding and the memory embeddings.
2. Rerank those with a cross-encoder and keep the ones above a threshold.

Reading it back with fresh eyes turned up something we already knew but had never acted on. We had measured cosine similarity on these notes before, at every representation we could think of (title, title plus description, body, whole note), and it could barely separate the relevant notes from the irrelevant ones for a given question. Short notes about one company's systems all live in the same small corner of embedding space.

Stage one only ever cut the candidate list down to something the cross-encoder could afford. The cross-encoder did all the actual work.

So the question was whether Jev could do this job better: higher quality, cheaper, and faster.

We could not evaluate on the data we had#

We tried the shortcut first. We already log what happens to every retrieved memory: injected into the prompt, rejected by the threshold, or opened by the agent while it worked. "Opened" looks like a free relevance label, and there were tens of thousands of them.

We checked how they were distributed before using them. 72% of opened notes were notes the incumbent had already injected. That says nothing about quality. The agent can only open what it was shown. Judging a new ranker on those labels is a closed loop in which the incumbent grades its own homework and wins by construction.

So we built the dataset the expensive way: 292 real production questions, and for each one, every note that existed in that organization at that moment. 12,927 question/note pairs.

Only 1.8% of those pairs turned out to be relevant. That is honest: most notes are irrelevant to most questions. It is also why we never once looked at accuracy, a metric a constant "no" would have won outright.

Hiring a judge#

With the candidates reconstructed and the free labels rejected, we needed 12,927 relevance judgments from something that had no stake in the outcome.

We used GPT-5.6 Terra at medium reasoning, through the Batch API, against a strict JSON schema. It saw a question and a chunk of candidate notes, shuffled, and graded each note 0 to 3: unrelated, topically adjacent, useful, directly answers part of the question. Relevant meant 2 or better. It never saw which arm produced a score, what any score was, whether a note had been injected in production, or what the agent eventually answered.

We graded 10% of the pairs twice, in different chunks, purely to find out how much of the judge's output was noise. We were about to make an architecture decision on effect sizes in the hundredths. If the labels themselves wobbled at that scale, the whole exercise was theatre. 93.0% exact agreement and 99.3% on the same side of the threshold said they didn't.

Two ways to ask, and four ways to know#

There are two ways to point a judgment model at a ranking problem, and we did not know which was right, so we ran both against the incumbent.

The fan-out request is the one that looks like what the model was designed for:

json{
  "model": "jev-1.13.0",
  "state": [
    "Question: why did the nightly sync start failing after the region cutover?",
    "Note n1: Deploys to the EU region require the migration flag to be set first.",
    "Note n2: The team prefers short answers with links over long explanations."
  ],
  "questions": {
    "n1": { "type": "noul", "instructions": "Note n1 helps answer the question." },
    "n2": { "type": "noul", "instructions": "Note n2 helps answer the question." }
  }
}

The expensive half of a rerank call is the query context, and it is identical for every candidate. Here it is sent once. We chunked at twenty notes per request rather than sending an organization's whole store. The 64k budget covers the state plus every question, note lengths vary a lot, and twenty keeps every request comfortably inside it while bounding the tail latency of a single call.

We measured four numbers:

• nDCG@5: ranking quality at the depth we actually inject.
• Per-question AUC: given one relevant and one irrelevant note from the same question, how often the scorer ranks the relevant one higher. This is scale free, which matters when comparing two vendors whose numbers are not commensurable.
• Recall@5: the proportion of all relevant items that land in the top 5.
• Silent-if-none: of the questions where nothing in the store was relevant, how often the arm injected nothing at all.

Because the two score scales are unrelated, everything downstream of a threshold was compared at a matched admission rate. The challenger's threshold was set so it admits the same share of pairs the incumbent admits at its production threshold. Anything we tuned was tuned on one set of customer organizations and reported on held-out ones.

Per-pair lost, fan-out won#

Per-pair came back below the incumbent, at −0.0333 on nDCG@5, while costing one request per candidate. Our reading is that relevance among short notes is partly a relative judgment. Shown a single note in isolation, the model has nothing to calibrate "helps answer this" against, so it grades charitably. Shown twenty, it can see which three actually bear on the question.

Fan-out cleared the non-inferiority bar with room to spare. Both intervals sit entirely above zero:

Ranking metrics are abstract, though. The table that ended the debate is what this does to the prompt the agent actually receives, at a matched admission rate and a cap of five notes:

More of the right notes, fewer of the wrong ones, and a better instinct for shutting up. Read the last row the other way around: on questions where the organization's memory had nothing useful to offer, the system we were running pushed something into the prompt more than half the time.

The operational numbers came out roughly even, which we had not expected:

150 milliseconds on a path that sits behind an LLM answer taking several seconds is not something a user can perceive. And that cost line counts only the ranking call. It leaves out the embedding calls, the embedding columns, and the backfill that keeps them warm.

A day on prompt tuning, for nothing#

At this point we had a winner and we got greedy. There are obviously easy points in the instruction wording, we thought. The criteria are vague, and the model is being asked "helps answer the question" with no guidance about what helps means.

So we built a tuner: nine variants of the fan-out instructions (stricter criteria, worked examples, different representations of the note), fitted on one set of organizations and measured on held-out ones.

Not one of them beat the plain baseline wording by more than about three points in either direction, which on this dataset is noise. The only variant that moved decisively moved the wrong way. Sharpening the criteria to insist a note be necessary rather than useful cost twenty points of recall.

That non-result changed how we think about this class of model, and it is the part we would most like other teams to steal. With a generative ranker, the prompt is the only dial you have, so of course you spend your time turning it. With a calibrated one, the threshold is a continuous, monotone dial that does what those prompt adjectives were groping toward, and you can sweep it offline against your labels in seconds.

How it ended#

What we would tell another team standing where we started:

1. A calibrated probability is a better interface than a score. Thresholds, cascades and "say nothing" all get easier when the number means what it says.
2. Tune the threshold, not the prompt. Nine variants moved nothing; one sweep picked the operating point.
3. Never judge a new ranker on labels the old one generated. Reconstruct the full candidate set, get independent labels, and measure how noisy those labels are before trusting a small effect.

One caveat we would rather state than have pointed out. This is one workload. Our numbers are not a benchmark and we would not hand them to you as one. The method (full candidate reconstruction, a blind judge with a measured error rate, a margin fixed in advance) cost about twelve dollars and a couple of days, and that is the part worth copying.

Jev is now making one of the decisions we care most about, on every question, in production. We are steadily moving the rest of them off text models too.