Table of Contents
Executive Summary: The Benchmark Does Not Measure Your Problem
Search for the best model for code review and you will find leaderboards. SWE-bench Verified scores, ranked from highest to lowest, with open-weight models increasingly close to proprietary ones.
Those tables answer a question you are not asking.
SWE-bench measures whether a model can resolve an issue - given a described problem and a repository, can it produce a patch that makes the tests pass. That is a genuine capability and a reasonable benchmark for autonomous coding agents.
Code review measures something different: given a diff, can the model identify real problems, and - the part nobody benchmarks - can it stay silent when there are none.
A missed defect costs you once: it reaches production, you find it, you fix it. A false positive costs you repeatedly - the engineer’s attention on that pull request, plus a fraction of their trust in every comment afterwards. Accumulate enough and the reviewer is mentally filtered out, at which point its correct findings are lost too. A model with a 30% false positive rate is a model with notifications turned off, and its benchmark score is irrelevant to that outcome.
This article deliberately does not publish a leaderboard. Model rankings change monthly and any table here would be wrong within a quarter. What follows instead is a framework for evaluating models on your own code, which does not expire.
A Note on Sources, Before Anything Else
We think readers deserve to know how this article was researched, because the research process is itself the most useful finding.
Attempting to gather current benchmark figures for coding models surfaced a consistent pattern: most published comparison tables come from aggregator sites that do not state their methodology, and a significant number of those sites sell related products.
We also found direct contradictions between sources that ought to agree. One aggregator listed an open model at 93.4% on SWE-bench Verified while, in the same body of material, describing a different open model at 80.6% as “the strongest published coding result from any open-weight model”. Both statements cannot be true as presented. The discrepancy may come from a distinction between “open weight” and “open source”, from different harnesses, or from one figure simply being wrong - and none of the sources made it possible to tell.
We are not going to reproduce numbers we cannot verify at source. Where this article discusses the landscape, it describes directional findings that multiple independent sources agree on, and says so.
If you need current figures, go to the primary sources: the official SWE-bench leaderboard, independent evaluation harnesses that publish methodology, and model cards from the labs themselves. Treat any table without a stated methodology as marketing.
Where the Landscape Actually Stands in 2026
Three things are directionally clear and reported consistently enough to rely on.
The capability gap has narrowed substantially
Two years ago, using an open-weight model for serious code work meant accepting a visible quality drop. That is no longer the general case. The strongest open models now sit in the same performance band as proprietary ones for coding tasks, and the difference at the top has become a matter of percentage points rather than of categories.
Whether the gap has fully closed is disputed, and the answer depends on which harness you trust. That it has narrowed dramatically is not disputed.
The price difference is an order of magnitude, not a percentage
This is the finding with the clearest practical implication. Leading open models are served at roughly a tenth of the output token price of leading proprietary models, and several are available under permissive licences that allow self-hosting.
For a workload like code review - high volume, running on every pull request, potentially across hundreds of repositories - a tenfold price difference changes what is economically viable rather than merely improving margins.
Useful models now fit on one GPU
Open models at parameter counts that run on a single consumer GPU now reach performance levels that were, until recently, the preserve of hosted frontier models.
That matters specifically for code review because it makes self-hosting a realistic option for a mid-sized engineering organisation, not just for companies with a machine learning platform team.
The caveat that applies to all three
Every one of these statements is a snapshot. The model landscape moves faster than any publishing cadence can track, and this article included. Verify before deciding, and prefer a framework you can re-run over a ranking you have to trust.
Why SWE-Bench Is the Wrong Proxy for Code Review
Two different tasks
| Issue resolution (what SWE-bench measures) | Code review (what you need) | |
|---|---|---|
| Input | Problem description, repository | A diff, in context |
| Goal | Produce a patch that passes tests | Identify real problems |
| Success | Tests go green | Findings are correct and complete and few |
| Correct null result | None - you must produce a patch | Say nothing |
| Failure mode | Wrong patch | Noise that erodes trust |
That fourth row is the crux. In issue resolution, producing nothing is failure. In code review, producing nothing is frequently the correct answer - most diffs are fine.
Why the correlation is weaker than it looks
A model optimised to act will act. Reinforcement on benchmarks that reward producing a working patch selects for a disposition toward intervention, and that disposition is precisely wrong for a reviewer whose most common correct output is silence.
This is not a criticism of those models or those benchmarks. It is an argument that a leaderboard built on one task tells you less than you think about performance on a structurally different one.
The asymmetry, quantified
Consider a repository with 200 pull requests per month, where genuinely 15% contain a defect worth flagging.
Model A catches 90% of real defects with a 30% false positive rate. Our own arithmetic: 27 correct findings, and roughly 51 false positives across the remaining diffs. Engineers read 78 comments to find 27 useful ones.
Model B catches 70% with a 5% false positive rate: 21 correct findings and roughly 9 false positives. Engineers read 30 comments to find 21 useful ones.
Model A has the better detection rate and would win on most benchmarks. Model B is the better reviewer, because Model A’s signal-to-noise ratio predicts that within a month its comments stop being read - at which point its detection rate falls to zero in practice regardless of what it is in theory.
A Practical Evaluation Framework
The only evaluation that answers your question is one run on your code. Here is how to do it in about a week.
Step 1: Build a golden set from your own history
Take 50 to 100 merged pull requests from your repository, chosen to reflect normal work rather than dramatic ones.
For each, label the genuine problems that human review caught - with links to the review comments, so the labelling is evidence rather than recollection. Also label the pull requests where review correctly found nothing, because those are the ones that test the silence behaviour.
This is a day of work and it is the foundation for everything that follows. Without it you are comparing vibes.
Step 2: Measure four things, not one
| Metric | Definition | Why it matters |
|---|---|---|
| Detection rate | Share of real defects found | Baseline value |
| False positive rate | Share of comments that are not real problems | Determines whether the tool survives |
| Signal density | Correct findings per pull request | Attention budget per diff |
| Cost per pull request | Actual token spend on a real diff | Economic comparison |
Detection rate alone is the trap. Report all four together or the comparison is meaningless.
Step 3: Test on your diffs, not public benchmarks
Your codebase has conventions, internal libraries, domain vocabulary and patterns that a public benchmark cannot represent. A model that performs well on Django and Flask repositories tells you little about its performance on your internal framework.
This is also where the difference between models with structural understanding and models reading raw diffs becomes visible. Tools that build an abstract syntax tree representation of the repository - the approach CodeRabbit takes - can evaluate a change to a shared function against its callers. A model handed a diff in isolation cannot, and its findings on cross-file impact will be guesses.
Step 4: Run a human trial
Numbers are necessary and insufficient. Run the candidate alongside human review for two weeks on live pull requests, then ask engineers one question:
“Do you read its comments?”
If the honest answer is no, the metrics do not matter. That question has ended more automated review pilots than any benchmark, and it is worth asking early rather than discovering the answer six months into a contract.
Step 5: Tune for precision first
Whatever you deploy, start narrow. Restrict it to the categories where it performed best in your evaluation - often security patterns and null-safety rather than style or architecture opinions.
A reviewer that comments rarely and is right earns the trust that lets you widen its scope. One that comments constantly never gets the chance to.
The Self-Hosting Question
The price difference makes self-hosting tempting. The decision is more involved than comparing token rates.
What self-hosting actually costs
GPU capacity, whether purchased or rented, and sized for peak rather than average - code review load is bursty, concentrated around working hours and release cycles.
Operational ownership. Someone updates the model, monitors the serving stack, handles incidents and manages capacity. That is a fraction of an engineer indefinitely, and it is the cost most often omitted from the comparison.
Model currency. Hosted providers upgrade for you. Self-hosting means evaluating and migrating to new models yourself, which is a recurring project rather than a one-off.
When self-hosting wins
- High volume, where per-token savings exceed operational cost
- Hard data residency requirements, where code cannot leave your infrastructure under any terms
- Highly sensitive source, where the risk calculus is not about price at all
When an API wins
- Moderate volume, where operational cost dominates the savings
- No platform team with capacity to own inference infrastructure
- A need for the newest models, where migration cadence matters
Read the licence
“Open” is not one thing. Some models carry genuinely permissive licences such as MIT, which allow commercial use and self-hosting without restriction. Others are open-weight but carry usage restrictions that may exclude commercial deployment or specific applications.
Read the licence for every model you shortlist. This is a legal question, and the answer is not implied by the word “open” appearing in a headline.
Privacy and Compliance
For code review specifically, this section deserves more weight than for almost any other AI application.
The prompt is your source code
That is not an analogy. Automated code review means sending your source, or substantial portions of it, to whatever performs the analysis.
For most software organisations, source code is the single most sensitive category of data they hold - more so than customer records in some cases, because it encodes the business logic that constitutes the company’s actual product.
The questions that belong in procurement
- Is our code used to train models? Practice varies, and so does the tier at which opting out becomes available. Our v0 review documents a product where default training opt-out begins at the $100 per user per month tier, while Devs.ai applies zero-training agreements with model providers across all plans including its free tier.
- What is retained, and for how long? Zero-data retention exists as an option from several vendors, usually at enterprise tiers.
- Will they sign something? A published policy is a statement of intent. A contract is an obligation. Compliance functions require the second.
- Who are the subprocessors? A vendor’s own policy does not bind the model provider underneath it unless they have secured equivalent terms.
Why this is often the real reason for self-hosting
In our reading, teams that self-host models for code review usually do so because of question one, not because of cost. The token savings are the justification presented to finance; the reason is that the code never leaves.
If that describes your situation, the evaluation framework above still applies - you are simply running it against models you host rather than APIs you call.
Build vs Buy
An open model is not an automated reviewer. It is one component of one.
What you build around the model
- Diff extraction from your version control platform
- Context assembly - which surrounding files matter for this change
- Prompt engineering, maintained as models change
- Response parsing into structured findings
- Comment posting, with deduplication so the same issue is not raised on every push
- Threshold tuning, per repository and per team
- Feedback capture, so the system can improve
That is a small internal product with an owner, a backlog and ongoing maintenance.
What a commercial tool provides
Git platform integration, repository-wide context, a review interface, and - in the better implementations - structural analysis rather than diff reading. CodeRabbit is the example we have looked at most closely, and its AST-based approach is a meaningful architectural difference rather than a marketing distinction.
How to decide
Buy if code review is not your differentiator, your volume is moderate, and your team’s time is better spent elsewhere. That describes most organisations.
Build if volume is very high, data residency is non-negotiable, or you have specific review requirements that no product addresses. Be honest about the third - “our codebase is unusual” is true of everyone and rarely justifies a build.
What to Re-Check in Six Months
We would rather state this article’s shelf life than pretend it does not have one.
Will be out of date: every benchmark figure, every price, the identity of the leading models, and which capabilities are available in which tier.
Should remain true: the distinction between issue resolution and diff review, the asymmetry between false positives and false negatives, the four-metric evaluation framework, the self-hosting decision structure, and the privacy questions.
That division is the reason this article contains a framework rather than a table. Run the framework against whatever the landscape looks like when you read this, and you will get a better answer than any leaderboard would have given you.
Frequently Asked Questions
Are open-source models good enough for code review in 2026?
The capability gap has narrowed substantially and leading open models sit in the same band as proprietary ones for coding tasks. But raw capability is the wrong question - evaluate on detection rate, false positive rate, signal density and cost against your own diffs.
Why is SWE-bench not enough?
It measures whether a model can resolve an issue by producing a working patch. Code review measures whether a model can identify real problems in a diff and stay silent when there are none. The second task rewards restraint; the first does not.
Is a higher benchmark score always better for review?
No. A model with higher detection but a high false positive rate produces more noise, and a reviewer that gets muted has an effective detection rate of zero regardless of its benchmark position.
How much volume justifies self-hosting?
Model it explicitly: GPU capacity sized for peak, plus the fractional engineer who owns the serving stack, against your projected token spend. The operational cost is the component most often omitted, and for moderate volumes it usually dominates the savings.
Does my code get used to train models?
It depends on the vendor and often on the tier. Ask for a contractual commitment rather than relying on a published policy, and ask about subprocessors - a vendor’s own terms do not automatically bind the model provider underneath it.
Should we build our own reviewer around an open model?
Only if volume is very high, data residency is non-negotiable, or you have requirements no product addresses. The model is one component; diff extraction, context assembly, deduplication and threshold tuning make up a small internal product with ongoing maintenance.
Where This Fits
Automated review is one layer in a delivery pipeline, and it works best when the layers around it are sound - covered in our testing and continuous delivery playbook. If you are choosing tools for writing code rather than reviewing it, that is a different decision with different criteria, covered in how to choose an AI coding assistant.
For the products themselves, see our reviews of CodeRabbit for AST-based review and Devs.ai for the governance and zero-training angle.
