If you've been evaluating AI coding tools recently, you've probably sat through a vendor pitch where someone points at a SWE-bench score and says something like "our model resolves 72% of real-world software issues." It sounds impressive. It's meant to. And it's not exactly wrong, it's just that the number is doing a lot of work that the benchmark wasn't designed to do.
I've watched enough of these evaluations play out on actual production codebases to have developed a fairly firm view: the leaderboard and your repo are measuring different things, and conflating them is an expensive mistake.
SWE-bench is a benchmark built from 2,294 real GitHub issues and their corresponding pull requests, drawn from 12 popular Python libraries. The setup is elegant: give a model the repository state at the time the issue was filed, and ask it to produce a patch. Then run the existing test suite. If the tests pass, it's a resolved issue. If they don't, it isn't.
There are three variants. The Full set has all 2,294 tasks. Lite is a curated subset of 300 tasks chosen for clarity and reliability. Verified, which was developed with OpenAI's involvement, is a 500-task subset created specifically to remove the brittle or vague tasks that were producing unreliable scoring, cases where the environment was flaky, the problem description was ambiguous, or valid solutions were being rejected for the wrong reasons.
The benchmark is genuinely well-constructed for what it's measuring. That's not the issue. The issue is what it's not measuring, and that gap is where your actual engineering problems live.
The 12 libraries in SWE-bench are some of the most widely studied Python projects on the internet. Scikit-learn, Django, Flask, Sympy, these aren't obscure internal tools. They've been discussed, documented, blogged about, and included in training datasets for years.
When a model scores well on tasks drawn from these repositories, it's genuinely hard to know how much of that performance comes from the model's reasoning ability versus its familiarity with the specific codebase from training. This is a known problem in ML benchmarking generally, and it's worth keeping in mind when you imagine that 70% figure applying to your Laravel 13 monolith and Next.js 16 frontend that nobody outside your company has ever seen.
Your codebase has no training signal. The model is working cold. That doesn't mean it won't be useful, it very often is, but the performance profile will be different.
SWE-bench requires runnable, deterministic tests to score a pass. That's not a flaw in the benchmark design, it's a sensible constraint. The problem is that a significant portion of real-world bugs either lack tests entirely, have tests that are too brittle to run reliably in an automated harness, or exist in parts of the codebase where the test coverage is thin because it grew organically rather than by design.
In my experience, the bugs that are genuinely hard, the ones your team has been staring at for three days, are disproportionately in exactly these areas. They involve state that's difficult to reproduce, timing issues, interactions between services, or logic that was never tested because it wasn't obvious it needed to be. SWE-bench doesn't score those, and neither will your AI tool.
A pass is a pass. A fail is a fail. There's no partial credit for a patch that identifies the root cause correctly but implements the fix incorrectly, or one that fixes 80% of the affected cases and misses an edge condition. There's no credit for a suggestion that a competent engineer could turn into a working fix in twenty minutes.
This binary scoring is probably necessary for a reproducible benchmark. But it means the leaderboard systematically understates the practical value of tools that produce near-misses, and it also misses the failure modes that matter most in production: patches that pass the tests but introduce a regression, or that fix the reported symptom while leaving the underlying problem intact.
This is the gap I think is genuinely underappreciated. SWE-bench measures a model's ability to fix a well-specified, self-contained issue in a well-tested codebase with a clear acceptance criterion. Most real engineering work doesn't look like that.
Real bugs arrive as: "the checkout flow is breaking for some users on mobile, we're not sure when it started." Real features arrive as: "we need this to integrate with the new CRM but we're still figuring out exactly what data we need." Real codebases have a Laravel 13 monolith sitting next to a Next.js 16 frontend, three years of accumulated technical debt, a README that was last updated in 2022, and a critical authentication module that one person wrote and nobody else fully understands.
A benchmark built from well-documented issues in well-tested libraries, scored against deterministic test runs, is measuring performance in conditions that are as favourable as software engineering ever gets. Your conditions are not those conditions.
These are reasoned estimates from having watched these tools work on real codebases, not cited statistics, so hold them loosely. Somewhere around 50 to 60 per cent of the well-scoped, test-covered bugs might be genuinely automated on an internal codebase in reasonable shape. Another 15 to 20 per cent will produce patches that meaningfully accelerate a human engineer without fully resolving the issue. The remaining 20 to 30 per cent will require substantial human judgment regardless of what the tool produces, either because the problem isn't well-specified, the codebase context is too deep, or the fix requires architectural decisions the tool can't make.
That's still genuinely useful. A 50 to 60 per cent automation rate on a well-maintained, well-tested subset of your bug backlog is a real productivity improvement. But it's not 70%, and the bugs it can't touch tend to be the ones that are costing you the most time.
When someone shows you a SWE-bench score, the number isn't useless. It tells you something real about the model's capability on a controlled task. But before you reorganise your development budget around it, these questions matter more:
Sites like Artificial Analysis track model performance across multiple dimensions and are worth checking before committing to a particular provider. But no third-party benchmark tells you what you actually need to know, which is how a specific tool performs on your specific codebase, with your specific test coverage, on the specific category of issues your team actually struggles with.
I remain firmly of the view that AI coding tools are worth serious investment right now. The good ones do produce real output on real problems. But the path to getting actual value from them runs through honest evaluation, not leaderboard extrapolation. Run a structured trial on a representative sample of your backlog. Measure what actually gets resolved, what gets accelerated, and what the tool consistently misses. That data will tell you far more than any benchmark score.
The 70% on SWE-bench is a real number. It's just not your number.
