Benchmarks Explained
"SWE-bench" is four different benchmarks wearing one name, and their scores are not interchangeable. When a provider announces a score today they almost always mean SWE-bench Verified — the human-validated 500-task subset. Here is what each variant measures and why it matters.
| Variant | Tasks | What it is | What to know |
|---|---|---|---|
| SWE-bench (full) | 2,294 | Every task from the original benchmark: real GitHub issues from 12 popular open-source Python repositories. | Human review later found a meaningful share of tasks were unfair — underspecified issue descriptions or tests that fail for unrelated reasons — so full-set scores understate real ability. |
| SWE-bench Verified | 500 | A human-validated subset released in 2024 with OpenAI: professional developers screened tasks and kept only those that are clearly solvable with fair tests. | The industry standard. When a provider announces a SWE-bench score today, this is almost always the variant they mean — and it's what our leaderboard tracks. |
| SWE-bench Lite | 300 | A smaller, cheaper-to-run subset focused on self-contained bug fixes. Popular for research papers and agent development where eval cost matters. | Easier task mix than Verified — Lite scores typically run higher and can't be compared to Verified numbers. |
| SWE-bench Multimodal | 517 | JavaScript repositories with visual elements — issues that include screenshots, rendering bugs, and UI problems. | Tests a different skill (visual + front-end debugging). Scores here are much lower across the board and form their own leaderboard. |
| # | Model | Provider | Verified score |
|---|---|---|---|
| 1 | Claude Fable 5 | Anthropic | 93.4% |
| 2 | Claude Mythos 5 | Anthropic | 93.4% |
| 3 | Claude Opus 4.8 | Anthropic | 88.6% |
| 4 | Claude Opus 4.7 | Anthropic | 80% |
| 5 | Claude Sonnet 4.6 | Anthropic | 79.6% |
Full rankings with pricing and context windows on the SWE-bench leaderboard.
The original benchmark scraped 2,294 issue-and-patch pairs from twelve open-source Python repositories. That scale was its strength and its flaw: nobody had checked every task by hand. When professional developers later screened the set, they found a meaningful share of tasks were effectively unsolvable — issue descriptions too vague to act on, or hidden tests that failed for reasons unrelated to the issue. A model could write a perfectly good patch and still be marked wrong.
SWE-bench Verified fixed this by keeping only the 500 tasks that human reviewers confirmed are fairly specified and fairly tested. Scores jumped when it launched — not because models improved overnight, but because the noise was removed. That is the key trap when reading older articles: a full-set score from 2024 and a Verified score from today are different measurements, and comparing them inflates the apparent progress.
For how to interpret the numbers themselves — what counts as frontier-class this year — see what is a good SWE-bench score.
Full SWE-bench contains all 2,294 original tasks, but human review found many were unsolvable as posed — vague issue descriptions or broken tests. SWE-bench Verified is the 500-task subset that professional developers validated as fair and solvable. Verified is now the standard: when a provider announces a score, they almost always mean Verified.
No. The same model scores noticeably higher on Verified than on the full set, because the unfair tasks were removed. Comparing a Verified score against an old full-set score makes the newer model look better than the real difference. Always compare within one variant.
Verified. Provider announcements, the official leaderboard, and our own SWE-bench leaderboard all report SWE-bench Verified. The current Verified leader is Claude Fable 5 at 93.4%.
Cost. Running an agent across hundreds of repository tasks is expensive, so Lite's 300 self-contained tasks became the standard for research papers and quick iteration. Its task mix is easier, so Lite scores run higher than Verified scores for the same model.
Three reasons: different variants (Verified vs Lite vs full), different agent scaffolds (the harness that lets the model browse files and run tests), and different attempt policies (single attempt vs best-of-N). Check all three before trusting a comparison.
Newsletter
We track new benchmark publications. When a model takes the top spot, you'll know first.
No spam. Useful updates only. Affiliate disclosures always clearly labeled.