What AI Benchmarks Don't Tell You Before You Build on a Model

SWE-bench, MMLU, GPQA and the rest are genuinely useful — they're the closest thing the industry has to a controlled comparison, and we rely on them constantly. But a benchmark score is measured once, on a fixed set of tasks, on a specific model checkpoint, on a specific day. Nothing about that measurement tells you whether the provider will still offer that model next quarter, at that price, without a silent quality regression. Treating the number as a permanent property of the model rather than a snapshot is the most common mistake we see people make when picking a model to build on.

We've written elsewhere about watching an entire provider's model lineup disappear from under a production feature we run ourselves, and about a specific model ID being retired with effectively no warning we caught in time. No benchmark table has a column for 'how likely is this exact model to still exist in six months.' It should, because it's often the more important question. A model that scores five points lower on a benchmark but comes from a provider with a longer track record of not yanking models out from under integrations can be the objectively better choice for anything you plan to keep running.

A price-per-million-tokens figure implies you can actually send that many tokens. In practice, free tiers and even some paid tiers cap total throughput per minute, and that cap can be shared across input and output in ways that aren't obvious until you hit it. We measured a real free-tier ceiling of 8,000 tokens per minute, shared across everything a request needs — meaning a system that looks like it has generous headroom on a per-request basis can support roughly one real request per minute before it starts failing, for the entire deployment, not per user. That number appears nowhere on a pricing page, and it's the difference between a model being production-ready and being a demo.

The most honest thing we can say about benchmark leaderboards — the kind we publish and the kind competitors publish — is that they are only as current as the last time someone checked them against new launches. We ran a live SWE-bench leaderboard that named the wrong leader for a stretch of weeks in 2026, simply because a new model's published score hadn't been pulled in yet. Nothing about the page looked broken; the numbers on it were just outdated by the time a reader saw them. The lesson we took from it, and the one worth applying to any leaderboard you read: check the 'last verified' date before you trust the ranking, and be more suspicious of a page with no visible verification date at all.

Even a perfectly current, perfectly accurate benchmark score is an average over a fixed test set. It can't tell you how a model handles an ambiguous prompt it's never seen, how often it refuses a borderline-but-legitimate request, whether its output formatting stays consistent across a long session, or how it behaves at the exact context length your actual workload uses. Those behaviors only show up once real traffic runs through the model, which is exactly why we keep a section on every model comparison for what the benchmark doesn't cover, rather than letting the score speak for the whole page.

Use it as a first filter, not a verdict: narrow a field of dozens of models down to a handful worth a closer look. Then verify current pricing and availability independently of the benchmark announcement, since the two are published on different timelines and drift apart. And build with the assumption that whatever model you pick will eventually be deprecated, repriced, or replaced — because on the evidence of running this site, that isn't a tail risk, it's closer to a certainty on a long enough timeline.