Benchmark numbers are easy to quote and hard to interpret. Use these questions on any number, whether it comes from a vendor, a research group, or us.
What should I ask about any benchmark number?
- Who ran it? A vendor testing its own product has an incentive that an independent tester does not.
- What exactly was the task, and does it resemble your work?
- How was success judged, and could a different judge reach a different result?
- How many items and how many runs? Small samples produce unstable numbers.
- Which version and settings were used? Results do not transfer across versions.
- Could the test material have been seen by the model during training? Public tests can leak into training data.
- Is the comparison like for like? Different tools given different help are not comparable.
- Is the difference between two numbers large compared with run-to-run variation?
What will we publish with any result?
When we publish benchmark results, we will publish the task set, the conditions, the sample size, the number of runs, the version tested, the date, and the limitations. Until then, our benchmarks page lists the protocol requirements and contains no results.
Frequently asked questions
Why can a benchmark be misleading?
Small samples, test material leaking into training data, different help given to different tools, or tasks unlike your own work can all distort a result.
Who should run a benchmark?
Independent testers have fewer incentives to flatter a product than its vendor does, though independence alone does not guarantee quality.
Has this site published benchmark results?
Not yet. The benchmarks page lists the requirements any result must meet.
Sources
This article is editorial analysis. It cites no external sources and contains no product performance claims, benchmark figures, or policy facts.