This explainer is editorial interpretation of a well-established concept. It contains no test results.
What is benchmark contamination?
A benchmark is a set of tasks used to compare AI systems. If a model was trained on data that includes those tasks or their answers, it may do well because it has seen them, not because it can solve new problems of the same kind.
How does contamination happen?
- Test sets are published online and later collected into training data.
- Discussions, solutions, or copies of the test appear on other sites.
- Training data and test data are not carefully separated.
How can I spot a contaminated result?
- A very high score on a long-public test with no explanation of how overlap was checked.
- Strong performance on the test but weaker performance on similar new tasks.
- No statement of the training data cut-off relative to the test's publication date.
What do testers do about it?
Common defences are keeping test material private, regularly creating fresh tasks, rewriting items, and checking overlap. Each has costs and limits, which is why a result should state its conditions.
See reading benchmark claims critically for a checklist, and the glossary for related terms.
Frequently asked questions
Does contamination mean a model cheated?
Not necessarily. Training data is often collected from the public internet, so test material can end up in it without anyone intending it. The effect on the score is the same either way.
Can contamination be detected?
Sometimes. Testers check for overlap between test items and training data, compare performance on original and rewritten items, or use tasks created after a model's training data was collected. None of these is conclusive on its own.
Are private benchmarks immune?
They reduce the risk, but they cannot be inspected by others, so you must trust the tester's method. Both public and private tests have limits.
Sources
This article is editorial analysis. It cites no external sources and contains no product performance claims, benchmark figures, or policy facts.