Research
Benchmarks
The requirements every benchmark we publish must meet. We have not published any benchmark results yet.
Results
Results
No benchmark results published
We would rather show an empty page than a number without a protocol. Results will appear here, with all of the information below, once testing has been done.
What every published benchmark includes
Requirements
| Element | What we publish |
|---|---|
| Task set | The tasks are published before results, with the reason each was chosen. |
| Conditions | Product version or model, settings, environment, and date. |
| Scoring | How success is judged, who judged it, and whether the check is automated or by a person. |
| Sample and runs | Number of tasks and number of runs per task, and how variation between runs is reported. |
| Independence | Who ran the test and any relationship to the vendor. |
| Limitations | What the benchmark cannot tell you, and how results could be misread. |
| Reproducibility | Enough detail that someone else could repeat it, or a statement of why they cannot. |