Reading global AI benchmarks
Created by Adrian Dunkley. This page explains how to assess a reported score. It does not publish a current model leaderboard.
Evidence status
The earlier numerical score, price, context-size and hardware tables are withheld. Their exact source records and evaluation configurations were not available for verification in this review. A general reference to a vendor’s model cards or a leaderboard does not substantiate every number in a comparison table. This is not a claim that the cited organisations did not run evaluations.
Check what was measured
Benchmark families address different tasks. Knowledge questions, coding issue resolution, mathematical problems, multimodal questions and long-document retrieval should not be treated as interchangeable measures of general competence. Examples include MMLU, GPQA, SWE-bench, MATH, MMMU and RULER; check each benchmark’s own current documentation and version.
A score on one dataset does not establish reliability in a Caribbean language variety, compliance with local law or safe behaviour in a particular business workflow.
Minimum record for a defensible comparison
| Field | What to preserve |
|---|---|
| Model identity | Exact release, provider and relevant mode or configuration |
| Source | Direct report, table or run link and its publication/access date |
| Evaluation | Dataset/version, split, exclusions and contamination controls |
| Method | Prompts, tools, retrieval, attempts, inference budget and scoring code |
| Result | Numerator/denominator, failures, uncertainty and missing observations |
| Cost | Actual pricing basis, usage, retries and human checking cost |
| Reproducibility | Available code, outputs and independent reproduction status |
Do not average scores from incompatible methods into a new ranking without explaining and justifying the calculation. A small numerical gap may be inconclusive; examine uncertainty and operational relevance rather than assuming a winner.
Use a local trial
Define the task and acceptance criteria before comparing candidates. Use appropriately protected examples, local-language reviewers and a manual baseline. Test difficult and harmful failure cases as well as ordinary successful tasks. Keep high-impact decisions under suitable qualified human control.
For long-context work, test whether the system uses the necessary evidence accurately, not merely whether it accepts a large input. For local deployment, measure the exact hardware/runtime configuration rather than inferring performance from model size.
Related guidance
The original page’s Creative Commons Attribution 4.0 notice and Adrian Dunkley attribution are retained for its original text. Third-party evaluation data keeps its own terms.