How to read AI benchmarks without falling for the “percent accurate” trap

Before comparing two numbers, identify the task, sample, reference answers, metric, settings, model version, prompt, and date. SacreBLEU exists partly because changes in tokenization and test preparation can move machine-translation scores. Its signature records settings so another evaluator can reproduce the calculation.

chrF compares character n-grams between a candidate and one or more references. chrF++ also incorporates word n-grams. This is useful for morphologically rich languages because it can reward shared word structure, but it still measures overlap. A valid alternative sentence can score lower, and a fluent sentence with the wrong fact can retain substantial overlap.

First, reference similarity is presented as factual accuracy. Second, a sample from one genre is generalized to every use. Third, a benchmark that frontier systems have nearly saturated is treated as if it still separates them well. Fourth, an average hides critical failures or weak subgroups. Fifth, a small rank difference is reported without uncertainty or a paired comparison.

The AI Index adds another warning: benchmark questions themselves can be defective. In one review summarized by the report, estimated invalid-question rates ranged from 2 percent on MMLU Math to 42 percent on GSM8K. A precise calculation on a flawed test is still a flawed result.

Suppose Model A scores 57.8 chrF++ and Model B scores 57.2 on a set of paired news articles. The defensible statement is that A had 0.6 more chrF++ points under that dataset and metric configuration. It is not defensible to say A is 57.8 percent accurate or 0.6 percent better Armenian.

Next inspect paired article differences, confidence intervals, truncations, and error types. If the interval for the average difference includes zero, the ordering is uncertain on the sampled articles. Even when the difference is statistically distinguishable, a human-facing product still needs tests for meaning, names, numbers, tone, terminology, and correction burden.

For an Armenian publication, the test packet should include the actual content shapes the system will write: updates, explainers, opportunities, instructions, headlines, summaries, uncertainty, negation, quotations, and numerical claims. Deduplicate against development material and keep a hidden challenge set.

Use automatic metrics as diagnostics, not final authority. Track critical meaning errors, unsupported additions, numerical changes, names, unnatural constructions, critic disagreement, and later corrections. A model that produces slightly lower overlap but far fewer serious failures may be the better route.