Methodology
The three named quantities
1. Stale-target rate. For a stored table of "best known" values pinned at a version, the fraction of its targets that are STALE or MALFORMED under the verdict standard below, at a stated search date. Reported with a Wilson 95% interval. UNCLEAR and leaderboard-only rows are reported separately and never counted. Search recall is imperfect, so the rate is a lower bound.
2. Priority lag. For an announced improvement, the number of days by which the earliest primary source stating an equal-or-better value precedes the claim: lag = claim date − source date. It is defined only for claims with such a source, and a source dated on or after the claim date does not count. Over the 17 third-party rows, the median is 85 days (range 6 to 11,483). The table analogue on the dashboard, "lag to pin", is the pin date minus the superseding source date. For the constants table it is a median of 31.5 days over 16 rows (range 2 to 2,587).
3. Non-novel announcement rate. For a claimant (a lab, system or platform announcing "new best" values), the fraction of its announced values for which an opened primary source dated before the announcement already gives an equal or better value. The strict variant counts only third-party sources that the claimant's own text does not print; a re-report of the claimant's own earlier value is not a false novelty claim.
Verdict standard
- STALE: a strictly better value exists. Requires a verbatim quote from a primary source that was opened (paper, canonical table, or the benchmark authors' own paper), or a public construction re-scored as beating the target by the benchmark's own validator.
- MALFORMED: the stored value is not a scoreable best-known value (a proven bound stored as a target to beat, an exact constant, a numerical estimate, a range string, a count of test cases, or a value its cited source does not print).
- UNCLEAR: not resolvable under the standard (source unreachable; snippet, blog or announcement only; or an unrefereed claim to solve a major open problem). Reported, not counted.
- CURRENT: no superseding source found. This means "none found", not "proven current".
- STALE-LEADERBOARD: a live leaderboard shows a better score, but no primary source or validator re-score exists. Reported, not counted.
Search snippets, page summaries, blogs and social-media posts never suffice for STALE. Every non-current verdict was confirmed by re-opening the source and reading the number. A blind second searcher re-judged random samples of 30, 11 and 22 targets; in the first sample it found three stale targets we had missed, so the rates are lower bounds.
Reproduce
The per-target files, with sources and verbatim quotes, are released with the paper under CC BY 4.0. A standard-library script recomputes the table counts, the pooled rate, the Wilson intervals, the lag medians and the agreement statistics from those files and prints PASS or FAIL against the paper text.