Stale "best known" targets in AI-math benchmarks
How many stored targets in public benchmarks of AI mathematical discovery have been superseded by the literature, or cannot be scored as written. Every figure in the tables is computed from the per-target audit files released with the paper (CC BY 4.0).
Per benchmark
| Benchmark | Pinned | n | Stale or unscoreable | Wilson 95% | Interval | UNCLEAR | Stale at pin | Median lag to pin (days) | Last audited |
|---|
"Stale or unscoreable" = STALE + MALFORMED. UNCLEAR and leaderboard-only rows are shown, never counted. "Stale at pin" and "median lag" use a mechanical rule on the released files: a STALE row whose source date (the first full date given) precedes the pin date; the lag is the pin date minus the source date. The paper's hand-checked stale-at-pin counts, which also read dates given in the notes, are 5 (notebook) and 8 (paper; one of them now UNCLEAR). HorizonMath's pinned commit has no stated date, so both cells are blank for it.
Non-current rows
Non-novel announcement rate per claimant
Registered forecast
Registered 2026-10-12, before any of the objects exist. Window: releases between 2026-10-13 and 2027-04-12. Objects: the next HorizonMath release or data update (at least 10 changed rows); the next AlphaEvolve results release (at least 10 new rows, or a paper); the next EinsteinArena results paper or announced snapshot; any new public AI-math benchmark with at least 30 numeric "best known" targets.
- P1. For each object, the stale-or-unscoreable fraction (strict standard, searched within 14 days of release, cut-off = release date) is at least 0.15.
- P2. At least one object has at least one row already superseded on its release date by a source at least 60 days older.
- P3. In the next HorizonMath release, at least one of the 11 rows flagged in the October 2026 audit remains uncorrected.
Adjudicated per object 21 days after its release, hard stop 2027-05-03. Thresholds do not move. A negative result is reported as "the field corrected itself"; no qualifying release by 2027-04-12 is reported as "no observation". Status: open.