Questions a child answers and a frontier model can miss.
Measured on Idapt. Every score below has a receipt: click any verdict for the model's actual answer.
Every item is a short factual question with one verifiable answer. Models are asked to reason freely and then commit on a final Answer: line; the grader reads that line, and falls back to loose extraction when a model ignores the format (recorded as a lower-confidence extraction tier, not as a wrong answer).
Outcomes are correct, incorrect, abstain (the model declined) and unparseable (nothing extractable). A refusal is never counted as a wrong answer.
Each item runs k times per model and the score is the majority verdict; the pass rate and the answer-consistency across samples are shown per cell, because one sample cannot distinguish knowing from guessing. Every score carries a 95% Wilson interval and its n.
Contamination: public. These questions are famous, so any model trained after they trended may have been taught the answer directly, and labs patch specific traps once they circulate. Read this battery as a timeline of which traps still bite and when each was fixed, not as a measure of intelligence.
These items are famous, so any model trained after they circulated may know the answers directly. Read scores as a timeline of what still bites, not a measure of intelligence.