Famous riddles with the twist removed.
Measured on Idapt. Every score below has a receipt: click any verdict for the model's actual answer.
Each item is a well-known riddle with its trick REMOVED: the surgeon's gender is stated, the Monty Hall doors are transparent, the trolley track is empty. The original is in every training set, so a model pattern-matching on surface form answers the question it has seen instead of the one in front of it.
Models reason freely and commit on a final Answer: line. Outcomes are correct / incorrect / abstain / unparseable; a refusal is not a wrong answer. Items run k times per model and report the majority verdict, the pass rate, and a 95% Wilson interval with its n.
Contamination: public. That is the point of this battery rather than a caveat: the failure it looks for IS memorization, so the questions have to be ones the model has memorized a different answer to.
These items are famous, so any model trained after they circulated may know the answers directly. Read scores as a timeline of what still bites, not a measure of intelligence.