Ask for a random number 50 times. Watch what happens.
Measured on Idapt. Every score below has a receipt: click any verdict for the model's actual answer.
Each item asks for a random choice (a number from 1 to 100, a coin flip, a colour) and is run 50 times per model. Nothing is graded: there is no correct answer to a request for a random number. What is recorded is WHICH value came back each time, and the distribution across the 50 samples is the measurement.
The consistency figure is the share of samples that agreed with the model's most frequent answer: a model that says 42 every time scores 1.0, a genuinely uniform sampler scores near 1/50. Neither is right or wrong; they are different behaviours, and the histogram shows which one you are buying.
Responses are capped at a few tokens, so a full sweep costs cents.
Contamination: not applicable — there is no answer to leak.
These items are famous, so any model trained after they circulated may know the answers directly. Read scores as a timeline of what still bites, not a measure of intelligence.