Follow the instruction exactly. Verified by counting.
Measured on Idapt. Every score below has a receipt: click any verdict for the model's actual answer.
Each item states a constraint that is verifiable in code: write exactly 17 words, never use the letter E, make the line initials spell BENCH. The response is checked programmatically against that constraint. There is no LLM judge, so there is no judge bias and no judge cost, and you can verify any cell yourself by counting.
Scoring is pass/fail per item; items run k times per model and report the majority verdict, the pass rate, and a 95% Wilson interval with its n.
Contamination: public, but weakly so. These constraints are ours rather than drawn from a published instruction-following suite, which matters: labs generate synthetic training data from the same public constraint taxonomies those suites use, so scores there partly measure the taxonomy. A constraint nobody trained on measures the skill.
These items are ours and were not drawn from a published suite, so training on them is unlikely.