Prompt library
Read out an A/B test without fooling yourself
Verdict-first with the deciding number keeps readouts from burying the decision in statistics theater. The peeking flag, post-hoc segment quarantine, and believability check encode the three most common ways teams convince themselves of effects that are not there.
Last reviewed July 17, 2026
The prompt
Write the readout for this A/B test.
Test setup (what changed, hypothesis, primary metric, planned duration and sample): {{setup}}
Results: {{results}}
Structure:
1. Verdict first: ship, do not ship, or extend, with the one number that decides it.
2. The primary metric: effect size with confidence interval, in absolute and relative terms. State whether the test reached its PLANNED sample; a test stopped early because it "looked significant" gets flagged as unreliable, with the reason (peeking inflates false positives).
3. Guardrails: every guardrail metric with its movement; any degradation gets weighed against the win explicitly.
4. Segments: only pre-registered segment cuts get causal language. Post-hoc segment findings go in a "hypotheses for the next test" list, clearly labeled; a segment dredged from 20 cuts is noise until retested.
5. The believability check: does the effect size pass the smell test given what changed? A 40 percent lift from a button color demands a data-quality audit before a celebration.
6. Decision and next step, including what we now believe about users that we did not before.
Never let a p-value substitute for the effect size and interval.Run in idaptOpens a new chat with the prompt prefilled. Nothing sends until you press send.
Fill in the variables
| Variable | What it is | Example |
|---|---|---|
| {{setup}} | The test's pre-registration | new onboarding checklist vs control; primary: week-1 activation; planned 2 weeks, 10k users per arm |
| {{results}} | The numbers | [paste the experiment dashboard export] |