Data
Read out an A/B test
Call a result, including when the answer is that it is too early to call.
What it asks you for
- The setuprequired
- Hypothesis, metric, and how it was split.
- The numbersrequired
- Per variant: exposure and conversions.
What it tells the AI to do
Read out this experiment.
Setup:
{{ setup }}
Results:
{{ results }}
State the effect with its uncertainty, not as a point estimate. "4.2% versus
3.8%" invites a decision the data may not support; "a 0.4 point lift, interval
-0.3 to +1.1" shows the reader how much the data actually supports.
Answer four questions before recommending anything:
1. Was the metric the one that matters, or the one that moved?
2. Is the sample big enough for this effect size, or is this a coin flip with a
narrative?
3. Did the split hold? Check exposure balance before you read the result.
4. Did anything else change during the window?
Then give one verdict: ship, kill, or keep running. "Keep running" is a real
answer and the right one more often than teams like. If the result is
inconclusive, say inconclusive rather than picking the direction the numbers
lean.Why this works
The failure mode in experiment readouts is a point estimate presented as a decision. "4.2% versus 3.8%" invites a conclusion the data may not support; the same result stated with its interval shows how much the data supports. This checks the four things that invalidate a result before it will recommend anything, and treats "keep running" as a real verdict.
Questions
- Will it tell me the test failed?
- It will tell you the test is inconclusive, which is different and more common. Picking the direction the numbers happen to lean is how underpowered tests become shipped features.
- experiment
- ab test
- statistics
- readout
More data skills
- Extract to a tablePull structured rows out of documents, pages, or notes.
- Analyse a spreadsheetLoad a CSV or spreadsheet and answer questions about it.
- Chart this dataTurn numbers into one chart that makes the point.
- Clean a datasetDeduplicate, normalise, and repair a messy table.
- Write a data dictionaryDocument your tables and their traps so queries stop guessing.