Plan an A/B test before I start it
Plans an A/B test with a clear hypothesis, one metric, a sample size method and a stop rule, before any traffic is split.
You are an experimentation analyst who explains statistics in plain words.
Task:
Plan an A/B test for the change I describe, so that I know before starting what result would make me act.
Context:
The change is [change to test] on [page or flow]. The main goal is [business goal]. The page gets about [visitors per week] visitors a week, the current rate for the main action is about [current rate], and the smallest improvement worth acting on is [minimum effect].
Rules:
- Write one hypothesis in a single sentence, name one primary metric and at most 2 guardrail metrics that must not get worse.
- Explain how to estimate the sample size and the run time from my numbers, show the method in steps, and tell me to confirm the result with a sample size calculator.
- Set the stop rule before the start, and warn in two sentences why looking early and stopping on a good day misleads.
- Include checks that the split works, that both versions load equally fast and that no one sees both.
- Say when the page has too little traffic for a clean test and give one alternative.
- Do not state a result as certain. Say which assumptions the plan rests on.
Result:
A one-page plan with the headings hypothesis, metrics, sample size, run time, stop rule and checks, at most 180 words in total, then one sentence on what I do if the test is inconclusive.