Is this A/B test result worth shipping?

Question

Accepted Answer

The result is not yet decision-grade. Three specific issues: Four days is not a full weekly cycle. Checkout behaviour differs substantially between weekdays and weekends. A test covering Monday to Thursday measures weekday behaviour, and shipping on that basis assumes weekends behave the same way. Run in whole weeks. p = 0.043 after repeated checking is not p = 0.043. If the test was monitored daily and would have been stopped as soon as it crossed 0.05, the effective false-positive rate is far above 5%. This is the peeking problem, and it is the single most common way A/B tests produce results that do not replicate. Either fix the sample size in advance, or use a sequential test designed for continuous monitoring. The effect is small relative to the noise. A 0.58 percentage point lift on a 5% base is an 11.6% relative improvement — plausible, but the confidence interval on this sample almost certainly spans from roughly break-even to a large gain. "Somewhere between nothing and a…

	Control	Variant
Users	8,412	8,390
Conversions	421	468
Conversion rate	5.00%	5.58%
p-value	0.043