INTERMEDIATEDifficulty: INTERMEDIATEStatistics

Explain a p-value to a non-technical stakeholder.

Short answer

The probability of seeing a result at least this extreme if the change had no effect at all — not the probability that the change works.

Full answer

"If the new checkout made no difference whatsoever, how often would random variation alone produce a gap this big? That is the p-value. Ours is 0.04, so about 4 times in 100."

Then the important part: "It does not mean there is a 96% chance the change works. It only tells us this result would be unusual if the change did nothing."

And then the part stakeholders actually need: "The more useful number is the confidence interval. Our best estimate is a 0.6 point lift, and the plausible range is roughly 0.05 to 1.2 points. So the change probably helps, and we cannot yet tell whether it helps a little or a lot."

What the interviewer is assessing

This question tests communication more than statistics. Interviewers are listening for three things:

Do you avoid the inversion? A p-value is P(data | no effect), not P(no effect | data). Getting this backwards is the single most common statistical error in industry.

Do you move to effect size? Stakeholders make decisions on magnitude, not significance. An analyst who stops at "it's significant" has not finished the job.

Do you avoid jargon without being condescending? "Null hypothesis" adds nothing here. Neither does over-explaining.

If you get stuck

  1. Hint 1. What is the p-value conditional on?
  2. Hint 2. Stakeholders want to know how big the effect is, not just whether it exists.