Understanding margin of error and statistical significance
Survey results come from a sample, so they are estimates. These two calculations tell you how much weight an estimate can carry: the margin of error describes the precision of one result, and a significance test tells you whether two results are really different.
What is the margin of error?
The margin of error is the range around a survey result in which the true value for your whole audience is likely to lie. If 62% of 500 respondents are satisfied, the margin of error at 95% confidence is ±4.25 percentage points. The true figure is most likely between 57.7% and 66.3%. That range is called the confidence interval.
How do I make the margin of error smaller?
Collect more responses. The margin shrinks with the square root of the sample size, so you need four times as many responses to halve it. Use the sample size calculator to find the number of responses that matches the margin you want.
What does statistically significant mean?
Two groups almost never give exactly the same result. A significance test checks whether the difference is larger than you would expect from chance alone. If it is, the difference is called statistically significant.
How do I read the p-value?
The p-value is the probability of seeing a difference at least this large if the two groups were in fact the same. A small p-value means chance is an unlikely explanation. At a 95% confidence level, a p-value of 0.05 or lower counts as significant.
A worked example
In group A, 300 of 500 respondents (60%) are satisfied. In group B, 250 of 500 (50%) are. The difference is 10 percentage points, the z-score is 3.18 and the p-value is 0.0015. That is well below 0.05, so the difference is statistically significant at 95% confidence.
Significant does not always mean important
With very large samples, even a tiny difference becomes statistically significant. Always look at the size of the difference as well, and ask whether it is large enough to act on.
