The margin of error is an important statistical concept that helps you avoid over-generalizing the results of a survey or poll. It does this by telling you how much deviation from your survey result you can expect when projecting your findings to the total population.
For example, imagine you conducted a survey of 1,000 Singaporean adults and you found that 70% of the respondents had positive views toward foldable smartphones. Based on these 1,000 samples, can we really say that 70% of the total Singaporean population has favourable views toward foldable phones?
This is where the margin of error comes in. Based on the information we have for this poll (i.e. sample size, sample proportion, etc), we can determine that the margin of error is +/- 3% (assuming a 95% confidence level). What this means is that, based on the 1,000 samples we have, we can be 95% confident that between 67% and 73% would have answered this way if we interviewed the total Singapore population or ran the poll 100 times.
So the margin of error essentially tells you how much confidence you can have in a dataset based on the survey result and number of samples you’ve collected.
Generally speaking, the higher your sample size, the lower your margin of error. And a lower margin of error means you can be more confident that an observation is true. On the other hand, when you have a small sample size, your margin of error widens. This, in turn, means you are less confident that your findings are representative of the total population.
I’ve written another post that goes into detail about the margin of error and how you can calculate it manually, which you can find here.
You can also find a margin of error (MoE) calculator on my site here.
What I want to do in this post is offer another way for you to think about the margin of error. through charts!
Some of us are visual learners, and if you’ve read other articles on MoE and you’re still scratching your head, this article may help. I’ve created a visual demonstration of how the margin of error narrows as you introduce more samples. To do this, I’ve used a dataset that contains around n=5,000 samples from a survey about consumer views toward foldable phones.
With this data, I created 11 randomly generated cohorts ranging in size from n=10 to n=5,000 samples. Then I plotted the results of a few questions in a chart to show how the sample proportion for a particular response varies as more samples are introduced.
Let’s see what this looks like.
The first question, which we’ll refer to as Q1, asked the respondents if they like or dislike foldable phones. Here’s the question text and response list that was shown to the respondents.
Q1: In general, how do you feel about foldable smartphones with bendable screens?
Single Select
For this chart, I’ve isolated the results for the respondents who had positive views of foldable phones, which includes the sum of respondents who said I like them a little, I like them a lot, or I love them. In the world of market research, we refer to the kind of analysis (i.e. categorization of ordinal/Likert scale responses) as TOP or BOTTOM box grouping.
Here’s the chart.
As you can see in the chart above, the sum of those with favourable views of foldable phones starts out high, at 90%, with a very low sample base (i.e. n=10). But at the smallest sample size, we have an enormous MoE of +/- 19%. This means the true result could be anywhere between 71% to 100% (technically 109%, but we can’t go beyond 100%) if we interviewed the total population.
But as we introduce more samples, we can also see that the sample proportion of those with favourable views steadily decreases, eventually stabilizing around the 70-75% range. Aside from the sample proportion stabilizing, we can also observe how the margin of error narrows as the sample size increases (illustrated with the grey area labelled UPPER / LOWER BOUNDS). And the narrower the MoE, the more confidence we have in our results.
Next, we’ll be plotting data for a question that asked which words/phrases the respondents would associate with foldable phones (we’ll call this Q2). Here’s the question text and response list for that question.
Q2: Which of these attributes would you associate with foldable smartphones that have bendable screens?
Multi Select
This was a multi-select question where the respondents could select as many or as few options as they liked. For this chart, I’ll be isolating the results for those who sele the option “cool” from the list.
Here’s the second chart.
Similar to the chart for Q1, we can see a high sample proportion at a low sample size (i.e. 70%), then it steadily decreases and eventually stabilizes around 40% as we introduce more samples.
One thing to note is that the trend we’ve seen where the results over-index at lower samples and then gradually decrease isn’t always going to be the case. For example, here’s a chart that shows the data for respondents who selected the option “gimmick” from the list.
In this example, we can see that the results under-index at the smaller sample sizes, starting around 10%. Then, as we introduce more samples, the selection rate for “gimmick” gradually increases, stabilizing around the 30% range.
So remember that you won’t always see the sample proportion rate start high and then decrease. What’s important here is that you understand how the margin of error narrows as you collect more samples, giving us more confidence in the results. And, we also see how sample proportions (i.e. the selection rate for a response) stabilize as you collect more samples. For most of these charts, the sample proportion starts to stabilize when we exceed n=100 samples.
This is also one of the reasons why we don’t see polls in the news with 10,000 or 20,000 samples. That’s because we don’t really get significant boosts in confidence with a survey of 10K or even 20K samples compared to n=1,000. For example, have a look at the image below. The difference in MoE between 1,000 and 10,000 samples isn’t huge, and most researchers or marketers will find an MoE of 3% acceptable. That, and the cost to collect 10,000 samples would be enormous, so it’s usually not worth going to sample sizes that high for a 1-2% decrease in MoE.
You may have noticed some small differences in the margin of error across the three charts shown above. For example, in chart 1 (i.e. Q1) the MoE at n=10 is +/- 19%, while in chart 2 (Q2 = cool) the MoE at the same sample size is +/- 28%.
This occurs because the MoE calculation factors in the sample proportion, and the MoE will always be highest when the sample proportion is 50%. And when you move away from 50% (either higher or lower) the margin of error will decrease.
You can see an example of this in the chart below. The percentages across the x-axis show different sample proportions (i.e. 10%, 20%, 30%, etc), which correspond to a survey result where X% of the respondents answered a particular way (e.g. 10%/20%/30% of respondents said their favourite colour is orange).
As you can you see, the MoE is highest at 50%, and it’s lowest at 10% and 90%. You can also see that the MoE is exactly the same for sample proportions that are the same distance above and below 50% (i.e. the MoE is exactly the same at 40% and 60%, or at 30% and 70%, etc).
When you see polls in the news that cite a margin of error for an entire survey, you may be wondering how they can do this without referencing a specific question or survey result. This is because most news sites that cite a poll will use the highest MoE (i.e. at 50% sample proportion) to refer to the entire poll. This is also the same reason some MoE calculators don’t even ask for the sample proportion, and they’ll just calculate MoE based on the sample size, population size and confidence level, assuming a 50% selection rate.
That’s it for today. I hope you found the margin of error charts helpful!
