SurveyGizmo Blog

From survey design all the way through taking action, you'll find tried and tested survey knowledge on each and every page of our blog.

Survey Sample Size: What Should It Be?

Liz Millikin Nov 30, 2016 5 Comments

Your survey sample size, or the number of people from your target population that you distribute your survey to, has a big impact on both your costs and your data.

Too few respondents means your data may not be representative of the population, leaving you with inaccurate results).

Too many respondents can be a drain on both your time and your costs, especially if you are using a panel company to purchase respondents or offering incentives to all survey takers.

Thankfully, there is a better way. Use a survey sample size calculator like this one to determine how many respondents you need to reach too maximize your time, budget, and ensure you’re able to act on the most accurate results possible.

Sample Size Calculator

Confidence Level:
Confidence Interval: % Enter Interval - e.g. 4 = 4%
Population:
  * Assumes a normal distribution of 50%
to calculate your sample size
 
 

How to use the sample-size calculator

  1. Select your confidence level
  2. Enter your confidence interval - entered as a decimal
    (e.g. 4% entered as .4)
  3. Enter your population
  4. Get your sample size

Confidence Level: Tells how sure you can be. The most common percentages are 95% and 99%

Confidence Interval: The margin of error. How close your sample is to the true value of your population.

 

6 Factors That Affect Your Survey Sample Size

If you're new to calculating survey sample sizes, then the results of the calculator above may surprise you. After all, how could it be possible that to survey very large populations…you need comparatively few responses.

These six factors can and do influence your ideal sample size.

1. Analytical Plan

Before you collect your data, it’s important to know what you are going to be doing with that data. This is your analytical plan.

There are multivariate analysis techniques, such as regression analysis, that require a certain number of observations per variable in order to be accurate. For this kind of analysis, you will need the requisite number of responses for each variable you plan to analyze.

Or, if there you are interested in finding statistically significant differences between subgroups in the sample, the sample size needs to be adjusted accordingly to ensure that you are reaching enough members from each subgroup.

For example, if you want to determine whether male or female college graduates feel more prepared for careers, you’ve got to be sure and have enough responses from both groups that the comparison actually makes sense.

2. Population Variability

If there is a lot of variability among individuals in your target population around the issue of interest, a larges sample size might be needed to give you an accurate representation.

For instance, if between 20% or 80% of your college graduate audience reports that they plan to take a vacation right after graduation, that’s a high level of variability. It means you need a larger sample size.

But if around 50% of your audience exhibits a particular behavior, then you don’t need as large of a sample to ensure statistical relevance.

3. Level of Confidence

The amount of risk of inaccuracy that you’re willing to tolerate in your survey results is referred to as the level of confidence. It’s typically expressed as a percentage, e.g. 95% confidence level or confidence interval.

Survey results are typically reported as point estimates. For example, 60% of people do not know how to change a car tire. The fact is, however, that because we are looking at the results of a survey that only included a sample of the population, we can only be confident that 60% of the people who took the survey feel this way. That said, this survey included a very, very large sample size: 2,000 drivers in the US. Therefore, we can be fairly confident that these answers are reasonably representative of the population of drivers in the US as a whole.

The amount of confidence we have that this percentage correlates to the responses that would be given by the entire target population is our level of confidence.

This percentage indicates the probability that the true value falls within the confidence interval boundaries.

The confidence level is inversely proportional to estimate accuracy. This means that the more confident you want to be in your data, the larger the interval that will contain the true value of the estimate, which leads to lower levels of precision.

4. Margin of Error

Also known as sampling error, this indicates the level of precision of the estimates that you want in your data.

When you see poll results presented as +/- 3%, this refers to their margin of error.

This percentage defines the lower and upper bounds of the confidence interval likely to contain the parameter estimate, and it is an indicator of the estimate’s reliability.

The larger the sample, the smaller the margin of error and the greater the estimate precision.

5. The Cost of Reaching Your Survey Audience

While everyone wants 100% accurate data, it comes as a literal cost. It’s common in research to have to make a trade-off between statistical accuracy and research costs. Larger samples are more expensive.

Factors such as low incidence and low response rates can also increase the costs of reaching the sample you need.

For example, let’s say you wanted to conduct an online survey with a sample of 1,000 respondents, which would give a statistical accuracy of +/- 3.1% at the 95% confidence level, and your budget was $8,000.

At the same time, a sample of 400 respondents would give a statistical accuracy of +/- 4.9% and cost $3,400.

In this case, a 135% increase in sample cost would only yield a 60% gain in statistical accuracy. A smaller sample size would be the way to go.

6. The Total Size of the Population You're Surveying

Most of the time the size of the total target population is unknown, and it’s assumed to be very large (i.e. greater than 100,000). But in studies where the sample is a large fraction of the population of interest, some adjustments will need to be made to the sample size to ensure it’s representative of the target population’s size and proportions.

 

A Checklist for Estimating Survey Sample Size

When calculating sample size, use our sample calculator as a guide. But, don’t forget to ask yourself the following questions:

  • What type of data analysis will I be doing with this data? Will I want to compare subgroups?
  • What’s the probability of the event I’m investigating occurring in this population? If no previous data exists, use 50% for a conservative estimate.
  • How much error can I tolerate (your confidence interval)? How much precision do I need?
  • How confident do I need to be that the true population value falls within my confidence interval?
  • What’s my budget? Can I afford the sample I really want/need?
  • What’s the population size? Large? Small? Finite? When the population size is unknown, assume it’s very large.
  • As you may have deduced, there’s no magic solution to determining the right sample size for a survey, but there are a lot of factors to consider carefully.

Hopefully these factors will give you a place to start, and you can also use this calculator from Relevant Insights to help you determine your ideal sample size and margins of error.

 

Note: This post was originally published in May, 2010. It has been updated, expanded, and republished since then with best practices and examples.

Spread the word
Try SurveyGizmo!

Data collection has never been this easy. Test drive a SurveyGizmo account – free for 7 days.

START BUILDING SURVEYS

Do you work on a team? Start your trial on the SurveyGizmo Team & Enterprise page.

By accessing and using this page, you agree to the Terms of Use. Your information will never be shared.

Be in the know!

Subscribe to get regular updates and best practices for survey creation, data collection, and data analysis.