Non-sampling Errors in Polls and Surveys

Potential Sources of Non-sampling errors

• Poor sampling method
• Non-random sample (Not all subjects have equal likelihood of being chosen)
• Self-selection effect
• Sampling frame does not represent the population
• Low response rate
• Questionnaire or measurement error
• Poorly written questions
• Question order effect
• Poor measurement protocols
• Behavioural effects
• Interviewer effect
• Wanting to look good
• Responses affected by weather, political or current events

Methods of Data Collection

• Surveys and Polls
• Experimental Studies
• Observational Studies

Polls and Surveys

• Poll
• Few questions
• Multi-choice type only
• Survey
• Many questions
• May have branches and skips
• May have a “tick multiple boxes” option
• May have open-ended, write a comment questions

Slide 4

Sampling

• Sampling Errors (Random process)
• Non-Sampling Methods
• Selection bias
• Non-response bias
• Self selection bias
• Question effects
• Behavioural considerations
• Interviewer effects
• Survey-format effects
• Transfer of findings

Slide 6

Sources of Non-sampling Errors

• Non-response bias
• When people who have been targeted to be surveyed do not respond:
• Potential bias if non-respondents are likley to behave differently to respondents with respect to the question being asked.
• e.g. Non-respondents in an employment survey are likely to be those who work long hours.
• Self-selection bias
• People decide themselves whether to be surveyed or not.
• Question effects
• Subtle variations in wording can have an effect on responses.
• e.g. “Should euthanasia be legal?”
• vs. “Should voluntary euthanasia be legal?”
• People are more likely to favour “voluntary” euthanasia.

Slide 8

• Behavioural considerations
• People tend to answer questions in a way they consider to be socially desirable.
• e.g. pregnant women being asked about their drinking habits may be reluctant to admit that they drink alcohol
• Interviewer effects
• Different interviewers asking the same question can obtain different results.
• e.g. the sex, race, religion , manner
• of the interviewer
• may influence how people respond to a particular question.
• Survey-format effects
• -question order e.g.
• “To what extent do you think teenagers are affected by peer pressure when drinking alcohol ?”
• followed by:
• “ Name the top 5 peer pressures you think teenagers face today.”
• -survey layout
• -interviewed by phone or in-person or mail.

continued...

• Transferring findings
• Taking the data from one population and transferring the results to another.
• e.g. Auckland opinions may not be a good indication of New Zealand opinions.

Non-Sampling Errors

• can be much larger than sampling errors
• are always present
• can be virtually impossible to correct for after the completion of survey
• virtually impossible to determine how badly they will affect the result
• good surveys try to minimize them in the design of the survey (e.g. do a pilot survey first)

Survey/Polls

• A report on a sample survey/poll should include:
• Who carried it out and who funded it
• target population (population of interest)
• sample selection method
• the sample size and the margin of error
• the date of the survey
• the exact question(s) being asked
• the results
• the claims (inferences) made