Non-sampling Errors in Polls and Surveys

Poor sampling method Non-random sample (Not all subjects have equal likelihood of being chosen) Self-selection effect Sampling frame does not represent the population Low response rate Questionnaire or measurement error Poorly written questions Question order effect Poor measurement protocols Behavioural effects Interviewer effect Wanting to look good Responses affected by weather, political or current events  

Potential Sources of Non-sampling errors

Methods of Data Collection

Surveys and Polls Experimental Studies Observational Studies

Polls and Surveys

Poll   Few questions Multi-choice type only

Survey   Many questions May have branches and skips May have a “tick multiple boxes” option May have open-ended, write a comment questions  

2017 08 23 (binary/octet-stream)

Sampling

Sampling Errors (Random process)

Non-Sampling Methods Selection bias Non-response bias Self selection bias Question effects Behavioural considerations Interviewer effects Survey-format effects Transfer of findings

2017 08 23 (1) (binary/octet-stream)

Sources of Non-sampling Errors

Non-response bias When people who have been targeted to be surveyed do not respond: Potential bias if non-respondents are likley to behave differently to respondents with respect to the question being asked. e.g. Non-respondents in an employment survey are likely to be those who work long hours.   Self-selection bias People decide themselves whether to be surveyed or not.   Question effects Subtle variations in wording can have an effect on responses. e.g. “Should euthanasia be legal?” vs. “Should voluntary euthanasia be legal?” People are more likely to favour “voluntary” euthanasia.  

Behavioural considerations People tend to answer questions in a way they consider to be socially desirable. e.g. pregnant women being asked about their drinking habits may be reluctant to admit that they drink alcohol     Interviewer effects Different interviewers asking the same question can obtain different results. e.g. the sex, race, religion , manner       of the interviewer may influence how people respond to a particular question.   Survey-format effects -question order e.g. “To what extent do you think teenagers are affected by peer pressure when drinking alcohol ?” followed by: “ Name the top 5 peer pressures you think teenagers face today.” -survey layout   -interviewed by phone or in-person or mail.

Transferring findings Taking the data from one population and transferring the results to another. e.g. Auckland opinions may not be a good indication of New Zealand opinions.    

continued...

Non-Sampling Errors

can be much larger than sampling errors are always present can be virtually impossible to correct for after the completion of survey virtually impossible to determine how badly they will affect the result good surveys try to minimize them in the design of the survey (e.g. do a pilot survey first)

Survey/Polls

A report on a sample survey/poll should include: Who carried it out and who funded it target population (population of interest) sample selection method the sample size and the margin of error the date of the survey the exact question(s) being asked the results the claims (inferences) made