2.4 Looking for Flaws: Evaluating Research

Sampling Bias

• Samples need to be reasonably representative of the population. Sample composition (demographic makeup of age/sex/income/etc.) is similar to population. If this does NOT happen, generalizations may be inaccurate or skewed.
• When there are doubts/concerns about the results of a study, first thing to examine is the composition of the sample.

Placebo effects

• Given to some subjects to control for the effects of subjects' expectations (which would be an extraneous variable).
• Expectations can influence feelings, reactions, and behaviors; be powerful determinants of perceptions and behaviors when under empirical study
• Should guard against placebo effects whenever subjects likely to have expectations that a treatment will affect them in a certain way by including a placebo condition in a study

Distortions in Self-Report Data

• *Questionnaires, interviews, personality inventories use self-report data and can have distortions:
• Social desirability bias
• Respondents misunderstand questionnaire items
• Order in which questions are asked can shape responses
• Memory errors can undermine accuracy
• Some people tend to agree OR disagree with nearly all statements
• Halo effect

Experimenter Bias

• Most researchers have an emotional investment in the outcome of the research
• Researchers can sometimes see what they want to see. Can unintentionally influence subjects' behaviors through subtle nonverbal signals
• Can be neutralized w/double-blind procedure