Threats to Internal Validity

occurs during the pre-test

exposure to pre-test impacts response to post-test

Instrumentation

changes in measure instrument

not with surveys

interview of a focus group or observation

person observing can change/adapt, get more experience

pilot data to check raters training and script

interrater reliability

Statistical regression to the mean (threat to statistical conclusion validity)

extreme score and then get another score is more likely to not get extreme

problematic when using scores to categorize

low IQ = low performer or high IQ= high performer

Selection Bias

result of not doing random assignment

the difference in dependent variable due to nonrandom assignment

problem is with quasi and correlational studies

Selection loss (mortality, attrition)

when the study loses some participants

problematic because there could be something meaningful in the people that dropped out

could also lead to some groups not being equally represented

Experimenter expectancy

experimenter bias

the experimenter creates expectations and treats participants differently

Reactivity: Participant Awareness Bias

people come into the study with their own theories about a particular behavior

they might:

try and make sense of the study

avoid negative evaluations

stupid, naive, embarrassed

please the researcher ( displease researcher)

Diffusion of treatment

The effect observed is due to treatment spreading across groups

e.g., teaching a strategy to one group and participants of that group tell the participants of the control group and the control group begin to use strategies

Compensatory equalization of treatment

the effect due to compensating control groups

e.g., give money to two schools and only tell one what to do with it and both schools show increase in dependent variable but can't be attribute to independent variable

Compensatory rivalry

the effect due to the control group wanting to prove research wrong ( John Henry effect)

the control group is the underdog

resentful demoralization

the control group wants to retaliate

the control group partakes in hypothesis guessing and can misconstrue the data

Hawthorne Effect

The effect in the dependent variable due to being assigned to the treatment group

changes were observed because changes were being made in the environment

the participant new about the observation and knowing this altered their reaction

Demand characteristics

features that communicate the information of the study (i.e., the hypothesis)

social roles of participants

good, faithful, negativistic, apprehensive

Other considerations *mundane realism - would this happen in real life

strategies to avoid participant awareness bias:

deception, cover stories ( high psychological realism) (high experimental realism)

double blind design, if not possible:

stay blind until last minute possible

use multiple researchers

make the independent variable and dependent variable hard to see

"accident" or "whoops" maneuver

- accidentally lost your test results and need to retake after true treatment

confederates

people part of the experiment not being tested

"multiple study" ruse

measure behavior that is hard to control

Measuring the dependent variable

option 1: behavior observations of participants' responses

typically the first choice

uncommon

option 2: self-report ( liking of another person)

limited by:

participants not knowing the answer

may base answers on inaccurate theories

may report what they think is desirable

option 3: behavioriod ( how much labor a participant says they would perform for someone else)

people might lie or overestimate

Next up

Previous