Threats to Internal Validity
occurs during the pre-test
exposure to pre-test impacts response to post-test
Instrumentation
changes in measure instrument
not with surveys
interview of a focus group or observation
person observing can change/adapt, get more experience
pilot data to check raters training and script
interrater reliability
Statistical regression to the mean (threat to statistical conclusion validity)
extreme score and then get another score is more likely to not get extreme
problematic when using scores to categorize
low IQ = low performer or high IQ= high performer
Selection Bias
result of not doing random assignment
the difference in dependent variable due to nonrandom assignment
problem is with quasi and correlational studies
Selection loss (mortality, attrition)
when the study loses some participants
problematic because there could be something meaningful in the people that dropped out
could also lead to some groups not being equally represented
Experimenter expectancy
experimenter bias
the experimenter creates expectations and treats participants differently
Reactivity: Participant Awareness Bias
people come into the study with their own theories about a particular behavior
they might:
try and make sense of the study
avoid negative evaluations
stupid, naive, embarrassed
please the researcher ( displease researcher)
Diffusion of treatment
The effect observed is due to treatment spreading across groups
e.g., teaching a strategy to one group and participants of that group tell the participants of the control group and the control group begin to use strategies
Compensatory equalization of treatment
the effect due to compensating control groups
e.g., give money to two schools and only tell one what to do with it and both schools show increase in dependent variable but can't be attribute to independent variable
Compensatory rivalry
the effect due to the control group wanting to prove research wrong ( John Henry effect)
the control group is the underdog
resentful demoralization
the control group wants to retaliate
the control group partakes in hypothesis guessing and can misconstrue the data
Hawthorne Effect
The effect in the dependent variable due to being assigned to the treatment group
changes were observed because changes were being made in the environment
the participant new about the observation and knowing this altered their reaction
Demand characteristics
features that communicate the information of the study (i.e., the hypothesis)
social roles of participants
good, faithful, negativistic, apprehensive
Other considerations *mundane realism - would this happen in real life
strategies to avoid participant awareness bias:
deception, cover stories ( high psychological realism) (high experimental realism)
double blind design, if not possible:
stay blind until last minute possible
use multiple researchers
make the independent variable and dependent variable hard to see
"accident" or "whoops" maneuver
- accidentally lost your test results and need to retake after true treatment
confederates
people part of the experiment not being tested
"multiple study" ruse
measure behavior that is hard to control
Measuring the dependent variable
option 1: behavior observations of participants' responses
typically the first choice
uncommon
option 2: self-report ( liking of another person)
limited by:
participants not knowing the answer
may base answers on inaccurate theories
may report what they think is desirable
option 3: behavioriod ( how much labor a participant says they would perform for someone else)
people might lie or overestimate
External Validity
Process:
step 1: define a target population of individuals, settings, or times
step 2: draw samples from those populations
two types:
representative sample: samples that correspond to a well known population
very rare
accidental samples or samples of convenience: achieved by a procedure designed to ensure representativeness
may or may not be representative but you might not know
Features of external validity
1. generalizing to particular target individuals , settings, and times
e.g., does this information hold true for other individuals under the same circumstance
2. generalizing across types of individuals, settings, and times
e.g., does this information hold true for different individuals under different circumstance
Threats to External Validity
1. Interactin of selection and treatment
population is adequately targeted, but the results are only applicable to the participants who show up.
(volunteers, exhibitionists, hypochondriacs, scientific dogooders)
Solution: Make participation as convenient as possible and attractive to get overall participation
2. Interaction of setting and treatment
this is where the effect of treatment is dependent on the setting
e.g., can you get the same results at a factory as you do at a university
Solution: Vary setting and see if the same patterns show up
3. Interaction of History and treatment
this is when different times results in different results
an example is the cohort effect
e.g., national tragedy
Solution: replication
