Before asseessing validity
statistical conclusion validity
do the variables actually covary?
Covariation
A necessary condition for inferring cause:
no variance, no relationship can be detected
covariation when there is an actual relationship?
GREAT
Covariation when there isn't an actual relationship?
NOT GOOD - type I error
No covariation when there isn't an actual relationship?
GREAT
No covariation when there is an actual relationship?
NOT GOOD - type II error
type I error
false positive
type II error
false negative
How to asses type I and II errors
alpha probability
P <= 0.05 of a type I error
Sophisticated answers
assess threats to statistical conclusion validity
low statistical power
1. Is the study sensitive enough to permit reasonable statements about covariation?
2. How much power does a study have to detect a difference when one actually exists?
two functions:
A prior ( i.e., planning a study)
conduct a power analysis to determine the sample size required for detecting an effect of the desired magnitude (e.g., small, medium, large)
power analysis calculator/ formula
Post-hoc ( i.e., evaluating a study's power)
most common approach significance testing
p<= .05
becoming more popular: confidence intervals, or the magnitude of the effect that could have been reasonably detected
Violated assumptions of statistical tests
most common assumptions
equivalent groups in the beginning
normality
equal variances
assumptions vary by statistical test
Fishing
Chances are high that if you test every possible relationship, something will be significant
looking for the change by looking at different angles
increases type I error
what to look for
post-hoc tests presented as a prior hypothesis
multiple tests when a single test would be sufficient
what not to do
present post hoc/exploratory analyses as a prior hypothesis
Reliability of measures
reliability of treatment implementation from participant to participant
random irrelevancies in the test conditions
regression to the mean
Random heterogeneity of participants
Criterion oriented Validity
General process:
Researcher administers the test, obtains a measure of the criterion on the same subjects and computes a correlation
criterion oriented validity is similar to the idea of nomological network
Convergent: measures of constructs that theoretically should be related to each other are, in fact, observed to be related to each other
testing for convergence across different measures or manipulations of the same thing
Divergent/discriminant: measures of constructs that theoretically should not be related to each other are, in fact, observed to not be related to each other
testing for divergence between measures and manipulations of related but conceptually distinct things
the Multi-trait Multi-method matrix
coefficients in the reliability diagonal should consistently be the highest in the matrix of the nomological network
basically, a trait should be more highly correlated with itself than with anything else
coefficients in the validity diagonals should be significantly different from zero and high enough to warrant further investigation
A validity coefficient should be higher than values lying in its column and row in the same heteromethod block
A validity coefficient should be than all coefficients in the heterotrait monomethod triangles
The same pattern of trait interrelationship should be seen in all triangles
