5. Reliability vs. Validity

5. Reliability vs. Validity

1. Reliability = whether the measure gives consistency results

   • Test-retest reliability: do the test, then go back and do the test as a later point and should find the same results (if you are measuring a stable thing)
   • Internal reliability: a measure based on the correlations between different items on the same test. It measures whether several items that propose to measure the same general construct, produce similar scored.
   • Inter-rater reliability: the degree of agreement among rater's. It gives a score of how much consensus there is in ratings given by judges.

2. Validity = whether it is measuring what it's supposed to be measuring

   • Convergent validity: if you have two different measurements and they're meant to be measuring the same thing, they should overlap
   • Discriminant validity: if you have two different measurements and they're meant to be measuring different thing, they shouldn't overlap
   • External validity: the extent to which the results of a study can be generalised to other situations and to other people.
   • Incremental validity: tells you what a previous measure has told you and more - additionally, relevant information

3. Types of personality measures

   • Explicit self rating: self description using personality questionnaires
     • Advantages: easy to collect lots of rich data, inexpensive, easy to conduct --> relates to the person's conscious view of self
     • Disadvantages: Social desirability basis
     • Standard way to measure personality
   • Implicit self-categorisation test (implicit association test): alternative between a condition where self is paired with one end of the scale or the other. Then you look at the difference in reaction times between when self is paired with one end over the other. E.g. quicker reaction time if you unconsciously think you're nervous, when self is paired with nervous words
     • Advantages: immune to social desirability, relates to person's unconscious view of self
     • Implicit vs. explicit: where both IAT and explicit measures are used, they correlates about r=0.2. Both predict external stimuli, which is more externally valid depends on the domain. Incremental validity = having both explicit and implicit allows you to predict better than either one alone.
   • Behavioural: spontaneous behaviour in a particular situation
     • Advantages: high degree of external validity if you are interested in predicting behaviour
     • Disadvantages: costly and slow to gather, may not be appropriate to your question
     • Animal research is mostly behavioural
     • Back et al (2009): Behavioural measures related to the explicit and implicit personality measures, however not very strongly.
   • Performance: how long would you sit with a spider crawling up your arm (as a measure of neuroticism)
     • Advantages: high degree of external validity if you're interested in predicting
     • Disadvantages: costly and slow to gather, may not be appropriate to your question

4. Gender differences

   • Vianello et al (2013): In implicit measures, gender differences were much smaller than for explicit measures.
   • Possible explanation: explicit self-concepts partly reflect societal norms about gender roles. Whereas, implicit self-concepts reflect self-related experiences