Measurement: Reliability and Validity

Measurement: Reliability and Validity

1. Reliability

   • Internal and External
     • Internal Reliability
       • Consistency of results across items within a test
       • Internal Consistnecy
         • Extent to which measure is consistent with itself
       • Split-half Reliability Test
         • Split test results into two equal halves -> Do questions that aim to measure the same thing have CORRELATED scores?
       • Test-retest
         • Measures stability over time - Give same participant the same test over time - Scores should be very similar
       • Parallel
         • Same kind of question as other test -> i.e. searching for the same measure -> Do scores correlate -> If two tests given to the same participant - should be highly correlated
     • External Reliability
       • Extent to which measure varies from one use to another
       • Inter-rater Reliability
         • Do professional score the same measures in the same way -> Someone who understands the thing being measured -> Do they rate the scores in the same way as others
         • Clinical diagnosis
         • Peer report
         • Behavioural observation and coding
     • Sources of Error
       • Unsystematic Errors -> cannot be correct retrospectiely
         • Response selection error -> Biased / unrepresentative sample
         • Poor test administation scoring -> subjectivity in recording results / research bias
       • Systemactic Errors -> easy to correct / harder to spot
         • Unconsidered conflating factor
         • Error in measurement tool
   • Consistently measures the same effect in the same circumstances
     • Measuring the same thing, should produce the same measure
   • Hindered by bad practice in measuring

2. Validity

   • Internal and External
     • External Validity
       • Essentially the generalisability of the results
       • Ecological
         • Generalise claim of effect / relationship to other settings
       • Population
         • Generalise claim of effect / relationship to other groups of people
       • Historical
         • Generalisability of effect / relationship over time
       • Improve with good sampling and large n
     • Internal Validity
       • Ability to claim whether some effect / relationship exists - Quantitative: Whether effect / relationship observed on DV was due to controlled manipulation of intended IV - Qualitative - Whether score on some trait / features reflects the score in real world 
       • Improve
         • Controlling for external effects
         • Standardised instructions
         • Remove instructor effects / bias
         • Counterbalancing -> participants each take part in as many conditions as possible
       • Face Validity
         • Superficial measure -> does it look right? - How valid the test 'appears' - Failures of intuition means this is a poor measure
       • Construct Validity
         • Is test score INTERPRETATION (and hence claim of effect / relationship) consistent with the theoretical background?
         • Divergent
           • Different measures of the same construct are uncorrelated
         • Convergent
           • Different measures of the same construct are correlated
         • Need to demonstrate phenomenon being measured actually exists
   • A TEST is valid if it measures what is claims to measure
     • And not something else -> i.e. effect from multiple causes, experiment bias, mistaking a trait (e.g. memory) as something else (e.g. memory)
   • Hindered by bad practice in sampling and investigating

3. Is the test / tool good at measuring something

   • Consistent
   • Stable
   • Precise

4. Is that thing what the test thinks it is

   • Certainty that other 'things' are excluded as causes of measure
   • Limited by reliability of measure

5. IS the test good at measuring the thing - Is there nothing else (i.e. bias) that may be causing the results of the test?

   • Is it a real effect that is true for all / most people or just coincidence?