Reliability and Validity - Research Methods

Reliability and Validity - Research Methods

1. Reliability

   • consistent findings
     • For example, if researchers replicate an experiment, using the same methods of observation and measurement, and obtain the same results, then both the method and the data they produce are seen as reliable.
     • Researchers can only trust results if they are reliable
     • A measuring instrument is reliable – it produces consistent measurements – but it is not valid. A valid measuring instrument produces accurate measurements and measures what it is supposed to measure.
   • Types
     • Internal reliability
       • How consistently a method measures within itself. E.g. all the questions on a psy. test should be measuring the same thing. To test, randomly split the test into two halves – the scores should be similar if the test is reliable
     • External reliability
       • How consistently a method measures between uses i.e. over time, population and location when repeated. A tape measure which gives the same reading for your height on a number of occasions is said to have high external reliability.
   • Measured
     • Inter-Observer Reliability (or: Inter-Rater) – whether two observers are being consistent in their observations. Data from each observer is correlated. Higher the correlation the more inter-rater reliability.
     • Test-Retest Reliability – whether the same test administered on two different occasions gives highly positively correlated results (time span is critical here too).
     • Parallel-Forms Reliability – whether two instruments administered to the same group of people give highly positively correlated results.
     • Internal Consistency Reliability – whether the results obtained from administering the instrument to a group of people on one occasion correlate highly positively – e.g. Split-Half Reliability.

2. Validity

   • accurate findings
     • Example: Effect of noise on memory experiment If participants performance on the memory test (dependent variable) was due to variations in noise level (independent variable) then the experiment is internally valid. However if performance was affected by extraneous variables such as the difference in room temperature between conditions, then the experiment does not have internal validity – the results are inaccurate.
   • Types
     • Internal validity
       • The experiment measures what it intended to measure. Are the effects observed genuinely caused by the independent variable?
       • Internal validity is the extent to which the findings can be attributed to the effect of the independent variable. An experiment is internally valid when the findings have not been affected by confounding variables
     • External validity
       • The extent to which the findings of an experiment can be generalised to settings outside the laboratory i.e. everyday situations. A research study has high external validity if its findings can be generalised to other populations, other settings and other times.
       • Populations – What if a sample was based on female undergraduate? Can the findings be generalised to males and people of different ages? Settings – what if the field study of a study on helping behaviour took place in Northwood Hills could the findings be generalised to other settings e.g. Manchester, Glasgow? Times (temporal) - Are the findings of Asch’s conformity study, conducted in the 1950s, still valid today Ecological validity refers to the extent the which the findings can be applied to other settings The highest level of external validity is when findings can be generalised to all population, all settings and all times
   • Validity of the test used.
     • Assessed using, face validity (eyeball test), content validity (content of measurement is appropriate for gathering data on a particular subject), concurrent validity (new test is compared to an old established test)