Reliablity - Internal and External and how they affect research

Page 1

Reliability in Observational Methods.

Definition of reliability - A measure of consistency both within a set of scores or item (internal reliability) and also over time, such that it is possible to obtain the same results on subsequent occasions when the measure is used (external reliability). The reliability of an experiment can be determined through replications.

So it refer to whether something is consistent.

Any tool used to measure something must be reliable. If the tool is measuring the same thing it should produce the same result of every occasion. If the result is different then we need to be sure that it is the thing that has changed or is different, and not our measuring tool.

Reliability of observations

Observations should be consistent, which means that ideally two observes should produce the same record.

The extent to which two (or more) observers agree is called INTER-RATE or INTER-OBSERVER RELIABILITY.

A general rules is that if there is more than 80% agreement on the observations, the data have inter-observer reliability.

Total Agreements > 80%

Total Observations

Improving Reliability

Observers should be trained in the use of a coding system/behaviour checklist.

They should practise using it and discuss their observations.

The investigator can then check the reliability of their observations.

Reliability in Self-Report Techniques

Internal reliability is a measure of the extent to which something is consistent within itself. However sometimes internal consistency is not important.

External reliability is a measure of consistency over several different occasions. The outcome should be the same after a week later, otherwise (say the interview)is not reliable.

Reliability also concerns whether 2 interviewers produce the same outcome.

Accessing reliability

Internal reliability - Split half method: A single group of participants all take a test once. Their answers to the test are divided in half. This is done by, for example, comparing all answers to odd number answers with all answers to even number answers.

The individuals scores can be compared by calculating a correlation coefficient.

External reliability - Test re-test method: A group of participants are given a test or questionnaire or interview once and then again sometime later. The answers can be compared and should be the same. If the tests produce scores, these can be compared by calculating a correlation coefficient.

Improving reliability - Possible to improve internal reliability by removing those items which are most inconsistent. The only way to do this is by trial and error - remove one test item and see if the split-half correlation coefficient improves.If it does, then the removed item should be permanently left out.

!