Descriptive Methods, Reliability & Validity

Descriptive Methods, Reliability & Validity

1. Construct

   • An attribute not directly observable

2. Measurement

   • Measurement instrument
     • Tool that quantifies/reflects a construct otherwise unobservable. E.g. Test
     • A good measurement instrument has..
       • High reliability
         • Gives same outcome on different occasions
           • Buttt that ain't possible 'cuz of measurement errors
             • Observed Score = True Score + Error
             • Systematic error
               • Contaminates measurements in same directn/predictable pattern
                 • Can be estimated & controlled, hence must distinguish from random error
             • Random error
               • Unpredictable pattern
             • Sources of error
               • Participants
               • Chance variability (i.e. events beyond control)
               • Situation - weather, temperature
               • Measurement instrument - some inherently better than others
         • Types of reliability
           • Internal reliability/consistency
             • Extent to which a measure if consistent within itself
               • To test this, use split-half method: measures extent to which all parts of a test contribute equally to the result
                 • By comparing results of one half of the test with the other 1/2. Can be split several ways. E.g. odd/even questions
                 • Effective for large questionnaires measuring same construct; questionnaires like MMPI measuring different behaviours not suitable
           • External reliabilities
             • Extent to which a measure varies from one use to another
               • Temporal stability/test-retest reliability
                 • Instrument gives same result across time (Time 1, Time 2)
               • Inter-rater reliability
                 • Same result across User 1, User 2 etc. of instrument
       • High validity
         • An instrument that actually measures what it claims to
         • Types of validity
           • In scientific research
             • Internal validity
               • Whether effects observed in study are due to the IV - there is a CAUSAL R/S btwn IV and DV (i.e. How reasonable is the causal claim?)
               • Threats
                 • Selection
                   • Groups differ systematically before study (Time of arrival etc.)
                 • Maturation
                   • Natural developmental changes over time (longitudinal studies)
                     • E.g. Testing effect of sports program on child's height -- but height will increase over time!
                 • Regression towards the mean
                   • http://study.com/academy/lesson/regression-to-the-mean-in-psychology-definition-example-quiz.html
                   • Due to random error
                     • Extreme scores tend to regress towards the mean in the 2nd measurement
                 • Mortality
                   • Differential attrition rate (loss of personnel, usually respondents): participants abandon study
                 • History
                   • Effect of environment
                     • Testing happiness before/after earthquake or in hot/cold room
                 • Testing
                   • Effect of pretest on post-test (e.g. learning effect)
                 • Instrumentation
                   • Effect of instrument (its reliability. Say, human observers)
                 • To reduce, (and hence incr validity)
                   • Replication
                     • Greater replication of results -> greater CONFIDENCE in 'em! :D
                     • Direct
                       • Just repeat the damn study.
                     • Systematic
                       • Still measuring same variable, but perhaps..
                         • Change operationalisation (the process of defining the measurement of a construct)
                         • Add variable (which may change result!)
                         • Use different measure
                   • Statistical control
                     • Equate participants on 3rd variable so they can be compared on variables of interest??
                   • True experiment
                     • Reasonable causal inference
                       • 3 criteria
                         • 1. Temporal precedence (cause PRECEDES outcome
                         • 3. Internal validity (X is SOLE cause of Y)
                         • 2. Co-variation (X & Y co-vary)
                     • Group equivalence; no systematic diff btwn comparison groups
                       • Use random ASSIGNMENT to the grps, NOT random sampling (which refers to selecting the ppl for populatn before grouping)
                     • Manipulation
                       • Creation of at least 2 comparison grps that differ on independent variable
                     • Quasi expt: absence of manipulatn & grp eq
             • External validity
               • Extent the findings can be generalised to other times, people and settings
                 • If findings only appear under specific conditions and are irrelevant to other contexts, they LACK ext val
                 • Across time (E.g. Can a survey from 1970 be applic to current generation?)
                   • E.g. If energy level of students is determined through results generated from tests done after exams, then the students are probably tired. => CANNOT generalise!
                 • Across situations
                   • If testing phone level usage in Singapore, but only testing in the CBD => obv level is higher
                 • Across people
                   • Representative sample must come from random selection to allow each indiv to have eq chance of being picked
                     • Most famous psychology studies done on the Western popl'n :(
           • In measurement (Test validity: How much meaning can be placed in the test results?)
             • Face validity
               • At face value, does the test appear to measure what it claims to?
                 • Least sophisticated measure of validity
                   • Can only be considered robust construct if reasonable lvl of agreemt exists among raters
                     • Raters like the test-takers, those who KNOW the test & about its topics
                 • 'I have recently thought of killing myself' has obvious face validity - measures suicidal cognitions - may be useful when measuring symptoms of depression
                   • Implications of clear face validity - more vulnerable to social desirability bias
                     • Individuals may alter responses to deny/hide problems or exaggerate positive images of themselves
             • Content validity
               • Extent to which instrument measures all major aspects of construct
                 • Checks whether a specific element enhances/detracts from the research
                   • E.g. Educational test that asks students about ALL the subjects they are taught - not missing out subjects, nor adding irrelevant ones
             • Construct validity
               • Extent to which test measures up to its claims
                 • Test has construct validity if it demonstrates association btwn test SCORES and predicted THEORETICAL trait. E.g. IQ tests - where IQ must be taken to exist
                   • E.g. Doctor asking patients to measure pain on scale of 1 - 10. Low construct val if patients rated their numbness, discomfort etc. together with pain.
                     • Divergent validity
                       • Assesses if constructs that should have no r/s, in actual fact do not have r/s
                         • Will show that self esteem is unrelated to say, intelligence
                       • Instruments/scales of unrelated constructs should NOT be correlated
                     • Convergent validity
                       • Assesses if constructs supposed to be related are in fact related
                         • Will show that the 4 small circles are related to self-esteem
                       • Instruments/scales or related constructs should correlate
             • Criterion validity
               • Assesses if test reflects a certain set of abilities
                 • Predictive validity
                   • Degree to which test predicts criterion that will occur in future
                     • Basis made on new IQ test that high-scorers at age 12 will be likelier to attain uni degrees. If that prediction is true, test has high pred val.
                 • Concurrent (at the same time) validity
                   • Degree to which test correlates to concurrent/existing instruments of the SAME CONSTRUCT
             • Statistical conclusion validity
               • Are the statistical results due to the true effect or just chance?
                 • Hence, must have appropriate study design to..
                   • Detect effect
                   • Reduce measurement error
   • Types
     • Nominal
       • Categorical, discrete - number assignmt is arbitrary i.e. not indicative of the items
       • For grouping and coding only
       • Provides least amt of info
     • Ordinal
       • Discrete, rank ordering of a construct
       • Numbers are slightly meaningful; represent degree of a construct (best to worst etc.)
       • Intervals btwn numbers need not be equal
       • E.g. Ranking of gold, silver bronze
     • Interval
       • Continuous, rank ordering of a construct
       • E.g. °C scale
       • Numbers meaningfully represent degree of construct
       • No true zero (arbitrary)
       • Intervals btwn numbers are equal - allows mathematical comparison
     • Ratio
       • True zero
       • Intervals btwn numbers are equal, allows math'l comparison
       • Continuous
       • Numbers meaningfully represent degree of construct
       • E.g. Speed of 400km/h

3. Goals of scientific research

   • 1. Describe phenomenon
     • 2. Identify relations
       • 3. Infer causality
         • Which is the cause and which the effect?
       • How are X & Y related? Is it a causal r/s?

4. Research methods

   • Experimental
     • Relational: examines causal relationships; which phenomenon is a cause and which is the effect.
       • Show that X affects Y, not the other way around. Not achievable by non-exp methods.
   • Non-experimental
     • Descriptive: depicts properties of phenomena
       • Relational: identifies r/s btwn phenomena

Images and files in the map