Scale construction 2

Scale construction 2

1. Identify and eliminate bad cases

   • plot a histogram to get frequency of responses create new varibale unstead of deleteing a data point, tells you whether you wanto to keep or delete for analysis
   • Duplicate cases: accidently enter data twice, happen in online cases
   • Missing data: miss or not answer. Can code it as missing in SPSS, choose no. that doesn't represent score
     • Can delete subject but not thought to be good can miss out on info or decrease sample, but if not answered 80% you may have to delete
   • Possiblies with missing data
     • Substitute mid-scale value as a replacement for missing dta (average so to speak)  + not hugely biasing score by doing this, keeps subject-may noy reflect reality-doesnt work for yes/no responses
     • Substitute sample mean score on that item  If not missing other data can substiture particpants men or meidan of similar scores items (better estimate from persons other reposnes fro msame construct)
     • Impute the value: estimating what that value is likely to be, use other item as a proxy for missing item using info tailored to that individual to get most accurate replacement value use coefficent in regression to plug in data and estimate value for mssing data (some prog do this auto)
   • Other kinds of bad data
     • Wrong key presses 22 instead or two. impossible values SPSS allows you to fix these or with frequency function or recode to change these numbers into the missing code

2. Attrition: when people drop out of studies

   • Need to check those who are attriting are somehow diff from those who complete. effect may be relevant to study (shows if you just delete you can't comapre these people) split file by filter important in longitundationaldesigns retain entire samples of comaprison, distribution
   • way in which they may differ poorer mental or physical health (depression no motivation to complete) Less education (drop out) Less social responsibility (dont particularly care) Antisocial behaviour (dont conform) Less or greater intelligence (both ways, higher powered jobs less time) Gender: women particpate more often than men Less reason for interest in research

3. Item Performance on Scales

   • want range of item 'difficulties' want focus on aspect for purposes to which scale will be put (may differ for different tests and for the same test used for diff purposes) ex/ intelligence tests: to diagnose retardation, for selection of military training

4. Measuring Item Performance

   • Proportions 'correct' or endorsed in direction of scale construct should vary from almost all to almost none There should be few or no items that no one responds with the extremes of the scale

5. Classical test theory

   • Any score (x) on a test is made up of - the true (T) score -plus error (e) So X = T + e T: factors that contribute to consistency i.e. stable attributes of the individual e: factors that contribute to inconsistency i.e. of the individual, test, (pos or neg)
   • any test given to a large group will show a variability of X scores which can be expressed as a variance Because e contributes to inconsistency of X, results will not remain stable if the test is re-administered 
   • T is never known with certainty
     • Axioms (meaning true without proof) - there is a true score - e is random had has a mean of zero, is uncorrelated with T and with the e of other items

6. Sources of Measurement Error

   • Unsystematic (unpredictable and inconsistent) :  -item selection (should be low in well-designed tests) -Test administration (conditions, examinee, examiner) -Test scoring (errors, subjectiveity) Systematic: consistently measures something else

7. Reliability

   • Measurement accuracy of a test Expresses the relative influence of T and e on X (the proportion of T variance) Range = 0 to 1 Unreliability = 1 - reliability  Limited by the objectivity of the test Increases with test length!
   • Types
     • Temporal Consistency: -retest reliability -parallel forms reliability Internal consistency  -Split-half reliability Inter-rater reliability
     • Retest reliability
       • If you give the same test to the same sample twice, the correlation between the two test scores is the retest reliability (temporal stability) Limitations: -Confounded with true triat change/developemnt -Not suitable for state measures -Confounded with retest effects (test learning)
     • Parallel Forms Reliability
       • If you have two diff versions of a tests, constructed the same way, just using diff items, they should correlate -Given same set of subjects two forms of the test -Find out how much the test forms correlate
     • Retest vs Parallel
       • Parallel forms introduces item-sampling differences as an additional source of e variance While average difficulty is the same, some subjects may find one form easier/harder than the other Retest uses identical items in both administrations
     • Internal Consistency
       • Estimate reliability without needing to re-administer test  A set of items that measure the same construct should be correlated with each other
       • SPSS can compute Cronbacks alpha (reliability analysis) throw out items with low item toal correlations r <.4 If any items are negatively phrased, reverse code them first!
     • Split-Half reliability
       • Split the items into two equal halves (odd and even) Score the two 'half tests (SPSS) Get bivariate correlations between these half tests Useful where retesting is expensive or potentially misleading (e.g. practise effects)
       • Gives an estimate for the instrument half as long as the full test Shorter tests are usually less reliable  Pearson r will underestimate reliability Adjust using Spearman-Brown fromula
       • Critized for lack of precision: why rely on a single split?
         • Coefficient alpha
           • Conceptualised as the mean of all possible spli-half coefficients, corrected by spearman-brown formula Cronbachs-High when the variance among items is low relative to the full scale variance
           • Indexes the degree to which a test measures a single factor Heterogeneous tests measuring multiple traits will produce low alphas (low correlation) Homogeneous, unifactorial tests will produce high values
         • Kuder-Richardson formula 20 : for dichotomous items
     • Inter-rater reliability
       • 2 or more independent judges assess a subject When might this be critical? peer reports clinical diagnosis  post morbid comprisons
       • The extent to which ratings between judges are similar is an estimate of true score What elese might it be? Agree ment of score vs ranking Supplements other reliability estimate but DOESNT replace them
   • Considerations
     • Unstable chacteristics (eg skin response) Speed test (split half on same test not meaningful) Restriction of range Criterion-references tests: mastery rather than continuous scores ( is the same decision reached on two occasions?)

8. Validity: How well a test represents the measured construct. Limited by relaibility

   • The test reliability is the upper threshold for the test validity Validity depends on accumulation of research findings.
   • Types
     • Content Validity
       • The degree to which Q's tasks or items are representative of the universe of behavior purposely sampled. (easy to measure for well defined traits eg spelling) Face validity: does it look valid to test users, examiners (social acceptibility, not psychometrics)
     • Criterion Validity
       • Test is efffective in predicting performance of a relevant outcome -concurrent: criterion measures collected around the same time -predictive: measures collected in the future Uses a regression for prediction Correlation can serve as a validity coefficient
     • Construct Validity
       • Related to tests claiming to measure complex, multifaceted, and theory bound psychological attributes -Convergent: a scale should be correlated with measures or constructs that are theoretically related to that scale -Discriminant- a scale should not be correlated with measures that theoretically should not be related to that scale
       • Factor analysis
         • Criterion of a 'simple structure' - High factor loadings are as high as possible (convergent validity) - Low factor loadings to be as low as possible (discriminant validity) -Few cross-loadings on multiple factors (discriminant)
       • Multi-traits multi-methods analyses
       • Examples
         • studies -analysis of item homogeneity -developmental and group changes on the test -correlation and FA of tests scores in relation to toher sources of info Are results consistent with theory? A test is valid IF it serves the purpose for which it is used