Karteikarten
research chapter 4
Karten in diesem Stapel (30)
- 1validityA study is said to have validity if it measures what it claims to measure
- 2ReliabilityReliability of a study means that the measures used yield consistency in results.
- 3ErrorError denotes the fluctuations that occur in measurements, which can have an effect on how reliable and valid they are. This holds even in the slightest difference in measurement of e.g. 0.00001 g of the standard supermarket scales. This difference is due to chance error, or also called noise
- 4The systematic errordiffers from the random error in a way that measurements deviate from the exact value in one direction. This results in mean values that are not similar to the actual value. This problem can be solved by subtracting the systematic error from the mean of the observations. The random error in contrast consists of measurements above and under an exact value, making the average and actual value similar.
- 5Test-retest reliabilityIf a group is measured at a point in time for a certain characteristic and then at a later point in time again, the measurements used are of high test-retest reliability if they yield similar results. This is often applied for the measurement of trait characteristics, which are said not to change much over time and therefore are expected to have high retest reliability.
- 6Alternate-form reliabilityThis type of reliability becomes important when the retest reliability is subject to inflation because a participant is already acquainted with the material being tested. A new version of this instrument needs to be created. Alternate-form reliability describes if these different versions are equal in the measurement of interest.
- 7Equivalent-forms reliabilityrefers to the measurement content of one version being compared to the content of another version in the sense of equivalence. This is being done in the same session, so the same point in time and a correlation is being calculated between them. The variance also needs to be similar of the different instruments to qualify as statistically equivalent.
- 8Stability coefficientsrefer to correlations obtained from equivalent versions with the same research subjects but at different points in time. The coefficients of equivalence in contrast represents this correlation but at roughly the same point in time. Cross-lagged correlation implies time-lagged treatment of one variable in relation to the other (A or B).
- 9internal-consistencyIf the same feature is measured by items of the instrument, then it is said to have internal-consistency, which is also often termed the reliability of components. This is denoted as R, and following approaches of internal-consistency are discussed: 1.Spearman-Brown equation 2. Kuder and Richardson’s Formula 20 3. Cronbach’s alpha coefficient. Homogenous items can be added to increase R. The WAIS holds an average internal-consistency reliability of 0.87, which is said to be very satisfying.
- 10Spearman-Brown equationWith the help of the Spearman-Brown equation it is possible to see by how many items the test should be lengthened to yield a certain level of internal-consistency reliability. The formula is:
RSB= nrii / 1+ (n-1) x rij
RSB= Reliability of sum of item scores (n) (e.g. n= 4)
rij = item-to-item correlation, which is the single item reliability and calculated by taking the mean of the intercorrelations among all items (e.g. n= 0.60)
then: RSB = 4(.60) / 1+ (4-1) x .60 = 0.85
If an item added has a lower reliability than the present items, this could result in a decrease of reliability. Table 4.1 (p. 95, Research Methods) allows obtaining these results by finding the number of items (n) needed or the internal-consistency reliability (RSB) in the chart. - 11The KR20 and Cronbach’s AlphaIn the Split-half method a test is split in half and the halves are correlated. This gives the split-half reliability. When the test has a dichotomous format, the KR20 (RKR20) can be applied. It takes all possible divisions in halves into account. The formula is
RKR20 = (n/n-1) x (σ2t – Σ PQ / σ2t )
n= number of test items
σ2t = variance of total test score with that being: σ2t = Σ (tp –t(bar)p)2 / p - 1
P= proportion of responses labeled 1
Q= proportion of responses labeled with 0
The Kuder-Richardson equation is an undercase of Cronbach’s alpha or the alpha coefficient (RCronbach). Cronbach’s alpha can be estimated by:
RCronbach = (n/n-1) x (S2t - Σ (S2t) / S2t)
S2t= the sum of the variances of the individual items.
The Spearman-Brown formula and the Cronbach’s alpha yield approximately the same answer when the variances of all items are equal. - 12Effective Reliability of JudgesIf we look at a group of judges and we want to make an estimation of reliability of their ratings, we are dealing with effective reliability. By using the formula of the Spearman-Brown procedure: RSB = n x rjj / 1 + (n-1)x rij
RSB= effective reliability
rjj= mean correlation among all judges; mean reliability
We can find out how many judges are needed (n) when a specific mean reliability (rjj) is given and when we know what our effective reliability (RSB ) should be at least. Cronbach’s alpha coefficient can also be calculated with this information given. With ANOVA the effective reliability can also be estimated. Named after C. Hoyt (1941), the formula is:
RHoyt = MSpersons – Msresidual / MSpersons
In addition The RHoyt equation and the equation to obtain RCronbach via ANOVA are the same. Intraclass correlation gives an estimation of judge-to-judge reliability which is calculated by:
rintraclass = rij = MSpersons – Msresidual / MSpersons + (n-1) x MSresidual
The RCronbach , RHoyt, and RSB for estimates of effective reliability all give very similar results, with only a minimal diff - 13Effective Cost of JudgesEffective cost helps to select the right type of judge for a certain budget which will ensure maximum effective reliability:
ECj= Cj (1-rij / rij)
Cj= cost per judges (Type j)
rij= average intercorrelation of judges (Type j) with one another.
For each judge type an effective reliability cost computation is done. Then a ranking of these effective reliability costs will be made. The selection will then start from best to worse judge, and will end once the planned in money was all spent or if the maximum number of judges have been reached. The product of effective cost and number of judges equals the total cost. When there is more than one type of judge the overall effective reliability of each type can be calculated so that they can be compared. - 14Effective Cost of ItemsAs discussed above, when we have several types of judges or just one, we can estimate the effective reliability. This idea can be transferred to choosing the best items for a test to yield the maximum internal-consistency reliability. Scoring a specific test item costs a certain amount of money with which we can calculate its effective cost. That way we can for example compare the effective cost for an essay item and for a multiple choice item, using the same formula of effective cost as for the case with judges.
- 15Interrater Agreement and ReliabilityIt is necessary to make a distinction between interrater agreement and interrater reliability. Percentage agreement is computed with:
Percentage agreement = (A / A + D) x 100
A= number of agreements among judges
D= number of disagreements among judges
Similarly, the net agreement would be calculated with:
Net agreement = (A – D / A + D) x 100
A differentiation between accuracy and variability cannot be made with these calculations. That way, the outcome of these calculations might not be in agreement with the original data, which makes it not the best procedure to use.Instead we can work with the product-moment correlation (r). For the upper example, we would use a 2 x 2 table of counts from which we would obtain the product-moment r, which is the phi (φ) coefficient. For the computation use:
φ = ( X2(1) / N )
Using this equation, the differences in results of Table 4.7 are being accurately reflected, unlike the percentage agreement which was quite insensitive to the actual differences in results. - 16Cohen’s KappaKappa (ĸ) deals with the problem of lack of variability in the percentage of agreement and makes adjustments according to that. With kappa you can compute the interrater reliability between judges. Regarding table 4.8 which illustrates the use of kappa, we have O, E and N with which we can compute it. O represents the observed number of agreements between the two judges. E denotes the expected number of agreements, taking chance agreement for the cells along the diagonal of the table into account. N stands for the number of cases. Multiplying the row total with the column total and all divided by number of cases gives the expected number for each individual cell. Cohen’s kappa is computed by:
ĸ = O – E / N – E - 17omnibus kappaOmnibus kappas are kappas that have a statistic of df > 1, omnibus statistical procedures being the general term for this. They are the opposite of focused statistical procedures that have 1-df. The omnibus kappa is faced with the problem of not being able to tell how reliable specific judgments are being made. The 1-df kappas and omnibus kappas share an almost nonexistent or little relation. Unity or approaching unity of kappa means reliability of all judgments.
- 18kappa with df = 1 of a 2 x 2Even a kappa with df = 1 of a 2 x 2 can be problematic. This is of main concern to kappa which, due to its inequality to the product-moment correlation, cannot receive the same variety of interpretive procedures as those available to the product-moment correlations.
- 19replication researchThe goal of replication in research is to be able to generalize results. This goes from generalization of time, measurements, to observers and manipulations. An exact replication of previous results will never be possible. Looking at it critically, only the same experimenter can work on the same experiment, and even that experimenter changes as he becomes older with time, just like the participants. This is why we use the term relative replications. To indicate reliability of how useful a replication is depends partly on three factors when conducting the replication: When, how and by whom.
- 20when in replication researchWhen: The contribution replications make long after the research question has been around become less, as the first replications add the most information.
- 21how in replication researchHow: The way in which a replication is done is crucial, as a precise replication is very much like the original design, whereas a varied replication deliberately has adjusted some aspects. A plus of a successful varied replication is that it makes for an extension of the generalizability. An unsuccessful precise replication hurts the theory.
- 22by whom in replication researchBy whom: This point focuses on correlated replicators, describing how the independence of replicators is limited. A researcher with special interest for that field of study will work differently than a researcher not necessarily passionate about this field. Sets of replications might not be as independent for the former as for the latter. The talk is of “pre-correlation” of researchers of the same field of interest.
- 23Content validityWe have content validity if items test what they are supposed to test. Often judges are used for this assessment, but the evaluation is usually more global and not quantitative, which means that subjective judgments are yielded with this assessment. If an exam in university for example has content validity, the test reflected the material to be studied very well.
- 24Criterion ValidityThis type of validity refers to the relationship between a test and a certain criterion, with a strong correlation meaning that criterion validity is present. It is hence possible to make predictions with a test about a certain criterion, such as success in a future job.
- 25Concurrent validitymeans that the criterion which we measure with a test is not occurring in the future, but is already present now. This form of validity illustrates the thin line sometimes present between the practical assessment of validity versus reliability, as it is debated that with concurrent validity you actually determine the reliability and not validity.
- 26Predictive validitymeans that the test has predictive qualities for a future event or behavior taking place, such as for example the SAT test which measures how students would perform in college. In the clinical setting, judges who assess some behavior in a pooled manner will form the criterion. This type of criterion might not be as reliable, but adding more judges, will hence produce an increase of the reliability of this pooled judgment.
- 27Convergent validitywhen we have a test which measures certain things as several other tests which are said to measure the same thing. In addition, a small correlation is wished for between the test and other tests that are said to not measure the same thing.
- 28multitrait-multimethod matrixWith the multitrait-multimethod matrix of intercorrelations a better overview of the convergent and discriminant validities of a construct can be made. With this method it is possible to see whether it is the same methods of measurements of the correlations that are responsible for high or low correlations instead of the convergent or discriminant validity. With the help of contrasts, combinations of convergent and discriminant can be made out.
- 29“selection” ratioNot only does the validity tell us that a test is useful, but in regard to personnel selection another factor comes into play. This is the “selection” ratio, which determines how many of the total number of applicants will be chosen by the test, giving a proportion. If the selection ratio is very high, a high validity does not contribute much to the personnel selection, whereas for a low ratio, not the highest validity would still help a lot in the selection process. The best accuracy in selection is yielded when we have a high validity coefficient and a small selection ratio. When the distance between these two influences grows even further this causes even better selection accuracy.
- 30—
Informationen
- Beschreibung
- Psychologie Karteikarten am research chapter 4, erstellt von Adrian Mayer am 30/04/2018.
- Erstellt von
- Adrian Mayer
- Aufrufe
- 0
- Bewertung