Measurement reliability and agreement in psychiatry

Statistical Methods in Medical Research

Volumn 7, Issue 3, 1998, Pages 301-317

  Shrout, Patrick E ^a  

^a NEW YORK UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

HUMAN; MEDICAL RESEARCH; MENTAL DISEASE; PSYCHIATRY; PSYCHOMETRY; RELIABILITY; REVIEW; STATISTICAL ANALYSIS;

HUMANS; MENTAL DISORDERS; MODELS, STATISTICAL; PSYCHIATRIC STATUS RATING SCALES; PSYCHOMETRICS; REPRODUCIBILITY OF RESULTS;

EID: 0031716722     PISSN: 09622802     EISSN: None     Source Type: Journal    
DOI: 10.1191/096228098672090967     Document Type: Review

References (77)

1. Bartko JJ, Carpenter WT Jr. On the methods and theory of reliability. Journal of Nervous and Mental Disease 1976; 163: 307-17.
2. Carey G, Gottesman II. Reliability and validity in binary ratings: areas of common misunderstanding in diagnosis and symptom ratings. Archives of General Psychiatry 1978; 35: 1454-59.
3. Grove WM, Andreason NC, McDonald-Scott P, Keller MB, Shapiro R. Reliability studies of psychiatric diagnosis: theory and practice. Archives of General Psychiatry 1981; 38: 408-13.
4. Maxwell AE. Coefficients of agreement between observers and their interpretation. British Journal of Psychiatry 1977; 130: 79-83.
5. Schmidt FL, Hunter JE. Interrater reliability coefficients cannot be computed when only one stimulus is rated. Journal of Applied Psychology 1989; 74: 368-70.
6. Shrout PE, Spitzer RL, Fleiss JL. Quantification of agreement in psychiatric diagnosis revisited. Archives of General Psychiatiy 1987; 44: 172-77.
7. Spitzer RL. Psychiatric diagnosis: are clinicians still necessary? Comprehensive Psychiatry 1983; 24: 399-411.
8. Cochran WG. Errors in measurement in statistics. Technometrics 1968; 10: 637-66.
9. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 1960; 20: 37-46.
10. Cronbach LJ, Gleser GC, Nanda H, Rajaratnam N. The dependability of behavioral measurements: theory of generalizability for scores and profiles. New York: John Wiley, 1972.
11. Fleiss JL. Estimating the reliability of interview data. Psychometrika 1970; 35: 143-62.
12. Kraemer HC. Ramifications of a population model for kappa as a coefficient of reliability. Psychometrika 1979; 44: 461-72.
13. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159-74.
14. Lord FM, Novick MR. Statistical theories of mental test scores . Reading, MA: Addison-Wesley, 1968.
15. Dunn G. Design and analysis of reliability studies. New York: Oxford University Press, 1989.
16. Kraemer HC. Measurement of reliability for categorical data in medical research. Statistical Methods in Medical Research 1992; 1: 183-99.
17. American Psychiatric Association. Diagnostic and statistical manual of mental disorders, 3rd edition. Washington, DC: American Psychiatric Association, 1980.
18. American Psychiatric Association. Diagnostic • and statistical mental of mental disorders, 4th edition. Washington, DC: American Psychiatric Association, 1994.
19. World Health Organization. Draft international classification of diseases and related health problems, 10th edition. Geneva: World Health Association, 1992.
20. Snedecor GW, Cochran WG. Statistical methods, 6th edition. Ames, IA: Iowa State University Press, 1967.
21. Fuller WA. Measurement error models. New York: John Wiley, 1987.
22. Barlow W. Measurement of interrater agreement with adjustment for covariates. Biometrics 1996; 52: 695-702.
23. Bartko JJ. The intraclass correlation as a measure of reliability. Psychological Reports 1966; 1966: 19.
24. McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychological Methods 1996; 1: 30-46.
25. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin 1979; 86: 420-28.
26. Kraemer HC, Bloch DA. Kappa coefficients in epidemiology: an appraisal of a reappraisal. Journal of Clinical Epidemiology 1988; 41: 959-68.
27. Spearman C. Correlation calculated from faulty data. British Journal of Psychology 1910; 3: 271-95.
28. Brown W. Some experimental results in the correlation of mental abilities. British Journal of Psychology 1910; 3: 296-322.
29. Donner A, Eliasziw M. Statistical implications of the choice between a dichotomous or continuous trait in studies of interobserver agreement. Biometrics 1994; 50: 550-55.
30. Bland JM, Altman DG. A note on the use of the intraclass correlation coefficient in the evaluation of agreement between two methods of measurement. Computations in Biology and Medicine 1990; 20: 337-40.
31. Guggenmoos-Holzmann I. How reliable are chance-corrected measures of agreement? Statistics in Medicine 1993; 12: 2191-205.
32. Spitznagel EL, Heizer JE. A proposed solution to the base rate problem in the kappa statistic. Archives of General Psychiatry 1985; 42: 725-28.
33. Lindell MK, Brandt CJ. Measuring interrater agreement for ratings of a single target. Applied Psychological Measurement 1997; 21: 271-78.
34. James LR, Demaree RG, Wolfe G. Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology 1984; 69: 85-98.
35. wg: An assessment of within-group rater agreement. Journal of Applied Psychology 1993; 78: 306-309.
36. Kozlowski SWJ, Hattrup K. A disagreement about within-group agreement: disentangling issues of consistency versus consensus. Journal of Applied Psychology 1992; 77: 161-67.
37. Mellenbergh GJ. Measurement precision in test score and item response models. Psychological Methods 1996; 1: 293-99.
38. Aickin M, Ritenbaugh C. Analysis of multivariate reliability structures and the induced bias in linear model estimation. Statistics in Medicine 1996; 15: 1647-61.
39. Langenbucher J, Labouvie E, Morgenstern J. Measuring diagnostic agreement. Journal of Consulting and Clinical Psychology 1996; 64: 1285-89.
40. Bartko JJ. Measures of agreement: a single procedure. Statistics in Medicine 1994; 13: 737-45.
41. Nickerson CAE. A note on 'a concordance correlation coefficient to evaluate reproducibility'. Biometrics 1997; 53: 1503-507.
42. Blackman NJ-M, Koval JJ. Estimating rater agreement in 2 × 2 tables: Correction for chance and intraclass correlation. Applied Psychological Measurement 1993; 17: 211-23.
43. Cantor AB. Sample-size calculations for Cohen's kappa. Psychological Methods 1996; 1: 150-55.
44. Fleiss JL, Cohen J, Everitt BS. Large sample standard errors of kappa and weighted kappa. Psychological Bulletin 1969; 72: 323-27.
45. Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for reliability studies. Statistics in Medicine 1998; 17: 101-10.
46. Fisher RA. Statistical methods for research workers, 13th edition. Riverside, NJ: Hafner, 1958.
47. Donner A. Sample size requirements for the comparison of two or more coefficients of inter-observer agreement. Statistics in Medicine 1998; 17: 1157-68.
48. Donner A, Eliasziw M, Klar N. Testing the homogeneity of kappa statistics. Biometrics 1996; 52: 176-83.
49. Cohen J. The cost of dichotomization. Applied Psychological Measurement 1983; 7: 249-53.
50. Fleiss JL. Statistical methods for rates and proportions, 2nd edition. New York: John Wiley, 1981.
51. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika 1951; 16: 297-334.
52. Miller MB. Coefficient alpha: a basic introduction from the perspectives of classical test theory and structural equation modeling. Structural Equation Modeling 1995; 2: 258-73.
53. O'Grady KE, Medoff DR. Rater reliability: a maximum likelihood confirmatory factor-analytic approach. Multivariate Behavioral Research 1991; 26: 363-87.
54. Raykov T. Estimating composite reliability for congeneric measures. Applied Psychological Measurement 1997; 21: 173-84.
55. Raykov T. Scale reliability, Cronbach's coefficient alpha, and violations of essential tau-equivalence with fixed congeneric components. Multivariate Behavioral Research 1997; 32: 329-53.
56. Li H, Rosenthal R, Rubin DB. Reliability of measurement in psychology: from Spearman-Brown to maximal reliability. Psychological Methods 1996; 1: 98-107.
57. Fagot RF. Reliability of ratings for multiple judges: intraclass correlation and metric scales. Applied Psychological Measurement 1991; 15: 1-11.
58. Fagot RF. An ordinal coefficient of relational agreement for multiple judges. Psychometrika 1994; 59: 241-51.
59. Konishi S, Khatri CG, Rao CR. Inferences on multivariate measures of interclass and intraclass correlations in familial data. Journal of the Royal Statistical Society 1991; 53: 649-59.
60. Mak TK. Analyzing intraclass correlation for dichotomous variables. Journal of the Royal Statistical Society 1988; 37: 344-52.
61. Bodian CA. Intraclass correlation for two-by-two tables under three sampling designs. Biometrics 1994; 50: 183-93.
62. Fleiss JL, Shrout PE. Approximate interval estimation for a certain intraclass correlation coefficient. Psychometrika 1978; 43: 259-62.
63. Barchard K, Hakstian AR. The robustness of confidence intervals for coefficient alpha under violation of the assumption of essential parallelism. Multivariate Behavioral Research 1997; 32: 169-91.
64. McClelland G, Nickerson C. E-mail on ICC(2,1) confidence interval, 1997.
65. Lyles RH, Chambless LE. Effects of model misspecification in the estimation of variance components and intraclass correlation for paired data. Statistics in Medicine 1995; 14: 1693-706.
66. Hale CA, Fleiss JL. Interval estimation under two study designs for kappa with binary classifications. Biometrics 1993; 49: 523-34.
67. Cornfield J. A statistical problem arising from retrospective studies. In: Jeyman J ed. Proceedings of the third Berkeley symposium on mathematical statistics and probability. Berkeley, CA: University of California Press, 1956.
68. Flack VF. Confidence intervals for the interrater agreement measure kappa. Communications in Statistics - Theory and Methods 1987; 16: 953-68.
69. Donner A, Eliasziw M. A goodness-of-fit approach to inference procedures for the kappa statistic: confidence interval construction, significance-testing and sample size estimation. Statistics in Medicine 1992; 11: 1511-19.
70. Bloch DA, Kraemer HC. 2×2 Kappa coefficients: measures of agreement or association. Biometrics 1988; 45: 269-87.
71. McGraw KO, Wong SP. Correction. Psychological Methods 1996; 1: 390.
72. Donner A, Eliasziw M. A hierarchical approach to inferences concerning interbserver agreement for multinomial data. Statistics in Medicine 1997; 16: 1097-106.
73. Commenges D, Jacqmin H. The intraclass correlation coefficient: distribution free definition and test. Biometrics 1994; 50: 517-26.
74. Shavelson RJ, Webb NM. Generalizability theory: a primer. Newbury Park, CA: Sage, 1991.
75. Graham P. Modeling covariate effects in observer agreement studies: the case of nominal scale agreement. Statistics in Medicine 1995; 14: 299-310.
76. Tanner MA, Young MA. Modelling agreement among raters. Journal of the American Statistical Association 1985; 80:175-80.
77. Fleiss JL, Shrout PE. Reliability considerations in planning diagnostic validity studies. New York: Guilford Press, 1989: 279-29.