A new method for assessing interexaminer agreement when multiple ratings are made on a single subject: Applications to the assessment of neuropsychiatric symptomatology

Psychiatry Research

Volumn 72, Issue 1, 1997, Pages 51-63

  Cicchetti, Domenic V ^a,c   Showalter, Donald ^b   Rosenheck, Robert ^b  

^a YALE UNIVERSITY SCHOOL OF MEDICINE   (United States)

^b Northeast Program Evaluation Center   (United States)

^c NONE   (United States)

Author keywords

Clinical rating scales; Psychometrics; Statistics

Indexed keywords

ARTICLE; BIOSTATISTICS; COMPUTER PROGRAM; MEASUREMENT; MODEL; PRIORITY JOURNAL; PSYCHOMETRY; RATING SCALE; RESEARCH; SYMPTOMATOLOGY; TECHNIQUE;

DELIRIUM, DEMENTIA, AMNESTIC, COGNITIVE DISORDERS; DIAGNOSIS, COMPUTER-ASSISTED; HUMANS; NEUROPSYCHOLOGICAL TESTS; OBSERVER VARIATION; PSYCHOMETRICS; REPRODUCIBILITY OF RESULTS; SOFTWARE;

EID: 0030800504     PISSN: 01651781     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0165-1781(97)00095-4     Document Type: Article

References (21)

1. Bell, M., Milstein, R., Beam-Goulet, J., Lysaker, P., Cicchetti, D.V., 1992. The positive and negative syndrome scale and the brief psychiatric rating scale: reliability, comparability and predictive validity. Journal of Nervous and Mental Disease 1180, 723-728.
2. Cicchetti, D.V., 1997. A computer program for assessing interexaminer agreement when multiple ratings are made on a single subject. Psychiatry Research 72, 65-68.
3. Cicchetti, D.V., 1976. Assessing inter-rater reliability for rating scales: resolving some basic issues. British Journal of Psychiatry 129, 452-456.
4. Cicchetti, D.V., Showalter, D., McCarthy, P., 1990. A computer program for calculating subject-by-subject kappa or weighted kappa coefficients. Educational and Psychological Measurement 50, 153-158.
5. Cicchetti, D.V., Sparrow, S.S., 1981. Developing criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior. American Journal of Mental Deficiency 86, 127-137.
6. Cicchetti, D.V., Volkmar, F., Klin, A., Showalter, D., 1995. Diagnosing autism using ICD-10 criteria: a comparison of neural networks and standard multivariate procedures. Child Neuropsychology 1, 26-37.
7. Cicchetti, D.V., Volkmar, F., Sparrow, S.S., Cohen, D., Fermanian, J., Rourke, B.P., 1992. Assessing the reliability of clinical scales when the data have both nominal and ordinal features: proposed guidelines for neuropsychological assessments. Journal of Clinical and Experimental Neuropsychology 14, 673-686.
8. Donner, A., Eliasziw, M., 1987. Sample size requirements for reliability studies. Statistics in Medicine 6, 441-448.
9. Fleiss, J.L., 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin 72, 323-327.
10. Fleiss, J.L., 1981. Statistical Methods for Rates and Proportions, 2nd ed. Wiley, New York.
11. Fleiss, J.L., Nee, J.C.M., Landis, J.R., 1979. The large sample variance of kappa in case of different sets of raters. Psychological Bulletin 86, 974-977.
12. Hays, W.L., 1981. Statistics, 2nd ed. Wiley, New York.
13. Heinrichs, D.W., Hanlon, E.T., Carpenter, W.T., Jr., 1984. The quality of life scale: an instrument for rating the schizophrenic deficit syndrome. Schizophrenia Bulletin 10, 388-398.
14. Kay, S.R., Opler, L.A., Lindenmayer, J.P., 1987. Reliability and validity of the positive and negative syndrome scale for schizophrenics. Psychiatry Research 23, 99-110.
15. Lindström, E., Wieselgren, I.-M., von Knorring, L., 1994. Interrater reliability of the structured clinical interview for the positive and negative syndrome scale for schizophrenia. Acta Psychiatrica Scandinavica 89, 192-195.
16. Overall, J.E., Klett, C.J., 1972. Applied Multivariate Analysis. McGraw-Hill, New York.
17. Shrout, P.E., Fleiss, J.L., 1979. Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin 86, 420-428.
18. Shrout, P.E., Spitzer, R.L., Fleiss, J.L., 1987. Quantification of agreement in psychiatric diagnosis revisited. Archives of General Psychiatry 44, 172-177.
19. Stevens, S.S., 1946. On the theory of scales of measurement. Science 103, 677-680.
20. Stevens, S.S., 1951. Mathematics, measurement and psychophysics. In: Stevens, S.S. (Ed.), Handbook of Experimental Psychology. Wiley, New York, pp. 1-49.
21. Stevens, S.S., 1968. Measurement, statistics and the schemapiric view. Science 161, 849-856.