Rating scales, scales of measurement, issues of reliability: Resolving some critical issues for clinicians and researchers

Journal of Nervous and Mental Disease

Volumn 194, Issue 8, 2006, Pages 557-564

  Cicchetti, Domenic ^a,b,e   Bronen, Richard ^a   Spencer, Susan ^a   Haut, Sheryl ^c   Berg, Anne ^d   Oliver, Patricia ^b   Tyrer, Peter ^b  

^a YALE UNIVERSITY SCHOOL OF MEDICINE   (United States)

^b IMPERIAL COLLEGE LONDON   (United Kingdom)

^c ALBERT EINSTEIN COLLEGE OF MEDICINE   (United States)

^d NORTHERN ILLINOIS UNIVERSITY   (United States)

^e Yale Home Office   (United States)

Author keywords

Measurement; Ratings; Reliability; Significance; Validity

Indexed keywords

CLINICAL RESEARCH; KAPPA STATISTICS; RATING SCALE; RELIABILITY; REVIEW; VALIDITY;

BEHAVIORAL RESEARCH; BIOMEDICAL RESEARCH; BIOMETRY; DEPRESSIVE DISORDER; GUIDELINES; HUMANS; OBSERVER VARIATION; PSYCHIATRIC STATUS RATING SCALES; PSYCHOMETRICS; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN;

EID: 33747329395     PISSN: 00223018     EISSN: None     Source Type: Journal    
DOI: 10.1097/01.nmd.0000230392.83607.c5     Document Type: Review

References (40)

1. Ashenfelter O, Quandt R (1999) Analyzing a wine tasting statistically. Chance. 12:16-20.
2. Baca-Garcia B, Saiz-Ruiz C, Rico J, Diaz-Sastre F, Cicchetti DV (2001) Assessment of reliability in the clinical evaluation of depressive symptoms among multiple investigators in a multi-center clinical trial. Psychiatry Res. 102:163-173.
3. Bartko JJ (1966) The intraclass correlation coefficient as a measure of reliability. Psychol Rep. 19:3-11.
4. Bartko JJ (1976) On various intraclass correlation reliability coefficients. Psychol Bull. 83:762-765.
5. Bolanowski SJ, Gesheider GA (Eds) (1991) Ratio Scaling of Psychological Magnitude: In Honor of the Memory of S. S. Stevens. Hillsdale (NJ): Lawrence Erlbaum Associates.
6. Bronen RA, Chan S, Cicchetti DV, Berg AT, Spencer SS (2004) Inter-rater agreement for MRI interpretation of epilepsy surgery patients. Paper presented at American Epilepsy Society meeting.
7. Buchler J (1940) Philosophical Writings of Pierce. New York: Dover.
8. Cicchetti DV (1976) Assessing inter-rater reliability for rating scales: Resolving some basic issues. Br J Psychiatry. 129:452-456.
9. Cicchetti DV (1988) When diagnostic agreement is high, but reliability is low: Some paradoxes occurring in joint independent neuropsychology assessments. J Clin Exp Neuropsychol. 10:605-622.
10. Cicchetti DV (2001) The precision of reliability and validity estimates re-visited: Distinguishing between clinical and statistical significance of sample size requirements. J Clin Exp Neuropsychol. 23:695-700.
11. Cicchetti DV (2004a) Who won the 1976 blind wine tasting of French Bordeaux and American Cabernets? Parametrics to the rescue. J Wine Res. 15:211-220.
12. Cicchetti DV (2004b) On designing experiments and analyzing data to assess the reliability and accuracy of blind wine tasting. J Wine Res. 15:221-226.
13. Cicchetti DV, Rourke BP (Eds) (2004) Methodological and Biostatistical Foundations of Clinical Neuropsychology and Medical and Health Disciplines (2nd ed, Vol 2). New York: Psychology Press: Taylor & Francis.
14. Cicchetti DV, Showalter D, Rosenheck (1997) A new method for assessing inter-examiner agreement when multiple ratings are made on a single subject: Applications to the assessment of neuropsychiatric symptomatology. Psychiatry Res. 72:51-63.
15. Cicchetti DV, Showalter D, Tyrer P (1985) The effect of number of rating scale categories upon levels of interrater reliability: A Monte Carlo investigation. Appl Psychol Meas. 9:31-36.
16. Cicchetti DV, Sparrow SS (1981) Developing criteria for establishing interrater reliability of specific items: Applications to assessment of adaptive behavior. Am J Ment Deficiency. 86:127-137.
17. Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas. 23:37-46.
18. Cohen J (1968) Weighted kappa: Nominal scale agreement with provisions for scaled disagreement or partial credit. Psychol Bull. 70:213-220.
19. Coombs CH (1953) Theory and methods of social measurement. In L Festinger, D Katz (Eds), Research Methods in the Behavioral Sciences (pp 471-535). Hillside (NJ): Lawrence Erlbaum Associates.
20. Embretson SE, Hershberger SL (Eds) (1999) The New Rules of Measurement: What Every Psychologist Should Know. Mahwah (NJ): Lawrence Erlbaum Associates.
21. Endicott J, Spitzer RL, Fleiss JL, Cohen J (1976) The Global Assessment Scale. Arch Gen Psychiatry. 33:766-771.
22. Fleiss JL (1975) Measuring agreement between two judges on the presence or absence of a trait. Biometrics. 31:651-659.
23. Fleiss JL (1981) Statistical Methods for Rates and Proportions (2nd ed). New York: Wiley.
24. Fleiss JL, Cohen J (1973) The equivalence of weighted kappa and the intraclass correlation coefficient as measures of agreement. Educ Psychol Meas. 33:613-619.
25. Fleiss JL, Levin B, Paik MC (2003) Statistical Methods for Rates and Proportions (3rd ed). New York: Wiley.
26. Grimm LG, Yarnold PR (1995) Reading and Understanding Multivariate Statistics. Washington DC: American Psychological Association.
27. Hamilton M (1960) A rating scale for depression. J Neurol Neurosurg Psychiatry. 23:56-62.
28. Hamilton M (1967) Development of a rating scale for primary depressive illness. Br J Soc Clin Psychol. 6:278-296.
29. Haut SR, Berg AT, Shinnar S, Cohen HW, Bazil CW, Sperling MR, Langfitt JT, Pacia SV, Walczak TS, Spencer SS (2002) Inter-rater reliability among epilepsy centers: Multicenter study of epilepsy. Epilepsia. 43:1396-1401.
30. Kraemer HC, Kazdin AE, Offord DR, Kessler RC, Jensen PS, Kupfer DJ (1997) Coming to terms with the concept of risk. Arch Gen Psychiatry. 54:337-343.
31. Landis JR, Koch GG (1977) The measure of observer agreement for categorical data. Biometrics. 3:159-174.
32. Oliver P, Cooray S, Tyrer P, Cicchetti D (2003) Use of the Global Assessment of Function scale in learning disability. Br J Psychiatry. 182(suppl 44):s32-s35.
33. Preston CC, Colman AM (2000) Optimal number of response categories in rating scales: Reliability, validity, discriminating power and respondent preferences. Acta Psychol. 104:1-15.
34. Rice LL, Wagstaff AK (1967) Client voice quality and expressive style as indexes of productive psychotherapy. J Consult Psychol. 3:557-563.
35. Shrout PE, Fleiss JL (1979) Intraclass correlations: Uses in assessing rater reliability. Psychol Bull. 86:420-428.
36. Stevens SS (1946) On the theory of scales of measurement. Science. 10:677-680.
37. Stevens SS (1951) Mathematics, measurement and psychophysics. In SS Stevens, Handbook of Experimental Psychology (pp 1-49) New York: Wiley.
38. Stevens SS (1968) Measurement, statistics and the schemapiric view. Science. 161:849-856.
39. Szalai JP (1993) The statistics of agreement on a single item or object by multiple raters. Percept Motor Skills. 7:377-378.
40. Volkmar FR, Cicchetti DV, Dykens E, Sparrow SS, Leckman JF, Cohen DJ (1988) An evaluation of the Autism Behavior Checklist. J Autism Devel Disord. 18:81-97.