Interrater agreement and interrater reliability: Key concepts, approaches, and applications

Research in Social and Administrative Pharmacy

Volumn 9, Issue 3, 2013, Pages 330-338

  Gisev, Natasa ^a   Bell, J Simon ^a,b,c   Chen, Timothy F ^a  

^a UNIVERSITY OF SYDNEY   (Australia)

^b UNIVERSITY OF SOUTH AUSTRALIA   (Australia)

^c UNIVERSITY OF EASTERN FINLAND   (Finland)

Author keywords

Health services research; Observer variation; Reproducibility of results; Research design

Indexed keywords

HUMAN; MANAGEMENT; METHODOLOGY; NOTE; OBSERVER VARIATION; REPRODUCIBILITY; STATISTICAL ANALYSIS; STATISTICS;

DATA INTERPRETATION, STATISTICAL; HUMANS; OBSERVER VARIATION; PHARMACY ADMINISTRATION; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN;

EID: 84877819479     PISSN: 15517411     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.sapharm.2012.04.004     Document Type: Note

References (40)

1. Kottner J., Audigé L., Brorson S., et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. J Clin Epidemiol 2011, 64:96-106.
2. de Vet H.C.W., Terwee C.B., Knol D.L., Bouter L.M. When to use agreement versus reliability measures. J Clin Epidemiol 2006, 59:1033-1039.
3. LeBreton J.M., Senter J.L. Answers to 20 questions about interrater reliability and interrater agreement. Organ Res Methods 2008, 11:815-852.
4. Tinsley H.E.A., Weiss D.J. Interrater reliability and agreement. Handbook of Applied Multivariate Statistics and Mathematical Modeling 2000, 95-124. Academic Press, San Diego, CA. H.E.A. Tinsley, S.D. Brown (Eds.).
5. Streiner D.L., Norman G.R. Health Measurement Scales: A Practical Guide to Their Development and Use 2008, Oxford University Press, New York, NY. 4th ed.
6. Stevens S.S. On the theory of scales of measurement. Science 1946, 103:677-680.
7. Holdford D. Content analysis methods for conducting research in social and administrative pharmacy. Res Social Adm Pharm 2008, 4:173-181.
8. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960, 20:37-46.
9. Siegel S., Castellen N.J. Nonparametic Statistics for the Behavioral Sciences 1988, McGraw-Hill Book Co, New York, NY. 2nd ed.
10. Landis J.R., Koch G.G. The measurement of observer agreement for categorical data. Biometrics 1977, 33:159-174.
11. Fleiss J.L., Levin B., Paik M.C. Statistical Methods for Rates and Proportions 2003, John Wiley & Son, Inc, New York, NY. 3rd ed.
12. Byrt T., Bishop J., Carlin J.B. Bias, prevalence and kappa. J Clin Epidemiol 1993, 46:423-429.
13. Feinstein A.R., Cicchetti D.V. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol 1990, 43:543-549.
14. Cicchetti D.V., Feinstein A.R. High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol 1990, 43:551-558.
15. Gallagher P., Baeyens J.-P., Topinkova E., et al. Inter-rater reliability of STOPP (Screening Tool of Older Persons' Prescriptions) and START (Screening Tool to Alert doctors to Right Treatment) criteria amongst physicians in six European countries. Age Ageing 2009, 38:603-606.
16. Guénette L., Moisan J., Préville M., Boyer R. Measures of adherence based on self-report exhibited poor agreement with those based on pharmacy records. J Clin Epidemiol 2005, 58:924-933.
17. LaFleur J., Larson B.S., Gunning K.M., et al. Agreement between pharmacists for problem identification: an initial quality measurement of cognitive services. Ann Pharmacother 2009, 43:1173-1180.
18. Kraemer H.C., Periyakoil V.S., Noda A. Kappa coefficients in medical research. Stat Med 2002, 21:2109-2129.
19. Sheskin D.J. Handbook of Parametric and Nonparametric Statistical Procedures 2007, Chapman & Hall/CRC, Boca Raton, FL. 4th ed.
20. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull 1968, 70:213-220.
21. Cicchetti D.V. Testing the normal approximation and minimal sample size requirements of weighted kappa when the number of categories is large. Appl Psychol Meas 1981, 5:101-104.
22. Streiner D.L. Learning how to differ: agreement and reliability statistics in psychiatry. Can J Psychiatry 1995, 40:60-66.
23. Soeken K.L., Prescott P.A. Issues in the use of kappa to estimate reliability. Med Care 1986, 24:733-741.
24. Fleiss J.L. Measuring nominal scale agreement among many raters. Psychol Bull 1971, 76:378-382.
25. Fleiss J.L., Cuzick J. The reliability of dichotomous judgments: unequal numbers of judges per subject. Appl Psychol Meas 1979, 3:537-542.
26. Arimone Y., Miremont-Salamé G., Haramburu F., et al. Inter-expert agreement of seven criteria in causality assessment of adverse drug reactions. Br J Clin Pharmacol 2007, 64:482-488.
27. Schmidt R.C. Managing delphi surveys using nonparametric statistical techniques. Decis Sci J 1997, 28:763-774.
28. Levitan B., Yee C.L., Russo L., Bayney R., Thomas A.P., Klincewicz S.L. A model for decision support in signal triage. Drug Saf 2008, 31:727-735.
29. Gisev N., Bell J.S., O'Reilly C.L., Rosen A., Chen T.F. An expert panel assessment of comprehensive medication reviews for clients of community mental health teams. Soc Psychiatry Psychiatr Epidemiol 2010, 45:1071-1079.
30. George J., Phun Y.-T., Bailey M.J., Kong D.C.M., Stewart K. Development and validation of the Medication Regimen Complexity Index. Ann Pharmacother 2004, 38:1369-1376.
31. Bland J.M., Altman D.G. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986, 1:307-310.
32. Bland J.M., Altman D.G. Measuring agreement in method comparison studies. Stat Methods Med Res 1999, 8:135-160.
33. Sabater-Hernández D., De La Sierra A., Sánchez-Villegas P., et al. Agreement between community pharmacy and ambulatory and home blood pressure measurement methods to assess the effectiveness of antihypertensive treatment: the MEPAFAR study. J Clin Hypertens (Greenwich) 2012, 14:236-244.
34. Haug J.B., Myhr R., Reikvam å. Pharmacy sales data versus ward stock accounting for the surveillance of broad-spectrum antibiotic use in hospitals. BMC Med Res Methodol 2011, 11:166.
35. McGraw K.O., Wong S.P. Forming inferences about some intraclass correlation coefficients. Psychol Methods 1996, 1:30-46.
36. Shrout P.E., Fleiss J.L. Intraclass correlations: uses in assessing rater reliability. Psychol Bull 1979, 86:420-428.
37. Fleiss J.L., Cohen J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ Psychol Meas 1973, 33:613-619.
38. George J., Vuong T., Bailey M.J., Kong D.C., Marriott J.L., Stewart K. Development and validation of the Medication-Based Disease Burden Index. Ann Pharmacother 2006, 40:645-650.
39. Taddio A., O'Brien L., Ipp M., Stephens D., Goldbach M., Koren G. Reliability and validity of observer ratings of pain using the visual analog scale (VAS) in infants undergoing immunization injections. Pain 2009, 147:141-146.
40. Tinsley H.E.A., Weiss D.J. Interrater reliability and agreement of subjective judgments. J Couns Psychol 1975, 22:358-376.