Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed

International Journal of Nursing Studies

Volumn 48, Issue 6, 2011, Pages 661-671

  Kottner, Jan ^a   Audige, Laurent ^b   Brorson, Stig ^c   Donner, Allan ^d   Gajewski, Byron J ^e   Hróbjartsson, Asbjørn ^f   Roberts, Chris ^g   Shoukri, Mohamed ^h   Streiner, David L ⁱ  

^a AO Clinical Investigation and Documentation ^*  (Germany)

^b AO Clinical Investigation and Documentation   (Switzerland)

^c GENTOFTE UNIVERSITY HOSPITAL   (Denmark)

^d WESTERN UNIVERSITY   (Canada)

^e UNIVERSITY OF KANSAS SCHOOL OF MEDICINE   (United States)

^f University of Copenhagen   (Denmark)

^g UNIVERSITY OF MANCHESTER   (United Kingdom)

^h UNIVERSITY OF WESTERN ONTARIO   (Canada)

ⁱ UNIVERSITY OF TORONTO   (Canada)

Author keywords

Agreement; Guideline; Interrater; Intrarater; Reliability; Reproducibility

Indexed keywords

EID: 79957909363     PISSN: 00207489     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ijnurstu.2011.01.016     Document Type: Article

References (118)

1. Altaye M., Donner A., Eliasziw M. A general goodness-of-fit approach for inference procedures concerning the kappa statistic. Statistics in Medicine 2001, 20(16):2479-2488.
2. American Educational Research Association, American Psychological Association, National Council on Measurement in Education, 1999. Standards for Educational and Psychological Testing. American Educational Research Association, Washington.
3. Audigé L., Bhandari M., Hanson B., Kellam J. A concept for the validation of fracture classifications. Journal of Orthopaedic Trauma 2005, 19(6):404-409.
4. Audigé L., Bhandari M., Kellam J. How reliable are reliability studies of fracture classifications?. Acta Orthopaedica Scandinavica 2004, 75(2):184-194.
5. Bååth C., Hall-Lord M.L., Idvall E., Wiberg-Hedman K., Larsson B.W. Interrater reliability using Modified Norton Scale, Pressure Ulcer Card, Short Form-Mini Nutritional Assessment by registered and enrolled nurses in clinical practice. Journal of Clinical Nursing 2008, 17(5):618-626.
6. Barlow W., Lai M.Y., Azen S.P. A comparison of methods for calculating a stratified kappa. Statistics in Medicine 1991, 10(9):1465-1472.
7. Barone J.E., Madlinger R.V. Should an Allen test be performed before radial artery cannulation?. The Journal of Trauma 2006, 61(12):468-470.
8. Bates-Jensen B.M., McCreath H.E., Pongquan V., Apeles N.C.R. Subepidermal moisture differentiates erythema and stage I pressure ulcers in nursing home residents. Wound Repair and Regeneration 2008, 16(2):189-197.
9. Beckman T.J., Ghosh A.K., Cook D.A., Erwin P.J., Mandrekar J.N. How reliable are assessments of clinical teaching?. Journal of General Internal Medicine 2004, 19(9):971-977.
10. Bhat R., Rockwood K. Inter-rater reliability of delirium rating scales. Neuroepidemiology 2005, 25(1):48-52.
11. Bland J.M., Altman D.G. A note on the use of the intraclass correlation coefficient in the evaluation of agreement between two methods of measurement. Computers in Biology and Medicine 1990, 20(5):337-340.
12. Bland J.M., Altman D.G. Measuring agreement in method comparison studies. Statistical Methods in Medical Research 1999, 8(2):135-160.
13. Bonnet D.G. Sample size requirements for estimating intraclass correlations with desired precision. Statistics in Medicine 2002, 21(9):1331-1335.
14. Bossuyt P.M., Reitsma J.B., Bruns D.E., Gatsonis C.A., Glasziou P.P., Irwig L.M., et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Annals of Internal Medicine 2003, 138(1):W1-W12.
15. Bot S.D., Terwee C.B., van der Windt D.A., Bouter L.M., Dekker J., De Vet H.C. Clinimetric evaluation of shoulder disability questionnaires: a systematic review. Annals of the Rheumatic Diseases 2004, 63(4):335-341.
16. Bours G.J., Halfens R.J., Lubbers M., Haalboom J.R. The development of a National Registration Form to measure the prevalence of pressure ulcers in the Netherlands. Ostomy Wound Management 1999, 45(11):28-40.
17. Brorson S., Hróbjartsson A. Training improves agreement among doctors using the Neer system for proximal humeral fractures in a systematic review. Journal of Clinical Epidemiology 2008, 61(1):7-16.
18. Buntinx F., Beckers H., De Keyser G., Flour M., Nissen G., Raskin T., et al. Inter-observer variation in the assessment of skin ulceration. Journal of Wound Care 1996, 5(4):166-170.
19. Cantor A.B. Sample-size calculations for Cohen's kappa. Psychological Methods 1996, 1(2):150-153.
20. Charter R.A. Combining reliability coefficients: possible application to meta-analysis and reliability generalization. Psychological Reports 2003, 93:643-647.
21. Cicchetti D. Sample size requirements for increasing the precision of reliability estimates: problems and proposed solutions. Journal of Clinical and Experimental Neuropsychology 1999, 21(4):567-570.
22. Cicchetti D. The precision of reliability and validity estimates re-visited: distinguishing between clinical and statistical significance of sample size requirements. Journal of Clinical and Experimental Neuropsychology 2001, 23(5):695-700.
23. Cicchetti D., Bronen R., Spencer S., Haut S., Berg A., Oliver P., et al. Rating scales, scales of measurement, issues of reliability. The Journal of Nervous and Mental Disease 2006, 194(8):557-564.
24. Colle F., Rannon F., Revel M., Fermanian J., Poiraudeau S. Impact of quality scales on levels of evidence inferred from a systematic review of exercise therapy and low back pain. Archives of Physical Medicine and Rehabilitation 2002, 83(12):1745-1752.
25. Darroch J.N., McCloud P.I. Category distinguishability and observer agreement. Australian & New Zealand Journal of Statistics 1986, 28(3):371-388.
26. De Vet H.C.W., Terwee C.B., Knol D.L., Bouter L.M. When to use agreement versus reliability measures. Journal of Clinical Epidemiology 2006, 59(10):1033-1039.
27. De Villiers M.R., de Villiers P.J.T., Kent A.P. The Delphi technique in health sciences education research. Medical Teacher 2005, 27(7):639-643.
28. Defloor T., Schoonhoven L. Inter-rater reliability of the EPUAP pressure ulcer classification system using photographs. Journal of Clinical Nursing 2004, 13(8):952-959.
29. D'Olhaberriague L., Litvan I., Mitsias P., Mansbach H.H. A reappraisal of reliability and validity studies in stroke. Stroke 1996, 27(12):2331-2336.
30. Donner A., Eliasziw M. Sample size requirements for reliability studies. Statistics in Medicine 1987, 6(4):441-448.
31. Donner A., Eliasziw M. A goodness-of-fit approach to inference procedures for the kappa statistic: confidence interval construction, significance testing and sample size estimation. Statistics in Medicine 1992, 11(11):1511-1519.
32. Donner A., Eliasziw M. Statistical implications of the choice between a dichotomous or continuous trait in studies of interobserver agreement. Biometrics 1994, 50(2):550-555.
33. Donner A., Klar N. The statistical analysis of kappa statistics in multiple samples. Journal of Clinical Epidemiology 1996, 49(9):1053-1058.
34. Dunn G. Statistical Evaluation of Measurement Errors: Design and Analysis of Reliability Studies 2004, Arnold, London. 2nd ed.
35. Elkum N., Shoukri M.M. Signal-to-noise ratio (SNR) as a measure of reproducibility: design, estimation, and application. Health Services and Outcomes Research Methodology 2008, 8:119-133.
36. European Pressure Ulcer Advisory Panel, 2005. EPUAP Statement on Prevalence and Incidence Monitoring of Pressure Ulcer Occurrence 2005. Retrieved March 8, 2009, from http://www.epuap.org/review6_3/page5.html.
37. Feinstein A.R., Cicchetti D.V. High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology 1990, 43(6):543-549.
38. Fink A., Kosecoff J., Chassin M., Brook R.H. Consensus methods: characteristics and guidelines for use. American Journal of Public Health 1984, 74(9):979-983.
39. Fleiss J.L., Levin B., Paik M.C. Statistical Methods for Rates and Proportions 2003, Wiley, New Jersey. 3rd ed.
40. Gajewski B.J., Hart S., Bergquist S., Dunton N. Inter-rater reliability of pressure ulcer staging: probit Bayesian Hierarchical Model that allows for uncertain rater response. Statistics in Medicine 2007, 26(25):4602-4618.
41. Gardener M.J., Altman D.G. Confidence intervals rather than P values: estimation rather than hypothesis testing. British Medical Journal 1986, 292(6522):746-750.
42. Giraudeau B., Mary J.Y. Planning a reproducibility study: how many subjects and how many replicates per subject for an expected width of the 95 percent confidence interval of the intraclass correlation coefficient. Statistics in Medicine 2001, 20(21):3205-3214.
43. Gjørup T. The kappa coefficient and the prevalence of a diagnosis. Methods of Information in Medicine 1988, 27(4):184-186.
44. Glaser E.M. Using behavioral science strategies for defining the state-of-the-art. Journal of Applied Behavioral Science 1980, 16:79-92.
45. Gould D., Goldstone L., Kelly D., Gammon J. Examining the validity of pressure ulcer risk assessment scales: a replication study. International Journal of Nursing Studies 2004, 41(3):331-339.
46. Gouttebarge V., Wind H., Kuijer P.P., Frings-Dresen M.H.W. Reliability and validity of functional capacity evaluation methods: a systematic review with reference to Blankenship system, Ergos work simulator, Ergo-Kit and Isernhagen work system. International Archives of Occupational and Environmental Health 2004, 77(8):527-537.
47. Gwet K.L. Computing inter-rater reliability and its variance in the presence of high agreement. The British Journal of Mathematical and Statistical Psychology 2008, 61(Pt 1):29-48.
48. Hall T.M., Robinson K.W., Fujinawa O., Akasaka K., Pyne E.A. Intertester reliability and diagnostic validity of the cervical flexion-rotation test. Journal of Manipulative and Physiological Therapy 2008, 31(4):293-300.
49. Hart S., Bergquist S., Gajewski B., Dunton N. Reliability testing of the national database of nursing quality indicators pressure ulcer indicator. Journal of Nursing Care Quality 2006, 21(3):256-265.
50. Hestbaek L., Leboeuf-Yde C. Are chiropractic tests for the lumbo-pelvic spine reliable and valid? A systematic review. Journal of Manipulative and Physiological Therapeutics 2000, 23(4):258-275.
51. House A.E., House B.J., Campbell M.B. Measures of interobserver agreement: calculation formulas and distribution effects. Journal of Behavioral Assessment 1981, 3(1):37-57.
52. Hutchings A., Raine R., Sanderson C., Black N. A comparison of formal consensus methods used for developing clinical guidelines. Journal of Health Services Research & Policy 2006, 11(4):218-224.
53. Hwang J.C., Yu A.C., Casper D.S., Starren J., Cimino J.J., Chiang M.F. Representation of ophthalmology concepts by electronic systems: intercoder agreement among physicians using controlled terminologies. Ophtalmology 2006, 113(4):511-519.
54. Innes E., Straker L. Reliability of work-related assessments. Work 1999, 13(2):107-124.
55. Kadam U.T., Jordan K., Croft P.R. A comparison of two consensus methods for classifying morbidities in a single professional group showed the same outcomes. Journal of Clinical Epidemiology 2006, 59(11):1169-1173.
56. Kirshner B., Guyatt G. A methodological framework for assessing health indices. Journal of Chronic Diseases 1985, 38(1):27-36.
57. Kobak K.A., Engelhardt N., Williams J., Lipsitz J.D. Rater training in multicenter clinical trials: issues and recommendations. Journal of Clinical Psychopharmacology 2004, 24(2):113-117.
58. Kobak K.A., Lipsitz J.D., Williams J.B.W., Engelhardt N., Bellew K.M. A new approach to rater training and certification in a multicenter clinical trial. Journal of Clinical Psychopharmacology 2005, 25(5):407-411.
59. Kobak K.A., Williams J.B.W., Engelhardt N. A comparison of face-to-face and remote assessment of inter-rater reliability on the Hamilton Depression Rating Scale via videoconferencing. Psychiatry Research 2008, 158(1):99-103.
60. Kottner J., Dassen T. Interpreting interrater reliability coefficients of the Braden scale: a discussion paper. International Journal of Nursing Studies 2008, 45(8):1239-1246.
61. Kottner J., Dassen T. An interrater reliability study of the Braden scale in two nursing homes. International Journal of Nursing Studies 2008, 45(10):1501-1511.
62. Kottner J., Dassen T., Tannen A. Inter- and intrarater reliability of the Waterlow pressure sore risk scale: a systematic review. International Journal of Nursing Studies 2009, 46(3):369-379.
63. Kottner J., Raeder K., Halfens R., Dassen T. A systematic review of interrater reliability of pressure ulcer classification systems. Journal of Clinical Nursing 2009, 18(3):315-336.
64. Kraemer H.C. Ramifications of a population model for κ as a coefficient of reliability. Psychometrika 1979, 44(4):461-472.
65. Kraemer H.C. Measurement of reliability for categorical data in medical research. Statistical Methods in Medical Research 1992, 1(2):183-199.
66. Kraemer H.C., Periyakoil V.S., Noda A. Kappa coefficients in medical research. Statistics in Medicine 2002, 21(14):2109-2129.
67. Landis J.R., Koch G.G. The measurement of observer agreement for categorical data. Biometrics 1977, 33(1):159-174.
68. Lee J., Koh D., Ong C.N. Statistical evaluation of agreement between two methods for measuring a quantitative variable. Computers in Biology and Medicine 1989, 19(1):61-70.
69. Lewicki L.J., Mion L.C., Secic M. Sensitivity and specificity of the Braden scale in the cardiac surgical population. Journal of Wound Ostomy and Continence Nursing 2000, 27(1):36-41.
70. Maclure M., Willett W.C. Misinterpretation and misuse of the kappa statistic. American Journal of Epidemiology 1987, 126(2):161-169.
71. McAlister F.A., Straus S.E., Sackett D.L. Why we need large, simple studies of clinical examination: the problem and a proposed solution. Lancet 1999, 354(13):1721-1724.
72. McGraw K.O., Wong S.P. Forming inferences about some intraclass correlation coefficients. Psychological Methods 1996, 1(1):30-46.
73. Mok J.M., Berven S.H., Diab M., Hackbarth M., Hu S.S., Deviren V. Comparison of observer variation in conventional and three digital radiographic methods used in the evaluation of patients with adolescent idiopathic scoliosis. Spine 2008, 33(6):681-686.
74. Müller R., Büttner P. A critical discussion of intraclass correlation coefficients. Statistics in Medicine 1994, 13(23-24):2465-2476.
75. Mulsant B.H., Kastango K.B., Rosen J., Stone R.A., Mazumdar S., Pollock B.G. Interrater reliability in clinical trials of depressive disorders. The American Journal of Psychiatry 2002, 159(9):1598-1600.
76. Nanda R., Gupta S., Kanapathipillai P., Liow R.Y.L., Rangan A. An assessment of the inter examiner reliability of clinical tests for subacromial impingement and rotator cuff integrity. European Journal of Orthopaedic Surgery and Traumatology 2008, 18:495-500.
77. Nunnally J.C., Bernstein I.H. Psychometric Theory 1994, McGraw-Hill, New York.
78. Nwosu C.R., Khan K.S., Chien P.F., Honest M.R. Is real-time ultrasonic bladder volume estimation reliable and valid? A systematic overview. Scandinavian Journal of Urology and Nephrology 1998, 32(5):325-330.
79. Oot-Giromini B. Pressure ulcer prevalence, incidence and associated risk factors in the community. Decubitus 1993, 6(5):24-32.
80. Peat J., Barton B. Medical Statistics: A Guide to Data Analysis and Critical Appraisal 2005, Blackwell, Oxford.
81. Perkins D.O., Wyatt R.J., Bartko J.J. Penny-wise and pound-foolish: the impact of measurement error on sample size requirements in clinical trails. Biological Psychiatry 2000, 47(8):762-766.
82. Polit D.F., Beck C.T. Nursing Research: Generating and Assessing Evidence for Nursing Practice 2008, Lippincott Williams & Wilkins, Philadelphia. 8th ed.
83. Ratanawongsa N., Thomas P.A., Marinopoulos S.S., Dorman T., Wilson L.M., Ashar B.H., et al. The reported validity and reliability of methods for evaluating continuing medical education: a systematic review. Academic Medicine 2008, 83(3):274-283.
84. Richardson F.M. Peer review of medical care. Medical Care 1972, 10(1):29-39.
85. Roberts C., McNamee R. A matrix of kappa-type coefficients to assess the reliability of nominal scales. Statistics in Medicine 1998, 17(4):471-488.
86. Roberts C., McNamee R. Assessing the reliability of ordered categorical scales using kappa-type statistics. Statistical Methods in Medical Research 2005, 14(5):493-514.
87. Rothery P. A nonparametric measure of intraclass correlation. Biometrika 1979, 66(3):629-639.
88. Rousson V., Gasser T., Seifert B. Assessing intrarater, interrater and test-retest reliability of continuous measurements. Statistics in Medicine 2002, 21(22):3431-3446.
89. Sainsbury A., Seebass G., Bansal A., Young J.B. Reliability of the Barthel Index when used with older people. Age Ageing 2005, 34(3):228-232.
90. Saito Y., Sozu T., Hamada C., Yoshimura I. Effective number of subjects and number of raters for inter-rater reliability studies. Statistics in Medicine 2006, 25(9):1547-1560.
91. Scinto J.D., Galusha D.H., Krumholz H.M., Meehan T.P. The case for comprehensive quality indicator reliability assessment. Journal of Clinical Epidemiology 2001, 54(11):1103-1111.
92. Shoukri M.M. Measures of Interobserver Agreement 2004, Chapman & Hall/CRC, Boca Raton.
93. Shoukri M.M., Asyali M.H., Donner A. Sample size requirements for the design of reliability study: review and new results. Statistical Methods in Medical Research 2004, 13:251-271.
94. Shrout P.E. Measurement reliability and agreement in psychiatry. Statistical Methods in Medical Research 1998, 7(3):301-317.
95. Shrout P.E., Fleiss J.L. Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin 1979, 86(2):420-428.
96. Slongo T., Audigé L., Schlickewei W., Clavert J.-M., Hunter J. Development and validation of the AO Pediatric Comprehensive Classification of Long Bone Fractures by the pediatric expert group of the AO Foundation in collaboration with AO Clinical Investigation and Documentation and the International Association for Pediatric Traumatology. Journal of Pediatric Orthopedics 2006, 26(1):43-49.
97. Steenbeek D., Ketelaar M., Galama K., Gortner J.W. Goal attainment in paediatric rehabilitation: a critical review of the literature. Developmental Medicine & Child Neurology 2007, 49(7):550-556.
98. Stevens S.S. On the theory of scales of measurement. Science 1946, 103(2684):677-680.
99. Stochkendahl M.J., Christensen H.W., Hartvigsen J., Vach W., Haas M., Hestbaek L., et al. Manual examination of the spine: a systematic critical literature review of reproducibility. Journal of Manipulative and Physiological Therapeutics 2006, 29(6):475-485.
100. Stratford P.W., Goldsmith C.H. Use of the standard error as a reliability index of interest: an applied example using elbow flexor strength data. Physical Therapy 1997, 77(7):745-750.
101. Streiner D.L. Clinimetrics versus psychometrics: an unnecessary distinction. Journal of Clinical Epidemiology 2003, 56(12):1142-1145.
102. Streiner D.L., Norman G.R. Health Measurement Scales: A Practical Guide to Their Development and Use 2008, Oxford University Press, Oxford. 4th ed.
103. Suen H.K. Agreement, reliability, accuracy, and validity: toward a clarification. Behavioral Assessment 1988, 10(4):343-366.
104. Sun Y., Stürmer T., Günther K.P., Brenner H. Reliability and validity of clinical outcome measurements of osteoarthritis of the hip and knee - a review of the literature. Clinical Rheumatology 1997, 16(2):185-198.
105. Swingler G.H. Observer variation in chest radiography of acute lower respiratory infections in children: a systematic review. BMC Medical Imaging 2001, 1(1):1.
106. Szklo M., Nieto F.J. Epidemiology Beyond the Basics 2007, Jones and Bartlett, Sudbury. 2nd ed.
107. Terwee C.B., Bot S.D., de Boer M.R., van der Windt D.A., Knol D.L., Dekker J., et al. Quality criteria were proposed for measurement properties of health status questionnaires. Journal of Clinical Epidemiology 2007, 60(1):34-42.
108. Thomsen N.O., Olsen L.H., Nielsen S.T. Kappa statistics in the assessment of observer variation: the significance of multiple observers classifying ankle fractures. Journal of Orthopaedic Science 2002, 7(2):163-166.
109. Topf M. Interrater reliability decline under covert assessment. Nursing Research 1988, 37(1):47-49.
110. Uebersax, J., 2002. Raw Agreement Indices. Retrieved April 1, 2008, from http://ourworld.compuserve.com.homepages/jsuebersax/raw.htm.
111. Vach W. The dependence of Cohen's kappa on the prevalence does not matter. Journal of Clinical Epidemiology 2005, 58(7):655-661.
112. Vacha-Haase T. Reliability generalisation: exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement 1998, 58(1):6-20.
113. Vella K., Goldfrad C., Rowan K., Bion J., Black N. Use of consensus development to establish national priorities in critical care. BMJ 2000, 320(7240):976-980.
114. Vikström A., Skånér Y., Strender L.E., Nilsson G.H. Mapping the categories of the Swedish primary health care version of ICD-10 to SNOMED CT concepts: rule development and intercoder reliability in a mapping trial. BMC Medical Informatics and Decision Making 2007, 7:9.
115. Walter S.D., Eliasziw M., Donner A. Sample size and optimal designs for reliability studies. Statistics in Medicine 1998, 17(1):101-110.
116. Whitfield J.W. Intra-class rank correlation. Biometrika 1949, 36(3-4):463-467.
117. Wickström G., Bendix T. The " Hawthorne effect": what did the original Hawthorne studies actually show?. Scandinavian Journal of Work, Environment & Health 2000, 26(4):363-367.
118. Zegers M., de Bruijne M.C., Wagner C., Groenewegen P.P., van der Wal G., de Vet H.C. The inter-rater agreement of retrospective assessments of adverse events does not improve with two reviewers per patient record. Journal of Clinical Epidemiology 2010, 63(1):94-102.