Pitfalls in the use of kappa when interpreting agreement between multiple raters in reliability studies

Physiotherapy (United Kingdom)

Volumn 100, Issue 1, 2014, Pages 27-35

  O'Leary, Shaun ^a,b   Lund, Marte ^a,c   Ytre Hauge, Tore Johan ^a,d   Holm, Sigrid Reiersen ^a,e   Naess, Kaja ^a,f   Dalland, Lars Nagelstad ^a,g   McPhail, Steven M ^h,i  

^a UNIVERSITY OF QUEENSLAND   (Australia)

^b ROYAL BRISBANE AND WOMEN'S HOSPITAL   (Australia)

^c Hjelp24 Norwegian Sports Medicine Clinic (Hjelp24 NIMI)   (Norway)

^d Medi 3 Clinic   (Norway)

^e UNIVERSITY HOSPITAL OF NORTH NORWAY   (Norway)

^f Hans and Olaf Physiotherapy Clinic   (Norway)

^g Eggedal Physiotherapy Clinic   (Norway)

^h Centre for Functioning and Health Research   (Australia)

ⁱ QUEENSLAND UNIVERSITY OF TECHNOLOGY   (Australia)

Author keywords

Agreement; Inter therapist; Kappa; Posture; Reliability; Scapular

Indexed keywords

AGREEMENT; INTER-THERAPIST; KAPPA; POSTURE; RELIABILITY; SCAPULAR;

ADULT; BIOMECHANICAL PHENOMENA; DATA INTERPRETATION, STATISTICAL; FEMALE; HUMANS; MALE; OBSERVER VARIATION; PHYSICAL THERAPY SPECIALTY; POSTURE; REPRODUCIBILITY OF RESULTS; SCAPULA;

EID: 84895073059     PISSN: 00319406     EISSN: 18731465     Source Type: Journal    
DOI: 10.1016/j.physio.2013.08.002     Document Type: Article

References (30)

1. Fedorak C., Ashworth N., Marshall J., Paull H. Reliability of the visual assessment of cervical and lumbar lordosis: how good are we?. Spine 2003, 28:1857-1859.
2. Kibler W.B., Uhl T.L., Maddux J.W., Brooks P.V., Zeller B., McMullen J. Qualitative clinical evaluation of scapular dysfunction: a reliability study. J Shoulder Elbow Surg 2002, 11:550-556.
3. Hickey B.W., Milosavljevic S., Bell M.L., Milburn P.D. Accuracy and reliability of observational motion analysis in identifying shoulder symptoms. Man Ther 2007, 12:263-270.
4. McClure P., Tate A.R., Kareha S., Irwin D., Zlupko E. A clinical method for identifying scapular dyskinesis, part 1: reliability. J Athl Train 2009, 44:160-164.
5. Tooth L.R., Ottenbacher K.J. The kappa statistic in rehabilitation research: an examination. Arch Phys Med Rehabil 2004, 85:1371-1376.
6. Sim J., Wright C.C. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther 2005, 85:257-268.
7. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960, 37-46.
8. McPhail S., Lane P., Russell T., Brauer S.G., Urry S., Jasiewicz J., et al. Telephone reliability of the Frenchay Activity Index and EQ-5D amongst older adults. Health Qual Life Outcomes 2009, 7:48.
9. McPhail S., Beller E., Haines T. Two perspectives of proxy reporting of health-related quality of life using the Euroqol-5D, an investigation of agreement. Med Care 2008, 46:1140-1148.
10. McPhail S., Haines T. Patients undergoing subacute rehabilitation have accurate expectations of their health-related quality of life at discharge. Health Qual Life Outcomes 2012, 10:94.
11. Landis J.R., Koch G.G. The measurement of observer agreement for categorical data. Biometrics 1977, 33:159-174.
12. Cicchetti D.V., Feinstein A.R. High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol 1990, 43:551-558.
13. Feinstein A.R., Cicchetti D.V. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol 1990, 43:543-549.
14. Byrt T., Bishop J., Carlin J.B. Bias, prevalence and kappa. J Clin Epidemiol 1993, 46:423-429.
15. Hoehler F.K. Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity. J Clin Epidemiol 2000, 53:499-503.
16. Fleiss J.L. Measuring nominal scale agreement among many raters. Psychol Bull 1971, 76:378-382.
17. O'Toole R.V., Cox G., Shanmuganathan K., Castillo R.C., Turen C.H., Sciadini M.F., et al. Evaluation of computed tomography for determining the diagnosis of acetabular fractures. J Orthop Trauma 2010, 24:284-290.
18. Behensky H., Giesinger K., Ogon M., Krismer M., Hannes B., Karlmeinrad G., et al. Multisurgeon assessment of coronal pattern classification systems for adolescent idiopathic scoliosis: reliability and error analysis. Spine (Phila Pa 1976) 2002, 27:762-767.
19. Riddle D.L., Freburger J.K. Evaluation of the presence of sacroiliac joint region dysfunction using a combination of tests: a multicenter intertester reliability study. Phys Ther 2002, 82:772-781.
20. Jull G., Sterling M., Falla D., Treleaven J., O'Leary S. Whiplash, headache and neck pain: research based directions for physical therapies 2008, Elsevier, Edinburgh.
21. Sahrmann S.A. Diagnosis and treatment of movement impairment syndromes 2002, Mosby, St. Louis.
22. Craven B., Morris A. Modified Ashworth scale reliability for measurement of lower extremity spasticity among patients with SCI. Spinal Cord 2009, 48:207-213.
23. Daley K., Mayo N., Wood-Dauphinée S. Reliability of scores on the Stroke Rehabilitation Assessment of Movement (STREAM) measure. Phys Ther 1999, 79:8-23.
24. Sakzewski L., Boyd R., Ziviani J. Clinimetric properties of participation measures for 5-to 13-year-old children with cerebral palsy: a systematic review. Dev Med Child Neurol 2007, 49:232-240.
25. McPhail S.M., Dalland L.N., Naess K., Lund M., Ytre-Hauge T.J., Holm S.R., et al. Intratherapist reliability in the rating of scapula posture in multiple planes of reference. ISRN Rehabil 2012, 2012.
26. Karduna A.R., McClure P.W., Michener L.A., Sennett B. Dynamic measurements of three-dimensional scapular kinematics: a validation study. J Biomech Eng 2001, 123:184-190.
27. McClure P.W., Michener L.A., Sennett B.J., Karduna A.R. Direct 3-dimensional measurement of scapular kinematics during dynamic movements in vivo. J Shoulder Elbow Surg 2001, 10:269-277.
28. Sobush D.C., Simoneau G.G., Dietz K.E., Levene J.A., Grossman R.E., Smith W.B. The Lennie test for measuring scapular position in healthy young adult females: a reliability and validity study. J Orthop Sports Phys Ther 1996, 23:39-50.
29. Nijs J., Roussel N., Struyf F., Mottram S., Meeusen R. Clinical assessment of scapular positioning in patients with shoulder pain: state of the art. J Manipulative Physiol Ther 2007, 30:69-75.
30. Bland J.M., Altman D.G. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986, 1:307-310.