Evaluating agreement: Conducting a reliability study

Journal of Bone and Joint Surgery

Volumn 91, Issue SUPPL. 3, 2009, Pages 99-106

  Karanicolas, Paul J ^a   Bhandari, Mohit ^a   Kreder, Hans ^b   Moroni, Antonio ^c   Richardson, Martin ^d   Walter, Stephen D ^a   Norman, Geoff R ^a   Guyatt, Gordon H ^a  

^a MCMASTER UNIVERSITY   (Canada)

^b UNIVERSITY OF TORONTO   (Canada)

^c UNIVERSITY OF BOLOGNA   (Italy)

^d UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

CLINICAL PRACTICE; CLINICAL RESEARCH; HUMAN; INSTRUMENT; MEASUREMENT ERROR; ORTHOPEDICS; PRIORITY JOURNAL; RELIABILITY; REVIEW; SAMPLE SIZE; STATISTICAL ANALYSIS; ARTICLE; MEDICAL RESEARCH; METHODOLOGY; REPRODUCIBILITY; STATISTICS;

BIOMEDICAL RESEARCH; HUMANS; ORTHOPEDICS; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; SAMPLE SIZE; STATISTICS AS TOPIC;

EID: 66349131005     PISSN: 00219355     EISSN: None     Source Type: Journal    
DOI: 10.2106/JBJS.H.01624     Document Type: Review

References (32)

1. Audigé L, Bhandari M, Kellam J. How reliable are reliability studies of fracture classifications? A systematic review of their methodologies. Acta Orthop Scand. 2004;75:184-94.
2. Streiner DL, Norman GR. Health measurement scales: a practical guide to their development and use. 3rd ed. Oxford: Oxford University Press; 2003.
3. Karanicolas PJ. The reliability study. In: Bhandari M, Joensson A, editors. Clinical research for surgeons. Stuttgart, Germany: Thieme Medical; 2009. p 91-9.
4. Leggin BG, Michener LA, Shaffer MA, Brenneman SK, Iannotti JP, Williams GR Jr. The Penn shoulder score: reliability and validity. J Orthop Sports Phys Ther. 2006;36:138-51.
5. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16:297-334.
6. van der Esch M, Steultjens M, Ostelo RW, Harlaar J, Dekker J. Reproducibility of instrumented knee joint laxity measurement in healthy subjects. Rheumatology (Oxford). 2006;45:595-9.
7. Irwig L, Macaskill P, Walter SD, Houssami N. New methods give better estimates of changes in diagnostic accuracy when prior information is provided. J Clin Epidemiol. 2006;59:299-307.
8. Flikkilä T, Nikkola-Shito A, Kaarela O, Pääkkö E, Raatikainen T. Poor interobserver reliability of AO classification of fractures of the distal radius. Additional computed tomography is of minor value. J Bone Joint Surg Br. 1998;80:670-2.
9. Boulos MN, Maramba I, Wheeler S. Wikis, blogs and podcasts: a new generation of Web-based tools for virtual collaborative clinical practice and education. BMC Med Educ. 2006;6:41.
10. Castel JM, Figueras A, Vigo JM. The internet as a tool in clinical pharmacology. Br J Clin Pharmacol. 2006;61:787-90.
11. Sachs NA, Nulud PL, Loeb, GE. Virtual Visit: improving communication for those who need it most. Stud Health Technol Inform. 2003;94:302-8.
12. Collaboration for Outcome Assessment in Surgical Trials. http://www.coastresearch.ca. 2007 Apr 16.
13. Luiz RR, Szklo M. More than one statistical strategy to assess agreement of quantitative measurements may usefully be reported. J Clin Epidemiol. 2005;58:215-6.
14. Ludbrook J. Statistical techniques for comparing measurers and methods of measurement: a critical review. Clin Exp Pharmacol Physiol. 2002;29:527-36.
15. Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther. 2005;85:257-68.
16. Hayes KW, Petersen CM. Reliability of classifications derived from Cyriax's resisted testing in subjects with painful shoulders and knees. J Orthop Sports Phys Ther. 2003;33:235-46.
17. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20:37-46.
18. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull. 1968;70:213-20.
19. Kraemer HC. Ramifications of a population model for kappa as a coefficient of reliability. Psychometrika. 1979;44:461-72.
20. McGinn T, Guyatt G, Cook R, Meade M. Measuring agreement beyond chance. In: Guyatt G, Rennie D, editors. Users' guides to the medical literature: a manual for evidence-based clinical practice. Chicago: AMA Press; 2001. p 461-70.
21. Fisher RA. Statistical methods for research workers. Edinburgh: Oliver and Boyd; 1925.
22. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86:420-8.
23. Donner A, Shoukri MM, Klar N, Bartfay E. Testing the equality of two dependent kappa statistics. Stat Med. 2000;19:373-87.
24. Donner A, Zou G. Testing the equality of dependent intraclass correlation coefficients. Statistician. 2002;51:367-79.
25. Shavelson RJ, Webb NM, Rowley GL. Generalizability theory. Am Psychol. 1989;44:922-32.
26. Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for reliability studies. Stat Med. 1998;17:101-10.
27. Giraudeau B, Mary JY. Planning a reproducibility study: how many subjects and how many replicates per subject for an expected width of the 95 per cent confidence interval of the intraclass correlation coefficient. Stat Med. 2001;20:3205-14.
28. Altman DG. Practical statistics for medical research. London: Chapman & Hall/CRC; 1990. Inter-rater agreement; p 403-9.
29. Brage ME, Rockett M, Vraney R, Anderson R, Toledano A. Ankle fracture classification: a comparison of reliability of three X-ray views versus two. Foot Ankle Int. 1998;19:555-62.
30. Fleiss JL. Statistical methods for rates and proportions. 2nd ed. New York: Wiley-Interscience; 1981.
31. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159-74.
32. Svanholm H, Starklint H, Gundersen HJ, Fabricius J, Barlebo H, Olsen S. Reproducibility of histomorphologic diagnoses with special reference to the kappa statistic. APMIS. 1989;97:689-98.