Reliability and agreement studies: A guide for clinical investigators

Gut

Volumn 64, Issue 7, 2015, Pages 1018-1027

  Hernaez, Ruben ^a  

^a JOHNS HOPKINS UNIVERSITY SCHOOL OF MEDICINE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; COMPUTER PROGRAM; DISEASE ACTIVITY; MEASUREMENT ERROR; METHODOLOGY; OUTCOME ASSESSMENT; PRIORITY JOURNAL; RELIABILITY; REPRODUCIBILITY; STATISTICAL ANALYSIS; STATISTICAL PARAMETERS; VALIDITY; GASTROENTEROLOGY; HUMAN; MEDICAL RESEARCH; STATISTICS;

BIOMEDICAL RESEARCH; GASTROENTEROLOGY; HUMANS; REPRODUCIBILITY OF RESULTS; STATISTICS AS TOPIC;

EID: 84934977649     PISSN: 00175749     EISSN: 14683288     Source Type: Journal    
DOI: 10.1136/gutjnl-2014-308619     Document Type: Article

References (89)

1. de Vet HC, Terwee CB, Knol D.L., et al. When to use agreement versus reliability measures. J Clin Epidemiol 2006; 59:1033-9.
2. de Vet HC, Terwee CB, Bouter LM. Current challenges in clinimetrics. J Clin Epidemiol 2003; 56:1137-41.
3. Streiner DL. Clinimetrics vs. psychometrics: an unnecessary distinction. J Clin Epidemiol 2003; 56:1142-5.
4. Mokkink LB, Terwee CB, Patrick D.L., et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res 2010; 19:539-49.
5. Mokkink LB, Terwee CB, Patrick D.L., et al. COSMIN checklist manual. http://www.cosmin.nl/images/upload/files/COSMIN%20checklist%20manual%20v9.pdf (accessed 1/5/2015).
6. Mosli MH, Feagan BG, Zou G.Y., et al. Reproducibility of histological assessments of disease activity in UC. Gut 2014 Oct 30. pii gutjnl-2014-307536. doi:10.1136/gutjnl-2014-307536.
7. Gisev N, Bell JS, Chen TF. Interrater agreement and interrater reliability: key concepts, approaches, and applications. Res Social Adm Pharm 2013; 9:330-8.
8. Mokkink LB, Terwee CB, Patrick D.L., et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol 2010; 63:737-45
9. Terwee CB, Mokkink LB, Knol DL, et al. Rating the methodological quality in systematic reviews of studies on measurement properties: a scoring system for the COSMIN checklist. Qual Life Res 2012; 21:651-7.
10. Lucas NP, Macaskill P, Irwig L., et al. The development of a quality appraisal tool for studies of diagnostic reliability (QAREL). J Clin Epidemiol 2010; 63:854-61.
11. Kottner J, Audige L, Brorson S., et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. J Clin Epidemiol 2011; 64:96-106.
12. Kottner J, Streiner DL. The difference between reliability and agreement. J Clin Epidemiol 2011; 64:701-2.
13. Bossuyt PM, Reitsma JB, Bruns D.E., et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med 2003; 138:W1-12.
14. Haber M, Barnhart HX. Coefficients of agreement for fixed observers. Stat Methods Med Res 2006; 15:255-71.
15. Higgins PA, Straub AJ. Understanding the error of our ways: mapping the concepts of validity and reliability. Nurs Outlook 2006; 54:23-9.
16. Dunn G. Statistical evaluation of measurement errors. 2nd edn. London: Hodder Arnold, 2004.
17. Windish DM, Diener-West M. A clinician-educator's roadmap to choosing and interpreting statistical tests. J Gen Intern Med 2006; 21:656-60.
18. Lang TA, Secic M. Reporting hypothesis testing. How to report statistics in medicine. 2nd edn. Sheridan Press, 2006:51.
19. Hanley JA, Negassa A, Edwardes M.D., et al. Statistical analysis of correlated data using generalized estimating equations: an orientation. Am J Epidemiol 2003; 157:364-75.
20. Loftus EV Jr, Harewood GC, Loftus C.G., et al. PSC-IBD: a unique form of inflammatory bowel disease associated with primary sclerosing cholangitis. Gut 2005; 54:91-6.
21. Woreta TA, Sutcliffe CG, Mehta S.H., et al. Incidence and risk factors for steatosis progression in adults coinfected with HIV and hepatitis C virus. Gastroenterology 2011; 140:809-17.
22. Graubard BI, Korn EL. Regression analysis with clustered data. Stat Med 1994; 13:509-22.
23. Uebersax JS. Kappa coefficients: A critical appraisal. http://john-uebersax.com/stat/kappa.htm (accessed 1/8/2015).
24. Yang Z, Zhou M. Kappa statistic for clustered matched-pair data. Stat Med 2014; 33:2612-33.
25. de Vet HC, Terwee CB, Mokkink L.B., et al. Reliability. Measurement in medicine. New York, USA: Cambridge University Press, 2008:115-19.
26. Banerjee M, Capozzoli M, McSweeny L., et al. Beyond kappa: a review of interrater agreement measurements. Can J Stat 1999; 27:3-23.
27. Dunn G. Setting the scene. Statistical evaluation of measurement errors. 2nd edn. London: Hodder Arnold, 2004:17.
28. Streiner DL, Norman GR. Reliability. Health measurement scales: a practical guide to their development and use. 4th edn. New York, USA: Oxford Scholarship Online, 2009.
29. Haley SM, Osberg JS. Kappa coefficient calculation using multiple ratings per subject: a special communication. Phys Ther 1989; 69:970-4.
30. Shoukri MM, Martin SW, Mian IU. Maximum likelihood estimation of the kappa coefficient from models of matched binary responses. Stat Med 1995; 14:83-99.
31. Roberts C. Modelling patterns of agreement for nominal scales. Stat Med 2008; 27:810-30.
32. Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for reliability studies. Stat Med 1998; 17:101-10.
33. Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther 2005; 85:257-68.
34. Donner A, Eliasziw M. Statistical implications of the choice between a dichotomous or continuous trait in studies of interobserver agreement. Biometrics 1994; 50:550-5.
35. Hoehler FK. Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity. J Clin Epidemiol 2000; 53:499-503.
36. Solovieva S, Vehmas T, Riihimaki H., et al. Hand use and patterns of joint involvement in osteoarthritis. A comparison of female dentists and teachers. Rheumatology (Oxford) 2005; 44:521-8.
37. Roberts C, McNamee R. A matrix of kappa-type coefficients to assess the reliability of nominal scales. Stat Med 1998; 17:471-88.
38. Kraemer HC. Measurement of reliability for categorical data in medical research. Stat Methods Med Res 1992; 1:183-99.
39. Guggenmoos-Holzmann I, Vonk R. Kappa-like indices of observer agreement viewed from a latent class perspective. Stat Med 1998; 17:797-812.
40. Uebersax JS, Grove WM. Latent class analysis of diagnostic agreement. Stat Med 1990; 9:559-72.
41. Uebersax JS. Modeling approaches for the analysis of observer agreement. Invest Radiol 1992; 27:738-43.
42. Uebersax JS, Grove WM. A latent trait finite mixture model for the analysis of rating agreement. Biometrics 1993; 49:823-35.
43. Bartholomew DJ, Knott M, Moustaki I. Latent variable models and factor analysis: a unified approach. 3rd edn. West Sussex, UK: Wiley, 2011.
44. Uebersax JS. Latent Structure Analyses. http://www.john-uebersax.com/stat/index.htm (accessed 1/8/2015).
45. Christensen AH, Gjorup T, Hilden J., et al. Observer homogeneity in the histologic diagnosis of Helicobacter pylori. Latent class analysis, kappa coefficient, and repeat frequency. Scand J Gastroenterol 1992; 27:933-9.
46. Agresti A. Modelling patterns of agreement and disagreement. Stat Methods Med Res 1992; 1:201-18.
47. Becker MP, Agresti A. Log-linear modelling of pairwise interobserver agreement on a categorical scale. Stat Med 1992; 11:101-14.
48. Valet F, Guinot C, Ezzedine K., et al. Quality assessment of ordinal scale reproducibility: log-linear models provided useful information on scale structure. J Clin Epidemiol 2008; 61:983-90.
49. McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychol Methods 1996; 1:30-46.
50. Weir JP. Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. J Strength Cond Res 2005; 19:231-40.
51. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull 1979; 86: 420-8.
52. de Vet HC, Terwee CB, Mokkink L.B., et al. Reliability. Measurement in medicine. New York, USA: Cambridge University Press, 2008:131-7.
53. Donner A, Eliasziw M. Sample size requirements for reliability studies. Stat Med 1987; 6:441-8.
54. Giraudeau B, Mary JY. Planning a reproducibility study: how many subjects and how many replicates per subject for an expected width of the 95 per cent confidence interval of the intraclass correlation coefficient. Stat Med 2001; 20:3205-14.
55. Bonett DG. Sample size requirements for estimating intraclass correlations with desired precision. Stat Med 2002; 21:1331-5.
56. Zou GY. Sample size formulas for estimating intraclass correlation coefficients with precision and assurance. Stat Med 2012; 31:3972-81.
57. Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess 1994; 6:284-90.
58. de Vet HC, Terwee CB, Mokkink L.B., et al. Reliability. Measurement in medicine. New York, USA: Cambridge University Press, 2008:120.
59. Muller R, Buttner P. A critical discussion of intraclass correlation coefficients. Stat Med 1994; 13:2465-76.
60. Chen CC, Barnhart HX. Assessing agreement with intraclass correlation coefficient and concordance correlation coefficient for data with repeated measures. Comput Stat Data Anal 2013; 60:132-45.
61. Streiner DL, Norman GR. Generalizability theory. Health measurement scales: a practical guide to their development and use. 4th edn. New York, USA: Oxford Scholarship Online, 2009.
62. Institute for Digital Research and Education. FAQ: What is the coefficient of variation? Consulted on 1/12/2015 ed. 2015.
63. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 1:307-10.
64. Schmidt ME, Steindorf K. Statistical methods for the validation of questionnaires-discrepancy between theory and practice. Methods Inf Med 2006; 45:409-13.
65. Altman DG. Relation between two continuous variables. Pract Stat Med Res 1999; 1:277-99.
66. Uebersax JS. Raw Agreement Indices. http://www.john-uebersax.com/stat/raw.htm (accessed 1/8/2015).
67. Samsa GP. Sampling distributions of p(pos) and p(neg). J Clin Epidemiol 1996; 49:917-19.
68. US Department of Health and Human Services FaDA. Statistical Guidance on Reporting from Studies Evaluating Diagnostic Tests. Consulted on 1/12/2015 ed. 2007.
69. de Vet HC, Terwee CB, Mokkink L.B., et al. Reliability. Measurement in medicine. New York, USA: Cambridge University Press, 2008:111-13.
70. Sedgwick P. Limits of agreement (Bland-Altman method). BMJ 2013; 346:f1630.
71. de Vet HC, Terwee CB, Mokkink L.B., et al. Reliability. Measurement in medicine. New York, USA: Cambridge University Press, 2008:113-15.
72. Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res 1999; 8:135-60.
73. Costa-Santos C, Bernardes J, Yres-de-Campos D, et al. The limits of agreement and the intraclass correlation coefficient may be inconsistent in the interpretation of agreement. J Clin Epidemiol 2011; 64:264-9.
74. Euser AM, Dekker FW, le CS. A practical approach to Bland-Altman plots and variation coefficients for log transformed variables. J Clin Epidemiol 2008; 61:978-82.
75. Carroll RJ, Pee D, Freedman L.S., et al. Statistical design of calibration studies. Am J Clin Nutr 1997; 65 (4 Suppl):1187S-9S.
76. Carroll RJ, Freedman L, Pee D. Design aspects of calibration studies in nutrition, with analysis of missing data in linear measurement error models. Biometrics 1997; 53:1440-57.
77. MacMahon S, Peto R, Cutler J., et al. Blood pressure, stroke, and coronary heart disease. Part 1, Prolonged differences in blood pressure: prospective observational studies corrected for the regression dilution bias. Lancet 1990; 335:765-74.
78. Knuiman MW, Divitini ML, Buzas J.S., et al. Adjustment for regression dilution in epidemiological regression analyses. Ann Epidemiol 1998; 8:56-63.
79. Guolo A. Robust techniques for measurement error correction: a review. Stat Methods Med Res 2008; 17:555-80.
80. Carroll RJ, Hardin J, Schmiediche H. Stata software for generalized linear measurement error models. Accessed on 1/14/2015 ed. 2015.
81. Rothman KJ, Greenland S. Precision and validity in epidemiologic studies. Modern epidemiology. 2nd edn. Philadelphia: Lippincott Williams and Wilkins, 1998:115.
82. Delgado-Rodriguez M, Llorca J. Bias. J Epidemiol Community Health 2004; 58:635-41.
83. Zhou XH. Correcting for verification bias in studies of a diagnostic test's accuracy. Stat Methods Med Res 1998; 7:337-53.
84. Bachmann LM, Ter RG, Weber W.E., et al. Multivariable adjustments counteract spectrum and test review bias in accuracy studies. J Clin Epidemiol 2009; 62:357-61.
85. de Vet HC, Terwee CB, Mokkink L.B., et al. Validity. Measurement in medicine. New York, USA: Cambridge University Press, 2008: 150-201.
86. de Vet HC, Terwee CB, Mokkink L.B., et al. Responsiveness. Measurement in medicine. New York, USA: Cambridge University Press, 2008:202-16.
87. de Vet HC, Terwee CB, Mokkink L.B., et al. Interpretability. Measurement in medicine. New York, USA: Cambridge University Press, 2008:227-74.
88. de Vet HC, Terwee CB. The minimal detectable change should not replace the minimal important difference. J Clin Epidemiol 2010; 63:804-5.
89. Terwee CB. COSMIN checklist with 4-point scale. http://www.cosmin.nl/images/upload/files/COSMIN%20checklist%20with%204-point%20scale%2022%20juni%202011.pdf (accessed 1/5/2015).