Reliability studies of diagnostic tests are not using enough observers for robust estimation of interobserver agreement: a simulation study

Journal of Clinical Epidemiology

Volumn 61, Issue 7, 2008, Pages 722-727

  Sadatsafavi, Mohsen ^a   Najafzadeh, Mehdi ^a   Lynd, Larry ^a   Marra, Carlo ^a  

^a UNIVERSITY OF BRITISH COLUMBIA   (Canada)

Author keywords

Computer simulation; Observer variation; Reliability; Reproducibility of results; Signal detection; Study design

Indexed keywords

ARTICLE; BREAST TUMOR; DECISION MAKING; DIAGNOSTIC ACCURACY; DIAGNOSTIC PROCEDURE; DIAGNOSTIC TEST; DIAGNOSTIC VALUE; MAMMOGRAPHY; MATHEMATICAL COMPUTING; MODEL; OBSERVER VARIATION; PERFORMANCE MEASUREMENT SYSTEM; PRIORITY JOURNAL; RELIABILITY; REPRODUCIBILITY; SIGNAL DETECTION; SIMULATION;

COMPUTER SIMULATION; DATA INTERPRETATION, STATISTICAL; DIAGNOSTIC TESTS, ROUTINE; HUMANS; OBSERVER VARIATION; REPRODUCIBILITY OF RESULTS; SAMPLE SIZE; STATISTICS AS TOPIC;

EID: 44649111822     PISSN: 08954356     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jclinepi.2007.10.023     Document Type: Article

References (20)

1. Altman D.G., and Bland J.M. Diagnostic tests. 1: sensitivity and specificity. BMJ 308 6943 (1994) 1552
2. Kraemer H.C. Measurement of reliability for categorical data in medical research. Stat Methods Med Res. 1 2 (1992) 183-199
3. Fleiss J.L. Statistical methods for rates and proportions. 2nd ed. (1981), John Wiley, New York
4. Gur D., Rockette H.E., Maitz G.S., King J.L., Klym A.H., and Bandos A.I. Variability in observer performance studies experimental observations. Acad Radiol 12 (2005) 1527-1533
5. Roe C.A., and Metz C.E. Variance-component modeling in the analysis of receiver operating characteristic index estimates. Acad Radiol 4 (1997) 587-600
6. Schouten H.J.A. Nominal scale agreement among observers. Psychometrika 51 (1986) 453-466
7. Fleiss J.L. Measuring nominal scale agreement among many raters. Psychol Bull. 76 (1971) 378-382
8. Uebersax J.S. Modeling approaches for the analysis of observer agreement. Invest Radiol 27 (1992 Sep) 738-743
9. Getty D.J., Pickett R.M., D'Orsi C.J., and Swets J.A. Enhanced interpretation of diagnostic images. Invest Radiol 23 (1988 Apr) 240-252
10. Metz C.E., and Shen J.H. Gains in accuracy from replicated readings of diagnostic images: prediction and assessment in terms of ROC analysis. Med Decis Making 12 (1992) 60-75
11. Moskowitz M. Screening for breast cancer: how effective are our tests? A critical review. CA Cancer J Clin 33 (1983) 26-39
12. Beam C.A., Layde P.M., and Sullivan D.C. Variability in the interpretation of screening mammograms by US radiologists. Findings from a national sample. Arch Intern Med 156 (1996) 209-213
13. Uebersax J.S. Diversity of decision-making models and the measurement of interrater agreement. Psychol Bull 101 (1987) 140-146
14. Uebersax J.S. Validity inferences from interobserver agreement. Psychol Bull. 104 (1988) 405-416
15. Swets J.A., Dawes R.M., and Monahan J. Psychological science can improve diagnostic decisions. Psychol Sci Public Interest 1 1 (2000) 1
16. Swets J.A. Signal detection theory and ROC analysis in psychology and diagnostics: Collected papers (1996), Erlbaum, Mahwah, NJ
17. Lusted L.B. Decision processes and observer error in medical diagnosis. Introduction to medical decision making (1968), Charles C Thomas, Spring field, IL 98-140
18. Berry K.J., and Mielke Jr. P.W. A generalization of Cohen's kappa agreement measure to interval measurement and multiple raters. Educ psychol meas 48 (1988) 921-933
19. Fleiss J.L., and Cohen J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ psychol meas 33 (1973) 613-619
20. Flury B. A first course in multivariate statistics (1997), Springer, New York