Assessment Methodologies and Statistical Issues for Computer-Aided Diagnosis of Lung Nodules in Computed Tomography: Contemporary Research Topics Relevant to the Lung Image Database Consortium

Academic Radiology

Volumn 11, Issue 4, 2004, Pages 462-475

  Dodd, Lori E ^a   Wagner, Robert F ^b   Armato III, Samuel G ^c   McNitt Gray, Michael F ^d   Beiden, Sergey ^e   Chan, Heang Ping ^f   Gur, David ^g   McLennan, Geoffrey ^h   Metz, Charles E ^c   Petrick, Nicholas ^j   Sahiner, Berkman ^f   Sayre, Jim ⁱ  

^a NATIONAL CANCER INSTITUTE   (United States)

^b CENTER FOR DRUG EVALUATION AND RESEARCH   (United States)

^c UNIVERSITY OF CHICAGO   (United States)

^d UNIVERSITY OF CALIFORNIA   (United States)

^e CARANA Corporation   (Russian Federation)

^f UNIVERSITY OF MICHIGAN MEDICAL SCHOOL   (United States)

^g UNIVERSITY OF PITTSBURGH SCHOOL OF MEDICINE   (United States)

^h UNIVERSITY OF IOWA   (United States)

ⁱ UNIVERSITY OF CALIFORNIA   (United States)

^j NONE

more..

Author keywords

Computer aided diagnosis (CAD); Database development; Lung cancer; Lung nodule; MRMC; ROC

Indexed keywords

ALGORITHM; CANCER DIAGNOSIS; CANCER SURVIVAL; COMPUTER ASSISTED DIAGNOSIS; COMPUTER ASSISTED TOMOGRAPHY; DATA BASE; DIAGNOSTIC IMAGING; HUMAN; IMAGE ANALYSIS; IMAGE QUALITY; IMAGING SYSTEM; LUNG CANCER; LUNG NODULE; MEDICAL DECISION MAKING; METHODOLOGY; PERFORMANCE; PRIORITY JOURNAL; REVIEW; STANDARD;

EID: 12144290346     PISSN: 10766332     EISSN: None     Source Type: Journal    
DOI: 10.1016/S1076-6332(03)00814-6     Document Type: Note

References (87)

1. Jemal A, Murray T, Samuels A, Ghafoor A, Ward E, Thun MJ. Cancer statistics 2003. CA Cancer J Clin 2003; 53:5-26.
2. Fry WA, Menck HR, Winchester DP. The National Cancer Data Base report on lung cancer. Cancer 1996; 77:1947-1955.
3. Martini N, Bains MS, Burt ME, et al. Incidence of local recurrence and second primary tumors in resected stage I lung cancer. J Thorac Cardiovasc Surg 1995; 109:120-129.
4. Flehinger BJ, Kimmel M, Melamed M. The effect of surgical treatment on survival from early lung cancer. Chest 1992; 101:1013-1018.
5. Kaneko M, Eguchi K, Ohmatsu H, et al. Peripheral lung cancer: screening and detection with low-dose spiral CT versus radiography. Radiology 1996; 201:798-802.
6. Henschke CI, McCauley DE, Yankelevitz DF, et al. Early Lung Cancer Action Project: overall design and findings from baseline screening. Lancet 1999; 354:99-105.
7. Sone S, Takashima S, Li F, et al. Mass screening for lung cancer with mobile spiral computed tomography scanner. Lancet 1998; 351:1242-1245.
8. Ohmatsu H, Kakinuma R, Kaneko M, Moriyama N, Kusumoto M, Eguchi K. Successful lung cancer screening with low-dose helical CT in addition to chest x-ray and sputum cytology: the comparison of two screening period with or without helical CT. Radiology 2000; 217(P): 242(abstr).
9. Rubin GD. Data explosion: The challenge of multidetector-row CT. Eur J. Radiol 2000; 36:74-80.
10. Naidich DP. Helical computed tomography of the thorax: clinical applications. Radiol Clin North Am 1994; 32:759-774.
11. Clarke LP, Croft BY, Staab E, Baker H, Sullivan DC. National Cancer Institute initiative: Lung image database resource for imaging research. Acad Radiol 2001; 8:447-450.
12. Giger ML. Current issues in CAD for Mammography. In: Doi K, Giger ML, Nishikawa RM, Schmidt RA, eds. Proceedings of the 3rd International Workshop on Digital Mammography, Digital Mammography '96. Amsterdam: Elsevier, 1996; 53-59.
13. Schultz DG, Wagner RF, Campbell G. Editorial response. Radiology 1997; 202:317-318.
14. Wagner RF, Beiden SV, Campbell G, Metz CE, Sacks WM. Assessment of medical imaging and computer-assist systems: Lessons from recent experience. Acad Radiol 2002; 9:1264-1277.
15. Swets JA, Pickett RM. Evaluation of diagnostic systems. New York: Academic Press, 1982.
16. Metz CE. Basic principles of ROC analysis. Semin Nucl Med 1978; 8:283-298.
17. Metz CE. ROC methodology in radiologic imaging. Invest Radiol 1986; 21:720-733.
18. Metz CE. Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol 1989; 24:234-245.
19. Metz CE. Fundamental(s) of ROC Analysis. In: Handbook of Medical Imaging. Beutel J, Kundel HL, Van Metter RL, eds. Physics and Psychophysics. Vol 1. Bellingham, WA: SPIE Press, 2000, 751-769.
20. Wagner RF, Beiden SV, Metz CE. Continuous versus categorical data for ROC analysis: some quantitative considerations. Acad Radiol 2001; 8:328-334.
21. Metz CE. Statistical analysis of ROC data in evaluating diagnostic performance. In: Herbert DE, Myers RH, eds. Multiple regression analysis: applications in the health sciences. New York: American Institute of Physics: American Association of Physicists in Medicine, 1986; 365-384.
22. Metz CE, Herman BA, Shen JH. Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data. Stat Med 1998; 17:1033-1053.
23. Pepe MS. An interpretation for the ROC curve and inference using GLM procedures. Biometrics 2000; 56:352-359.
24. Zou KH, Hall WJ. Two transformation models for estimating an ROC curve derived from continuous data. J Appl Stat 2000; 27:621-631.
25. Shapiro DE. The interpretation of diagnostic tests. Stat Methods Med Res 1999; 8:113-134.
26. Tosteson AA, Begg CB. A general regression methodology for ROC curve estimation. Med Dec Making 1988; 8:204-215.
27. Toledano AY, Gatsonis C. Ordinal regression methodology for ROC curves derived from correlated data. Stat Med 1996; 15:1807-1826.
28. Cal T, Pepe MS. Semiparametric receiver operating characteristic analysis to evaluate biomarkers for disease. J Am Stat Assoc 2002; 97:1099-1107.
29. Dodd LE, Pepe MS. Semi-parametric regression for the area under the receiver-operating characteristic curve. J Am Stat Assoc 2003; 98:397-405.
30. Dodd LE, Pepe MS. Partial AUC estimation and regression. Biometrics 2003 Sep; 59:614-623.
31. Pepe MS. An interpretation for the ROC curve and inference using GLM procedures. Biometrics 2000; 56:352-359.
32. Nishikawa RM, Yarusso LM. Variations in measured performance of CAD schemes due to database composition and scoring protocol. Proc SPIE 1998; 3338:840-844.
33. Giger ML. Current issues in CAD for mammography. In: Doi K, Giger ML, Nishikawa RM, Schmidt RA, eds. Digital mammography '96. Philadelphia: Elsevier Science, 1996; 53-59.
34. Bunch PC, Hamilton JF, Sanderson GK, Simmons AH. A free response approach to the measurement and characterization of radiographic observer performance. Proc SPIE 1977; 127:124-135.
35. Chakraborty DP. Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Med Phys 1989; 16:561-568.
36. Chakraborty DP, Winter L. Free-response methodology: alternate analysis and a new observer-performance experiment. Radiology 1990; 174:873-881.
37. Metz CE, Starr SJ, Lusted LB. Observer performance in detecting multiple radiographic signals: prediction and analysis using a generalized ROC approach. Radiology 1976; 121:337-347.
38. Starr SJ, Metz CE, Lusted LB, Goodenough DJ. Visual detection and localization of radiographic images. Radiology 1975; 116:533-538.
39. Swensson RG. Unified measurement of observer performance in detecting and localizing target objects on images. Med Phys 1996; 23: 1709-1725.
40. Swensson RG. Using localization data from image interpretations to improve estimates of performance accuracy. Med Decis Making 2000; 20:170-185.
41. Chakraborty DP. The FROC, AFROC and DROC variants of the ROC analysis. In: Beutel J, Kundel HL, Van Metter RL, eds. Handbook of medical imaging. Vol 1. Physics and psychophysics. Bellingham, WA: SPIE Press, 2000, 771-796.
42. Petrick N, Sahiner B, Chan H-P, Helvie MA, Paquerault S, Hadjiiski LM. Breast cancer detection: evaluation of a mass-detection algorithm for computer-aided diagnosis: experience in 263 patients. Radiology 2002; 224:217-224.
43. Metz CE. Evaluation of digital mammography by ROC analysis. In: Doi K, Giger M, Nishikawa R, Schmidt RA, eds. Digital mammography '96. Amsterdam, The Netherlands: Elsevier Science, 1996; 61-68.
44. Metz CE. Evaluation of CAD methods. In: Doi K, MacMahon H, Giger ML, Hoffmann KR, eds. Computer-aided diagnosis in medical imaging. Excerpta Medica International Congress Series, Vol. 1182. Amsterdam: Elsevier Science, 1999, 543-554.
45. Chakraborty DP. Statistical power in observer performance studies: comparison of the receiver operating characteristic and free-response methods in tasks involving localization. Acad Radiol 2002; 9:147-156.
46. Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method. Invest Radiol 1992; 27:723-731.
47. Roe CA, Metz CE. Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: validation with computer simulation. Acad Radiol 1997; 4:298-303.
48. Chakraborty DP, Berbaum KS. Comparing inter-modality diagnostic accuracies in tasks involving lesion localization: a jackknife AFROC approach. Radiology 2002; 225(Suppl P):259.
49. Chakraborty, DP. Proposed solution to the FROC problem and an invitation to collaborate. In: Chakraborty, DP, Krupinski, EA, (eds.), Medical imaging 2003: image perception, observer performance, and technology assessment, Vol 5034, Proc SPIE, 204-212
50. Obuchowski NA, Lieber ML, Powell KA. Data analysis for detection and localization of multiple abnormalities with application to mammography. Acad Radiol 2000; 7:516-525.
51. Rutter CM. Bootstrap estimation of diagnostic accuracy with patient-clustered data. Acad Radiol 2000; 7:413-419.
52. Efron B, Tibshirani RJ. An introduction to the bootstrap. New York: Chapman and Hall, 1993.
53. Chakraborty D, Obuchowski NA, Lieber ML, Powell KA. Point-counter-point. Acad Radiol 2000; 7:553-556.
54. Begg CB, Greenes RA. Assessment of diagnostic tests when disease verification is subject to selection bias. Biometrics 1983; 39:207-215.
55. Gray R, Begg CB, Greenes RA. Construction of receiver operating characteristic curves when disease verification is subject to selection bias. Med Decis Making 1984; 4:151-164.
56. Zhou XH. Correcting for verification bias in studies of a diagnostic test's accuracy. Stat Methods Med Res 1998; 7:337-353.
57. Revesz G, Kundel HL, Bonitatibus M. The effect of verification on the assessment of imaging techniques. Invest Radiol 1983; 18:194-198.
58. Walter SD, Irwig LM. Estimation of test error rates, disease prevalence and relative risk from misclassified data: a review. Clin Epidemiol 1988; 41:923-937.
59. Vacek PM. The effect of conditional dependence on the evaluation of diagnostic tests. Biometrics 1985; 41:959-968.
60. Torrance-Rynard VL, Walter SD. Effects of dependent errors in the assessment of diagnostic test performance. Stat Med 1997; 97:2157-2175.
61. Qu Y, Tan M, Kutner MH. Random effects models in latent class analysis for evaluating accuracy of diagnostic tests. Biometrics 1996; 52: 707-810.
62. Espeland MA, Handleman SL. Using latent class models to characterize and assess relative-error in discrete measurements. Biometrics 1989; 45:587-599.
63. Hui SL, Zhou XH. Evaluation of diagnostic tests without a gold standard. Stat Methods Med Res 1998; 7:354-370.
64. Albert PS, McShane LS, Shih JH. Latent class modeling approaches for assessing diagnostic error without a gold standard: with applications to p53 immunohistochemical assays in bladder tumors. Biometrics 2001; 57:610-619.
65. Albert PS, Dodd LE.A cautionary note on the robustness of latent class models for estimating diagnostic error without a gold standard. (submitted)
66. Dendukuri NM, Lawrence JL. Bayesian approaches to modeling the conditional dependence between multiple diagnostic tests'. Biometrics 2001; 57:158-167.
67. Henkelman RM, Kay I, Bronskill MJ. Receiver operator characteristic (ROC) analysis without truth. Med Decis Making 1990; 10:24-29.
68. Begg CB, Metz CE. Consensus diagnoses and "gold standards". Med Decis Making 1990; 10:29-30.
69. Beiden SV, Campbell G, Meier KL, Wagner RF. On the problem of ROC analysis without truth: the EM algorithm and the information matrix. Proc SPIE 2000; 3981:126-134.
70. University of Chicago ROC software. Available at: http://www-radiology.uchicago.edu/krl/toppage11.htm#software.
71. Metz CE, Wang PL, Kronman HB. A new approach for testing the significance of differences between ROC curves measured from correlated data. In: Deconinck F, ed. Information processing in medical imaging. The Hague: Nijhoff, 1984; 432-445.
72. Kundel HL, Polansky M. Comparing observer performance with mixture distribution analysis when there is no external gold standard. Proc SPIE 1998; 3340:78-84.
73. Polansky M. Agreement and accuracy mixture distribution analysis. In: Handbook of medical imaging. Beutel J, Kundel HL, Van Metter RL, eds. Physics and psychophysics Vol 1. Bellingham, WA: SPIE Press, 2000, 797-835.
74. Kundel HL, Polansky M, Phelan M. Evaluating imaging systems in the absence of truth: a comparison of ROC and mixture distribution analysis in computer aided diagnosis in mammography. Proc SPIE 2000; 4324:22.
75. Hastie T, Tibshirani RJ, Friedman JThe elements of statistical learning-data mining, inference and prediction. New York: Springer-Verlag, 2001.
76. Kundel HL, Polansky M. Comparing observer performance with mixture distribution analysis when there is no external gold standard. Proc SPIE 1998; 3340:78-84.
77. Begg CB, NcNeil BJ. Assessment of radiologic tests: control of bias and other design considerations. Radiology 1988; 167:565-569.
78. Beiden SV, Maloof MA, Wagner RF. A general model for finite-sample effects in training and testing of competing classifiers. IEEE Trans Patt Anal Mach Intel 2003; 25:1561-1569.
79. Beiden SV, Wagner RF, Campbell G. Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects receiver operating characteristic analysis. Acad Radiol 2000; 7:341-349.
80. Beiden SV, Wagner RF, Campbell G, Metz CE, Jiang Y. Components-of-variance models for random-effects ROC analysis: the case of unequal variance structures across modalities. Acad Radiol 2001; 8:605-615.
81. Beiden SV, Wagner RF, Campbell G, Chan H-P. Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis. Acad Radiol 2001; 8:616-622.
82. Wagner RF, Beiden SV, Campbell G. Multiple-reader studies, digital mammography, computer-aided diagnosis-and the Holy Grail of imaging physics (I). Proc SPIE 2001; 4320:611-618.
83. Obuchowski NA. Multireader receiver operating characteristic studies: a comparison of study designs. Acad Radiol 1995; 2:709-716.
84. Gatsonis CA, Begg CB, Wieand S. Advances in statistical methods for diagnostic radiology: a symposium. Acad Radiol 1995; 2(suppl 1): S1-S84.
85. Gifford HC, King MA. Case sampling in LROC: a Monte Carlo analysis. Proc SPIE 2001; 4324:143-152.
86. Chan H-P, Sahiner B, Wagner RF, Petrick N. Classifier design for computer-aided diagnosis: effects of finite sample size on the mean performance of classical and neural network classifiers. Med Phys 1999; 26:2654-2668.
87. Sahiner B, Chan H-P, Petrick N, Wagner RF, Hadjiiski L. Feature selection and classifier performance in computer-aided diagnosis: the effect of finite sample size. Med Phys 2000; 27:1509-1522.