Training variability in the evaluation of automated classifiers

Progress in Biomedical Optics and Imaging - Proceedings of SPIE

Volumn 7624, Issue , 2010, Pages

  Chen, Weijie ^a   Gallas, Brandon D ^a  

^a Office of Science and Engineering Laboratories   (United States)

Author keywords

AUC; bootstrap; classification; resampling; training variability

Indexed keywords

AUTOMATION; COMPUTER AIDED DIAGNOSIS; DIAGNOSIS; MEDICAL IMAGING; RANDOM PROCESSES; STATISTICAL TESTS; UNCERTAINTY ANALYSIS;

ANALYTICAL EXPRESSIONS; AUTOMATED CLASSIFIERS; BOOTSTRAP; PERFORMANCE ESTIMATION; RECEIVER OPERATING CHARACTERISTIC CURVES; RESAMPLING; TRADITIONAL APPROACHES; TRAINING AND TESTING;

CLASSIFICATION (OF INFORMATION);

EID: 79955761255     PISSN: 16057422     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1117/12.844605     Document Type: Conference Paper

References (21)

1. D. Bamber, "The area above the ordinal dominance graph and the area below the receiver operating characteristic curve," Journal of Mathematical Psychology 12, pp. 387 415, 1975.
2. J. A. Hanley and B. J. McNeil, "The meaning and use of the area under a receiver operating characteristic (ROC) curve," Radiology 143(1), pp. 29 36, 1982.
3. E. R. DeLong, D. M. DeLong, D. L. Clarke-Pearson, "Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach," Biometrics 44(3), pp. 837 845, 1988. 000634 lx Biometric Society.
4. A. P. Bradley, "The use of the area under the ROC curve in the evaluation of machine learning algorithms," Pattern Recognition 30(7), p. 1145, 1997.
5. K. Fukunaga and R. R. Hayes, "Effects of sample size in classifier design," Pattern Analysis and Machine Intelligence, IEEE Transactions on 11(8), pp. 873 885, 1989.
6. K. Fukunaga and R. R. Hayes, "Estimation of classifier performance," Pattern Analysis and Machine Intelligence, IEEE Transactions on 11(10), pp. 1087 1101, 1989.
7. K. Fukunaga, Introduction to statistical pattern recognition, Computer science and scientific computing, Academic Press, Boston, 2nd ed., 1990.
8. S. J. Raudys and A. K. Jam, "Small sample size effects in statistical pattern recognition: recommendations for practitioners," Pattern Analysis and Machine Intelligence, IEEE Transactions on 13(3), pp. 252 264, 1991.
9. A. K. Jam, R. P. W. Duin, M. Jianchang, "Statistical pattern recognition: a review," Pattern Analysis and Machine Intelligence, IEEE Transactions on 22(1), pp. 4 37, 2000.
10. S. V. Beiden, M. A. Maloof, R. F. Wagner, "A general model for finite-sample effects in training and testing of competing classifiers," Pattern Analysis and Machine Intelligence, IEEE Transactions on 25(12), pp. 1561 1569, 2003.
11. R. F. Wagner, W. A. Yousef, W. Chen, "Finite training of radiologists and statistical learning machines: Parallel lessons," in Advances in Medical Physics: 2008, A. B. Wolbarst, K. L. Mossman, W. R. Hendee, eds., Medical Physics Pub., Madison, WI, 2008.
12. R. F. Wagner, "From medical images to multiple-biomarker microarrays," Medical Physics 34(12), pp. 4944 4951, 2007.
13. D. D. Dorfman, K. S. Berbaum, C. E. Metz, "Receiver operating characteristic rating analysis-generalization to the population of readers and patients with the jackknife method," Investigative Radiology 27(9), pp. 723 731, 1992.
14. A. N. Obuchowski and H. E. Rockette, "Hypothesis testing of diagnostic accuracy for multiple readers and multiple tests: An ANOVA approach with dependent observations," Communications in Statistics-Simulation and Computation 24(2), pp. 285 308, 1995.
15. S. V. Beiden, R. F. Wagner, G. Campbell, "Components-of-variance models and multiple-bootstrap experiments: An alternative method for random-effects, receiver operating characteristic analysis," Academic Radiology 7(5), pp. 341 349, 2000.
16. H. H. Barrett, M. A. Kupinski, E. Clarkson, "Probabilistic foundations of the MRMC method," in Medical Imaging 2005: Image Perception, Observer Performance, Technology Assessment, 5749, p. 21, SPIE, (San Diego, CA, USA), 2005.
17. B. D. Gallas, "One-shot estimate of MRMC variance: AUC," Academic Radiology 13(3), p. 353, 2006.
18. B. D. Gallas, A. Bandos, F. W. Samuelson, R. F. Wagner, "A framework for random-effects ROC analysis: Biases with the bootstrap and other variance estimators," Communications in Statistics-Theory and Methods 38, pp. 2586 2603, 2009.
19. D. Bamber, "The area above the ordinal dominance graph arid the area below the receiver operating characteristic graph," J. Math. Psych. 12, pp. 387 415, 1975.
20. W. A. Yousef, R. F. Wagner, M. H. Loew, "Assessing classifiers from two independent data sets using ROC analysis: A nonparametric approach," Pattern Analysis and Machine Intelligence, IEEE Transactions on 28(11), pp. 1809 1817, 2006.
21. B. Efron and R. Tibshirani, "Improvements on cross-validation: The .632+ bootstrap method," Journal of the American Statistical Association 92(438), pp. 548 560, 1997.