A General Model for Finite-Sample Effects in Training and Testing of Competing Classifiers

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 25, Issue 12, 2003, Pages 1561-1569

  Beiden, Sergey V ^a   Maloof, Marcus A ^b   Wagner, Robert F ^a  

^a CENTER FOR DRUG EVALUATION AND RESEARCH   (United States)

^b GEORGETOWN UNIVERSITY   (United States)

Author keywords

Bootstrap methods; Classifier design and evaluation; Components of variance models; Discriminant analysis; Pattern recognition; ROC analysis

Indexed keywords

COMPONENTS-OF-VARIANCE MODELS; DISCRIMINANT ANALYSIS; RECEIVER OPERATING CHARACTERISTIC (ROC);

CLASSIFICATION (OF INFORMATION); COMPUTER SIMULATION; CORRELATION METHODS; FINITE ELEMENT METHOD; MATHEMATICAL MODELS; MONTE CARLO METHODS; PROBLEM SOLVING; RANDOM PROCESSES; SET THEORY;

PATTERN RECOGNITION;

EID: 0347252304     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2003.1251149     Document Type: Article

References (29)

1. K. Fukunaga, Introduction to Statistical Pattern Recognition, second ed. Boston: Academic Press, 1990.
2. K. Fukunaga and R.R. Hayes, "Effects of Sample Size in Classifier Design," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 11, no. 8, pp. 873-885, Aug. 1989.
3. K. Fukunaga and R.R. Hayes, "Estimation of Classifier Performance," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 11, no. 10, pp. 1087-1101, Oct. 1989.
4. S.J. Raudys and A.K. Jain, "Small Sample Size Effects in Statistical Pattern Recognition: Recommendations for Practitioners," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, pp. 252-264, 1991.
5. A.K. Jain, R.P.W. Duin, and J. Mao, "Statistical Pattern Recognition: A Review," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 1, pp. 4-37, Jan. 2000.
6. S.J. Raudys, Statistical and Neural Classifiers: An Integrated Approach to Design. New York: Springer-Verlag, 2001.
7. H.-P. Chan, B. Sahiner, R.F. Wagner, and N. Petrick, "Classifier Design for Computer-Aided Diagnosis: Effects of Finite Sample Size on the Mean Performance of Classical and Neural Network Classifiers," Medical Physics, vol. 26, no. 12, pp. 2654-2668, 1999.
8. D.D. Dorfman, K.S. Berbaum, and C.E. Metz, "Receiver Operating Characteristic Rating Analysis: Generalization to the Population of Readers and Patients with the Jackknife Method," Investigative Radiology, vol. 27, pp. 723-731, 1992.
9. S.V. Beiden, R.F. Wagner, and G. Campbell, "Components-of-Variance Models and Multiple-Bootstrap Experiments: An Alternative Method for Random-Effects Receiver Operating Characteristic Analysis," Academic Radiology, vol. 7, pp. 341-349, 2000.
10. S.V. Beiden, R.F. Wagner, G. Campbell, C.E. Metz, and Y. Jiang, "Components-of-Variance Models for Random-Effects ROC Analysis: The Case of Unequal Variance Structures across Modalities," Academic Radiology, vol. 8, pp. 605-615, 2001.
11. S.V. Beiden, R. F. Wagner, G. Campbell, and H.-P. Chan, "Analysis of Uncertainties of Estimates of Variance Components in Multivariate ROC Analysis," Academic Radiology, vol. 8, pp. 616-622, 2001.
12. C.E. Metz, "ROC Methodology in Radiologic Imaging," Investigative Radiology, vol. 21, pp. 720-733, 1986.
13. C.E. Metz, "Some Practical Issues of Experimental Design and Data Analysis in Radiological ROC Studies," Investigative Radiology, vol. 24, pp. 234-245, 1989.
14. Y. Jiang, C.E. Metz, and R.M. Nishikawa, "A Receiver Operating Characteristic Partial Area Index for Highly Sensitive Diagnostic Tests," Radiology, vol. 201, pp. 745-750, 1996.
15. C.E. Metz, "Statistical Analysis of ROC Data in Evaluating Diagnostic Performance," Multiple Regression Analysis: Applications in the Health Sciences, D. Herbert and R. Meyers, eds. New York: Am. Inst. of Physics: Am. Assoc. of Physicists in Medicine, pp. 365-384, 1986.
16. C.E. Metz, Y. Jiang, H. MacMahon, R.M. Nishikawa, and X. Pan, "ROC Software," Kurt Rossmann Laboratories for Radiologic Image Research, Univ. of Chicago, http://www.radiology.uchicago.edu/krl/roc_soft.htm, 2003.
17. C.A. Roe and C.E. Metz, "Variance-Component Modeling in the Analysis of Receiver Operating Characteristic Index Estimates," Academic Radiology, vol. 4, pp. 587-600, 1997.
18. B. Efron and R.J. Tibshirani, An Introduction to the Bootstrap. New York: Chapman and Hall, 1993.
19. T. Hastie, R.J. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction. New York: Springer-Verlag, 2001.
20. M.A. Maloof, S.V. Beiden, and R.F. Wagner, "Analysis of Competing Classifiers in Terms of Components of Variance of ROC Accuracy Measures," Technical Report CS-02-01, Dept. of Computer Science, Georgetown Univ., Washington, D.C., http://www.cs.georgetown.edu/maloof/pubs/cstr-02-01.pdf, Jan. 2002.
21. L.P. Clarke, B.Y. Croft, E. Staab, H. Baker, and D.C. Sullivan, "National Cancer Institute Initiative: Lung Image Database Resource for Imaging Research," Academic Radiology, vol. 8, pp. 447-450, 2001.
22. L.E. Dodd et al., "An Overview of Assessment Methodologies and Related Statistical Issues for Computer-Assist Modalities in Lung Imaging: A Status Report of the Lung Image Database Consortium (LIDC)," Academic Radiology, to be submitted, July 2003.
23. A.P. Bradley, "The Use of the Area under the ROC Curve in the Evaluation of Machine Learning Algorithms," Pattern Recognition, vol. 30, no. 7, pp. 1145-1159, 1997.
24. B. Efron, The Jackknife, the Bootstrap, and Other Resampling Plans. Philadelphia: SIAM, 1982.
25. B. Efron, "Estimating the Error Rate of an Prediction Rule: Improvement on Cross-Validation," J. Am. Statistical Assoc., vol. 78, no. 382, pp. 316-331, 1983.
26. G.J. McLachlan, Discriminant Analysis and Statistical Pattern Recognition. New York: John Wiley & Sons, 1992.
27. B.D. Ripley, Pattern Recognition and Neural Networks. Cambridge, U.K.: Cambridge Univ. Press, 1996.
28. B. Efron and R.J. Tibshirani, "Improvements on Cross-Validation: The .632+ Bootstrap Method," J. Am. Statistical Assoc.: Theory and Methods, vol. 92, no. 438, pp. 548-560, 1997.
29. R.F. Wagner, H.-P. Chan, B. Sahiner, N. Petrick, and J.T. Mossoba, "Components of Variance in ROC Analysis of CADx Classifier Performance. II: Applications of the Bootstrap," Proc. SPIE, vol. 3661, pp. 523-532, 1999.