The use of the area under the ROC curve in the evaluation of machine learning algorithms

Pattern Recognition

Volumn 30, Issue 7, 1997, Pages 1145-1159

  Bradley, Andrew P ^a  

^a UNIVERSITY OF ALBERTA   (Canada)

Author keywords

Accuracy measures; Cross validation; Standard error; The area under the ROC curve (AUC); The ROC curve; Wilcoxon statistic

Indexed keywords

RECEIVER OPERATING CHARACTERISTICS (ROC);

DECISION THEORY; LEARNING ALGORITHMS; MEDICAL APPLICATIONS;

LEARNING SYSTEMS;

EID: 0031191630     PISSN: 00313203     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0031-3203(96)00142-2     Document Type: Article

References (36)

1. K. Fukunaga, Introduction to Statistical Pattern Recognition, 2nd Edn. Academic Press, San Diego, California (1990).
2. C. W. Therrien, in Decision Estimation and Classification: An Introduction to Pattern Recognition and Related Topics. Wiley, New York (1989).
3. R. E. Walpole and R. H. Myers, Probability and Statistics for Engineers and Scientists. Macmillan, New York (1990).
4. D. D. Dorfmann and E. Alf, Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals - rating method data, J. Math. Psychology 6, 487-496 (1969).
5. J. A. Swets, ROC analysis applied to the evaluation of medical imaging techniques, Invest. Radiol. 14, 109-121 (1979).
6. J. A. Hanley and B. J. McNeil, The meaning and use of the area under a receiver operating characteristic (ROC) curve, Radiology 143, 29-36 (1982).
7. J. A. Hanley and B. J. McNeil, A method of comparing the areas under receiver operating characteristic curves derived from the same cases, Radiology 148, 839-843 (1983).
8. J. H. Friedman, An overview of predictive learning and function approximation, From statistics to neural networks: Theory and pattern recognition applications, V. Cherkassky, J. H. Friedman and H. Wechsler, eds, NATO ASI Series: Computer and Systems Sciences, Vol. 136, pp. 1-61. Springer, Berlin (1993).
9. R. Walker, P. Jackway, B. Lovell and D. Longstaff, Classification of cervical cell nuclei using morphological segmentation and textural feature extraction, Proc. 2nd Australian and New Zealand Conf. on Intelligent Information Systems, pp. 297-301 (1994).
10. M. J. Ray, G. A. T. Hawson, S. J. E. Just, G. McLachlan and M. O'Brien, Relationship of platelet aggregation to bleeding after cardiopulmonary bypass, Ann. Thoracic Surgery 57, 981-986 (1994).
11. W. H. Wolberg and O. L. Mangasarian, Multisurface method of pattern separation for medical diagnosis applied to breast cytology, Proc. Nat. Acad. Sci., U.S.A. #87#12, 9193-9196 (1990).
12. J. W. Smith, J. E. Everhart, W. C. Dickson, W. C. Knowler and R. S. Johannes, Using the (ADAP) learning algorithm to forecast the onset of diabetes mellitus, Proc. Symp. on Computer Applications and Medical Care, pp.261-265. IEEE Computer Society Press, Silver Spring, Maryland (1988).
13. R. Detrano, A. Janosi, W. Steinbrunn, M. Pfisterer, J. Schmid, S. Sandu, K. Guppy, S. Lee and V. Froelicher, International application of a new probability algorithm for the diagnosis of coronary artery disease, Am. J. Cardiol. 64, 304-310 (1989).
14. J. H. Gennari, P. Langley and D. Fisher, Models of incremental conceot formation, Artif. Intell. 40, 11-61 (1989).
15. B. C. Lovell and A. P. Bradley, The multiscale classifier, IEEE Trans. Pattern Analysis Mach. Intell. 18, 124-137 (1996).
16. P. A. Devijver and J. Kittler, Pattern Recognition: A Statistical Approach. Prentice-Hall, London (1982).
17. T. W. Rauber, M. M. Barata and A. S. Steiger-Garcao, A Toolbox for the Analysis and Visualisation of Sensor Data in Supervision, Intelligent Robots Group Technical report, Universidade Nova de Lisboa, Portugal (1993).
18. J. T. Tou and R. C. Gonzalez, Pattern Recognition Principles. Addison-Wesley, Reading, Massachusetts (1981).
19. D. J. Hand, Discrimination and Classification. Wiley, New York (1981).
20. J. R. Quinlan, C4.5:Programs for Machine Learning. Morgan Kaufmann, San Mateo (1993).
21. M. L. Minsky and S. A. Papert, Perceptrons. MIT Press, Cambridge, Massachusetts (1969).
22. D. E. Rumelhart, G. E. Hinton and R. J. Williams, Learning Internal Representations by Error Propagation, Parallel Distributed Computing'. Explorations in the Microstructure of Cognition, D. E. Rumelhart and J. L. McClelland, eds. MIT Press, Cambridge, Massachusetts (1986).
23. F. Rosenblatt, Principles of Neurodynamics. Spartan Press, Washington D.C. (1961).
24. J. M. Twomey and A. E. Smith, Power curves for pattern classification networks, Proc. IEEE Int. Conf. on Neural Networks, San Francisco, California, pp. 950-955 (1993).
25. D. Nguyen and B. Widrow, Improving the Learning Speed of 2-Layer Neural networks by Choosing Initial Values of Adaptive Weights, Proc. Int. Joint Conf. on Neural Networks 3, 21-26 (1990).
26. S. M. Weiss and C. A. Kulikowski, Computer Systems That Learn: Classification and Prediction Methods from Statistics, Neural Networks, Machine Learning and Expert Systems. Morgan Kaufmann, San Mateo (1991).
27. L. Breiman, J. Friedman, R. Olshen and C. Stone, Classification and Regression Trees. Wadsworth, Belmont (1984).
28. J. H. Friedman, Introduction to computational learning and statistical prediction, Tutorial, Twelth Int. Conf. on Machine Learning, Lake Tahoe, California (1995).
29. B. Efron, Bootstrap methods: Another look at the jackknife, Ann. Statist. 7, 1-26 (1979).
30. J. A. Swets and R. M. Pickets, Evaluation of Diagnostic Systems: Methods from Signal Detection Theory. Academic Press, New York (1982).
31. C. R. Hicks, Fundamental Concepts in the Design of Experiments. Saunders College Publishing, London (1993).
32. A. P. Bradley, B. C. Lovell, M. Ray and G. Hawson, On the methodology for comparing learning algorithms: A case study, in Proc. Second Australian and New Zealand Conf. on Intelligent Information Systems, pp. 37-41. IEEE Publications, Brisbane, Australia (1994).
33. C. Schaffer, Overfitting avoidance as bias, Machine Learning 10, 153-178 (1993).
34. R. F. Raubertas, L. E. Rodewald, S. G. Humiston and P. G. Szilagyi, ROC Curves for Classification Trees, Med. Decision Making 14, 169-174 (1994).
35. D. M. Green and J. A. Swets, Signal Detection Theory and Psychophysics. Wiley, New York (1966, 1974).
36. B. PicinBono, On deflection as a performance criterion in dection, IEEE Trans. Aerospace Electronic Systems 31, 1072-1081 (1995).