Warning: Statistical benchmarking is addictive. Kicking the habit in machine learning

Journal of Experimental and Theoretical Artificial Intelligence

Volumn 22, Issue 1, 2010, Pages 67-80

  Drummond, Chris ^a   Japkowicz, Nathalie ^b  

^a NATIONAL RESEARCH COUNCIL CANADA   (Canada)

^b UNIVERSITY OF OTTAWA   (Canada)

Author keywords

Algorithm evaluation; Benchmarking; Machine learning; Null hypothesis tests

Indexed keywords

ALGORITHM EVALUATION; ALGORITHM PERFORMANCE; DATA SETS; MACHINE LEARNING COMMUNITIES; MACHINE-LEARNING; NULL HYPOTHESIS; SET OF RULES;

BENCHMARKING; LEARNING SYSTEMS; STATISTICAL TESTS;

LEARNING ALGORITHMS;

EID: 77951200774     PISSN: 0952813X     EISSN: 13623079     Source Type: Journal    
DOI: 10.1080/09528130903010295     Document Type: Article

References (48)

1. Bay, S.D., Kibler, D., Pazzani, M.J., and Smyth, P. (2000), 'The UCI KDD Archive of Large Data Sets for Data Mining Research and Experimentation', SIGKDD Explorations, 2(2), 81-85.
2. Blake, C.L., and Merz, C.J. (1998), UCI Repository of Machine Learning Databases, Irvine, CA, USA: University of California. www.ics.uci.edu/-mlearn/ MLRepository.html
3. Bradley, A.P. (1997), 'The Use of the Area Under the ROC Curve in the Evaluation of Machine Learning Algorithms', Pattern Recognition, 30(7), 1145-1159.
4. Brumfiel, G. (2002). Physicist Found Guilty of Misconduct. Published online in Nature DOI:10.1038/news020923-9.
5. Buja, A., and Cook, D. (1999). Inference for Data Visualization. Talk given at the Joint Statistical Meetings.
6. Caruana, R., and Niculescu-Mizil, A. (2004), 'Data Mining in Metric Space: An Empirical Analysis of Supervised Learning Performance Criteria', in Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, pp. 69-78.
7. Caruana, R., and Niculescu-Mizil, A. (2006), 'An Empirical Comparison of Supervised Learning Algorithms', in Proceedings of the Twenty-Third International Conference on Machine Learning, Banff, Alberta, Canada, pp. 161-168.
8. Chow, S.L. (1998), 'Precise of Statistical Significance: Rationale, Validity, and Utility', Behavioral and Brain Sciences, 21, 169-239.
9. Cohen, J. (1994), 'The Earth is Round (p4.05)', American Psychologist, 49, 997-1003.
10. Cohen, P., and Howe, A. (1988), 'How Evaluation Guides AI Research', AI Magazine, 9(4), 35-43.
11. Couzin, J. (2006), 'Breakthrough of the Year: Breakdown of the Year: Scientific Fraud', Science, 314(5807), p. 1853.
12. Davis, J., and Goadrich, M. (2006), 'The Relationship Between Precision-recall and ROC Curves', in Proceedings of the Twenty Third International Conference on Machine Learning, pp. 233-240.
13. Drummond, C., Elazmeh, W., and Japkowicz, N. (eds.) (2006), Proceedings of the Twenty-first National Conference on Artificial Intelligence: Workshop on Evaluation Methods for Machine Learning. American Association for Artificial Intelligence Technical Report WS-06-06, Menlo Park, CA, USA.
14. Drummond, C., Elazmeh, W., Japkowicz, N., and Macskassy, S.A. (eds.) (2008), Proceedings of the Twenty-Fifth International Conference on Machine Learning: Evaluation Methods for Machine Learning III. http://www.site.uottawa. ca/ICML08WS
15. Drummond, C., and Holte, R.C. (2005), 'Learning to Live with False Alarms', in Proceedings of the Eleventh International Conference on Knowledge Discovery and Data Mining: Workshop on Data Mining Methods for Anomaly Detection, pp. 21-24.
16. Drummond, C., and Holte, R.C. (2006), 'Cost Curves: An Improved Method for Visualizing Classifier Performance', Machine Learning, 65(1), 95-130. (Pubitemid 44451195)
17. Evgeniou, T., Pontil, M., and Elisseeff, A. (2004), 'Leave-one-out Error, Stability, and Generalisation of Voting Combination of Classifiers', Machine Learning, 55(1), 71-97.
18. Ferri, C., Flach, P., Herná ndez-Orallo, J., and Lachiche, N. (eds.) (2004), Proceedings of the Sixteenth European Conference on Artificial Intelligence: Workshop on ROC Analysis in AI. http://www.dsic.upv.es/-flip/ ROCAI2004/accepted.html
19. Giles, J. (2004). Scientists Behaving Badly. Published online in Nature DOI:10.1038/news040301-9.
20. Greiner, R., Silver, B., Becker, S., and Gruninger, M. (1988), 'A Review of Machine Learning at AAAI-87', Machine Learning, 3(1), 79-92.
21. Hagen, R.L. (1997), 'In Praise of the Null Hypothesis Statistical Test', American Psychologist, 52, 15-24.
22. Hagood, M.J. (1941), 'The Notion of the Hypothetical Universe', in The Significance Test Controversy: A Reader (Chap. 4), eds. Denton E. Morrison and Ramon E. Henkel, Chicago: Aldine, pp. 65-78.
23. Hand, D.J. (2006), 'Classifier Technology and the Illusion of Progress', Statistical Science, 21(1), 1-15.
24. Hettich, S., and Bay, S.D. (1999), The UCI KDD Archive. Irvine, CA, USA: Department of Information and Computer Science, University of California. http://kdd.ics.uci.edu
25. Holte, R.C. (1993), 'Very Simple Classification Rules Perform Well on Most Commonly Used Datasets', Machine Learning, 11(1), 63-91.
26. Huang, J., and Ling, C.X. (2007), 'Constructing New and Better Evaluation Measures for Machine Learning', in Proceedings of the Twentieth International Joint Conference on Artificial Intelligence, pp. 859-864.
27. Japkowicz, N. (2006), 'Why Question Machine Learning Evaluation Methods? (An illustrative review of the shortcomings of current methods)', in Proceedings of the Twenty-First National Conference on Artificial Intelligence: Workshop on Evaluation Methods for Machine Learning, AAAI Technical Report WS-06-06, pp. 6-11. (Pubitemid 46053316)
28. Kibler, D., and Langley, P. (1998), 'Machine Learning as an Experimental Science', in Proceedings of the Third European Working Session on Learning, pp. 81-92.
29. Kodratoff, Y. (1994), 'Guest Editor's Introduction: The Comprehensibility Manifesto', AI Communications, 7(2), 83-85.
30. Kuhn, T. (1962), The Structure of Scientific Revolutions, Chicago: University of Chicago Press.
31. Langley, P. (1988), 'Machine Learning as an Experimental Science', Machine Learning, 3, 5-8.
32. Meyer, M., and Vlachos, P. (1989), Statlib. Pittsburgh, PA, USA: Department of Statistics, Carnegie Mellon University. http://lib.stat.cmu.edu/
33. Michalski, R.S. (1973), 'Discovering Classification Rules using Variable-valued Logic System Vl1', in Proceedings of the Third International Joint Conference on Artificial Intelligence, pp. 162-172.
34. Michalski, R.S. (1983), 'A Theory and Methodology of Inductive Learning', Artificial Intelligence, 20, 111-161.
35. Mitchell, T. (1977), 'Version Spaces: A Candidate Elimination Approach to Rule Learning', in Proceedings of the Fifth International Joint Conference on Artificial Intelligence, pp. 305-310.
36. Narasimhamurthy, A., and Kuncheva, L. (2007), 'A Framework for Generating Data to Simulate Changing Environments', in Proceedings of the Twenty-Fifth IASTED International Multi-Conference on Artificial Intelligence and Applications, pp. 384-389.
37. Oblinger, D., Lau, T., Gil, Y., and Bauer, M. (eds.) (2005), Proceedings of the Twentieth National Conference on Artificial Intelligence: Workshop on Human Comprehensible Machine Learning. American Association for Artificial Intelligence. Technical Report WS-05-104 Menlo Park, CA, USA.
38. Pazzani, M., Merz, C., Murphy, P., Ali, K., Hume, T., and Brunk, C. (1994), 'Reducing Misclassification Costs', in Proceedings of the Eleventh International Conference on Machine Learning, pp. 217-225.
39. Platt, J.R. (1964), 'Strong Inference', Science, 146(3642), 347-353.
40. Provost, F., and Fawcett, T. (2001), 'Robust Classification for Imprecise Environments', Machine Learning, 42, 203-231. (Pubitemid 32188799)
41. Provost, F., Fawcett, T., and Kohavi, R. (1998), 'The Case Against Accuracy Estimation for Comparing Induction Algorithms', in Proceedings of the Fifteenth International Conference on Machine Learning, pp. 43-48.
42. Saitta, L., and Neri, F. (1998), 'Learning in the 'Real World', Machine Learning, 30(2-3), 133-163.
43. Salzberg, S.L. (1997), 'On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach', Data Mining and Knowledge Discovery, 1, 317-327.
44. Simon, H.A. (1993), 'Artificial Intelligence as an Experimental Science', in Proceedings of the Eleventh National Conference on Artificial Intelligence, p. 853.
45. Simon, H.A. (1995), 'Artificial Intelligence: An Empirical Science', Artificial Intelligence, 77, 95-127.
46. Thearling, K. and Stein, R. (eds.) (1998), Proceedings of the Fourth International Conference on Knowledge Discovery and Data Mining: Workshop on Keys to the Commercial Success of Data Mining. http://www.thearling.com/ workshop/workshop.htm
47. Tsang, D. (2000). Social Science Data Archive Libraries, Irvine, CA, USA: University of California. http://data.lib.uci.edu/
48. Tukey, J.W. (1977), Exploratory Data Analysis. Reading, MA, USA: Addison-Wesley.