Evaluating learning algorithms and classifiers

International Journal of Intelligent Information and Database Systems

Volumn 1, Issue 1, 2007, Pages 37-52

  Lavesson, Niklas ^a   Davidsson, Paul ^a  

^a BLEKINGE INSTITUTE OF TECHNOLOGY   (Sweden)

Author keywords

classification; evaluation; supervised learning

Indexed keywords

EID: 73649111816     PISSN: 17515858     EISSN: 17515866     Source Type: Journal    
DOI: 10.1504/IJIIDS.2007.013284     Document Type: Article

References (52)

1. Akaike, H. (1973) ‘Information theory and an extension of the maximum likelihood principle’, in Petrov, B.N. and Csáki, F. (Eds.): 2nd International Symposium on Information Theory, Akadémia Kiadó, Budapest, pp.267–281.
2. Anderson, E. (1935) ‘The irises of the Gaspé Peninsula’, Bulletin of the American Iris Society, Vol. 59, pp.2–5.
3. Andersson, A., Davidsson, P. and Lindén, J. (1999) ‘Measure-based classifier performance evaluation’, Pattern Recognition Letters, Vol. 20, No. 11–13, pp.1165–1173.
4. Bailey, T.L. and Elkan, C. (1993) ‘Estimating the accuracy of learned concepts’, 13th International Joint Conference on Artificial Intelligence, Chambéry, France, pp.895–901.
5. Burges, C.J.C. (1998) ‘A tutorial on support vector machines for pattern recognition’, Data Mining and Knowledge Discovery, Vol. 2, No. 2, pp.121–167.
6. Caruana, R. and Niculescu-Mizil, A. (2004) ‘Data mining in metric space: an empirical analysis of supervised learning performance criteria’, 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Seattle, Washington, USA, pp.69–78.
7. Dietterich, T.G. (1998) ‘Approximate statistical test for comparing supervised classification learning algorithms’, Neural Computation, Vol. 10, No. 7, pp.1895–1923.
8. Domingos, P. (1999) ‘The role of Occam’s razor in knowledge discovery’, Data Mining and Knowledge Discovery, Vol. 3, No. 4, pp.409–425.
9. Duda, R.O., Hart, P.E. and Stork, D.G. (2000) Pattern Classification, 2nd ed., John Wiley & Sons, USA.
10. Efron, B. (1983) ‘Estimating the error rate of a prediction rule: improvement on cross-validation’, American Statistical Association, Vol. 78, No. 382, pp.316–330.
11. Efron, B. and Tibshirani, R.J. (1993) ‘An introduction to the bootstrap’, Monographs on Statistics and Applied Probability, Chapman & Hall, New York City, New York, USA.
12. Egan, J.P. (1975) ‘Signal detection theory and ROC analysis’, Cognition and Perception, Academic Press, London.
13. Fawcett, T. (2003) ROC Graphs – Notes and Practical Considerations for Data Mining Researchers, Technical Report HPL-2003–2004, Intelligent Enterprise Technologies Laboratories, HP Laboratories, Palo Alto.
14. Giraud-Carrier, C., Vilalta, R. and Brazdil, P. (2004) ‘Introduction to the special issue on meta-learning’, Machine Learning, Vol. 54, No. 3, pp.187–193.
15. Gordon, D.F. and Desjardins, M. (1995) ‘Evaluation and selection of biases in machine learning’, Machine Learning, Vol. 20, pp.5–22.
16. Grünwald, P. (2005) ‘Minimum description length tutorial’, in Grünwald, P.I., Myung, J. and Pitt, M. (Eds.): Advances in Minimum Description Length – Theory and Applications, MIT Press, Cambridge, Massachusetts, USA, pp.23–81.
17. Hand, D.J. and Till, R.J. (2001) ‘A simple generalisation of the area under the ROC curve for multiple class classification problems’, Machine Learning, Vol. 45, pp.171–186.
18. Jain, A.K., Dubes, R.C. and Chen, C-C. (1987) ‘Bootstrap techniques for error estimation’, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 9, No. 5, pp.628–623.
19. King, R.D., Feng, C. and Sutherland, A. (1995) ‘STATLOG: comparison of classification algorithms on large real-world problems’, Applied Artificial Intelligence, Vol. 9, No. 3, pp.259–287.
20. Kohavi, R. (1995) ‘A study of cross-validation and bootstrap for accuracy estimation and model selection’, in Kaufmann, M. (Ed.): 14th International Joint Conference on Artificial Intelligence, Montréal, Québec, Canada, pp.1137–1145.
21. Kolmogorov, A.N. (1965) ‘Three approaches to the quantitative definition of information’, Problems in Information Transmission, Vol. 1, No. 1, pp.1–7.
22. Lavesson, N. and Davidsson, P. (2006) ‘Quantifying the impact of learning algorithm parameter tuning’, National Conference on Artificial Intelligence, AAAI Press, Boston, Massachusetts, USA, pp.395–400.
23. Mitchell, T.M. (1980) The Need for Biases in Learning Generalizations, Technical Report CBM-TR-117, Rutgers University, New Brunswick, New Jersey, USA.
24. Mitchell, T.M. (1997) ‘Machine learning’, Computer Science Series, McGraw-Hill, Singapore, international edition.
25. Murata, N., Yoshizawa, S. and Amari, S-I. (1994) ‘Network information criterion – determining the number of hidden units for an artificial neural network model’, IEEE Transactions on Neural Networks, Vol. 5, No. 6, pp.865–872.
26. Myung, J. and Pitt, M. (2002) Model Evaluation, Testing and Selection, Sage, London.
27. Nakhaeizadeh, G. and Schnabl, A. (1997) ‘Development of multi-criteria metrics for evaluation of data mining algorithms’, 3rd International Conference on Knowledge Discovery and Data Mining, Newport Beach, AAAI Press, Menlo Park, California, USA, pp.37–42.
28. Provost, F. and Fawcett, T. (1997) ‘Analysis and visualization of classifier performance: comparison under imprecise class and cost distributions’, 3rd International Conference on Knowledge Discovery and Data Mining, Huntington Beach, AAAI Press, Menlo Park, California, USA.
29. Provost, F., Fawcett, T. and Kohavi, R. (1998) ‘The case against accuracy estimation for comparing induction algorithms’, in Kaufmann, M. (Ed.): 15th International Conference on Machine Learning, Madison, WI, USA, pp.445–453.
30. Quenouille, M.H. (1956) ‘Notes on bias in estimation’, Biometrika, Vol. 61, No.353–360.
31. Raftery, A.E. (1986) ‘Choosing models for cross-classifications’, American Sociological Review, Vol. 51, pp.145–146.
32. Ratsaby, J., Meir, R. and Maiorov, V. (1996) ‘Towards robust model selection using estimation and approximation error bounds’, 9th Annual Conference on Computational Learning Theory, ACM Press, New York City, New York, USA, pp.57–67.
33. Rissanen, J. (1978) ‘Modeling by shortest data description’, Automatica, Vol. 14, pp.465–471.
34. Russell, S. and Norvig, P. (2003) ‘Artificial intelligence: a modern approach’, Prentice Hall Series in Artificial Intelligence, 2nd ed., Prentice-Hall, Upper Saddle River, New Jersey, USA.
35. Salzberg, S.L. (1999) ‘On comparing classifiers: a critique of current research and methods’, Data Mining and Knowledge Discovery, Vol. 1, pp.1–12.
36. Sathiya Keerthi, S., Chapelle, O. and DeCoste, D. (2006) ‘Building support vector machines with reduced classifier complexity’, Machine Learning Research, Vol. 7, pp.1493–1515.
37. Schaffer, C. (1993) ‘Overfitting avoidance as bias’, Machine Learning, Vol. 10, No. 2, pp.153–178.
38. Schaffer, C. (1994) ‘A conservation law for generalization performance’, 11th International Conference on Machine Learning, Morgan Kaufmann, New Brunswick, New Jersey, USA, pp.259–265.
39. Schuurmans, D. (1997) ‘A new metric-based approach to model selection’, 14th National Conference on Artificial Intelligence, AAAI Press, Providence, Rhode Island, USA, pp.552–558.
40. Schwartz, G. (1978) ‘Estimating the dimension of a model’, Annals of Statistics, Vol. 6, No. 2, pp.461–464.
41. Stone, M. (1974) ‘Cross-validatory choice and assessment of statistical predictions’, Royal Statistical Society, Vol. 36, pp.111–147.
42. Sugiyama, M. and Ogawa, H. (2001) ‘Subspace information criterion for model selection’, Neural Computation, Vol. 13, No. 8, pp.1863–1890.
43. Tornay, S.C. (1938) Ockham: Studies and Selections, Open Court Publishing Company, La Salle, Illinois, USA.
44. Vapnik, V. (1995) The Nature of Statistical Learning Theory, Springer-Verlag, New York, USA.
45. Vapnik, V. and Chervonenkis, A. (1971) ‘On the uniform convergence of relative frequencies of events to their probabilities’, Theory of Probability and its Applications, Vol. 16, pp.264–280.
46. Vilalta, R. and Drissi, Y. (2002) ‘Perspective view and survey of meta learning’, Artificial Intelligence Review, Vol. 18, No. 2, pp.77–95.
47. Weiss, G.M. and Provost, F. (2001) The Effect of Class Distribution on Classifier Learning: An Empirical Study, Technical Report ML-TR-44, Department of Computer Science, Rutgers University, New Brunswick, New Jersey, USA.
48. Witten, I.H. and Frank, E. (2005) Data Mining: Practical Machine Learning Tools and Techniques, 2nd ed., Morgan Kaufmann, San Francisco, California, USA.
49. Wolpert, D.H. (2001) The Supervised Learning no free Lunch Theorems, Technical Report, NASA Ames Research Center, Moffett Field, California, USA.
50. We assume that each V is a finite set but the framework as such is not limited in theory to this assumption.
51. This process is sometimes called model selection. The term, originating from statistics, is frequently but also ambiguously used in the machine learning community. Consequently, it is often followed by some definition, e.g., “the objective is to select a good classifier from a set of classifiers” (Kohavi, 1995), “a mechanism for […] selecting a hypothesis among a set of candidate hypotheses based on some pre-specified quality measure” (Ratsaby et al., 1996).
52. In terms of our framework, an overfitted classifier scores a very high accuracy on Ts but it has an unacceptably low accuracy when classifying other instances. The goal is to find a classifier with an acceptable accuracy for typical instances of the problem domain.