Classifier technology and the illusion of progress

Statistical Science

Volumn 21, Issue 1, 2006, Pages 1-14

  Hand, David J ^a  

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

Author keywords

Empirical comparisons; Error rate; Flat maximum effect; Misclassification rate; Population drift; Principle of parsimony; Problem uncertainty; Selectivity bias; Simplicity; Supervised classification

Indexed keywords

EID: 33745886270     PISSN: 08834237     EISSN: None     Source Type: Journal    
DOI: 10.1214/088342306000000060     Document Type: Review

References (47)

1. ADAMS, N. M. and HAND, D. J. (1999). Comparing classifiers when the misallocation costs are uncertain. Pattern Recognitions 32 1139-1147.
2. BENTON, T. C. (2002). Theoretical and empirical models. Ph.D. dissertation, Dept. Mathematics, Imperial College London.
3. BREIMAN, L. (2001). Statistical modeling: The two cultures (with discussion). Statist. Sci. 16 199-231.
4. DUIN, R. P. W. (1996). A note on comparing classifiers. Pattern Recognition Letters 17 529-536.
5. EFRON, B. (2001). Comment on "Statistical modeling: The two cultures," by L. Breiman. Statist. Sci. 16 218-219.
6. FAWCETT, T. and PROVOST, F. (1997). Adaptive fraud detection. Data Mining and Knowledge Discovery 1 291-316.
7. FISHER, R. A. (1936). The use of multiple measurements in taxonomic problems. Annals of Eugenics 7 179-188.
8. FRIEDMAN, C. P. and WYATT, J. C. (1997). Evaluation Methods in Medical Informatics. Springer. New York.
9. FRIEDMAN, J. H. (1997). On bias, variance, 0/1 loss, and the curse of dimensionality. Data Mining and Knowledge Discovery 1 55-77.
10. GALLAGHER, J. C., HEDLUND, L. R., STONER, S. and MEEGER, C. (1988). Vertebral morphometry: Normative data. Bone and Mineral 4 189-196.
11. HAND, D. J. (1981). Discrimination and Classification. Wiley, Chichester.
12. HAND, D. J. (1996). Classification and computers: Shifting the focus. In COMPSTAT-96: Proceedings in Computational Statistics (A. Prat, ed.) 77-88. Physica, Berlin.
13. HAND, D. J. (1997). Construction and Assessment of Classification Rules. Wiley, Chichester.
14. HAND, D. J. (1998). Strategy, methods, and solving the right problems. Comput. Statist. 13 5-14.
15. HAND, D. J. (1999). Intelligent data analysis and deep understanding. In Causal Models and Intelligent Data Management (A. Gammerman, ed.) 67-80. Springer, Berlin.
16. HAND, D. J. (2001). Modelling consumer credit risk. IMA J. Management Mathematics 12 139-155.
17. HAND, D. J. (2001). Reject inference in credit operations. In Handbook of Credit Scoring (E. Mays, ed.) 225-240. Glenlake, Chicago.
18. HAND, D. J, (2004). Academic obsessions and classification realities: Ignoring practicalities in supervised classification. In Classification, Clustering and Data Mining Applications (D. Banks, L. House, F. R. McMorris, P. Arabie and W. Gaul, eds.) 209-232. Springer, Berlin.
19. HAND, D. J. (2005). Supervised classification and tunnel vision. Applied Stochastic Models in Business and Industry 21 97-109.
20. HAND, D. J. and HENLEY, W. E. (1997). Statistical classification methods in consumer credit scoring: A review. J. Roy. Statist. Soc. Ser. A 160 523-541.
21. HOADLEY, B. (2001). Comment on "Statistical modeling: The two cultures," by L. Breiman. Statist. Sci. 16 220-224.
22. HOLTE, R. C. (1993). Very simple classification rules perform well on most commonly used datasets. Machine Learning 11 63-90.
23. JAMAIN, A. (2004). Meta-analysis of classification methods. Ph.D. dissertation, Dept. Mathematics, Imperial College London.
24. JAMAIN, A. and HAND, D. J. (2005). Mining supervised classification performance studies: A meta-analytic investigation. Technical report, Dept. Mathematics, Imperial College London.
25. KELLY, M. G and HAND, D. J. (1999). Credit scoring with uncertain class definitions. IMA J. Mathematics Management 10331-345.
26. KELLY, M. G., HAND, D. J. and ADAMS, N. M. (1998). Defining the goals to optimise data mining performance. In Proc. Fourth International Conference on Knowledge Discovery and Data Mining (R. Agrawal, P. Stolorz and G. Piatetsky-Shapiro, eds.) 234-238. AAAI Press, Menlo Park, CA.
27. KELLY, M. G., HAND, D. J. and ADAMS, N. M. (1999). The impact of changing populations on classifier performance. In Proc. Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (S. Chaudhuri and D. Madigan, eds.) 367-371. ACM, New York.
28. KELLY, M. G., HAND, D. J. and ADAMS, N. M. (1999). Supervised classification problems: How to be both judge and jury. In Advances in Intelligent Data Analysis. Lecture Notes in Comput. Sci. 1642 235-244. Springer, Berlin.
29. KLINKENBERG, R. and THORSTEN, J. (2000). Detecting concept drift with support vector machines. In Proc. 17th International Conference on Machine Learning (P. Langley, ed.) 487-494. Morgan Kaufmann, San Francisco.
30. LANE, T. and BRODLEY, C. E. (1998). Approaches to online learning and concept drift for user identification in computer security. In Proc. Fourth International Conference on Knowledge Discovery and Data Mining (R. Agrawal, P. Stolorz and G. Piatetsky-Shapiro, eds.) 259-263. AAAI Press, Menlo Park, CA.
31. LEWIS, E. M. (1990). An Introduction to Credit Scoring. Athena, San Rafael, CA.
32. LI, H. G. and HAND, D. J. (2002). Direct versus indirect credit scoring classifications. J. Operational Research Society 53 647-654.
33. McLACHLAN, G. J. (1992). Discriminant Analysis and Statistical Pattern Recognition. Wiley, New York.
34. MINGERS, J. (1989). An empirical comparison of pruning methods for decision tree induction. Machine Learning 4 227-243.
35. NEWMAN, D. J., HETTICH, S., BLAKE, C. L. and MERZ, C. J. (1998). UCI repository of machine learning databases. Dept, Information and Computer Sciences, Univ. California, Irvine. Available at www.ics.uci.edu/-mlearn/ MLRepository.html.
36. PROVOST, F. and FAWCETT, T. (2001). Robust classification for imprecise environments. Machine Learning 42 203-231.
37. RENDELL, A. L. and SESHU, R. (1990). Learning hard concepts through constructive induction: Framework and rationale. Computational Intelligence 6 247-270.
38. RIPLEY, B. D. (1996). Pattern Recognition and Neural Networks. Cambridge Univ. Press.
39. ROSENBERG, E. and GLEIT, A. (1994). Quantitative methods in credit management: A survey. Oper. Res. 42 589-613.
40. SALZBERG, S. L. (1997). On comparing classifiers: Pitfalls to avoid and a recommended approach. Data Mining and Knowledge Discovery 1 317-328.
41. SHAVLIK, J., MOONEY, R. J. and TOWELL, G. (1991). Symbolic and neural learning algorithms: An experimental comparison. Machine Learning 6 111-143.
42. THOMAS, L. C. (2000). A survey of credit and behavioural scoring: Forecasting financial risk of lending to consumers. International J. Forecasting 16 149-172.
43. VON WINTERFELDT, D. and EDWARDS, W. (1982). Costs and payoffs in perceptual research. Psychological Bulletin 91 609-622.
44. WEBB, A. R. (2002). Statistical Pattern Recognition, 2nd ed. Wiley, Chichester.
45. WEISS, S. M., GALEN, R. S. and TADEPALLI, P. V. (1990). Maximizing the predictive value of production rules. Artificial Intelligence 45 47-71.
46. WIDMER, G. and KUBAT, M. (1996). Learning in the presence of concept drift and hidden contexts. Machine Learning 2369-101.
47. ZAHAVI, J. and LEVIN, N. (1997). Issues and problems in applying neural computing to target marketing. J. Direct Marketing 11(4) 63-75.