The impact of overfitting and overgeneralization on the classification accuracy in data mining

Soft Computing for Knowledge Discovery and Data Mining

Volumn , Issue , 2008, Pages 391-431

  Pham, Huy Nguyen Anh ^a   Triantaphyllou, Evangelos ^a  

^a Louisiana State University   (United States)

Author keywords

classification; false negative; falsepositive; homogeneity degree; homogenous set; optimization; overfitting; overgeneralization; prediction

Indexed keywords

EID: 84890008245     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1007/978-0-387-69935-6_16     Document Type: Chapter

References (61)

1. Abdi, H., (2003), "A neural network primer," Journal of Biological Systems, vol. 2, pp. 247-281.
2. Ali, K., C. Brunk, and M. Pazzani, (1994), "On learning multiple descriptions of a concept," Proceedings of Tools with Artificial Intelligence, New Orleans, LA, USA, pp. 476-483.
3. Artificial Neural Network Toolbox 6.0 and Statistics Toolbox 6.0, Matlab Version 7.0, website: http://www.mathworks.com/products/
4. Boros, E., P. L. Hammer, and J. N. Hooker, (1994), "Predicting Cause-Effect Rela-tionships from Incomplete Discrete Observations," Journal on Discrete Mathe-matics, vol. 7, no. 4, pp. 531-543.
5. Bracewell, R., (1999), "The Impulse Symbol," Chapter 5 in The Fourier Transform and Its Applications, 3rd ed. New York: McGraw-Hill, pp. 69-97.
6. Breiman, L., (1996), "Bagging predictors," Journal of Machine Learning, vol. 24, pp. 123-140.
7. Breiman, L., (2001), "Random forests," Journal of Machine Learning, vol. 45, no. 1, pp. 5-32.
8. Breiman, L., J. H. Friedman, R. A. Olshen, and C. J. Stone, (1984), "Classification and Regression Trees," Chapman Hall/CRC Publisher, pp. 279-293.
9. Byvatov, E., and G. Schneider, (2003), "Support vector machine applications in bioinformatics," Journal of Application Bioinformatics, vol. 2, no.2, pp. 67-77.
10. Clark, P., and R. Boswell, (1991), "Rule induction with CN2: Some recent improve-ments," Y. Kodratoff, editor, Machine Learning -EWSL-91, Berlin, Springer-Verlag, pp. 151-163.
11. Clark, P., and T. Niblett, (1989), "The CN2 Algorithm," Journal of Machine Learn-ing, vol. 3, pp. 261-283.
12. Cohen S., L. Rokach, O. Maimon, (2007), "Decision-tree instance-space decomposi-tion with grouped gain-ratio,", Information Science, Volume 177, Issue 17, pp. 3592-3612.
13. Cohen, W. W., (1995), "Fast effective rule induction," Machine Learning: Pro-ceedings of the Twelfth International Conference, Tahoe City, CA., USA, pp. 115-123.
14. Cortes, C., and V. Vapnik, (1995), "Support-vector networks," Journal of Machine Learning, vol. 20, no. 3, pp. 273-297.
15. Cover, T. M., and P. E. Hart, (1967), "Nearest Neighbor Pattern Classification," Institute of Electrical and Electronics Engineers Transactions on Information Theory, vol. 13, no. 1, pp. 21-27.
16. Cristianini, N., and S. T. John, (2000), "An Introduction to Support Vector Ma-chines and other kernel-based learning methods," Cambridge University Press.
17. Dasarathy, B. V., and B. V. Sheela, (1979), "A Composite Classifier System Design: Concepts and Methodology," Proceedings of the IEEE, vol. 67, no. 5, pp. 708-713.
18. Dietterich, T. G., and G. Bakiri, (1994), "Solving multiclass learning problems via error-correcting output codes," Journal of Artificial Intelligence Research, vol. 2, pp. 263-286.
19. Duda, R. O., and P. E. Hart, (1973), "Pattern Classification and Scene Analysis," Wiley Publisher, pp. 56-64.
20. Duda. O. R., E. H. Peter, G. S. David , (2001), "Pattern Classification," Chapter 4: Nonparametric Techniques in Wiley Interscience Publisher, pp. 161-199.
21. Dudani, S., (1976), "The Distance-Weighted k-Nearest-Neighbor Rule," IEEE Transactions on Systems, Man and Cybernetics, vol. 6, no. 4, pp. 325-327.
22. Friedman, N., D. Geiger, and M. Goldszmidt, (1997), "Bayesian Network Classi-fiers," Journal of Machine Learning, vol. 29, pp. 131-161.
23. Geman, S., E. Bienenstock, and R. Doursat, (1992), "Neural Networks and the Bias/Variance Dilemma," Journal of Neural Computation, vol. 4, pp. 1-58.
24. Hecht-Nielsen, R., (1989), "Theory of the Backpropagation neural Network," In-ternational Joint Conference on neural networks, Washington, DC, USA, pp. 593-605.
25. Huzefa, R., and G. Karypis, (2005), "Profile Based Direct Kernels for Remote Homology Detection and Fold Recognition," Journal of Bioinformatics, vol. 31, no. 23, pp. 4239-4247.
26. Karp, R. M., (1972), "Reducibility Among Combinatorial Problems," Proceedings of Sympos. IBM Thomas J. Watson Res. Center, Yorktown Heights, New York: Plenum, pp. 85-103.
27. Keller, J. M., M. R. Gray, and J. A. Givens, Jr, (1985), "A Fuzzy K-Nearest Neigh-bor Algorithm," Journal of IEEE Transactions on Systems, Man, and Cyber-netics, vol. 15, no. 4, pp. 580-585.
28. Kohavi R., (1996), "Scaling up the accuracy of naive-Bayes classifiers: a decision-tree hybrid," Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, Portland, OR, USA, pp. 202-207.
29. Kohavi, R., and G. John, (1997), "Wrappers for Feature Subset Selection," Journal of Artificial Intelligence: special issue on relevance, vol. 97, no. 1-2, pp. 273-324.
30. Kokol, P., M. Zorman, M. M. Stiglic, and I. Malcic, (1998), "The limitations of decision trees and automatic learning in real world medical decision making," Proceedings of the 9th World Congress on Medical Informatics MEDINFO'98, vol. 52, pp. 529-533.
31. Kononenko, I., (1991), "Semi-naïve Bayesian classifier," Y. Kodratoff Editor, Pro-ceedings of sixth European working session on learning, Springer-Verlag, pp. 206-219.
32. Kwok, S., and C. Carter, (1990), "Multiple decision trees: uncertainty," Journal of Artificial Intelligence, vol.4, pp. 327-335.
33. Langley, P., and S. Sage, (1994), "Induction of Selective Bayesian Classifiers," Proceedings of UAI-94, Seattle, WA, USA, pp. 399-406.
34. Mansour, Y., D. McAllester, (2000), "Generalization Bounds for Decision Trees," Proceedings of the 13th Annual Conference on Computer Learning Theory, San Francisco, Morgan Kaufmann, USA, pp. 69-80.
35. Moody, J. E., (1992), "The Effective Number of Parameters: An Analysis of Gen-eralization and Regularization in Nonlinear Learning Systems," Journal of Ad-vances in Neural Information Processing Systems, vol. 4, pp. 847-854.
36. Nock, R., and O. Gascuel, (1995), "On learning decision committees," Proceed-ings of the Twelfth International Conference on Machine Learning, Morgan Kaufmann, Taho City, CA., USA, pp. 413-420.
37. Oliver, J. J., and D. J.Hand, (1995), "On pruning and averaging decision trees," Proceedings of the Twelfth International Conference on Machine Learning, Morgan Kaufmann, Taho City, CA., USA, pp. 430-437.
38. Pazzani, M.J., (1995), "Searching for dependencies in Bayesian classifiers," Pro-ceedings of AI STAT'95, pp. 239-248.
39. Podgorelec, V., P. Kokol, B. Stiglic, and I. Rozman, (2002), "Decision trees: an overview and their use in medicine," Journal of Medical Systems, Kluwer Aca-demic/Plenum Press, vol. 26, no. 5, pp. 445-463
40. Quinlan, J. R., (1987), "Simplifying decision trees," International Journal of Man-Machine Studies, vol. 27, pp. 221-234.
41. Quinlan, J. R., (1993), "C4.5: Programs for Machine Learning," Morgan Kaufmann Publisher San Mateo, CA., USA, pp. 35-42.
42. Rada, M., (2004), "Seminar on Machine Learning," a presentation of a course taught at University of North Texas.
43. Rokach L., O. Maimon, O. Arad, (2005), "Improving Supervised Learning by Sample Decomposition," Journal of Computational Intelligence and Applications, vol. 5, no. 1, pp. 37-54.
44. Sands D., (1998), "Improvement theory and its applications," Gordon A. D., and A. M. Pitts Editors, Higher Order Operational Techniques in Semantics, Pub-lications of the Newton Institute, Cambridge University Press, pp. 275-306.
45. Schapire, R. E, (1990), "The strength of weak learnability," Journal of Machine Learning, vol. 5, pp. 197-227.
46. Shawe-Taylor. J., and C. Nello, (1999), "Further results on the margin distribution," Proceedings of COLT99, Santa Cruz, CA., USA, pp. 278-285.
47. Smith, M., (1996), "Neural Networks for Statistical Modeling," Itp New Media Publisher, ISBN 1-850-32842-0, pp. 117-129.
48. Spizer, M., L. Stefan, C. Paul, S. Alexander, and F. George, (2006), "IsoSVM { Distinguishing isoforms and paralogs on the protein level," Journal of BMC Bioinformatics, vol. 7:110
49. website: http://www.biomedcentral.com/content/pdf/1471-2105-7-110.pdf.
50. Tan, P. N., S. Michael, and K. Vipin, (2005), "Introduction to Data Mining," Chap-ters 4 and 5, Addison-Wesley Publisher, pp. 145-315.
51. Triantaphyllou, E., (2007), "Data Mining and Knowledge Discovery Via a Novel Logic-Based Approach," A monograph, Springer, Massive Computing Series, 420 pages, (in print).
52. Triantaphyllou, E., and G. Felici, (Editors), (2006), "Data Mining and Knowledge Discovery Approaches Based on Rule Induction Techniques," Springer, Massive Computing Series, 796 pages.
53. Triantaphyllou, E., L. Allen, L. Soyster, and S. R. T. Kumara, (1994), "Generating Logical Expressions From Positive and Negative Examples via a Branch-and-Bound approach," Journal of Computers and Operations Research, vol. 21, pp. 783-799.
54. Vapnik, V., (1998), "Statistical Learning Theory," Wiley Publisher, pp. 375-567.
55. Webb, G. I., (1996), "Further experimental evidence against the utility of Occam's razor," Journal of Artificial Intelligence Research, vol. 4, pp. 397-417.
56. Webb, G. I., (1997), "Decision Tree Grafting," Proceedings of the 15th Interna-tional Joint Conference on Artificial Intelligence (IJCAI'97), vol. 2, pp. 23-29.
57. Weigend, A., (1994), "On overfitting and the effective number of hidden units," Proceedings of the 1993 Connectionist Models Summer School, pp. 335-342.
58. Wikipedia Dictionary, (2007), website: http://en.wikipedia.org/wiki/ Homogenous.
59. Wolpert, D. H, (1992), "Stacked generalization," Journal of Neural Networks, vol. 5, pp. 241-259.
60. Zavrsnik, J., P. Kokol, I. Maleiae, K. Kancler, M. Mernik, and M. Bigec, (1995), "ROSE: decision trees, automatic learning and their applications in cardiac medicine," MEDINFO'95, Vancouver, Canada, pp. 201-206.
61. Zhou Z. and C. Chen, (2002), "Hybrid decision tree," Journal of Knowledge-Based Systems, vol. 15, pp. 515 -528.