Empirical analysis and evaluation of approximate techniques for pruning regression bagging ensembles

Neurocomputing

Volumn 74, Issue 12-13, 2011, Pages 2250-2264

  Hernández Lobato, Daniel ^a   Martínez Muñoz, Gonzalo ^b   Suárez, Alberto ^b  

^a UNIVERSITÉ CATHOLIQUE DE LOUVAIN   (Belgium)

^b UNIVERSIDAD AUTÓNOMA DE MADRID   (Spain)

Author keywords

Bagging; Boosting; Ensemble learning; Ensemble pruning; Regression; Semidefinite programming

Indexed keywords

BAGGING; BOOSTING; ENSEMBLE LEARNING; ENSEMBLE PRUNING; REGRESSION; SEMI-DEFINITE PROGRAMMING;

ALGORITHMS; MATHEMATICAL PROGRAMMING; REGRESSION ANALYSIS;

COST BENEFIT ANALYSIS;

ANALYSIS OF COVARIANCE; ANALYSIS OF VARIANCE; ANALYTICAL ERROR; ARTICLE; BOOTSTRAPPING; COMPUTER PREDICTION; LEARNING ALGORITHM; MACHINE LEARNING; PRIORITY JOURNAL; PROCESS OPTIMIZATION; REGRESSION ANALYSIS; SYSTEM ANALYSIS;

EID: 79956208533     PISSN: 09252312     EISSN: 18728286     Source Type: Journal    
DOI: 10.1016/j.neucom.2011.03.001     Document Type: Article

References (57)

1. Avnimelech R., Intrator N. Boosting regression estimators. Neural Computation 1999, 11(2):499-520.
2. Banfield R.E., Hall L.O., Bowyer K.W., Kegelmeyer W.P. Ensemble diversity measures and their application to thinning. Information Fusion 2005, 6(1):49-62.
3. Bauer E., Kohavi R. An empirical comparison of voting classification algorithms: bagging, boosting, and variants. Machine Learning 1999, 36(1-2):105-139.
4. C.L. Blake, C.J. Merz, UCI repository of machine learning databases 〈〉, 1998. http://www.ics.uci.edu/~mlearn/MLRepository.html.
5. Borchers B. CSDP: a C library for semidefinite programming. Optimization Methods and Software 1999, 11:613-1623.
6. Breiman L. Bagging predictors. Machine Learning 1996, 24(2):123-140.
7. L. Breiman, Bias, variance, and arcing classifiers, Technical Report 460, Statistics Department, University of California, 1996.
8. Breiman L. Heuristics of instability and stabilization in model selection. The Annals of Statistics 1996, 24(6):2350-2383.
9. Breiman L. Arcing classifiers. The Annals of Statistics 1998, 26(3):801-849.
10. L. Breiman, Using adaptive bagging to debias regressions, Technical Report, Statistics Department, University of California 1999.
11. Breiman L. Random forests. Machine Learning 2001, 45(1):5-32.
12. Brown G., Wyatt J.L., Tino P. Managing diversity in regression ensembles. Journal of Machine Learning Research 2005, 6:1621-1650.
13. Caruana R., Niculescu-Mizil A. Ensemble selection from libraries of models. Proceedings of the 21st International Conference on Machine Learning 2004, ACM Press, New York, NY, USA.
14. Chambers J.M., Hastie T.J. Statistical Models in S 1992, Chapman & Hall, London.
15. Cormen T.H., Leiserson C.H., Rivest R.L. Introduction to Algorithms 1990, The MIT Press.
16. Demšar J. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 2006, 7:1-30.
17. Dietterich T.G. An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization. Machine Learning 2000, 40(2):139-157.
18. Drucker H. Improving regressors using boosting techniques. Proceedings of the 14th International Conference on Machine Learning 1997, 107-115. Morgan Kaufmann.
19. Freund Y., Schapire R.E. A decision-theoretic generalization of on-line learning and an application to boosting. Proceedings of the Second European Conference on Computational Learning Theory 1995, 23-37.
20. Friedman J.H. Multivariate adaptive regression splines. The Annals of Statistics 1991, 19(1):1-67.
21. Friedman J.H. Greedy function approximation: a gradient boosting machine. The Annals of Statistics 2001, 29(5):1189-1232.
22. Garey M.R., Johnson D.S. Computers and Intractability; A Guide to the Theory of NP-Completeness 1990, W. H. Freeman & Co., New York, NY, USA.
23. Goemans M.X., Williamson D.P. Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the Association for Computing Machinery 1995, 42(6):1115-1145.
24. Q. Han, Y. Ye, J. Zhang, An improved rounding method and semidefinite programming relaxation for graph partition, Technical Report, Iowa City, Iowa 52242, 2000.
25. Hastie T., Tibshirani R., Friedman J.H. The Elements of Statistical Learning 2001, Springer.
26. Hernández-Lobato D., Martínez-Muñoz G., Suárez A. Pruning in ordered regression bagging ensembles. IEEE International Joint Conference on Neural Networks 2006, 1266-1273.
27. Krogh A., Hertz J.A. A simple weight decay can improve generalization. Advances in Neural Information Processing Systems 1992, vol. 4:950-957. Morgan Kaufmann Publishers, Inc.
28. Kung F.H. Fitting logistic growth curve with predetermined carrying capacity. Proceedings of the Statistical Computing Section, American Statistical Association 1986, 340-343.
29. Liu Y., Yao X. Ensemble learning via negative correlation. Neural Networks 1999, 12(10):1399-1404.
30. Margineantu D.D., Dietterich T.G. Pruning adaptive boosting. Proceedings of the 14th International Conference on Machine Learning 1997, 211-218. Morgan Kaufmann.
31. Martínez-Muñoz G., Hernández-Lobato D., Suárez A. An analysis of ensemble pruning techniques based on ordered aggregation. IEEE Transactions on Pattern Analysis and Machine Intelligence 2009, 31:245-259.
32. Martínez-Muñoz G., Sánchez-Martínez A., Hernández-Lobato D., Suárez A. Class-switching neural network ensembles. Neurocomputing 2008, 71(13-15):2521-2528.
33. Martínez-Muñoz G., Suárez A. Aggregation ordering in bagging. Proceedings of the IASTED International Conference on Artificial Intelligence and Applications 2004, 258-263. Acta Press.
34. Martínez-Muñoz G., Suárez A. Pruning in ordered bagging ensembles. ICML '06: Proceedings of the 23rd International Conference on Machine Learning 2006, 609-616. ACM Press, New York, NY, USA.
35. Martínez-Muñoz G., Suárez A. Using boosting to prune bagging ensembles. Pattern Recognition Letters 2007, 28(1):156-165.
36. Miller A. Subset Selection in Regression 2002, Chapman & Hall, CRC.
37. Nanni L., Lumini A. Ensemble generation and feature selection for the identification of students with learning disabilities. Expert Systems with Applications 2009, 36(2, Part2):3896-3900.
38. Nocedal J., Wright S.J. Numerical Optimization 1999, Springer.
39. Perrone M.P., Cooper L.N. When networks disagree. ensemble methods for hybrid neural networks. Neural Networks for Speech and Image Processing 1993, 126-142. Chapman & Hall.
40. Politis D., Romano J., Wolf M. Subsampling 1999, Springer-Verlag, New York.
41. Quinlan J.R. Bagging, boosting, and C4.5. Proceedings of the 13th National Conference on Artificial Intelligence, Cambridge, MA 1996, 725-730.
42. R Development Core Team, R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria, ISBN 3-900051-07-0, 2005.
43. Reid S., Grudic G. Regularized linear models in stacked generalization. Multiple Classifier Systems, Lecture Notes in Computer Science 2009, vol. 5519:112-121. Springer.
44. Rodríguez J.J., Kuncheva L.I., Alonso C.J. Rotation forest: a new classifier ensemble method. IEEE Transactions on Pattern Analysis and Machine Intelligence 2006, 28(10):1619-1630.
45. Ruta D., Gabrys B. Classifier selection for majority voting. Information Fusion 2005, 6(1):63-81.
46. Schapire R.E. The strength of weak learnability. Machine Learning 1990, 5:197-227.
47. Tibshirani R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B (Methodological) 1996, 58(1):267-288.
48. Ting K.M., Witten I.H. Issues in stacked generalization. Journal of Artificial Intelligence Research 1999, 10:271-289.
49. Ueda N., Nakano R. Generalization error of ensemble estimators. Proceedings of International Conference on Neural Networks 1996, 90-95.
50. Venables W.N., Ripley B.D. Modern Applied Statistics with S 2002, Springer, New York, ISBN 0-387-95457-0. fourth ed.
51. Wilcoxon F. Individual comparisons by ranking methods. Biometrics Bulletin 1968, 1(6):80-83.
52. Witten I.H., Frank E. Data Mining: Practical Machine Learning Tools and Techniques 2005, Morgan Kaufmann, San Francisco. second ed.
53. Wolpert D.H. Stacked generalization. Neural Networks 1992, 5(2):241-259.
54. Zhang C.-X., Zhang J.-S. Rotboost: a technique for combining rotation forest and Adaboost. Pattern Recognition Letters 2008, 29(10):1524-1536.
55. Zhang Y., Burer S., Street W.N. Ensemble pruning via semi-definite programming. Journal of Machine Learning Research 2006, 7:1315-1338.
56. Zhou Z.-H., Wu J., Tang W. Ensembling neural networks: many could be better than all. Artificial Intelligence 2002, 137(1-2):239-263.
57. Zou H., Hastie T. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society B 2005, 67:301-320.