Classifier ensembles: Select real-world applications

Information Fusion

Volumn 9, Issue 1, 2008, Pages 4-20

  Oza, Nikunj C ^a   Tumer, Kagan ^b  

^a NASA AMES RESEARCH CENTER   (United States)

^b OREGON STATE UNIVERSITY   (United States)

Author keywords

Classifier ensembles; Ensemble applications

Indexed keywords

ALGORITHMS; DECISION THEORY; OBJECT RECOGNITION; PROBLEM SOLVING; REMOTE SENSING;

CLASSIFIER ENSEMBLES; ENSEMBLE APPLICATIONS; REAL WORLD APPLICATIONS; STATISTICAL CLASSIFICATIONS;

REAL TIME SYSTEMS;

EID: 35348915328     PISSN: 15662535     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.inffus.2007.07.002     Document Type: Article

References (107)

1. Face recognition grand challenge and the biometrics experimentation environment. Available from: .
2. A. Agogino, K. Tumer, Efficient agent-based cluster ensembles, in: Proceedings of the Fifth International Joint Conference on Autonomous Agents and Multi-Agent Systems, Hakodate, Japan, May 2006.
3. Battiti R., and Colla A.M. Democracy in neural nets: voting schemes for classification. Neural Networks 7 4 (1994) 691-709
4. Bauer E., and Kohavi R. An empirical comparison of voting classification algorithms: bagging, boosting, and variants. Machine Learning 36 (1999) 105-139
5. Benediktsson J.A., Sveinsson J.R., Ersoy O.K., and Swain P.H. Parallel consensual neural networks with optimally weighted outputs. Proceedings of the World Congress on Neural Networks (1994), INNS Press 129-137
6. Benediktsson J.A., Swain P.H., and Ersoy O.K. Neural network approaches versus statistical methods in classification of multisource remote sensing. IEEE Transactions of Geoscience and Remote Sensing 28 (1990) 540-552
7. Bishop C.M. Neural Networks for Pattern Recognition (1995), Oxford University Press, New York
8. L. Breiman, Stacked regression, Technical Report 367, Department of Statistics, University of California, Berkeley, 1993.
9. Breiman L. Bagging predictors. Machine Learning 24 2 (1996) 123-140
10. Breiman L., Friedman J.H., Olshen R.A., and Stone C.J. Classification and Regression Trees (1984), Wadsworth and Brooks, Pacific Grove, California
11. J.B.D. Cabrera, C. Gutierrez, R.K. Mehra, Ensemble methods for anomaly detection and distributed intrusion detection in mobile ad-hoc networks, Information Fusion (Special Issue on Applications of Ensemble Methods) in press, doi:10.1016/j.inffus.2007.03.001.
12. Cabrera J.B.D., Gutierrez C., and Mehra R.K. Infrastructures and algorithms for distributed anomaly-based intrusion detection in mobile ad-hoc networks. Proceedings of the IEEE Conference on Military Communications (2005), IEEE, Atlantic city, NJ, USA 1831-1837
13. Chan P.K., and Stolfo S.J. A comparative evaluation of voting and meta-learning on partitioned data. Proceedings of the Twelfth International Machine Learning Conference (1995), Morgan Kaufmann, Tahoe City, CA 90-98
14. Chawla N., and Bowyer K. Designing multiple classifier systems for face recognition. In: Oza N., Polikar R., Kittler J., and Roli F. (Eds). Proceedings of the Sixth International Workshop on Multiple Classifier Systems (2005), Springer, Berlin 407-416
15. Chellappa R., Wilson C., and Sirohey S. Human and machine recognition of faces: a survey. Proceedings of the IEEE vol. 835 (1995), Institute for Electrical and Electronics Engineers 705-740
16. Cordella L., Limongiello A., and Sansone C. Network intrusion detection by a multi-stage classification system. In: Kittler J., Roli F., and Windeatt T. (Eds). Proceedings of the Fifth International Workshop on Multiple Classifier Systems (2004), Springer, Berlin 324-333
17. Dhillon I.S., and Modha D.S. Concept decompositions for large sparse text data using clustering. Machine Learning 42 1 (2001) 143-175
18. Dietterich T.G. An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization. Machine Learning 40 (2000) 139-158
19. Drucker H., Cortes C., Jackel L.D., LeCun Y., and Vapnik V. Boosting and other ensemble methods. Neural Computation 6 6 (1994) 1289-1301
20. Drucker H., Schapire R., and Simard P. Improving performance in neural networks using a boosting algorithm. In: Hanson S.J., Cowan J.D., and Giles C.L. (Eds). Advances in Neural Information Processing Systems-5 (1993), Morgan Kaufmann 42-49
21. Duda R.O., Hart P.E., and Stork D.G. Pattern Classification. second ed. (2001), Wiley, New York, NY
22. Ekins S. Progress in predicting human ADME parameters in silico. Journal of Pharmacology Toxicology Methods 44 (2000) 251-272
23. Erdoǧan H., Erçil A., Ekenel H.K., Bilgin S.Y., Eden I., Kirisçi M., and Abut H. Multi-modal person recognition for vehicular applications. In: Oza N., Polikar R., Kittler J., and Roli F. (Eds). Proceedings of the Sixth International Workshop on Multiple Classifier Systems (2005), Springer, Berlin 366-375
24. Eskin E., Arnold A., Prerau M., Portnoy L., and Stolfo S. A geometric framework for unsupervised anomaly detection: detecting intrusions in unlabeled data. Applications of Data Mining in Computer Security (2002), Kluwer
25. Fauvel M., Chanuscot J., and Benediktsson J.A. Decision fusion for the classification of urban remote sensing images. IEEE Transactions of Geoscience and Remote Sensing 44 (2006) 2828-2838
26. Fierrez-Aguilar J., Garcia-Romero D., Ortega-Garcia J., and Gonzalez-Rodriguez J. Exploiting general knowledge in user-dependent fusion strategies for multimodal biometric verification. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2004) vol. 5 (2004), Institute for Electrical and Electronics Engineers 617-620
27. Fierrez-Aguilar J., Garcia-Romero D., Ortega-Garcia J., and Gonzalez-Rodriguez J. Bayesian adaptation for user-dependent multimodal biometric authentication. Pattern Recognition 38 8 (2005) 1317-1319
28. Fierrez-Aguilar J., Ortega-Garcia J., Garcia-Romero D., and Gonzalez-Rodriguez J. A comparative evaluation of fusion strategies for multimodal biometric verification. In: Kittler J., and Nixon M.S. (Eds). Fourth International Conference on Audio- and Video-Based Biometric Person Authentication (AVBPA) (2003), Springer, Berlin 830-837
29. Fierrez-Aguilar J., Garcia-Romero D., Ortega-Garcia J., and Gonzalez-Rodriguez J. Speaker verification using adapted user-dependent multilevel fusion. In: Oza N., Polikar R., Kittler J., and Roli F. (Eds). Proceedings of the Sixth International Workshop on Multiple Classifier Systems (2005), Springer, Berlin 356-365
30. P.J. Flynn, K.W. Bowyer, P.J. Phillips, Assessment of time dependency in face recognition, in: International Conference on Audio and Video Based Biometric Person Authentication, 2003, pp. 44-51.
31. Freund Y., and Schapire R. A decision-theoretic generalization of on-line learning and an application to boosting. Proceedings of the Second European Conference on Computational Learning Theory (1995), Springer-Verlag 23-37
32. Freund Y., and Schapire R. Experiments with a new boosting algorithm. Proceedings of the Thirteenth International Conference on Machine Learning (1996), Morgan Kaufmann, Bari, Italy 148-156
33. Freund Y., and Schapire R.E. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55 1 (1997) 119-139
34. J.H. Friedman. On bias, variance, 0/1-loss, and the curse-of-dimensionality, 1997. Available from: .
35. Fumera G., and Roli F. Analysis of linear and order statistics combiners for fusion of imbalanced classifiers. In: Roli F., and Kittler J. (Eds). Proceedings of the Third International Workshop on Multiple Classifier Systems. LNCS vol. 2364 (2002), Springer, Berlin 252-261
36. G. Giacinto, F. Roli, Ensembles of neural networks for soft classification of remote sensing images, in: Proceedings of the European Symposium on Intelligent Techniques, Bari, Italy, March 1997, pp. 166-170.
37. G. Giacinto, R. Perdisci, M. Del Rio, F. Roli, Intrusion detection in computer networks by a modular ensemble of one-class classifiers, Information Fusion (Special Issue on Applications of Ensemble Methods), in press, doi:10.1016/j.inffus.2006.10.002.
38. Gislason P.A., Benediktsson J.A., and Sveinsson J.R. Decision fusion for the classification of urban remote sensing images. Pattern Recognition Letters 27 (2006) 294-300
39. Granger C.W.J. Combining forecasts-twenty years later. Journal of Forecasting 8 3 (1989) 167-173
40. Hampshire J.B., and Waibel A.H. The Meta-Pi network: building distributed representations for robust multisource pattern recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 14 7 (1992) 751-769
41. Hansen L.K., and Salamon P. Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence 12 10 (1990) 993-1000
42. S. Hashem, B. Schmeiser, Approximating a function and its derivatives using MSE-optimal linear combinations of trained feedforward neural networks, in: Proceedings of the Joint Conference on Neural Networks, New Jersey, vol. 87, 1993, pp. I:617-620.
43. Hawkins D.M., Basak S.C., and Shi X. Qsar with few compounds and many features. Journal of Chemical Information and Computer Sciences 41 (2001) 663-670
44. Haykin S. Neural Networks: A Comprehensive Foundation (1994), Macmillan, New York
45. Ho T.K. The random space method for constructing decision forests. IEEE Transactions on Pattern Analysis and Machine Intelligence 20 8 (1998) 832-844
46. Ho T.K., Hull J.J., and Srihari S.N. Decision combination in multiple classifier systems. IEEE Transactions on Pattern Analysis and Machine Intelligence 16 1 (1994) 66-76
47. Huang P., Harris C., and Nixon M. Comparing different template features for recognizing people by their gait. In: Carter J., and Nixon M. (Eds). Proceedings of the British Machine Vision Conference (1998), British Machine Vision Association, Southampton, UK
48. Y.-A. Huang, W. Lee, A cooperative intrusion detection system for ad hoc networks, in: ACM Workshop on Security of Ad Hoc and Sensor Network (SASN03), 2003.
49. Jacobs R. Method for combining experts' probability assessments. Neural Computation 7 5 (1995) 867-888
50. Jain A.K., and Ross A. Learning user-specific parameters in a multibiometric system. Proceedings of the International Conference on Image Processing vol. 1 (2002), Institute for Electrical and Electronics Engineers 57-60
51. A. Kale, A. Rajagopalan, N. Cuntoor, V. Kruger, Gait-based recognition of humans using continuous HMMs, in: Proceedings of the International Conference on Face and Gesture Recognition, Washington, DC, USA, 2002, pp. 321-326.
52. Karypis G., and Kumar V. A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM Journal on Scientific Computing 20 1 (1998) 359-392
53. G. Karypis, R. Aggarwal, V. Kumar, S. Shekhar, Multilevel hypergraph partitioning: applications in VLSI domain, in: Proceedings of the Design and Automation Conference, 1997.
54. Kawaguchi N., Matsubara S., Kishida I., Irie Y., Murao H., Yamaguchi Y., Takeda K., and Itakura F. Construction and analysis of the multi-layered in-car spoken dialogue corpus. DSP in Vehicular and Mobile Systems (2005), Springer, New York Chapter 1
55. Kohonen T. Self-organized formation of topologically correct feature maps. Biological Cybernetics 43 (1982) 59-69
56. Krogh A., and Vedelsby J. Neural network ensembles, cross validation and active learning. In: Tesauro G., Touretzky D.S., and Leen T.K. (Eds). Advances in Neural Information Processing Systems-7 (1995), MIT Press 231-238
57. A. Kumar, D. Zhang, Integrating palmprint with face for user authentication, in: Proceedings of the Workshop on Multimodal User Authentication (MMUA 2003), 2003, pp. 107-112.
58. Kumar S., Ghosh J., and Crawford M. Hierarchical fusion of multiple classifiers for hyperspectral data analysis. Pattern Analysis and Applications 5 (2002) 210-220
59. Kuncheva L.I. A theoretical study on six classifier fusion strategies. IEEE Transactions on Pattern Analysis and Machine Intelligence 24 2 (2002) 281-286
60. M. Le Blanc, R. Tibshirani, Combining estimates in regression and classification, Technical report, University of Toronto, 1993.
61. K. Leung, C. Leckie, Unsupervised anomaly detection in network intrusion detection using clusters, in: Proceedings of the 28th Australasian Computer Science Conference (ACSC2005), 2005.
62. Li P., Chan K., Fu S., and Krishnan S.M. An abnormal ecg beat detection approach for long-term monitoring of heart patients based on hybrid kernel machine ensemble. In: Oza N.C., Polikar R., Kittler J., and Roli F. (Eds). Proceedings of the Sixth International Workshop on Multiple Classifier Systems (2005), Springer, Berlin 346-355
63. Lincoln W.P., and Skrzypek J. Synergy of clustering multiple back propagation networks. In: Touretzky D. (Ed). Advances in Neural Information Processing Systems-2 (1990), Morgan Kaufmann 650-657
64. Little J., and Boyd J. Recognizing people by their gait: the shape of motion. Videre 1 (1998) 1-32
65. Mani G. Lowering the variance of decisions by using artificial neural network portfolios. Neural Computation 3 (1991) 484-486
66. Mermelstein P., and Davis S.B. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Transactions on Acoustics, Speech, and Signal Processing 28 (1990) 357-366
67. Merz C.J., and Pazzani M.J. Combining neural network regression estimates with regularized linear weights. In: Mozer M.C., Jordan M.I., and Petsche T. (Eds). Advances in Neural Information Processing Systems-9 (1997), MIT Press 564-570
68. Nikulin A.E., Dolenko B., Bezabeh T., and Somorjai R. Near-optimal region selection for feature space reduction: novel preprocessing methods for classifying MR spectra. NMR in Biomedicine 11 (1998) 209-216
69. Nilsson N.J. Learning Machines: Foundations of Trainable Pattern-Classifying Systems (1965), McGraw-Hill, NY
70. Oza N.C., and Tumer K. Input decimated ensembles: decorrelation through dimensionality reduction. In: Kittler J., and Roli F. (Eds). Proceedings of the Second International Workshop on Multiple Classifier Systems (2001), Springer, Berlin 238-249
71. Parmanto B., Munro P.W., and Doyle H.R. Reducing variance of committee prediction with resampling techniques. Connection Science (Special Issue on Combining Artificial Neural Networks: Ensemble Approaches) 8 3-4 (1996) 405-426
72. Perrone M.P., and Cooper L.N. When networks disagree: ensemble methods for hybrid neural networks. In: Mammone R.J. (Ed). Neural Networks for Speech and Image Processing (1993), Chapman-Hall Chapter 10
73. Poh N., and Bengio S. An investigation of f-ratio client-dependent normalisation on biometric authentication tasks. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2005) vol. 1 (2005), Institute for Electrical and Electronics Engineers 721-724
74. Poh N., and Bengio S. EER of fixed and trainable fusion classifiers: a theoretical study with application to biometric authentication tasks. In: Oza N., Polikar R., Kittler J., and Roli F. (Eds). Proceedings of the Sixth International Workshop on Multiple Classifier Systems (2005), Springer, Berlin 74-85
75. L. Portnoy, E. Eskin, S. Stolfo, Intrusion detection with unlabeled data using clustering, in: Proceedings of ACM CSS Workshop on Data Mining Applied to Security (DMSA-2001), 2001.
76. Prackeviciene E., Baumgartner R., and Somorjai R. Using domain knowledge in the random subspace method: application to the classification of biomedical spectra. In: Oza N.C., Polikar R., Kittler J., and Roli F. (Eds). Proceedings of the Sixth International Workshop on Multiple Classifier Systems (2005), Springer, Berlin 336-345
77. Rajan S., and Ghosh J. Exploiting class hierarchies for knowledge transfer in hyperspectral data. In: Oza N.C., Polikar R., Kittler J., and Roli F. (Eds). Proceedings of the Sixth International Workshop on Multiple Classifier Systems (2005), Springer, Berlin 417-428
78. Richard M.D., and Lippmann R.P. Neural network classifiers estimate Bayesian a posteriori probabilities. Neural Computation 3 4 (1991) 461-483
79. Ruck D.W., Rogers S.K., Kabrisky M.E., Oxley M.E., and Suter B.W. The multilayer perceptron as an approximation to a Bayes optimal discriminant function. IEEE Transactions on Neural Networks 1 4 (1990) 296-298
80. Samal A., and Iyengar P. Automatic recognition and analysis of human faces and facial expressions: a survey. Pattern Recognition 25 1 (1992) 65-77
81. Sarhan A.E., and Greenberg B.G. Estimation of location and scale parameters by order statistics from singly and doubly censored samples. Annals of Mathematical Statistics Science 27 (1956) 427-451
82. Scholkopf B., Platt J.C., Shawe-Taylor J., Smola A.J., and Williamson R.C. Estimating the support of a high-dimensional distribution. Neural Computation 13 (2001) 1443-1471
83. Serpico S.B., Pellegretti P., and Bruzzone L. Feature-selection for remote-sensing data classification. Image and Signal Processing for Remote Sensing 2315 (1994), SPIE, Rome, Italy 569-577
84. In: Sharkey A.J.J. (Ed). Connection Science: Special Issue on Combining Artificial Neural Networks: Ensemble Approaches 8 3-4 (1996)
85. P.A. Shoemaker, M.J. Carlin, R.L. Shimabukuro, C.E. Priebe, Least squares learning and approximation of posterior probabilities on classification problems by neural network models, in: Proc. 2nd Workshop on Neural Networks, WNN-AIND91, Auburn, February 1991, pp. 187-196.
86. N. Short, Remote sensing tutorial, 2000. Available from: .
87. Snelick R., Uludag U., Mink A., Indovina M., and Jain A. Large scale evaluation of multimodal biometric authentication using state-of-the-art systems. IEEE Transactions on Pattern Analysis and Machine Intelligence 27 (2005) 450-455
88. Strehl A., and Ghosh J. Cluster ensembles - a knowledge reuse framework for combining partitionings. Proceedings of AAAI 2002 (2002), AAAI, Edmonton, Canada 93-98
89. J. Suutala, J. Roning, Methods for person identification on a pressure-sensitive floor: experiments with multiple classifiers and reject option, Information Fusion (Special Issue on Applications of Ensemble Methods), in press, doi:10.1016/j.inffus.2006.11.003.
90. Svetnik V., Liaw A., Tong C., and Wang T. Application of Breiman's random forest to modeling structure-activity relationships of pharmaceutical molecules. In: Kittler J., Roli F., and Windeatt T. (Eds). Proceedings of the Fifth International Workshop on Multiple Classifier Systems (2004), Springer, Berlin 334-343
91. Tax D.M.J., and Duin R.P.W. Support vector data description. Machine Learning 54 1 (2004) 45-66
92. Toh K.A., Jiang X., and Yau W.Y. Exploiting local and global decisions for multi-modal biometrics verification. IEEE Transactions on Signal Processing 52 (2004) 3059-3072
93. Tumer K., and Ghosh J. Order statistics combiners for neural classifiers. Proceedings of the World Congress on Neural Networks (1995), INNS Press, Washington, DC 31-34
94. Tumer K., and Ghosh J. Analysis of decision boundaries in linearly combined neural classifiers. Pattern Recognition 29 2 (1996) 341-348
95. Tumer K., and Ghosh J. Error correlation and error reduction in ensemble classifiers. Connection Science (Special Issue on Combining Artificial Neural Networks: Ensemble Approaches) 8 3-4 (1996) 385-404
96. Tumer K., and Ghosh J. Linear and order statistics combiners for pattern classification. In: Sharkey A.J.C. (Ed). Combining Artificial Neural Nets: Ensemble and Modular Multi-Net Systems (1999), Springer-Verlag, London 127-162
97. Tumer K., and Ghosh J. Robust order statistics based ensembles for distributed data mining. In: Kargupta H., and Chan P. (Eds). Advances in Distributed and Parallel Knowledge Discovery (2000), AAAI/MIT Press 185-210
98. Tumer K., and Ghosh J. Robust combining of disparate classifiers through order statistics. Pattern Analysis and Applications 5 (2002) 189-200
99. K. Tumer, Linear and order statistics combiners for reliable pattern classification, Ph.D. thesis, The University of Texas, Austin, TX, May 1996.
100. Tumer K., and Oza N.C. Input decimated ensembles. Pattern Analysis and Applications 6 (2003) 65-77
101. Weiser M., and Brown J. The coming age of calm technology. Beyond Calculation: The Next Fifty Years of Computing (1997), Springer-Verlag, New York
102. Wolpert D.H. Stacked generalization. Neural Networks 5 (1992) 241-259
103. Wolpert D.H. On bias plus variance. Neural Computation 9 6 (1997) 1211-1243
104. Xu L., Krzyzak A., and Suen C.Y. Methods of combining multiple classifiers and their applications to handwriting recognition. IEEE Transactions on Systems, Man and Cybernetics 22 3 (1992) 418-435
105. S. Yu, Feature selection and classifier ensembles: a study of hyperspectral remote sensing, Ph.D. thesis, Universiteit Antwerpen, 2003.
106. Y. Zhang, W. Lee, Intrusion detection in wireless ad-hoc networks, in: Proceedings of the 6th International Conference on Mobile Computing and Networking, August 2000.
107. Zhang Y., Lee W., and Huang Y.-A. Intrusion detection techniques for mobile wireless networks. Wireless Networks 9 5 (2003) 545-556