Ensemble based systems in decision making

IEEE Circuits and Systems Magazine

Volumn 6, Issue 3, 2006, Pages 21-44

  Polikar, Robi ^a  

^a Rowan University   (United States)

Author keywords

Classifier combination; Classifier diversity; Classifier fusion; Classifier selection; Data fusion; Incremental learning; Multiple classifier systems

Indexed keywords

ALGEBRA; ALGORITHMS; COMPUTATIONAL METHODS; DECISION MAKING; ERROR CORRECTION; LEARNING SYSTEMS; PARAMETER ESTIMATION;

CLASSIFIER COMBINATION; CLASSIFIER DIVERSITY; CLASSIFIER FUSION; CLASSIFIER SELECTION; DATA FUSION; INCREMENTAL LEARNING; MULTIPLE CLASSIFIER SYSTEMS;

EMBEDDED SYSTEMS;

EID: 33748611921     PISSN: 1531636X     EISSN: None     Source Type: Journal    
DOI: 10.1109/MCAS.2006.1688199     Document Type: Review

References (96)

1. B.V. Dasarathy and B.V. Sheela, "Composite classifier system design: Concepts and methodology," Proceedings of the IEEE, vol. 67, no. 5, pp. 708-713, 1979.
2. L.K. Hansen and P. Salamon, "Neural network ensembles," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, no. 10, pp. 993-1001, 1990.
3. R.E. Schapire, "The strength of weak learnability," Machine Learning, vol. 5, no. 2, pp. 197-227, 1990.
4. R.A. Jacobs, M.I. Jordan, S.J. Nowlan, and G.E. Hinton, "Adaptive mixtures of local experts," Neural Computation, vol. 3, pp. 79-87, 1991.
5. M.J. Jordan and R.A. Jacobs, "Hierarchical mixtures of experts and the EM algorithm," Neural Computation, vol. 6, no. 2, pp. 181-214, 1994.
6. D.H. Wolpert, "Stacked generalization," Neural Networks, vol. 5, no. 2, pp. 241-259, 1992.
7. J.A. Benediktsson and P.H. Swain, "Consensus theoretic classification methods," IEEE Trans. on Systems, Man and Cybernetics, vol. 22, no. 4, pp. 688-704, 1992.
8. L. Xu, A. Krzyzak, and C.Y. Suen, "Methods of combining multiple classifiers and their applications to handwriting recognition," IEEE Transactions on Systems, Man and Cybernetics, vol. 22, no. 3, pp. 418-435, 1992.
9. T.K. Ho, J.J. Hull, and S.N. Srihari, "Decision combination in multiple classifier systems," IEEE Trans. on Pattern Analy. Machine Intel., vol. 16, no. 1, pp. 66-75, 1994.
10. G. Rogova, "Combining the results of several neural network classifiers," Neural Networks, vol. 7, no. 5, pp. 777-781, 1994.
11. L. Lam and C.Y. Suen, "Optimal combinations of pattern classifiers," Pattern Recognition Letters, vol. 16, no. 9, pp. 945-954, 1995.
12. K. Woods, W.RJ. Kegelmeyer, and K. Bowyer, "Combination of multiple classifiers using local accuracy estimates," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 4, pp. 405-410, 1997.
13. L.I. Kuncheva, "Change-glasses' approach in pattern recognition," Pattern Recognition Letters, vol. 14, no. 8, pp. 619-623, 1993.
14. I. Bloch, "Information combination operators for data fusion: A comparative review with classification," IEEE Transactions on Systems, Man, and Cybernetics Part A: Systems and Humans., vol. 26, no. 1, pp. 52-67, 1996.
15. S.B. Cho and J.H. Kim, "Combining multiple neural networks by fuzzy integral for robust classification," IEEE Transactions on Systems, Man and Cybernetics, vol. 25, no. 2, pp. 380-384, 1995.
16. L.I. Kuncheva, J.C. Bezdek, and R. Duin, "Decision templates for multiple classifier fusion: An experimental comparison," Pattern Recognition, vol. 34, no. 2, pp. 299-314, 2001.
17. H. Drucker, C. Cortes, L.D. Jackel, Y. LeCun, and V. Vapnik, "Boosting and other ensemble methods," Neural Computation, vol. 6, no. 6, pp. 1289-1301, 1994.
18. R. Battiti and A.M. Colla, "Democracy in neural nets: Voting schemes for classification," Neural Networks, vol. 7, no. 4, pp. 691-707, 1994.
19. L.I. Kuncheva, "Classifier ensembles for changing environments," 5th Int. Workshop on Multiple Classifier Systems, Lecture Notes in Computer Science, F. Roli, J. Kittler, and T. Windeatt, Eds., vol. 3077, pp. 1-15, 2004.
20. F. Smieja, "Pandemonium system of reflective agents," IEEE Transactions on Neural Networks, vol. 7, no. 1, pp. 97-106, 1996.
21. L.I. Kuncheva, Combining Pattern Classifiers, Methods and Algorithms, New York, NY: Wiley Interscience, 2005.
22. E. Alpaydin and M.I. Jordan, "Local linear perceptrons for classification," IEEE Transactions on Neural Networks, vol. 7, no. 3, pp. 788-792, 1996.
23. F. Roli, G. Giacinto, and G. Vernazza, "Methods for designing multiple classifier systems," 2nd Int. Workshop on Multiple Classifier Systems, in Lecture Notes in Computer Science, J. Kittler and F. Roli, Eds., vol. 2096, pp. 78-87, 2001.
24. G. Giacinto and F. Roli, "Approach to the automatic design of multiple classifier systems," Pattern Recognition Letters, vol. 22, no. 1, pp. 25-33, 2001.
25. L. Breiman, "Bagging predictors," Machine Learning, vol. 24, no. 2, pp. 123-140, 1996.
26. Y. Freund and R.E. Schapire, "Decision-theoretic generalization of on-line learning and an application to boosting," Journal of Computer and System Sciences, vol. 55, no. 1, pp. 119-139, 1997.
27. F.M. Alkoot and J. Kittler, "Experimental evaluation of expert fusion strategies," Pattern Recognition Letters, vol. 20, no. 11-13, pp. 1361-1369, Nov. 1999.
28. J. Kittler, M. Hatef, R.P.W. Duin, and J. Mates, "On combining classifiers," IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 20, no. 3, pp. 226-239, 1998.
29. M. Muhlbaier, A. Topalis, and R. Polikar, "Ensemble confidence estimates posterior probability," 6th Int. Workshop on Multiple Classifier Sys., in Lecture Notes in Comp. Science, N. Oza, R. Polikar, J. Kittler, and F. Roli, Eds., vol. 3541, pp. 326-335, 2005.
30. J. Grim, J. Kittler, P. Pudil, and P. Somol, "Information analysis of multiple classifier fusion," 2nd Int. Workshop on Multiple Classifier Systems, in Lecture Notes in Computer Science, J. Kittler and F. Roli, Eds., vol. 2096, pp. 168-177, 2001.
31. L.I. Kuncheva, "A theoretical study on six classifier fusion strategies," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 2, pp. 281-286, 2002.
32. S.B. Cho and J.H. Kim, "Multiple network fusion using fuzzy logic," IEEE Transactions on Neural Networks, vol. 6, no. 2, pp. 497-501, 1995.
33. M. Grabisch and J.M. Nicolas, "Classification by fuzzy integral: performance and tests," Fuzzy Sets and Systems, vol. 65, no. 2-3, pp. 255-271, 1994.
34. Y. Lu, "Knowledge integration in a multiple classifier system," Applied Intelligence, vol. 6, no. 2, pp. 75-86, 1996.
35. L.I. Kuncheva, "Using measures of similarity and inclusion for multiple classifier fusion by decision templates," Fuzzy Sets and Systems, vol. 122, no. 3, pp. 401-407, 2001.
36. L.I. Kuncheva, "Switching between selection and fusion in combining classifiers: An experiment," IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, vol. 32, no. 2, pp. 146-156, 2002.
37. D. Tax, M. van Breukelen, R.P.W. Duin, and J. Kittler, "Combining multiple classifiers by averaging or by multiplying?," Pattern Rec., vol. 33, no. 9, pp. 1475-1485, 2000.
38. L.I. Kuncheva and C.J. Whitaker, "Feature Subsets for Classifier Combination: An Enumerative Experiment," 2nd Int. Workshop on Multiple Classifier Systems, in Lecture Notes in Comp. Science, J. Kittler and F. Roli, Eds., vol. 2096, pp. 228-237, 2001.
39. N.C. Oza and K. Tumer, "Input decimation ensembles: Decorrelation through dimensionality reduction," 2nd Int. Workshop on Multiple Classifier Systems, in Lecture Notes in Computer Science, J. Kittler and F. Roli, Eds., vol. 2096, pp. 238-247, 2001.
40. N.S.V. Rao, "On fusers that perform better than best sensor," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 8, pp. 904-909, 2001.
41. K. Tumer and J. Ghosh, "Analysis of decision boundaries in linearly combined neural classifiers," Pattern Recognition, vol. 29, no. 2, pp. 341-348, 1996.
42. G. Fumera and F. Roli, "A Theoretical and Experimental Analysis of Linear Combiners for Multiple Classifier Systems," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27. no. 6, pp. 942-956, 2005.
43. Various Authors, "Proceedings of International Workshop on Multiple Classifier Systems (2000-2005)," F. Roli, J. Kittler, T. Windeatt, N. C. Oza, and R. Polikar, Eds. Berlin, Germany: Springer, 2005.
44. T.K. Ho, "Random subspace method for constructing decision forests," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 8, pp. 832-844, 1998.
45. L. Kuncheva and C. Whitaker, "Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy," Machine Learning, vol. 51, no. 2, pp. 181-207, 2003.
46. G. Brown, "Diversity in neural network ensembles," Ph.D. dissertation, University of Birmingham, 2004.
47. D.H. Wolpert and W.G. Macready, "No free lunch theorems for optimization," IEEE Transactions on Evolutionary Computation, vol. 1, no. 1, pp. 67-82, 1997.
48. B. Rosen, "Ensemble learning using decorrelaled neural networks," Connection Science, vol. 8, no. 3, pp. 373-384, 1996.
49. L. Breiman, "Random Forests," Machine Learning, vol. 45, no. 1, pp. 5-32, Oct. 2001.
50. L. Breiman, "Pasting small votes for classllcatlon in large databases and on-line," Machine Learning, vol. 36, pp. 85-103, 1999.
51. N.V. Chawla, L.O. Hall, K.W. Bowycr, J. Moore, and W.P. Kegelmoyer, "Distributed pasting of small votes," 3rd Int. Workshop on Multiple Classifier Systems, In Lecture Notes in Computer Science, F. Roli and J. Kittler, Eds., vol. 2364. p. 52-61, Springer, 2002.
52. R.E. Schaplre, Y. Freund, P. Bartlett, and W.S. Lee, "Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods," Annals or Statistics, vol. 26, no. 5, pp. 1651-1686, 1998.
53. N.C. Oza, "Boosting with Averaged Weight Vectors," 4th Int. Workshop on Multiple Classifier Systems, in Lecture Notes in Computer Science, T. Windeatt and F. Roli, Eds., vol. 2709, pp. 15-24, Springer, 2003.
54. N.C. Oza, "AveBoost2: Boosting for Noisy Data," 5th Int. Workshop on Multiple Classifier Systems. In Lecture Notes in Computer Science, vol. 3077, pp. 31-40, F. Roli, J. Kittler, and T. Windeatt, Eds. Springer, 2004.
55. R. Polikar, L. Udpa, S.S. Udpa, and V. Honavar, "Learn++: An incremental learning algorithm for supervised neural networks," IEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews, vol. 31, no. 4, pp. 497-508, 2001.
56. M.I. Jordan and L. Xu, "Convergence results for the EM approach to mixtures of experts architectures," Neural Networks, vol. 8, no. 9. pp. 1409-1431, 1995.
57. P.J. Boland, "Majority system and the Condorcet jury theorem," Statistician, vol. 38, no. 3, pp. 181-189, 1989.
58. L. Shapley and B. Grofman, "Optimizing group judgmental accuracy In the presence of interdependencles," Public Choice (Historical Archive), vol. 43, no. 3, pp. 329-343, 1984.
59. N. Littlestone and M. Warrnuth, "Weighted majority algorithm," Information and Computation, vol. 108, pp. 212-261, 1994.
60. Y.S. Huang and C.Y. Suen, "Behavior-knowledge space method for combination of multiple classifiers," Proc. of IEEE Computer Vision and Pattern Recog., pp. 347-352, 1993.
61. Y.S. Huang and C.Y. Suen, "A method of combining multiple experts for the recognition of unconstrained handwritten numerals," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 17, no. 1, pp. 90-94, 1995.
62. H. Ichihashi, T. Shirai, K. Nagasaka, and T. Mlyoshl, "Neuro-fuzzy ID3: a method of inducing fuzzy decision trees with linear programming for maximizing entropy and an algebraic method for incremental learning," Fuzzy Sets and Systems, vol. 81, no. 1, pp. 157-167, 1996.
63. R.O. Duda, P.E. Hart, and D. Stork, "Algorithm Independent Techniques," in Pattern Classification, 2 ed New York: Wiley, pp. 453-516, 2001.
64. A.P. Dempster, "Upper and lower probabilities induced by multivalued mappings," Annals of Mathematical Statistics, vol. 38, no. 2, pp. 325-339, 1967.
65. G. Shafer, A Mathematical Theory of Evidence. Princeton, NJ: Princeton Univ. Press, 1976.
66. T.G. Dietterich, "Ensemble methods in machine learning," 1st Int. Workshop on Multiple Classifier Systems, in Lecture Notes in Computer Sceince, J. Kitler and F. Roli, Eds., vol. 1857, pp. 1-15, 2000.
67. T.G. Dietterich, "Experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization," Machine Learning, vol. 40, no. 2, pp. 139-157, 2000.
68. L. Breiman, "Arcing classifiers," Annals of Statistics, vol. 26, no. 3, pp. 801-849, 1998.
69. E. Bauer and R. Kohavi, "An empiricial comparison of voting classification algorithms: bagging, boosting and variants," Machine Learning, vol. 36, pp. 105-142, 1999.
70. J.R. Quinlan, "Bagging, boosting and C4.5, 13th Int. Conf. on Artificial Intelligence, pp. 725-730, 1996.
71. R.A. Jacobs, "Methods for combining experts' probability assessments," Neural Computation, vol. 7, no. 5, p. 867, 1995.
72. R. Duin and D. Tax, "Experiments with Classifier Combining Rules," 1st Int. Workshop on Multiple Classifier Systems, in Lecture Notes in Computer Science, J. Kittler and F. Roli, Eds., vol. 1857, pp. 16-29, Springer, 2000.
73. J. Ghosh, "Multiclassifier Systems: Back to the Future," 3rd Int. Workshop on Multiple Classifier Systems, in Lecture Notes in Computer Science, F. Roli and J. Kittler, Eds., vol. 2364, pp. 1-15, 2002.
74. R. French, "Catastrophic forgetting in connectionist networks: causes, consequences and solutions," Trends in Cognitive Sciences, vol. 3, no. 4, pp. 128-135, 1999.
75. M. Muhlbaier, A. Topalis, and R. Polikar, "Learn++.MT: A New Approach to Incremental Learning," 5th Int. Workshop on Multiple Clas sifier Systems, In Lecture Notes in Computer Sceince, F. Roli, J. Kittler, and T. Windeatt, Eds., vol. 3077, pp. 52-61, Springer, 2004.
76. R. Polikar, L. Udpa, S. Udpa, and V. Honavar, "An Incremental learning algorithm with confidence estimation for automated identification of NDE signals," IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control, vol. 51, no. 8, pp. 990-1001, 2004.
77. M. Muhlbaier, A. Topalis, and R. Polikar, "Incremental learning from unbalanced data," IEEE Int. Joint Conf. on Neural Networks vol. 2, pp. 1057-1062, Budapest, Hungary, 2004.
78. H. Altincay and M. Demirekler, "Speaker identification by combining multiple classifiers using Dempster-Shafer theory of evidence," Speech Communication, vol. 41, no. 4, pp. 531-547, 2003.
79. Y. Bi, D. Bell, H. Wang, G. Guo, and K. Greer, "Combining multiple classifiers using dempster's rule of combination for text categorization," in Lecture Notes in Artificial Intelligence, vol. 3131, pp. 127-138, Barcelona, Spain: Springer, 2004.
80. T. Denoeux, "Neural network classifier based on Dempster-Shafer theory," IEEE Trans. on Sys, Man, and Cyber. A: Sys. and Humans., vol. 30, no. 2, pp. 131-150, 2000.
81. G.A. Carpenter, S. Martens, and O.J. Ogas, "Self-organizing information fusion and hierarchical knowledge discovery: a new framework using ARTMAP neural networks," Neural Networks, vol. 18, no. 3, pp. 287-295, Apr. 2005.
82. D. Parikh, M.T. Kim, J. Oagaro, S. Mandayam, and R. Polikar, "Combining classifiers for multisensor data fusion," IEEE Int. Conf. on Systems, Man and Cybernetics, vol. 2, pp. 1232-1237, The Hague, Netherlands, 2004.
83. B.F. Buxton, W.B. Langdon, and S.J. Barrett, "Data fusion by intelligent classifier combination," Measurement and Control, vol. 34, no. 8, pp. 229-234, 2001.
84. G.J. Briem, J.A. Benediktsson, and J.R. Sveinsson, "Use of multiple classifiers in classification of data from multiple data sources," in International Geoscience and Remote Sensing Symposium (IGARSS), vol. 2, pp. 882-884, Sydney, 2001.
85. W. Fan, M. Gordon, and P. Pathak, "On linear mixture of expert approaches to information retrieval," Decision Support Systems, vol. In Press, Corrected Proof.
86. S. Jianbo, W. Jun, and X. Yugeng, "Incremental learning with balanced update on receptive fields for multi-sensor data fusion," IEEE Transactions on Systems, Man and Cybernetics, Part B, vol. 34, no. 1, pp. 659-665, 2004.
87. S. Gunter and H. Bunke, "An evaluation of ensemble methods in handwritten word recognition based on feature selection," Int. Conf. on Pattern Recog., vol. 1, pp. 388-392, 2004.
88. S. Krause and R. Polikar, "An Ensemble of Classifiers Approach for the Missing Feature Problem," In Int. Joint Conf on Neural Networks, vol. 1, pp. 553-556, Portland, OR, 2003.
89. P.M. Long and V.B. Vega, "Boosting and Microarray Data," Machine Learning, vol. 52, no. 1-2, pp. 31-44, July 2003.
90. E. Allwein, R.E. Schapire, and Y. Singer, "Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers," Journal of Machine Learning Research, vol. 1, pp. 113-141, 2000.
91. T.G. Dietterich and G. Bakiri, "Solving Multiclass Learning Problems via Error-Correcting Output Codes," Journal of Artificial Intel. Research, vol. 2, pp. 263-286, 1995.
92. S. Rajan and J. Ghosh, "An empirical comparison of hierarchical vs. two-level approaches to multiclass problems," in Lecture Notes in Computer Science, vol. 3077, pp. 283-292, F. Roli, J. Kittler, and T. Windeatt, Eds. Springer, 2004.
93. R.S. Smith and T. Windeatt, "Decoding Rules for Error Correcting Output Code Ensembles," 6th Int. Workshop on Multiple Classifier Sys., Lecture Notes in Computer Science, N. C. Oza, R. Polikar, F. Roli, and J. Kittler, Eds., vol. 3541, pp. 53-63, Springer, 2005.
94. H.G. Ayad and M.S. Kamel, "Cluster-Based Cumulative Ensembles," 6th Int. Workshop on Multiple Classifier Sys., Lecture Notes in Computer Science, N. C. Oza, R. Polikar, F. Roli, and J. Kittler, Eds., vol. 3541, pp. 236-245, Springer, 2005.
95. S. Monti, P. Tamayo, J. Mesirov, and T. Golub, "Consensus Clustering: A Resampling-Based Method for Class Discovery and Visualization of Gene Expression Microarray Data," Machine Learning, vol. 52, no. 1-2, pp. 91-118, July 2003.
96. A. Strehl and J. Ghosh, "Cluster Ensembles-A Knowledge Reuse Framework for Combining Multiple Partitions," Journal of Machine Learning Research, vol. 3, pp. 583-617, 2002.