Information theoretic combination of pattern classifiers

Pattern Recognition

Volumn 43, Issue 10, 2010, Pages 3412-3421

  Meynet, Julien ^a   Thiran, Jean Philippe ^b  

^a Yahoo France R and D   (France)

^b EPFL   (Switzerland)

Author keywords

Classifier combination; Diversity; Information theory; Machine learning; Mutual information; Pattern recognition

Indexed keywords

CLASSIFIER COMBINATION; ENSEMBLE OF CLASSIFIERS; MACHINE LEARNING TECHNIQUES; MACHINE-LEARNING; MUTUAL INFORMATIONS; OPTIMAL ENSEMBLE; PATTERN CLASSIFIER; SELECTION BASED; SELECTION TECHNIQUES;

INFORMATION THEORY; LEARNING ALGORITHMS; LEARNING SYSTEMS; PATTERN RECOGNITION;

CLASSIFIERS;

EID: 77953650340     PISSN: 00313203     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patcog.2010.04.013     Document Type: Article

References (42)

1. T.G. Dietterich, Ensemble Methods in Machine Learning, in: Lecture Notes in Computer Science, vol. 1857, 2000, pp. 1-15.
2. Y. Freund, Y. Mansour, R. Schapire, Why averaging classifiers can protect against overfitting, in: Proceedings of the Eighth International Workshop on Artificial Intelligence and Statistics, 2001.
3. Breiman L. Bagging predictors. Machine Learning 24 2 (1996) 123-140
4. Freund Y., and Schapire R. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55 (1997) 119-139
5. Breiman L. Random forests. Machine Learning 45 1 (2001) 5-32
6. Kuncheva L.I. Combining Pattern Classifiers Methods and Algorithms (2004), Wiley, New York, NY, USA
7. Kittler J., Hatef M., Duin R.P.W., and Matas J. On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence 20 3 (1998) 226-239
8. Kuncheva L.I., and Whitaker C.J. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine Learning 51 2 (2003) 181-207
9. P. Cunningham, J. Carney, Diversity versus quality in classification ensembles based on feature selection, in: European Conference on Machine Learning (ECML), 2000, pp. 109-116.
10. Giacinto G., and Roli F. Design of effective neural network ensembles for image classification purposes. Image Vision Computing 19 9-10 (2001)
11. C. Shipp, L. Kuncheva, An investigation into how adaboost affects classifier diversity, in: Information Processing and Management of Uncertainty in Knowledge-based Systems (IPMU), Annecy, France, 2002.
12. L. Kuncheva, C. Whitaker, Using diversity with three variants of boosting: aggressive, conservative, and inverse, in: Proceedings of International Workshop on Multiple Classifier Systems, Lecture Notes in Computer Science, vol. 2364, 2002, pp. 81-90.
13. Shipp C.A., and Kuncheva L.I. Relationships between combination methods and measures of diversity in combining classifiers. Information Fusion 3 (2002) 135-148
14. Hadjitodorov S.T., Kuncheva L.I., and Todorova L.P. Moderate diversity for better cluster ensembles. Information Fusion 7 3 (2006) 264-275
15. Brown G., Wyatt J., Harris R., and Yao X. Diversity creation methods: a survey and categorisation. Journal of Information Fusion 6 1 (2005) 5-20
16. Windeatt T. Diversity measures for multiple classifier system analysis and design. Information Fusion 6 (2005) 21-36
17. Melville P., and Mooney R.J. Creating diversity in ensembles using artificial data. Information Fusion 6 1 (2005) 99-111
18. A. Krogh, J. Vedelsby, Neural network ensembles, cross validation, and active learning, in: Seventh Conference on Neural Information Processing Systems, 1995, pp. 234-238.
19. P. Cunningham, Overfitting and diversity in classification ensembles based on feature selection, Technical Report TCD-CS-2000-07, Department of Computer Science, Trinity College Dublin, 2000.
20. Yule G. On the association of attributes in statistics. Biometrika 2 (1903) 121-134
21. D. Skalak, The sources of increased accuracy for two proposed boosting algorithms, in: AAAI '96 Workshop on Integrating Multiple Learned Models for Improving and Scaling Machine Learning Algorithms, 1996.
22. R. Kohavi, D. Wolpert, Bias plus variance decomposition for zero-one loss functions, in: ICML, 1996, pp. 275-283.
23. L.I. Kuncheva, C.J. Whitaker, Ten measures of diversity in classifier ensembles: limits for two classifiers, in: IEE Workshop on Intelligent Sensor Processing, February 2001, IEE.
24. Tang E.K., Suganthan P.N., and Yao X. An analysis of diversity measures. Machine Learning 65 1 (2006) 247-271
25. Principe J.C., Xu D., and Fisher J.W. Learning from examples with information theoretic criteria. Journal of VLSI Signal Processing Systems 26 (2000) 61-77
26. J. Fisher, III, J. Principe, A methodology for information theoretic feature extraction, in: IEEE International Conference on Neural Networks (IJCNN'98), vol. 3, Anchorage, AK, 1998, pp. 1712-1716.
27. Hild II K.E., Erdogmus D., Torkkola K., and Principe J.C. Feature extraction using information-theoretic learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 28 9 (2006) 1385-1392
28. Sindhwani V., Rakshit S., Deodhare D., Erdogmus D., Principe J.C., and Niyogim P. Feature selection in mlps and svms based on maximum output information. IEEE Transactions on Neural Networks 15 (2004) 937-949
29. Butz T., and Thiran J.-P. From error probability to information theoretic (multi-modal) signal processing. Signal Processing 85 5 (2005) 875-902
30. Cover T., and Thomas J. Elements of Information Theory (1991), Wiley, New York
31. Fano R.M. Transmission of Information: A Statistical Theory of Communication (1961), MIT Press, Wiley, Cambridge
32. Erdogmus D., and Principe J.C. Lower and upper bounds for misclassification probability based on renyi's information. Journal of VLSI Signal Processing 37 (2004) 305-317
33. Kuncheva L.I., Whitaker C.J., Shipp C.A., and Duin R.P.W. Limits on the majority vote accuracy in classifier fusion. Pattern Analysis and Applications 6 (2003) 22-31
34. Lam L., and Suen S.Y. Application of majority voting to pattern recognition: an analysis of its behavior and performance. IEEE Transactions on Systems, Man, and Cybernetics 27 (1997) 553-568
35. Narasimhamurthy. Theoretical bounds of majority voting performance for a binary classification problem. IEEE Transactions on Pattern Analysis and Machine Intelligence 27 (2005) 1988-1995
36. Ruta D., and Gabrys B. A theoretical analysis of the limits of majority voting errors for multiple classifier systems. Pattern Analysis and Applications 5 4 (2002) 333-350
37. Shapley L., and Grofman B. Optimizing group judgemental accuracy in the presence of interdependencies. Public Choice 43 (1984) 329-343
38. J. Meynet, J.-P. Thiran, Information theoretic combination of classifiers with application to AdaBoost, in: 7th international Workshop on Multiple Classifier Systems (MCS), Prague, 2007, ITS.
39. J. Meynet, J.-P. Thiran, Ensembles of SVMs using an Information Theoretic Criterion, Technical Report, 2008.
40. R. Duin, P. Juszczak, P. Paclik, E. Pekalska, D. de Ridder, D.M.J. Tax, Prtools4, a matlab toolbox for pattern recognition, Delft University of Technology, 2004.
41. D.J. Newman, A. Asuncion, UCI machine learning repository, 2007.
42. Breiman L., Friedman J., Olsen R.A., and Stone C.J. Classification and Regression Trees (1984), Wadsworth International Group, CA, USA