Bayes error rate estimation using classifier ensembles

International Journal of Smart Engineering System Design

Volumn 5, Issue 2, 2003, Pages 95-109

  Tumer, Kagan ^b   Ghosh, Joydeep ^a  

^a The University of Texas at Austin   (United States)

^b NASA AMES RESEARCH CENTER   (United States)

Author keywords

Bayes error; Combining; Ensembles; Error bounds; Error estimate

Indexed keywords

APPROXIMATION THEORY; LEARNING SYSTEMS; NEURAL NETWORKS; PROBABILITY; STATISTICAL METHODS;

ERROR ESTIMATION;

ERROR ANALYSIS;

EID: 0038202195     PISSN: 10255818     EISSN: None     Source Type: Journal    
DOI: 10.1080/10255810305042     Document Type: Article

References (68)

1. Ali, K. M., and M. J. Pazzani. 1995. On the link between error correlation and error reduction in decision tree ensembles. Technical Report 95-38, Department of Information and Computer Science, University of California, Irvine.
2. Barron, A. R. 1993. Universal approximation bounds for superpositions of a sigmoidal function theory. IEEE Transactions on Information Theory 39(3): 930-945.
3. Battiti, R. 1994. Using mutual information for selecting features in supervised neural net learning. IEEE Transactions on Neural Networks 5(4): 537-550.
4. Beck, S., and J. Ghosh. 1992. Noise sensitivity of static neural classifiers. In Proceedings of SPIE Conf. on Applications of Artificial Neural Networks, Vol. 1709, pages 770-779, Orlando, Florida, April 1992.
5. Benediktsson, J.A., and J.R. Sveinsson. 2000. Consensus based classification of multisource remote sensing data. In Proceedings of the First International Workshop on Multiple Classifier Systems, eds. J. Kittler and F. Roli, 280-289, Berlin: Springer.
6. Bishop, C.M. 1995. Neural Networks for Pattern Recognition. New York, NY: Oxford University Press.
7. Blake, C., E. Keogh, and C. J. Merz. 1998. UCI repository of machine learning databases, (URL: http://www.ics.uci.edu/̃mlearn/MLRepository.html).
8. Bollacker, K. D., and J. Ghosh. 1996. Linear feature extractors based on mutual information. In Proceedings of the 13th International Conference on Pattern Recognition, Pages IV: 720-724.
9. Breiman, L. 1996. Bagging predictors. Machine Learning 24(2): 123-140.
10. Buturović, L. J. 1993. Improving k-nearest neighbor density and error estimates. Pattern Recognition 26(4): 611-616.
11. Cohn, D., R. Atlas, and L. Ladner. 1994. Improving generalization with active learning. Machine Learning 15(2): 201-221.
12. Cover, T. M., and P. E. Hart. 1967. Nearest neighbor pattern classification. IEEE Transactions on Information Theory 13: 21-27.
13. Cover, T. M., and J. A. Thomas. 1991. Elements of Information Theory. New York, NY: John Wiley and Sons.
14. Devijver, P. A., and J. Kittler, 1982. Pattern Recognition: A Statistical Approach. Englewood Cliffs, NJ: Prentice-Hall.
15. Devroye, L., L. Gyorfi, and G. Lugosi. 1996. A Probabilistic Theory of Pattern Recognition. Berlin: Springer Verlag.
16. Dietterich, T.G. 2000. Ensemble methods in machine learning. In Proceedings of the First International Workshop on Multiple Classifier Systems, eds. J. Kittler and F. Roli 1-15. Berlin: Springer.
17. Djouadi, A., Ö. Snorrason, and F. D. Garber. 1990. The quality of training-sample estimates of the Battacharyya coefficient. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(1): 92-97.
18. Domingos, P. 2000. Bayesian averaging of classifiers and the overfitting problem. In Proceedings of the Seventeeth International Conference on Machine Learning, 223-230.
19. Drucker, H., C. Cortes, L. D. Jackel, Y. LeCun, and V. Vapnik. 1994. Boosting and other ensemble methods. Neural Computation 6(6): 1289-1301.
20. Duda, R.O., P. E. Hart, and D. G. Stork. 2001. Pattern Classification, 2nd edition. New York, NY: John Wiley and Sons.
21. Duin, R. P. W., and D. M. J. Tax. 2000. Experiments with classifier combining rules. In Proceedings of the First International Workshop on Multiple Classifier Systems, eds. J. Kittler and F. Roli, 16-29. Berlin: Springer.
22. Fraser, A. M., and H. L. Swinney. 1986. Independent coordinates for strange attractors from mutual information. Physical Review A 33: 1134-1140.
23. Freund, Y., and R. Schapire. 1996. Experiments with a new boosting algorithm. In Proceedings of the Thirteenth International Conference on Machine Learning, pages 148-156. Morgan Kaufmann.
24. Fukunaga, K. 1985. The estimation of the Bayes error by the k-nearest neighbor approach. In Progress in Pattern Recognition 2, eds. L. N. Kanal and A. Rosenfeld, 169-187. Amsterdam: North-Holland.
25. Fukunaga, K. 1990. Introduction to Statistical Pattern Recognition, 2nd edition. Boston: Academic Press.
26. Fukunaga, K., and D. Hummels. 1987. Bayes error estimation using Parzen and k-NN procedures. IEEE Transactions on Pattern Analysis and Machine Intelligence, 9: 634-643.
27. Fukunaga, K., and D. Hummels. 1987. Bias of the nearest neighbor error estimate. IEEE Transactions on Pattern Analysis and Machine Intelligence, 9(1): 103-112.
28. Garber, F. D., and A. Djouadi. 1988. Bounds on the Bayes classification error based on pairwise risk functions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 10(3): 281-288.
29. Geman, S., E. Bienenstock, and R. Doursat. 1992. Neural networks and the bias/variance dilemma. Neural Computation, 4(1): 1-58.
30. Ghosh, J. 2002. Multiclassifier systems: Back to the future. In Multiple Classifier Systems, eds. F. Roli and J. Kittler, 1-15, LNCS Vol. 2364. Berlin: Springer.
31. Ghosh, J., L. Deuser, and S. Beck. 1992. A neural network based hybrid system for detection, characterization and classification of short-duration oceanic signals. IEEE Journal of Ocean Engineering, 17(4): 351-363.
32. Ghosh, J., and K. Tumer. 1994. Structural adaptation and generalization in supervised feedforward networks. Journal of Artificial Neural Networks, 1 (4): 431-458.
33. Ghosh, J., K. Tumer, S. Beck, and L. Deuser. 1996. Integration of neural classifiers for passive sonar signals. In Control and Dynamic Systems - Advances in Theory and Applications, ed. C.T. Leondes, volume 77, 301-338. San Diego: Academic Press.
34. Giacinto, G., and F. Roli. 2000. Dynamic classifier selection. In Proceedings of the First International Workshop on Multiple Classifier Systems, 177-189. Berlin: Springer.
35. Hashem, S. 1993. Optimal Linear Combinations of Neural Networks. Ph.D. thesis, Purdue University.
36. Haykin, S. 1994. Neural Networks: A Comprehensive Foundation, New York, NY: Macmillan.
37. Jacobs, R. 1995. Method for combining experts' probability assessments. Neural Computation 7(5): 867-888.
38. Kargupta, H., and P. Chan, editors. 2000. Advances in Distributed and Parallel Knowledge Discovery. Cambridge, MA: AAAI/The MIT Press.
39. Kittler, J., and F. Roli, editors. 2000. Multiple Classifier Systems: Proceedings of the First International Workshop, Cagliari, Italy, June 2000. Berlin: Springer.
40. Krogh, A., and J. Vedelsby. 1995. Neural network ensembles, cross validation and active learning. In Advances in Neural Information Processing Systems-7, eds. G. Tesauro, D. S. Touretzky, and T. K. Leen, 231-238. Cambridge, MA: The M.I.T. Press.
41. Lowe, D., and A. R. Webb. 1991. Optimized feature extraction and the Bayes decision in feed-forward classifier networks. IEEE Trans. PAMI 13: 355-364.
42. Noordewier, M. O., G. G. Towell, and J. W. Shavlik. 1991. Training knowledge-based neural networks to recognize genes in DNA sequences. In Advances in Neural Information Processing Systems-3, eds. R.P. Lippmann, J.E. Moody, and D.S. Touretzky, 530-536. San Mateo, CA: Morgan Kaufmann.
43. Opitz, D. W., and J. W. Snavlik. 1996. Generating accurate and diverse members of a neural-network ensemble. In Advances in Neural Information Processing Systems-8, eds. D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, 535-541. Cambridge, MA: The M.I.T. Press.
44. Park, J., and I. W. Sandberg, 1993. Universal approximation and radial basis function networks. Neural Computation, 5: 305-316.
45. Perrone, M. P. 1993. Improving Regression Estimation: Averaging Methods for Variance Reduction with Extensions to General Convex Measure Optimization. Ph.D. thesis, Brown University.
46. Perrone, M. P., and L. N. Cooper. 1993. When networks disagree: Ensemble methods for hybrid neural networks. In Neural Networks for Speech and Image Processing, ed. R. J. Mammone, chapter 10. London: Chapmann-Hall.
47. Prechelt, L. 1994. PROBEN1 - A set of benchmarks and benchmarking rules for neural network training algorithms. Technical Report 21/94, Fakultät für Informatik, Universität Karlsruhe, D-76128 Karlsruhe, Germany, September 1994. Anonymous FTP: /pub/papers/techreports/1994/1994-21.ps.Z on ftp.ira.uka.de.
48. Richard, M. D., and R. P. Lippmann. 1991. Neural network classifiers estimate Bayesian a posteriori probabilities. Neural Computation, 3(4): 461-483.
49. Rosen, B. 1996. Ensemble learning using decorrelated neural networks. Connection Science, Special Issue on Combining Artificial Neural Networks: Ensemble Approaches 8(3 & 4): 373-384.
50. Ruck, D. W., S. K. Rogers, M. E. Kabrisky, M. E. Oxley, and B. W. Suter. 1990. The multilayer Perceptron as an approximation to a Bayes optimal discriminant function. IEEE Transactions on Neural Networks 1(4): 296-298.
51. Sharkey, A. 1990. Combining Artificial Neural Nets. Berlin: Springer-Verlag.
52. Sharkey, A. J. C. 1996. On combining artificial neural nets. Connection Science, Special Issue on Combining Artificial Neural Networks: Ensemble Approaches 8(3 & 4): 299-314.
53. Sharkey, A. J. C., N. E. Sharkey, and G. O. Chandroth. 1995. Neural nets and diversity. In Proceedings of the 14th International Conference on Computer Safety, Reliability and Security, Pages 375-389, Belgirate, Italy.
54. Sharkey, A. J. C., and N. E. Sharkey. 1997. Combining diverse neural nets. The Knowledge Engineering Review, 12(3): 231-247.
55. Sharkey, A. J. C., N. E. Sharkey, U. Gerecke, and G. O. Chandroth. 2000. The 'test and select' approach to ensemble combination. In Proceedings of the First International Workshop on Multiple Classifier Systems, eds. J. Kittler and F. Roli, 30-44. Berlin: Springer.
56. Shoemaker, P. A., M. J. Carlin, R. L. Shimabukuro, and C. E. Priebe. 1991. Least squares learning and approximation of posterior probabilities on classification problems by neural network models. In Proc. 2nd Workshop on Neural Networks (WNN-AIND91), Pages 187-196, Auburn, February 1991.
57. Towell, G. G., and J. W. Shavlik. 1992. Interpretation of artificial neural networks: Mapping knowledge-based neural networks into rules. In Advances in Neural Information Processing Systems-4, eds. J.E. Moody, S.J. Hanson, and R.P. Lippmann, 977-984. New York: Morgan Kaufmann, 1992.
58. Tumer, K., K. D. Bollacker, and J. Ghosh. 1998. A mutual information based ensemble method to estimate the Bayes error. In Intelligent Engineering Systems through Artificial Neural Networks, eds. C. Dagli et al., volume 8, 17-22. New York: ASME Press.
59. Tumer, K., and J. Ghosh. 1996a. Analysis of decision boundaries in linearly combined neural classifiers. Pattern Recognition, 29(2): 341-348.
60. Tumer, K., and J. Ghosh. 1996b. Error correlation and error reduction in ensemble classifiers. Connection Science 8(3 & 4): 385-404.
61. Tumer, K., and J. Ghosh. 1999. Linear and order statistics combiners for pattern classification. In Combining Artificial Neural Nets: Ensemble and Modular Multi-Net Systems, ed. A. J. C. Sharkey, 127-162. London: Springer-Verlag.
62. Tumer, K., and J. Ghosh. 2002. Robust combining of disparate classifiers through order statistics. Pattern Analysis and Applications 5(2): 189-200.
63. Tumer, K., and N. C. Oza. 2003. Input decimnated ensembles. Pattern Analysis and Applications (to appear).
64. Tumer, K., N. Ramanujam, R. Richards-Kortum, and J. Ghosh. 1997. Spectroscopic detection of cervical pre-cancer through radial basis function networks. In Advances in Neural Information Processing Systems-9, eds. M. C. Mozer, M. I. Jordan, and T. Petsche, 981-987. Cambridge, MA: The M.I.T. Press.
65. Wolpert, D. H. 1996. The existence of a priori distinctions between learning algorithms. Neural Computation 8: 1391-1420.
66. Wolpert, D. H. 1996. The lack of a priori distinctions between learning algorithms. Neural Computation 8: 1341-1390.
67. Young, T. Y., and Calvert, T. W. 1974. Classification, Estimation and Pattern Recognition. New York, NY: Elsevier.
68. Zheng, Z., and G. I. Webb. 1998. Stochastic attribute selection committees. In Proceedings of the Eleventh Australian Joint Conference on Artificial Intelligence (AI'98), Pages 321-332.