Feature extraction by non-parametric mutual information maximization

Journal of Machine Learning Research

Volumn 3, Issue , 2003, Pages 1415-1438

  Torkkola, Kari ^a  

^a MOTOROLA INC   (United States)

Author keywords

Feature extraction; Feature transform; Mutual information; Non parametric estimation; Parzen estimation; Quadratic divergence measures; Renyi entropy

Indexed keywords

DIVERGENCE MEASURES; FEATURE TRANSFORM; MUTUAL INFORMATIONS; NON-PARAMETRIC ESTIMATIONS; PARZEN ESTIMATIONS; RENYI ENTROPY;

FEATURE EXTRACTION; RADIAL BASIS FUNCTION NETWORKS;

MATHEMATICAL TRANSFORMATIONS;

EID: 1942450610     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (37)

1. M.E. Aladjem. Nonparametric discriminant analysis via recursive optimization of Patrick-Fisher distance. IEEE Transactions on SMC, 28(2):292-299, April 1998.
2. A. Antos, L. Devroye, and L. Gyorfi. Lower bounds for bayes error estimation. IEEE Transactions on PAMI, 21(7):643-645, July 1999.
3. R. Battiti. Using mutual information for selecting features in supervised neural net learning. Neural Networks, 5(4):537-550, July 1994.
4. C. M. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, Oxford, New York, 1995.
5. K.D. Bollacker and J. Ghosh. Linear feature extractors based on mutual information. In Proc. 13th ICPR, pages 720-724, August 25-29 1996.
6. B.V. Bonnlander and A.S. Weigend. Selecting input variables using mutual information and nonparametric density estimation. In Proc. 1994 International Symposium on Artificial Neural Networks, pages 42-50, Tainan, Taiwan, 1994.
7. R. Collobert and S. Bengio. SVMTorch: Support Vector Machines for Large-Scale Regression Problems. Journal of Machine Learning Research, 1:143-160, 2001.
8. S. Dasgupta. Experiments with random projection. In Proc. 16th Conf. on Uncertainty in Artificial Intelligence, pages 143-151, Stanford, CA, June30 - July 3 2000.
9. P.A. Devijver and J. Kittler. Pattern recognition: A statistical approach. Prentice Hall, London, 1982.
10. R.M. Fano. Transmission of Information: A Statistical theory of Communications. Wiley, New York, 1961.
11. K. Fukunaga. Introduction to statistical pattern recognition (2nd edition). Academic Press, New York, 1990.
12. M. Girolami, A. Cichocki, and S-I. Amari. A common neural network model for unsupervised exploratory data analysis and independent component analysis. IEEE Transactions on Neural Networks, 9(6):1495 - 1501, November 1998.
13. X. Guorong, C. Peiqi, and W. Minhui. Bhattacharyya distance feature selection. In Proc. 13th ICPR, volume 2, pages 195 - 199. IEEE, 25-29 Aug. 1996.
14. M.E. Hellman and J. Raviv. Probability of error, equivocation and the chernoff bound. IEEE Transactions on Information Theory, 16:368-372, 1970.
15. A.O. Hero, B. Ma, O. Michel, and J. Gorman. Alpha-divergence for classification, indexing and retrieval. Technical Report CSPL-328, University of Michigan Ann Arbor, Communications and Signal Processing Laboratory, May 2001.
16. A. Hillion, P. Masson, and C. Roux. A non-parametric approach to linear feature extraction; Application to classification of binary synthetic textures. In Proc. 9th ICPR, pages 1036-1039, Rome, Italy, November 14-17 1988.
17. J.N. Kapur. Measures of information and their applications. Wiley, New Delhi, India, 1994.
18. J.N. Kapur and H.K. Kesavan. Entropy optimization principles with applications. Academic Press, San Diego, London, 1992.
19. T. Kohonen, J. Kangas, J. Laaksonen, and K. Torkkola. LVQ PAK: A program package for the correct application of Learning Vector Quantization algorithms. In Proc. IJCNN, volume I, pages 725-730, Piscataway, NJ, 1992. IEEE.
20. D. Koller and M. Sahami. Toward optimal feature selection. In Proc. 13th International Conference on Machine Learning, pages 284-292, Bari, Italy, 1996.
21. N. Kumar and A.G. Andreou. Heteroscedastic discriminant analysis and reduced rank HMMs for improved speech recognition. Speech Communication, 26(4):283-297, 1998.
22. T. Okada and S. Tomita. An optimal orthonormal system for discriminant analysis. Pattern Recognition, 18 (2):139-144, 1985.
23. E. Parzen. On the estimation of probability density function and the mode. The Annals of Mathematical Statistics, 33:1065, 1962.
24. E.A. Patrick and F.P. Fisher II. Nonparametric feature selection. IEEE Transactions on Information Theory, 15:577-584, 1969.
25. J.C. Principe, J.W. Fisher III, and D. Xu. Information theoretic learning. In Simon Haykin, editor, Unsupervised Adaptive Filtering. Wiley, New York, NY, 2000.
26. A. Renyi. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, pages 547-561. University of California Press, 1961.
27. K. Rose, E. Gurewitz, and G.C. Fox. Statistical mechanics and phase transitions in clustering. Phys. Rev. Lett., 65(7):945-948, 1990.
28. G. Saon and M. Padmanabhan. Minimum Bayes error feature selection for continuous speech recognition. In Advances in Neural Information Processing Systems 13, pages 800-806. MIT Press, 2001.
29. B. Scholkopf, A. Smola, and K-R. Muller. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10:1299-1319, 1998.
30. C. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27:379-423, 623-656, July, October 1948.
31. F. Topsøe. Some inequalities for information divergence and related measures of discrimination. IEEE Transactions on Information Theory, 46:1602-1608, 2000.
32. K. Torkkola. Learning discriminative feature transforms to low dimensions in low dimensions. In Advances in neural information processing systems 14, Vancouver, BC, Canada, December 3-8 2001a. MIT Press.
33. K. Torkkola. Nonlinear feature transforms using maximum mutual information. In Proc. IJCNN, pages 2756-2761,Washington DC, USA, July 15-19 2001b.
34. K. Torkkola. On feature extraction by mutual information maximization. In Proc. ICASSP, Orlando, FL, May 13-17 2002.
35. K. Torkkola and W. Campbell. Mutual information in learning feature transformations. In Proc. 17th International Conference on Machine Learning, pages 1015-1022, Stanford, CA, USA, June 29 - July 2 2000.
36. J. Weston, S. Mukherjee, O. Chapelle, M. Pontil, T. Poggio, and V. Vapnik. Feature selection for SVMs. In Advances in Neural Information Processing Systems 13. MIT Press, 2001.
37. H. Yang and J. Moody. Data visualization and feature selection: New algorithms for nongaussian data. In Advances in Neural Information Processing Systems 12, pages 687-693. MIT Press, 2000.