Learning from examples with information theoretic criteria

Journal of VLSI Signal Processing Systems for Signal, Image, and Video Technology

Volumn 26, Issue 1, 2000, Pages 61-77

  Principe, Jose C ^a   Xu, Dongxin ^a,b   Zhao, Qun ^a   Fisher III, John W ^a  

^a UNIVERSITY OF FLORIDA   (United States)

^b INSTITUTE OF GEOLOGY AND GEOPHYSICS   (China)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTABILITY AND DECIDABILITY; DATA COMPRESSION; FEATURE EXTRACTION; LEARNING ALGORITHMS; LEARNING SYSTEMS; QUADRATIC PROGRAMMING; VECTORS;

BLIND SOURCE SEPARATION; RENYI'S QUADRATIC ENTROPY; SUPERVISED LEARNING;

INFORMATION THEORY;

EID: 0034245254     PISSN: 09225773     EISSN: None     Source Type: Journal    
DOI: 10.1023/a:1008143417156     Document Type: Article

References (38)

1. V. Vapnik, Statistical Learning Theory, Wiley, 1998.
2. H. Barlow, "Unsupervised Learning," Neural Computation, vol. 1, 1989, pp. 295-311.
3. P. Foldiak, "Adaptive Network for Optimal Linear Feature Extraction," IEEE Int. Joint Conf. Neural Net., vol. 1, 1989, pp. 401-405.
4. B. Olshausen and D. Fields, "Sparse Coding with an Overcomplete Basis Set: A Strategy Employed by V1," Vision Research, vol. 37, 1997, pp. 3311-3325.
5. R. Linsker, "An Application of the Principle of Maximum Information Preservation to Linear Systems," in Advances in Neural Information Processing Systems, vol. 1, Morgan-Kaufman, 1988, pp. 485-494.
6. C. Shannon and W. Weaver, The Mathematical Theory of Communication, University of Illinois Press, 1949.
7. T. Cover and J. Thomas, Elements of Information Theory, Wiley, 1991.
8. G. Deco and D. Obradovic, An Information-Theoretic Approach to Neural Computing, New York: Springer, 1996.
9. M. Plumbley and F. Fallside, "An Information Theoretic Approach to Unsupervised Networks," Int. J. Conf. on Neural Nets, Washington, DC, 1989, vol. 2, p. 598.
10. J.W. Fisher III, "Nonlinear Extensions to the Minimum Average Correlation Energy Filter," Ph.D. Dissertation, Dept. of ECE, University of Florida, 1997.
11. D. Xu, "Energy, Entropy and Information Potential for Neural Computation," Ph.D. Dissertation, U. of Florida, 1999.
12. D. Xu, J. Principe, J. Fisher, and H.-C. Wu, "A Novel Measure for Independent Component Analysis (ICA)," in Proc. ICASSP'98, vol. II, 1998, pp. 1161-1164.
13. D. Xu, J. Fisher, and J. Principe, "Mutual Information Approach to Pose Estimation," in Proc. SPIE, vol. 3370, 1998, pp. 218-229. Algorithms for Synthetic Aperture Radar Imagery V.
14. A. Bell and T. Sejnowski, "An Information-Maximization Approach to Blind Separation and Blind Deconvolution," Neural Computation, vol. 7, 1995, pp. 1129-1159.
15. J. Fisher, A. Ihler, and P. Viola, "Learning Informative Statistics: A Nonparametric Approach," Proc. of Neural Information Proc. Systems, vol. 12, in press.
16. J.N. Kapur, Measures of Information and Their Applications, John Wiley & Sons, 1994.
17. S. Amari, A. Chichocki, and H. Yang, "A New Learning Algorithm for Blind Source Separation," Advances of Information Processing Systems, vol. 8, 1996, pp. 757-763.
18. E. Jaynes, "Information Theory and Statistical Mechanics," Physical Review, vol. 106, 1957, pp. 620-630.
19. K. Diamantaras and S. Kung, Principal Component Neural Networks: Theory and Applications, Wiley, 1996.
20. S. Haykin, Adaptive Filter Theory, Prentice Hall, 1986.
21. J. Principe, D. Xu, and J. Fisher, "Information Theoretic Learning," in Unsupervised Adaptive Filtering, Haykin (Ed.), Wiley, 2000, pp. 265-319.
22. S. Haykin, Neural Networks, A Comprehensive Foundation, Macmillan Publishing Company, 1998.
23. J. Lin, "Divergence Measures Based on Shannon Entropy," IEEE Trans. Inform. Theory, vol. 37, no. 1, 1991, pp. 145-151.
24. J. Principe, "From Linear Adaptive to Information Filtering," in IEEE Workshop Neural Nets for Sig. Proc., Key note address, Cambridge, England, Aug. 1998.
25. R. Fano, Transmission of information, MIT Press, 1961.
26. M. Hellman and J. Raviv, "Probability of Error, Equivocation and the Chernoff Bound," IEEE Trans. Inform. Theory, vol. IT-16, no. 4, 1970, pp. 368-372.
27. A. Renyi, "Some Fundamental Questions of Information Theory," in Selected Papers of Alfred Renyi, vol. 2, Budapest: Akademie Kiado, 1976.
28. I. Grassberger and I. Proccacia, "Measuring the Strangeness of Strange Attractors," Physica D, vol. 9, 1983, pp. 189-208.
29. P. Viola, N. Schraudolph, and T. Sejnowski, "Empirical Entropy Manipulation for Real-World Problems," in Proc. Neural Info. Proc. Sys. (NIPS 8) Conf., 1995, pp. 851-857.
30. E. Parzen, "On the Estimation of a Probability Density Function and the Mode," Ann. Math. Stat., vol. 33, 1962, p. 1065.
31. D.E. Rumelhart, G.E. Hinton, and J.R. Williams, "Learning Representations by Back-Propagating Errors," Nature (London), vol. 323, 1986, pp. 533-536.
32. C. Diks, W. Zwet, F. Takens, and J. DeGoede, "Detecting Differences Between Delay Vector Distributions," Physical Rev E, vol. 53, no. 3, 1996, pp. 2169-2176.
33. H.C. Wu and J. Principe, "Novel Quadratic Entropy Measures and their Application to Blind Source Separation/Extraction," in IEEE Workshop Neural Networks Sig. Proc. 1999, accepted.
34. MSTAR (public) Targets, CDROM, Veda Inc. Ohio, 1997.
35. V. Velten, T. Ross, J. Mossing, S. Worrell, and M. Bryant, "Standard SAR/ATR Evaluation Experiments Using the MSTAR Public Release Data Set," Research Report, Wright State U., 1998.
36. Q. Zhao and J. Principe, "From Hyperplanes to Large Margin Classifiers: Applications to SAR/ATR," in Proc. SPIE 13th Annual Int. Sym. Aerospace/Defense Sensing, Simulation and Control, 1999, vol. 3718.
37. T. Friess, "Support Vector Neural Networks: The Kernel Adatron with Bias and Soft Margin," Research Report, U. of Sheffield, UK, 1998.
38. M. Gori and F. Scarselli, "Are Multilayer Perceptrons Adequate for Pattern Recognition and Verification?" IEEE Trans. Pattern Analysis and Machine Intell., vol. 20, no. 11, 1998, pp. 1121-1132.