Gradient-based manipulation of nonparametric entropy estimates

IEEE Transactions on Neural Networks

Volumn 15, Issue 4, 2004, Pages 828-837

  Schraudolph, Nicol N ^a  

^a SALK INSTITUTE FOR BIOLOGICAL STUDIES   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADAPTIVE SYSTEMS; INFORMATION THEORY; MAXIMUM LIKELIHOOD ESTIMATION; OPTIMIZATION; PROBABILITY DENSITY FUNCTION;

AFFINE INVARIANT ENTROPY; ENTROPY MANIPULATION; EXPECTATION MAXIMIZATION; KERNEL DENSITY; PARZEN WINDOWS;

LEARNING SYSTEMS;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; AUTOMATED PATTERN RECOGNITION; COMPARATIVE STUDY; COMPUTER SIMULATION; DECISION SUPPORT SYSTEM; ENTROPY; EVALUATION; HAND; HISTOLOGY; HUMAN; INFORMATION RETRIEVAL; INFORMATION SCIENCE; LEARNING; METHODOLOGY; SAMPLE SIZE; SIGNAL PROCESSING; STATISTICAL MODEL;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; DECISION SUPPORT TECHNIQUES; ENTROPY; HAND; HUMANS; INFORMATION STORAGE AND RETRIEVAL; INFORMATION THEORY; MODELS, STATISTICAL; NEURAL NETWORKS (COMPUTER); PATTERN RECOGNITION, AUTOMATED; PROBABILITY LEARNING; SAMPLE SIZE; SIGNAL PROCESSING, COMPUTER-ASSISTED;

EID: 3843094964     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2004.828766     Document Type: Article

References (43)

1. L. B. Almeida, T. Langlois, J. D. Amaral, and A. Plakhov, "Parameter adaptation in stochastic optimization," in On-Line Learning in Neural Networks, D. Saad, Ed. Cambridge, U.K.: Cambridge Univ. Press, 1999, ch. 6, pp. 111-134.
2. J. J. Atick, "Could information theory provide an ecological theory of sensory processing?," Network, vol. 3, pp. 213-251, 1992.
3. H. B. Barlow, "Possible principles underlying the transformation of sensory messages," in Sensory Communication, W. A. Rosenbluth, Ed. Cambridge, MA: MIT Press, 1961.
4. H. B. Barlow, "Unsupervised learning," Neural Computat., vol. 1, no. 3, pp. 295-311, 1989.
5. S. Becker and G. E. Hinton, "A self-organizing neural network that discovers surfaces in random-dot stereograms," Nature, vol. 355, pp. 161-163, 1992.
6. J. Beirlant, E. J. Dudewicz, L. Györfi, and E. C. van der Meulen, "Non-parametric entropy estimation: An overview," Int. J. Math. Statist. Sci., vol. 6, no. 1, pp. 17-39, 1997.
7. A. J. Bell and T. J. Sejnowski, "An information-maximization approach to blind separation and blind deconvolution," Neural Computat., vol. 7, no. 6, pp. 1129-1159, 1995.
8. J. A. Bilmes, "A Gentle Tutorial of the EM Algorithm and Its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models," International Computer Science Institute, Univ. California Berkeley, CA, Technical Report TR-97-021, 1997.
9. J. S. Bridle, "Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters," in Advances in Neural Information Processing Systems, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1990, vol. 2, pp. 211-217.
10. P. Comon, "Independent component analysis, a new concept?," Signal Process., vol. 36, no. 3, pp. 287-314, 1994.
11. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm (with discussion)," J. Roy. Statist. Soc. Ser. B, vol. 39, pp. 1-38, 1977.
12. R. Duda and P. Hart, Pattern Classification and Scene Analysis. New York: Wiley, 1973.
13. D. Erdogmus, K. E. Hild II, and J. C. Principe, "Online entropy manipulation: stochastic information gradient," IEEE Signal Processing Lett., vol. 10, pp. 242-245, Aug. 2003.
14. D. Erdogmus and J. C. Principe, "Generalized information potential criterion for adaptive system training," IEEE Trans. Neural Networks, vol. 13, pp. 1035-1044, Sept. 2002.
15. J. H. Friedman and J. W. Tukey, "A projection pursuit algorithm for exploratory data analysis," IEEE Trans. Comput., vol. 23, pp. 881-889, 1974.
16. P. J. Huber, "Projection pursuit," Ann. Statist., vol. 13, no. 2, pp. 435-475, 1985.
17. J. N. Kapur, Measures of Information and Their Applications. New York: Wiley, 1992.
18. J. N. Kapur and H. K. Kesavan, Entropy Optimization Principles With Applications. New York: Academic, 1992.
19. J. Kivinen and M. K. Warmuth, "Additive versus exponentiated gradient updates for linear prediction," in Proc. 27th Annu. ACM Symp. Theory Computing, New York, May 1995, pp. 209-218.
20. R. Linsker, "Self-organization in a perceptual network," Computer, pp. 105-117, March 1988.
21. M.Masa-aki Sato and S. Ishii, "On-line EM algorithm for the normalized Gaussian network," Neural Computat., vol. 12, no. 2, pp. 407-432, 2000.
22. L. C. Parra, "Symplectic nonlinear component analysis," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 1996, pp. 437-443.
23. E. Parzen, "On the estimation of a probability density function and mode," Ann. Math. Statist., vol. 33, pp. 1065-1076, 1962.
24. J. C. Principe, D. Xu, and J. W. Fisher III, "Information-theoretic learning," in Unsupervised Adaptive Filtering: Blind Source Separation, S. Haykin, Ed. New York: Wiley, 2000, vol. 1, ch. 7, pp. 265-319.
25. A. Renyi, "On measures of entropy and information," in Proc. 4th Berkeley Symp. Mathematical Statistics and Probability, Berkeley, CA, 1961, Reprinted in Selected Papers of Alfred Renyi Akademia Kiado, Budapest, 1961, pp. 547-561.
26. A. Renyi, Selected Papers of Alfred Renyi. Budapest: Akademia Kiado, 1976, vol. 2.
27. J. Schmidhuber, "Learning factorial codes by predictability minimization," Neural Computat., vol. 4, no. 6, pp. 863-879, 1992.
28. J. Schmidhuber, M. Eldracher, and B. Foltin, "Semilinear predictability minimization produces well-known feature detectors," Neural Computat., vol. 8, no. 4, pp. 773-786, 1996.
29. N. N. Schraudolph, "Optimization of Entropy With Neural Networks," Ph.D. dissertation, Univ. California, San Diego, CA, 1995.
30. N. N. Schraudolph, "Local gain adaptation in stochastic gradient descent," in Proc. Int. Conf. Artificial Neural Networks, Edinburgh, Scotland, 1999, pp. 569-574.
31. N. N. Schraudolph, "Fast curvature matrix-vector products for second-order gradient descent," Neural Computat., vol. 14, no. 7, pp. 1723-1738, 2002.
32. N. N. Schraudolph, M. Eldracher, and J. Schmidhuber, "Processing images by semi-linear predictability minimization," Network: Computat. Neural Syst., vol. 10, no. 2, pp. 133-169, 1999.
33. N. N. Schraudolph and T. J. Sejnowski, "Unsupervised discrimination of clustered data via optimization of binary information gain," in Advances in Neural Information Processing Systems, S. J. Hanson, J. D. Cowan, and C. L. Giles, Eds. San Mateo, CA: Morgan Kaufmann, 1993, vol. 5, pp. 499-506.
34. F. M. Silva and L. B. Almeida, "Speeding up back-propagation," in Advanced Neural Computers, R. Eckmiller, Ed. Amsterdam: Elsevier, 1990, pp. 151-158.
35. A. F. M. Smith and U. E. Makov, "A quasi-Bayes sequential procedure for mixtures," J. Roy. Statist. Soc. B, vol. 40, no. 1, pp. 106-112, 1978.
36. T. Tollenaere, "SuperSAB: fast adaptive back propagation with good scaling properties," Neural Networks, vol. 3, pp. 561-573, 1990.
37. K. Torkkola, "Feature extraction by nonparametric mutual information maximization," J. Machine Learning Research, vol. 3, pp. 1415-1438, 2003.
38. D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, Eds., Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 1996, vol. 8.
39. M. M. van Hulle, "The formation of topographic maps that maximize the average mutual information of the output responses to noiseless input signals," Neural Computat., vol. 9, no. 3, pp. 595-606, 1997.
40. M. M. van Hulle, "Kernel-based equiprobabilistic topographic map formation," Neural Computat., vol. 10, no. 7, pp. 1847-1871, 1998.
41. P. A. Viola, "Alignment by Maximization of Mutual Information," Ph.D. dissertation, Massachusetts Inst. Technol., 1995.
42. P. A. Viola, N. N. Schraudolph, and T. J. Sejnowski, "empirical entropy manipulation for real-world problems," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, pp. 851-857.
43. P. A. Viola and W. M. Wells III, "Alignment by maximization of mutual information," in Fifth Int. Conf. Computer Vision, Cambridge, MA, 1995, pp. 16-23.