Generalized information potential criterion for adaptive system training

IEEE Transactions on Neural Networks

Volumn 13, Issue 5, 2002, Pages 1035-1044

  Erdogmus, Deniz ^a,b   Principe, Jose C ^a,b  

^a IEEE   (United States)

^b UNIVERSITY OF FLORIDA   (United States)

Author keywords

Minimum error entropy; Parzen windowing; Renyi's entropy; Supervised training

Indexed keywords

INFORMATION CONTENT;

COMPUTER SIMULATION; CONVOLUTION; ENTROPY; INFORMATION ANALYSIS; OPTIMIZATION; SIGNAL PROCESSING; TIME SERIES ANALYSIS;

ADAPTIVE SYSTEMS;

EID: 0036737108     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2002.1031936     Document Type: Article

References (31)

1. G. Deco and D. Obradovic, An Information-Theoretic Approach to Neural Computing. New York: Springer-Verlag, 1996.
2. R. Linsker, "Toward an organizing principle for a layered perceptual network," in Neural Information Processing Systems, D. Anderson, Ed. New York: Amer. Inst. Phys., 1988, pp. 485-494.
3. J. Kapur and H. Kesavan, Entropy Optimization Principles and Applications: Associated Press, 1992.
4. R. V. Hartley, "Transmission of information," Bell Syst. Tech. J., vol. 7, 1928.
5. C. E. Shannon, "A mathematical theory of communications," Bell Syst. Tech. J., vol. 27, pp. 379-423, 623-656, 1948.
6. S. Amari, Differential-Geometrical Methods in Statistics. Berlin, Germany: Springer-Verlag, 1985.
7. P. Viola, N. Schraudolph, and T. Sejnowski, "Empirical entropy manipulation for real-world problems," in Proc. Neural Inform. Processing Syst. (NIPS 8) Conf., 1995, pp. 851-857.
8. P. Comon, "Independent component analysis, a new concept?," Signal Processing, vol. 36, pp. 287-314, 1994.
9. H. Yang and S. Amari, "Adaptive online learning algorithms for blind separation: Maximum entropy and minimum mutual information," Neural Comput., vol. 9, pp. 1457-1482, 1997.
10. E. Parzen, "On estimation of a probability density function and mode," in Time Series Analysis Papers. San Francisco, CA: Holden-Day, 1967.
11. J. C. Principe, D. Xu, and J. Fisher, "Information theoretic learning," in Unsupervised Adaptive Filtering, S. Haykin, Ed. New York: Wiley, 2000, vol. I, pp. 265-319.
12. K. E. Hild II, D. Erdogmus, and J. C. Principe, "Blind source separation using Renyi's mutual information," IEEE Signal Processing Lett., vol. 8, pp. 174-176, June 2001.
13. D. Xu, J. C. Principe, J. Fisher, and H. Wu, "A novel measure for independent component analysis (ICA)," in Proc. ICASSP'98, vol. II, Seattle, WA, pp. 1161-1164.
14. K. Torkkola and W. M. Campbell, "Mutual information in learning feature transformations," presented at the Proc. Int. Conf. Machine Learning, Stanford, CA, 2000.
15. V. Vapnik, The Nature of Statistical Learning Theory. New York: Springer-Verlag, 1995.
16. C. Diks, J. Houwelingen, F. Takens, and J. deGoede, "Detecting differences between delay vector distributions," Phys. Rev. E, vol. 53, pp. 2169-2176, 1996.
17. P. Grassberger and I. Procaccia, "Characterization of strange attractors," Phys. Rev. Lett., vol. 50, no. 5, pp. 346-349, 1983.
18. D. Erdogmus and J. C. Principe, "Comparison of entropy and mean square error criteria in adaptive system training using higher order statistics," presented at the Proc. Independent Components Analysis (ICA), Helsinki, Finland, 2000.
19. _, "An entropy minimization algorithm for short-term prediction of chaotic time series," IEEE Trans. Signal Processing, vol. 50, pp. 1780-1786, July 2002, submitted for publication.
20. A. Renyi, Probability Theory. New York: Elsevier, 1970.
21. I. Csiszár and J. Körner, Information Theory: Coding Theorems for Discrete Memoryless Systems. New York: Academic, 1981.
22. D. Erdogmus, K. E. Hild II, and J. C. Principe, "Blind source separation using Renyi's marginal entropy," Neurocomputing (Special Issue on Blind Source Separation and Independent Component Analysis), 2001.
23. S. C. Douglas and S.-Y. Kung, "Kuicnet algorithms for blind deconvolution," in Proc. Neural Networks Signal Processing VIII (NNSP'98), Cambridge, U.K., 1998, pp. 3-12.
24. S. Haykin, Introduction to Adaptive Filters. New York: MacMillan, 1984.
25. D. Rumelhart, G. Hinton, and R. Williams, "Learning internal representations by error back-propagation," Nature, vol. 323, pp. 533-536, 1986.
26. D. G. Luenberger, Linear and Nonlinear Programming. Reading, MA: Addison-Wesley, 1973.
27. A. Aarts and J. Korst, Simulated Annealing and Boltzmann Machines: A Stochastic Approach to Combinatorial Optimization and Neural Computing. New York: Wiley, 1989.
28. R. Y. Rubinstein, Simulation and the Monte Carlo Method. New York: Wiley, 1981.
29. W. Edmonson, K. Srinivasan, C. Wang, and J. Principe, "A global least square algorithm for adaptive IIR filtering," IEEE Trans. Circuits Syst., vol. 45, pp. 379-384, Mar. 1996.
30. D. Kaplan and L. Glass, Understanding Nonlinear Dynamics. New York: Springer-Verlag, 1995.
31. J. M. Kuo, "Nonlinear Dynamic Modeling with Artificial Neural Networks," Ph.D. dissertation, Univ. Florida, Gainesville, 1993.