Convergence properties and data efficiency of the minimum error entropy criterion in adaline training

IEEE Transactions on Signal Processing

Volumn 51, Issue 7, 2003, Pages 1966-1978

  Erdogmus, Deniz ^a   Principe, Jose C ^a  

^a University of Florida   (United States)

Author keywords

Adaptive systems; Convergence of numerical methods; Linear systems; Minimum entropy methods

Indexed keywords

ADAPTIVE SYSTEMS; ALGORITHMS; CONVERGENCE OF NUMERICAL METHODS; EIGENVALUES AND EIGENFUNCTIONS; ENTROPY; LINEAR SYSTEMS;

MINIMUM ERROR ENTROPY (MEE);

SIGNAL PROCESSING;

EID: 0038538359     PISSN: 1053587X     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSP.2003.812843     Document Type: Article

References (29)

1. N. Wiener, Extrapolation, Interpolation, and Smoothing of Stationary Time Series with Engineering Applications. Cambridge, MA: MIT Press, 1949.
2. S. Haykin, Adaptive Filter Theory. Englewood Cliffs. NJ: Prentice-Hall, 1996.
3. B. Farhang-Boroujeny, Adaptive Filters: Theory and Applications. New York: Wiley, 1998.
4. B. Widrow and S. D. Stearns, Adaptive Signal Processing. Englewood Cliffs, NJ: Prentice-Hall, 1985.
5. A. H. Sayed, "Advances and challenges in adaptive filtering," in Proc. Int. Conf. Acoust., Speech, Signal Process., Tutorial Lecture Notes 4, 2002.
6. D. Erdogmus and J. C. Principe, "Comparison of entropy and mean square error criteria in adaptive system training using higher order statistics," in Proc. ICA, Helsinki, Finland, 2000, pp. 75-80.
7. _, "An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems," IEEE Trans. Signal Processing, vol. 50, pp. 1780-1786, July 2002.
8. _, "Generalized information potential criterion for adaptive system training," IEEE Trans. Neural Networks, vol. 13, pp. 1035-1044, Sept. 2002.
9. J. C. Principe, D. Xu, and J. Fisher, "Information theoretic learning," in Unsupervised Adaptive Filtering, vol.I: Blind Source Separation, S. Haykin, Ed. New York: Wiley, 2000, pp. 265-319.
10. K. E. Hild, II, D. Erdogmus, and J. C. Principe, "Blind source separation using Renyi's mutual information," IEEE Signal Processing Lett., vol. 8, pp. 174-176, June 2001.
11. D. Erdogmus, K. E. Hild, II, and J. C. Principe, "Blind source separation using Renyi's α-marginal entropy," Neurocomput. - Special Issue on Blind Source Separation, vol. 49, no. 1, pp. 25-38, 2002.
12. D. Erdogmus, J. C. Principe, and L. Vielva, "Blind deconvolution with minimum Renyi's entropy," in Proc. EUSIPCO, vol. 2, Toulouse, France, 2002, pp. 71-74.
13. R. Y. Rubinstein, Simulation and the Monte Carlo Method. New York: Wiley, 1981.
14. A. Bell and T. Sejnowski, "An information-maximization approach to blind separation and blind deconvolution," Neural Comput., vol. 7, pp. 1129-1159, 1995.
15. D. T. Pham, "Mutual information approach to blind separation of stationary sources," IEEE Trans. Inform. Theory, vol. 48, pp. 1935-1946, Aug. 2002.
16. S. Haykin, Ed., Blind Deconvolution. Englewood Cliffs, NJ: Prentice-Hall, 1994.
17. A. O. Hero, B. Ma, O. J. J. Michel, and J. Gorman, "Applications of entropic spanning graphs," IEEE Signal Processing Mag., vol. 19, pp. 85-95, 2002.
18. K. Torkkola, "Visualizing class structure in data using mutual information," in Proc. NNSP, Sydney, Australia, 2000, pp. 376-385.
19. A. K. C. Wong and P. K. Sahoo, "A gray-level threshold selection method based on maximum entropy principle," IEEE Trans. Syst., Man, Cybern., vol. 19, pp. 866-971, July 1989.
20. R. G. Baraniuk, P. Flandrin, A. J. E. M. Janssen, and O. J. J. Michel, "Measuring time-frequency information content using the Renyi entropies," IEEE Trans. Inform. Theory, vol. 47, pp. 1391-1409, July 2001.
21. J. W. Fisher, "Nonlinear extensions to the minimum average correlation energy filter," Ph.D. dissertation, Univ. Florida, Gainesville, FL, 1997.
22. H. Akaike, "A new look at the statistical model identification," IEEE Trans. Automat. Contr., vol. AC-19, pp. 716-723, 1974.
23. A. P. Liavas and P. A. Regalia, "On the behavior of information theoretic criteria for model order selection," IEEE Trans. Signal Processing, vol. 49, pp. 1689-1695, Aug. 2001.
24. A. Renyi, Probability Theory. New York: Elsevier, 1970.
25. S. I. Amari, Differential-Geometrical Methods in Statistics. Berlin, Germany: Springer-Verlag, 1985.
26. P. Viola, N. N. Schraudolph, and T. J. Sejnowski, "Empirical entropy manipulation for real-world problems," in Proc. Advances Neural Inform. Processing Syst. VIII, 1996, pp. 851-857.
27. E. Parzen, "On estimation of a probability density function and mode," in Time Series Analysis Papers. San Francisco, CA: Holden-Day, 1967.
28. D. Erdogmus and J. C. Principe, "Convergence analysis of the information potential criterion in adaline training," in Proc. NNSP, Falmouth, MA, 2001, pp. 123-132.
29. D. Erdogmus and J. C. Principe, "An on-line adaptation algorithm for adaptive system training with minimum error entropy: Stochastic information gradient," in Proc. ICA, CA, 2001, pp. 7-12.