Mean-square convergence analysis of ADALINE training with minimum error entropy criterion

IEEE Transactions on Neural Networks

Volumn 21, Issue 7, 2010, Pages 1168-1179

  Chen, Badong ^a   Zhu, Yu ^a   Hu, Jinchun ^a,b  

^a TSINGHUA UNIVERSITY   (China)

^b NANJING UNIVERSITY OF AERONAUTICS AND ASTRONAUTICS   (China)

Author keywords

ADALINE training; energy conservation relation; mean square convergence analysis; minimum error entropy (MEE) criterion

Indexed keywords

ADALINE; ADAPTATION ALGORITHMS; ADAPTIVE LINEAR NEURONS; CONVERGENCE ANALYSIS; ENERGY CONSERVATION RELATION; MEAN SQUARE STABILITY; MEAN-SQUARE; MINIMUM ERROR ENTROPY CRITERIONS; NONCONVEX; STEADY STATE PERFORMANCE; THEORETICAL PREDICTION; TRANSIENT BEHAVIOR; UNIFIED APPROACH; WEIGHT UPDATE;

CONVERGENCE OF NUMERICAL METHODS; ENTROPY; INFORMATION THEORY;

WIRELESS TELECOMMUNICATION SYSTEMS;

ALGORITHM; ANIMAL; ARTICLE; ARTIFICIAL NEURAL NETWORK; BIOLOGICAL MODEL; COMPUTER SIMULATION; ENTROPY; INFORMATION SCIENCE; NERVE CELL; PHYSIOLOGY; STATISTICAL MODEL;

ALGORITHMS; ANIMALS; COMPUTER SIMULATION; ENTROPY; INFORMATION THEORY; LINEAR MODELS; MODELS, NEUROLOGICAL; NEURAL NETWORKS (COMPUTER); NEURONS;

EID: 77954565185     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2010.2050212     Document Type: Article

References (34)

1. S. Haykin, Adaptive Filtering Theory, 3rd ed. NewYork: Prentice Hall, 1996.
2. S. C. Douglas and T. H. Y. Meng, "Stochastic gradient adaptation under general error criteria," IEEE Trans. Signal Process., vol.42, no.6, pp. 1335-1351, Jun. 1994.
3. T. Y. Al-Naffouri, A. Zerguine, and M. Bettayeb, "A unifying view of error nonlinearities in LMS adaptation," in Proc. IEEE ICASSP, May 1998, pp. 1697-1700.
4. E. Walach and B. Widrow, "The least mean fourth (LMF) adaptive algorithm and its family," IEEE Trans. Inform. Theory, vol.IT-30, no.2, pp. 275-283, Mar. 1984.
5. S. C. Pei and C. C. Tseng, "Least mean p-power error criterion for adaptive FIR filter," IEEE J. Selected Areas Commun., vol.12, no.9, pp. 1540-1547, Dec. 1994.
6. O. Tanrikulu and J. A. Chambers, "Convergence and steady-state properties of the least-mean mixed norm (LMMN) adaptive algorithm," IEE Proc. Vision Image Signal Process., vol.143, no.3, pp. 137-142, Jun. 1996.
7. J. C. Principe, D. Xu, Q. Zhao, and J. W. Fisher, "Learning from examples with information theoretic criteria," J. VLSI Signal Process. Syst., vol.26, no.1-2, pp. 61-77, Aug. 2000.
8. D. Erdogmus and J. C. Principe, "From linear adaptive filtering to nonlinear information processing," IEEE Signal Process. Mag., vol.23, no.6, pp. 15-33, Nov. 2006.
9. D. Erdogmus and J. C. Principe, "An error entropy minimization algorithm for supervised training of nonlinear adaptive systems," IEEE Trans. Signal Process., vol.50, no.7, pp. 1780-1786, Jul. 2002.
10. D. Erdogmus and J. C. Principe, "Generalized information potential criterion for adaptive system training," IEEE Trans. Neural Netw., vol.13, no.5, pp. 1035-1044, Sep. 2002.
11. D. Erdogmus and J. C. Principe, "Convergence properties and data efficiency of the minimum error entropy criterion in ADALINE training," IEEE Trans. Signal Process., vol.51, no.7, pp. 1966-1978, Jul. 2003.
12. B. Chen, J. Hu, L. Pu, and Z. Sun, "Stochastic gradient algorithm under (h,)-entropy criterion," Circuits Syst. Signal Process., vol.26, pp. 941- 960, 2007.
13. D. Erdogmus, E. H. Kenneth, and J. C. Principe, "Online entropy manipulation: Stochastic information gradient," IEEE Signal Process. Lett., vol.10, no.8, pp. 242-245, Aug. 2003.
14. R. A. Morejon and J. C. Principe, "Advanced search algorithms for information-theoretic learning with kernel-based estimators," IEEE Trans. Neural Netw., vol.15, no.4, pp. 874-884, Jul. 2004.
15. S. Han, S. Rao, D. Erdogmus, K. H. Jeong, and J. C. Principe, "A minimum-error entropy criterion with self-adjusting step-size (MEESAS)," Signal Process., vol.87, no.11 pp. 2733-2745, 2007.
16. A. H. Sayed, Fundamentals of Adaptive Filtering. New York: Wiley, 2003.
17. T. Y. Al-Naffouri and A. H. Sayed, "Adaptive filters with error nonlinearities: Mean-square analysis and optimum design," EURASIP J. Appl. Signal Process., vol.4, pp. 192-205, Dec. 2001.
18. T. Y. Al-Naffouri and A. H. Sayed, "Transient analysis of datanormalized adaptive filters," IEEE Trans. Signal Process., vol.51, no.3, pp. 639-652, Mar. 2003.
19. T. Y. Al-Naffouri and A. H. Sayed, "Transient analysis of adaptive filters with error nonlinearities," IEEE Trans. Signal Process., vol.51, no.3, pp. 653-663, Mar. 2003.
20. N. R. Yousef and A. H. Sayed, "A unified approach to the steady-state and tracking analysis of adaptive filters," IEEE Trans. Signal Process., vol.49, no.2, pp. 314-324, Feb. 2001.
21. T. M. Cover and J. A. Thomas, Element of Information Theory. New York: Wiley, 1991.
22. A. Renyi, "On measures of entropy and information," in Proc. 4th Berkeley Symp. Math. Statist. Probability, vol.1. 1961, pp. 547-561.
23. M. Salicru, M. L. Menendez, D. Morales, and L. Pardo, "Asymptotic distribution of (h,)-entropies," Commun. Statist. (Theory Methods), vol.22, pp. 2015-2031, 1993.
24. L. Pardo, Statistical Inference Based on Divergence Measures. London, U.K.: Chapman and Hall, 2006.
25. J. Berilant, E. J. Dudewicz, L. Gyorfi, and E. C. van der Meulen, "Nonparametric entropy estimation: An overview," Int. J. Math. Statist. Sci., vol.6, no.1, pp. 17-39, 1997.
26. B. W. Silverman, Density Estimation for Statistics and Data Analysis. London, U.K.: Chapman and Hall, 1986.
27. U. Ozertem, I. Uysal, and D. Erdogmus, "Continuously differentiable sample-spacing entropy estimation," IEEE Trans. Neural Netw., vol.19, no.11, pp. 1978-1984, Nov. 2008.
28. G. A. Clark, S. K. Mitra, and S. R. Parker, "Block implementation of adaptive digital filters," IEEE Trans. Acoust. Speech Signal Process., vol.ASSP-29, no.3, pp. 744-752, Jun. 1981.
29. W. B. Mikhael and F. H. Wu, "Fast algorithms for block FIR adaptive digital filtering," IEEE Trans. Circuits Syst., vol.CAS-34, no.10, pp. 1152-1160, Oct. 1987.
30. S.C. Douglas, "Analysis of the multiple-error and block least-meansquare adaptive algorithms," IEEE Trans. Circuits Syst.-II: Analog Digital Signal Process., vol.42, no.2, pp. 92-101, Feb. 1995.
31. H.-C. Shin and A. H. Sayed, "Mean-square performance of a family of affine projection algorithms," IEEE Trans. Signal Process., vol.52, no.1, pp. 90-102, Jan. 2004.
32. N. J. Bershad and M. Bonnet, "Saturation effects in LMS adaptive echo cancellation for binary data," IEEE Trans. Acoust. Speech Signal Process., vol.38, no.10, pp. 1687-1696, Oct. 1990.
33. T. Y. Al-Naffouri, A. Zerguine, and M. Bettayeb, "Convergence analysis of the LMS algorithm with a general error nonlinearity and an iid input," in Proc. Asilomar Conf. Signals Syst. Comput., vol.1. 1998, pp. 556- 559.
34. C. G. Lopes and A. H. Sayed, "Diffusion least-mean squares over adaptive networks: Formulation and performance analysis," IEEE Trans. Signal Process., vol.56, no.7, pp. 3122-3136, Jul. 2008.