Information theoretic interpretation of error criteria

Zidonghua Xuebao/ Acta Automatica Sinica

Volumn 35, Issue 10, 2009, Pages 1302-1309

  Chen, Ba Dong ^a   Hu, Jin Chun ^a   Zhu, Yu ^a   Sun, Zeng Qi ^a  

^a TSINGHUA UNIVERSITY   (China)

Author keywords

Adaptive filtering; Entropy; Error criteria; Estimation; Kullback Leibler information divergence (KL divergence)

Indexed keywords

ENTROPY CRITERIA; ERROR COST FUNCTION; ERROR CRITERION; KL-DIVERGENCE; KULLBACK-LEIBLER INFORMATION; MONTE CARLO SIMULATION; POWER ERRORS; STATISTICAL ESTIMATION;

ADAPTIVE FILTERING; ADAPTIVE FILTERS; COMPUTER SIMULATION; COST FUNCTIONS; ESTIMATION; INFORMATION THEORY; MONTE CARLO METHODS;

ENTROPY;

EID: 70749126726     PISSN: 02544156     EISSN: None     Source Type: Journal    
DOI: 10.3724/SP.J.1004.2009.01302     Document Type: Article

References (36)

1. Sherman S. Non-mean-square error criteria. IRE Transactions on Information Theory, 1958, 4(3): 125-126.
2. Hall E B, Wise G L. On optimum estimation with respect to a large family of cost functions. IEEE Transactions on Information Theory, 1991, 37(3): 691-693.
3. Haykin S. Adaptive Filtering Theory (Third Edition). New York: Prentice Hall, 1996.
4. Kailath T, Sayed A H, Hassibi B. Linear Estimation. New Jersey: Prentice Hall, 2000.
5. Douglas S C, Meng T H Y. Stochastic gradient adaptation under general error criteria. IEEE Transactions on Signal Processing, 1994, 42(6): 1335-1351.
6. Pei S C, Tseng C C. Least mean p-power error criterion for adaptive FIR filter. IEEE Journal on Selected Areas in Communications, 1994, 12(9): 1540-1547.
7. Widrow B, Walach E. Adaptive Inverse Control. New York: Prentice Hall, 1996.
8. Wang Z, Bovik A C. Mean squared error: love it or leave it? IEEE Signal Processing Magazine, 2009, 26(1): 98-117.
9. Erdogmus D, Principe J C. From linear adaptive filtering to nonlinear information processing. IEEE Signal Processing Magazine, 2006, 23(6): 14-33.
10. Brown J L. Asymmetric non-mean-square error criteria. IRE Transactions on Automatic Control, 1962, 7(1): 64-66.
11. Zakai M. General error criteria. IEEE Transactions on Information Theory, 1964, 10(1): 94-95.
12. Walach E, Widrow B. The least mean fourth (LMF) adaptive algorithm and its family. IEEE Transactions on Information Theory, 1984, 30(2): 275-283.
13. Min S, Nikias C L. Signal processing with fractional lower order moments: stable processes and their applications. Proceedings of the IEEE, 1993, 81(7): 986-1010.
14. Cover T M, Thomas J A. Element of Information Theory. Chichester: Wiley and Son Inc., 1991.
15. Weidemann H L, Stear E B. Entropy analysis of parameter estimation. Information and Control, 1969, 14(6): 493-506.
16. Weidemann H L, Stear E B. Entropy analysis of estimating systems. IEEE Transactions on Information Theory, 1970, 16(3): 264-270.
17. Minamide N. An extension of the entropy theorem for parameter estimation. Information and Computation, 1982, 53(1-2): 81-90.
18. Erdogmus D, Hild K E, Principe J C. Online entropy manipulation: stochastic information gradient. IEEE Signal Processing Letters, 2003, 10(8): 242-245.
19. Erdogmus D, Principe J C. Generalized information potential criterion for adaptive system training. IEEE Transactions on Neural Networks, 2002, 13(5): 1035-1044.
20. Erdogmus D, Principe J C. Convergence properties and data efficiency of the minimum error entropy criterion in ada-line training. IEEE Transactions on Signal Processing, 2003, 51(7): 1966-1978.
21. Erdogmus D, Principe J C. Comparison of entropy and mean square error criteria in adaptive system training using higher order statistics. In: Proceedings of the 2nd International Workshop on Independent Component Analysis and Blind Signal Separation. Berlin, German: Springer, 2000. 75-80.
22. Principe J C, Xu D, Zhao Q, Fisher J W. Learning from examples with information theoretic criteria. The Journal of VLSI Signal Processing, 2000, 26(1-2): 61-77.
23. Chen B D, Hu J C, Pu L, Sun Z Q. Stochastic gradient algorithm under (h, φ)-entropy criterion. Circuits, Systems, and Signal Processing, 2007, 26(6): 941-960.
24. Han S, Rao S, Erdogmus D, Jeong K H, Principe J C. A minimum-error entropy criterion with self-adjusting step-size (MEE-SAS). Signal Processing, 2007, 87(11): 2733-2745.
25. Wolsztynski E, Thierry E, Pronzato L. Minimum entropy estimation in semi-parametric models. Signal Processing, 2005, 85(5): 937-949.
26. Silva L M, Felgueiras C S, Alexandre L A, Marques de Sa J. Error entropy in classification problems: a univariate data analysis. Neural Computation, 2006, 18(9): 2036-2061.
27. Kapur J N, Kesavan H K. Entropy Optimization Principles with Applications. New York: Academic Press, 1992.
28. Karmeshu J. Entropy Measures, Maximum Entropy Principle and Emerging Applications. Berlin: Springer-Verlag, 2003.
29. Ling T, Taniar D. Adaptive estimated maximum-entropy distribution model. Information Science, 2007, 177(15): 3110-3128.
30. Bryson A E, Ho Y C. Applied and Optimal Control, Optimization, Estimation, and Control. New York: Hemisphere Publishing, 1975. 47-48.
31. Varanasi M K, Aazhang B. Parametric generalized Gaussian density estimation. The Journal of the Acoustical Society of America, 1989, 86(4): 1404-1415.
32. Kokkinakis K, Nandi A K. Exponent parameter estimation for generalized Gaussian probability density functions with application to speech modeling. Signal Processing, 2005, 85(9): 1852-1858.
33. Lo J T, Wanner T. Existence and uniqueness of risksensitivity estimates. IEEE Transactions on Automatic Control, 2002, 47(11): 1945-1948.
34. Xiao Y, Tadokoro Y, Shida K. Adaptive algorithm based on least mean p-power error criterion for Fourier analysis in additive noise. IEEE Transactions on Signal Processing, 1999, 47(4): 1172-1181.
35. Chambers J, Avlonitis A. A robust mixed-norm adaptive filter algorithm. IEEE Signal Processing Letters, 1997, 4(2): 46-48.
36. Cho S H, Kim S D. Adaptive filters based on the high order error statistics. In: Proceedings of IEEE Asia Pacific Conference on Circuits and Systems. Seoul, South Korea: IEEE, 1996. 109-112.