Online entropy manipulation: Stochastic information gradient

IEEE Signal Processing Letters

Volumn 10, Issue 8, 2003, Pages 242-245

  Erdogmus, Deniz ^a   Hild II, Kenneth E ^a   Principe, Jose C ^a  

^a UNIVERSITY OF FLORIDA   (United States)

Author keywords

Shannon's entropy; Stochastic gradient for entropy

Indexed keywords

ENTROPY; OPTIMIZATION; RANDOM PROCESSES;

ONLINE ENTROPY;

SIGNAL PROCESSING;

EID: 0041663375     PISSN: 10709908     EISSN: None     Source Type: Journal    
DOI: 10.1109/LSP.2003.814400     Document Type: Article

References (23)

1. C. E. Shannon, "A mathematical theory of communication," Bell Syst. Tech. J., vol. 27, pp. 379-423, 1948.
2. A. Hyvarinen, J. Karhunen, and E. Oja, Independent Component Analysis. New York: Wiley, 2001.
3. S. Haykin, Ed., Unsupervised Adaptive Filtering. New York: Wiley, 2000, vol. 1, Blind Source Separation.
4. S. Haykin, Ed., Unsupervised Adaptive Filtering. New York: Wiley, 2000, vol. 2, Blind Deconvolution.
5. A. Rényi, Probability Theory. Amsterdam, The Netherlands: North-Holland, 1970.
6. D. Donoho, "On minimum entropy deconvolution," in Applied Time Series Analysis II. New York: Academic, 1981, pp. 565-609.
7. A. Benveniste, M. Goursat, and G. Ruget, "Robust identification of a nonminimum phase system: Blind adjustment of a linear equalizer in data communications," IEEE Trans. Automat. Contr., vol. 25, pp. 385-399, June 1980.
8. D. Erdogmus and J. C. Principe, "An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems," IEEE Trans. Signal Processing, vol. 50, pp. 1780-1786, July 2002.
9. _, "Generalized information potential criterion for adaptive system training," IEEE Trans. Neural Networks, vol. 13, pp. 1035-1044, Sept. 2002.
10. A. Bell and T. Sejnowski, "An information-maximization approach to blind separation and blind deconvolution," Neural Comput., vol. 7, pp. 1129-1159, 1995.
11. G. Miller and D. Horn, "Probability density estimation using entropy maximization," Neural Comput., vol. 10, no. 7, pp. 1925-1938, 1998.
12. A. Touzni, I. Fijalkow, M. G. Larimore, and J. R. Treichler, "A globally convergent approach for blind MIMO adaptive deconvolution," IEEE Trans. Signal Processing, vol. 49, pp. 1166-1178, June 2001.
13. E. T. Jaynes, Papers on Probability, Statistics and Statistical Physics, R. D. Rosenkrantz, Ed. Dordrecht, The Netherlands: D. Reidel, 1983.
14. A. Hyvarinen, "New approximations of differential entropy for independent component analysis and projection pursuit," Helsinki Univ. Technol., Helsinki, Finland, Rep. A47, 1997.
15. P. Viola, N. N. Schraudolph, and T. J. Sejnowski, "Empirical entropy manipulation for real-world problems," Advances in Neural Information Processing Systems, vol. 8, pp. 851-857, 1995.
16. K. Torkkola, "On feature extraction by mutual information maximization," in Proc. ICASSP, Orlando, FL, 2002, pp. 821-824.
17. J. C. Principe, J. W. Fisher, and D. Xu, "Information theoretic learning," in Unsupervised Adaptive Filtering, S. Haykin, Ed. New York: Wiley, 2000, pp. 265-319.
18. D. Erdogmus, K. E. Hild II, and J. C. Principe, "Blind source separation using Renyi's α-marginal entropies," Neurocomput., vol. 49, no. 1, pp. 25-38, Dec. 2002.
19. E. Parzen, "On estimation of a probability density function and mode," in Time Series Analysis Papers. San Francisco, CA: Holden-Day, 1967.
20. S. R. Sain and D. W. Scott, "On locally adaptive density estimation," J. Amer. Stat. Assoc., vol. 91, pp. 1525-1534, 1996.
21. L. Devroye and G. Lugosi, Combinatorial Methods in Density Estimation. New York: Springer, 2001.
22. D. Erdogmus, Y. N. Rao, J. C. Principe, J. Zhao, and K. E. Hild II, "Simultaneous extraction of principal components using Givens rotations and output variances," in Proc. ICASSP, Orlando, FL, 2002, pp. 1069-1072.
23. K. E. Hild II, D. Erdogmus, and J. C. Principe, "Blind source separation of time-varying, instantaneous mixtures using an on-line algorithm," in Proc. ICASSP, Orlando, FL, 2002, pp. 993-996.