A reproducing kernel Hilbert space framework for information-theoretic learning

IEEE Transactions on Signal Processing

Volumn 56, Issue 12, 2008, Pages 5891-5902

  Xu, Jian Wu ^a   Paiva, António R C ^a   Principe, Jose C ^a  

^a University of Florida   (United States)

Author keywords

Cross information potential; Information theoretic learning (ITL); Kernel function; Probability density function; Reproducing kernel Hilbert space (RKHS)

Indexed keywords

ALGEBRAIC COMPUTATIONS; INFORMATION POTENTIAL; INFORMATION THEORETIC LEARNING; KERNEL APPROACHES; KERNEL FUNCTION; KERNEL METHODS; REPRODUCING KERNEL HILBERT SPACE (RKHS); REPRODUCING KERNEL HILBERT SPACES; SQUARE INTEGRABLE; STATISTICAL DESCRIPTORS; STOCHASTIC FUNCTIONS;

ALGEBRA; DENSITY FUNCTIONAL THEORY; HILBERT SPACES; INFORMATION THEORY; PROBABILITY;

PROBABILITY DENSITY FUNCTION;

EID: 70449354098     PISSN: 1053587X     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSP.2008.2005085     Document Type: Article

References (45)

1. E. H. Moore, "On properly positive Hermitian matrices", Bull. Amer. Math. Soc., vol. 23, no. 59, pp. 66-67, 1916.
2. N. Aronszajn, "The theory of reproducing kernels and their applications", Cambridge Phil. Soc. Proc., vol. 39, pp. 133-153, 1943.
3. E. Parzen, "Statistical inference on time series by Hilbert space methods", Statistics Dept., Stanford Univ., Stanford, CA, Tech. Rep. 23, 1959.
4. E. Parzen, "An approach to time series analysis", Ann. Math. Stat., vol. 32, no. 4, pp. 951-989, Dec. 1961.
5. E. Parzen, "Extraction and detection problems and reproducing kernel Hilbert spaces", SIAM J. Contr., vol. 1, pp. 35-62, 1962.
6. T. Kailath, "RKHS approach to detection and estimation problems-Part I: Deterministic signals in Gaussian noise", IEEE Trans. Inf. Theory, vol. IT-17, pp. 530-549, Sep. 1971.
7. T. Kailath and H. Weinert, "An RKHS approach to detection and estimation problems-Part II: Gaussian signal detection", IEEE Trans. Inf. Theory, vol. IT-21, pp. 15-23, Jan. 1975.
8. T. Kailath and D. Duttweiler, "An RKHS approach to detection and estimation problems-Part III: Generalized innovations representations and a likelihood-ratio formula", IEEE Trans. Inf. Theory, vol. IT-18, pp. 730-745, Nov. 1972.
9. D. Duttweiler and T. Kailath, "RKHS approach to detection and estimation problems-Part IV: Non-Gaussian detection", IEEE Trans. Inf. Theory, vol. IT-19, pp. 19-28, Jan. 1973.
10. D. Duttweiler and T. Kailath, "RKHS approach to detection and estimation problems-Part V: Parameter estimation", IEEE Trans. Inf. Theory, vol. IT-19, pp. 29-37, Jan. 1973.
11. G. Wahba, Spline Models for Observational Data. Philadelphia, PA: SIAM, 1990, vol. 49.
12. R. J. DeFigueiredo, "A generalized Fock space framework for nonlinear system and signal analysis", IEEE Trans. Circuits Syst., vol. CAS-30, pp. 637-647, Sep. 1983.
13. V. Fock, The Theory of Space Time and Gravitation. London, U. K.: Pergamon, 1959.
14. V. N. Vapnik, Statistical Learning Theory. New York: Wiley, 1998.
15. C. Burges, "A tutorial on support vector machines for pattern recognition", Data Mining Knowl. Disc., vol. 2, no. 2, pp. 121-167, 1998. (Pubitemid 128695475)
16. M. G. Genton, "Classes of kernels for machine learning: A statistics perspective", J. Mach. Learn. Res., vol. 2, pp. 299-312, 2001.
17. T. M. Cover, "Classification and generalization capabilities of linear threshold units", Rome Air Force, Italy, Tech. Rep. RADC-TDR-64-32, Feb. 1964.
18. B. Schölkopf and A. Smola, Learning With Kernels. Cambridge, MA: MIT Press, 2002.
19. B. Schölkopf, A. Smola, and K.-R. Müller, "Nonlinear component analysis as a kernel eigenvalue problem", Neur. Comput., vol. 10, pp. 1299-1319, 1998. (Pubitemid 128463674)
20. F. R. Bach and M. I. Jordan, "Kernel independent component analysis", J. Mach. Learn. Res., vol. 3, pp. 1-48, 2002.
21. J. C. Principe, D. Xu, and J. W. Fisher, "Information theoretic learning", in Unsupervised Adaptive Filtering, S. Haykin, Ed. New York: Wiley, 2000, pp. 265-319.
22. D. Erdogmus and J. C. Principe, "From linear adaptive filtering to nonlinear information processing", IEEE Signal Process. Mag., vol. 23, pp. 14-33, Nov. 2006.
23. A. Rényi, "On measures of entropy and information", in Selected Papers of A. Rényi. Budapest, Hungary: Akademiai Kiado, 1976, vol. 2, pp. 565-580.
24. E. Gokcay and J. C. Principe, "Information theoretic clustering", IEEE Trans. Pattern Anal. Machine Intell., vol. 24, pp. 158-171, Feb. 2002. (Pubitemid 34198206)
25. R. Jenssen and T. Eltoft, "An information theoretic perspective to Mercer kernel-based clustering", Neurocomputing, 2008. to be published.
26. I. Hild, K. E. D. Erdogmus, K. Torkkola, and J. Principe, "Feature extraction using information-theoretic learning", IEEE Trans. Pattern Anal. Machine Intell., vol. 28, pp. 1385-1392, Sep. 2006. (Pubitemid 44132646)
27. D. Erdogmus and J. C. Principe, "An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems", IEEE Trans. Signal Process., vol. 50, no. 7, pp. 1780-1786, Jul. 2002. (Pubitemid 34882696)
28. M. Lazaro, I. Santamaria, D. Erdogmus, K. Hild, C. Pantaleon, and J. Principe, "Stochastic blind equalization based on pdf fitting using Parzen estimator", IEEE Trans. Signal Process., vol. 53, no. 2, pp. 696-704, Feb. 2005. (Pubitemid 40181763)
29. R. Jenssen, D. Erdogmus, J. Principe, and T. Eltoft, "Some equivalences between kernel methods and information theoretic methods", J. VLSI Signal Process., vol. 45, pp. 49-65, 2006. (Pubitemid 46132343)
30. A. N. Kolmogorov, "Sur la notion de la moyenne", Atti della Reale Accademia Nazionale dei Lincei, Serie VI, Rendiconti, vol. 12, pp. 388-391, 1930.
31. E. Parzen, "On estimation of a probability density function and mode", Ann. Math. Statist., vol. 33, no. 3, pp. 1065-1076, Sept. 1962.
32. S. Kullback and R. A. Leibler, "On information and sufficiency", Ann. Math. Statist., vol. 22, no. 1, pp. 79-86, 1951.
33. D. Xu, J. C. Principe, J. W. Fisher, and H. C. Wu, "A novel measure for independent component analysis", in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 1998, vol. 2, pp. 12-15.
34. R. Jenssen, D. Erdogmus, J. C. Principe, and T. Eltoft, "The Laplacian PDF distance: A cost function for clustering in a kernel feature space", in Advances in Neural Information Processing Systems (NIPS). Cambridge, MA: MIT Press, 2004. pp. 625-632.
35. J. Mercer, "Functions of positive and negative type, and their connection with the theory of integral equations", Phil. Trans. Roy. Soc. London, vol. 209, pp. 415-446, 1909.
36. R. Jenssen, J. C. Principe, D. Erdogmus, and T. Eltoft, "The Cauchy-Schwarz divergence and Parzen windowing: Connections to graph theory and Mercer kernels", J. Franklin Inst., vol. 343, pp. 614-629, 2006. (Pubitemid 44665871)
37. I. Santamaria, P. Pokharel, and J. C. Príncipe, "Generalized correlation function: Definition, properties, and application to blind equalization", IEEE Trans. Signal Process., vol. 54, no. 6, pp. 2187-2197, Jun. 2006. (Pubitemid 43811412)
38. E. Kreyszig, Introductory Functional Analysis with Applications. New York: Wiley, 1978.
39. A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, and A. Smola, "A kernel method for the two-sample-problem", in Advances in Neural Information Processing Systems (NIPS), Vancouver, BC, Canada, 2006.
40. S.-I. Amari and H. Nagaoka, Methods of Information Geometry. Providence, RI: AMS/Oxford Univ. Press, 2000.
41. B. Pistone and C. Sempi, "An infinite-dimensional geometric structure on the space of all probability measures equivalent toa given one", Ann. Statist., vol. 23, no. 5, pp. 1543-1561, 1995. (Pubitemid 126033100)
42. C. R. Rao, "Information and accuracy attainable in the estimation of statistical parameters", Bull. Cal. Math. Soc., vol. 37, pp. 81-91, 1945.
43. T. Jebara, R. Kondor, and A. Howard, "Probability product kernels", J. Mach. Learn. Res., vol. 5, pp. 819-844, 2004.
44. T. Jebara and R. Kondor, "Bhattarcharyya and expected likelihood kernels", presented at the Conf. Learning Theory (COLT), Washington, DC, 2003.
45. M. Hein and O. Bousquet, "Hilbertian metrics and positive definite kernels on probability measures", in Proc. 10th Int. Workshop on Artificial Intelligence Statistics (AISTATS), 2005, pp. 136-143.