The Cauchy-Schwarz divergence and Parzen windowing: Connections to graph theory and Mercer kernels

Journal of the Franklin Institute

Volumn 343, Issue 6 SPEC. ISS., 2006, Pages 614-629

  Jenssen, Robert ^a   Principe, Jose C ^b   Erdogmus, Deniz ^c   Eltoft, Torbjørn ^a  

^a UNIVERSITY OF TROMSØ   (Norway)

^b University of Florida   (United States)

^c OREGON HEALTH AND SCIENCE UNIVERSITY   (United States)

Author keywords

Cauchy Schwarz divergence; Graph cut; Information theory; Mercer kernel theory; Parzen windowing; Spectral methods

Indexed keywords

INFORMATION THEORY; LEARNING SYSTEMS; PARAMETER ESTIMATION; PROBABILITY DENSITY FUNCTION; SPECTRUM ANALYSIS;

CAUCHY-SCHWARZ (CS) DIVERGENCE; GRAPH CUT; MERCER KERNEL THEORY; PARZEN WINDOWING; SPECTRAL METHODS;

GRAPH THEORY;

EID: 33750503776     PISSN: 00160032     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jfranklin.2006.03.018     Document Type: Article

References (56)

1. Principe J., Xu D., and Fisher J. Information theoretic learning. In: Haykin S. (Ed). Unsupervised Adaptive Filtering vol. I (2000), Wiley, New York (Chapter 7)
2. Shannon C.E. A mathematical theory of communication. Bell Syst. Tech. J. 27 (1948) 379-423 623-653
3. A. Renyi, On measures of entropy and information, Selected Papers of Alfred Renyi, vol. 2, Akademiai Kiado, Budapest, 1976, pp. 565-580
4. Erdogmus D., Agrawal R., and Principe J.C. A mutual information extension to the matched filter. Signal Process. 85 5 (2005) 927-935
5. Lazaro M., Santamaria I., Erdogmus D., Hild II K.E., Pantaleon C., and Principe J.C. Stochastic blind equalization based on PDF fitting using Parzen estimator. IEEE Trans. Signal Process. 53 2 (2005) 696-704
6. Erdogmus D., Hild K.E., Rao Y.N., and Principe J.C. Minimax mutual information approach for independent component analysis. Neural Comput. 16 (2004) 1235-1252
7. Erdogmus D., and Principe J.C. Convergence properties and data efficiency of the minimum error-entropy criterion in adaline training. IEEE Trans. Signal Process. 51 7 (2003) 1966-1978
8. Erdogmus D., Hild K.E., and Principe J.C. Blind source separation using Renyi's α-marginal entropies. Neurocomputing 49 (2002) 25-38
9. Santamaria I., Erdogmus D., and Principe J.C. Entropy minimization for supervised digital communications channel equalization. IEEE Trans. Signal Process. 50 5 (2002) 1184-1192
10. Erdogmus D., and Principe J.C. Generalized information potential criterion for adaptive system training. IEEE Trans. Neural Networks 13 5 (2002) 1035-1044
11. Erdogmus D., and Principe J.C. An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems. IEEE Trans. Signal Process. 50 7 (2002) 1780-1786
12. Principe J.C., Xu D., Zhao Q., and Fisher J.W. Learning from examples with information theoretic criteria. J. VLSI Signal Process. 26 1 (2000) 61-77
13. Kullback S., and Leibler R.A. On information and sufficiency. Ann. Math. Stat. 22 1 (1951) 79-86
14. R. Jenssen, An information theoretic approach to machine learning, Ph.D. Thesis, University of Tromsø, Tromsø, Norway, 2005.
15. Chung F.R.K. Spectral Graph Theory (1997), American Mathematical Society, Providence, RI
16. Wu Z., and Leahy R. An optimal graph theoretic approach to data clustering: theory and its applications to image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 15 11 (1993) 1101-1113
17. Shawe-Taylor J., and Cristianini N. Kernel Methods for Pattern Analysis (2004), Cambridge University Press, Cambridge
18. Müller K.R., Mika S., Rätsch G., Tsuda K., and Schölkopf B. An introduction to kernel-based learning algorithms. IEEE Trans. Neural Networks 12 2 (2001) 181-201
19. F. Perez-Cruz, O. Bousquet, Kernel methods and their potential use in signal processing, IEEE Signal Processing Magazine, May 2004, pp. 57-65.
20. Schölkopf B., and Smola A.J. Learning with Kernels (2002), MIT Press, Cambridge
21. Cortes C., and Vapnik V.N. Support vector networks. Mach. Learn. 20 (1995) 273-297
22. Vapnik V.N. The Nature of Statistical Learning Theory (1995), Springer, New York
23. Cristianini N., and Shawe-Taylor J. An Introduction to Support Vector Machines (2000), Cambridge University Press, Cambridge
24. Burges C.J.C. A tutorial on support vector machines for pattern recognition. Knowl. Discovery Data Min. 2 2 (1998) 121-167
25. Roweis S., and Saul L. Nonlinear dimensionality reduction by locally linear embedding. Science 290 (2000) 2323-2326
26. Tenenbaum J., deSilva V., and Langford J.C. A global geometric framework for nonlinear dimensionality reduction. Science 290 (2000) 2319-2323
27. Belkin M., and Niyogi P. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput. 15 (2003) 1373-1396
28. Ng A.Y., Jordan M., and Weiss Y. On spectral clustering: analysis and an algorithm. Advances in Neural Information Processing Systems vol. 14 (2002), MIT Press, Cambridge 849-856
29. Weiss Y. Segmentation using eigenvectors. a unifying view. Proceedings of IEEE International Conference on Computer Vision. Corfu, Greece (September 20-25, 1999) 975-982
30. Schölkopf B., Smola A.J., and Müller K.R. Nonlinear component analysis as a kernel eigenvalue problem. Neural Comput. 10 (1998) 1299-1319
31. Silverman B.W. Density Estimation for Statistics and Data Analysis (1986), Chapman & Hall, London
32. Scott D.W. Multivariate Density Estimation (1992), Wiley, New York
33. Wand M.P., and Jones M.C. Kernel Smooting (1995), Chapman & Hall, London
34. Parzen E. On the estimation of a probability density function and the mode. Ann. Math. Stat. 32 (1962) 1065-1076
35. Simonoff J. Smoothing Methods in Statistics (1996), Springer, New York
36. Park B., and Marron J.S. Comparison of data driven bandwidth selectors. J. Am. Stat. Assoc. 85 (1990) 66-72
37. Sheater S.J., and Jones M.C. A reliable data-based bandwidth selection method for kernel density estimation. J. R. Stat. Soc. Ser. B 53 (1991) 683-690
38. Jones M.C., Marron J.S., and Sheater S.J. A brief survey of bandwidth selection for density estimation. J. R. Stat. Soc. 87 (1996) 227-233
39. Abramson I. On bandwidth variation in kernel estimates. Ann. Stat. 10 4 (1982) 1217-1223
40. Hall P., Hu T.-C., and Marron J.S. Improved variable window kernel estimates of probability densities. Ann. Stat. 23 (1995) 1-10
41. Chernoff H. A measure of asymptotic efficiency for tests of a hypothesis based on a sum of observations. Ann. Math. Stat. 23 (1952) 493-507
42. Jenssen R., Principe J.C., and Eltoft T. Information cut and information forces for clustering. Proceedings of IEEE International Workshop on Neural Networks for Signal Processing. Toulouse, France (17-19 September, 2003) 459-468
43. Shi J., and Malik J. Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22 8 (2000) 888-905
44. Ding C.H.Q., He X., Zha H., Gu M., and Simon H.D. A min-max cut algorithm for graph partitioning and data clustering. Proceedings of IEEE International Conference on Data Mining. San Jose, USA (November 29-December 2, 2001) 107-114
45. Gdalyahu Y., Weinshall D., and Werman M. Self-organization in vision: stochastic clustering for image segmentation, perceptual grouping, and image database organization. IEEE Trans. Pattern Anal. Mach. Intell. 23 10 (2001) 1053-1074
46. Scanlon J., and Deo N. Graph-theoretic algorithms for image segmentation. IEEE International Symposium on Circuits and Systems. Orlando, Florida (1999) VI141-VI144
47. Mercer J. Functions of positive and negative type and their connection with the theory of integral equations. Philos. Trans. R. Soc. London Ser. A (1909) 415-446
48. K.R. Müller, A.J. Smola, G. Rätsch, B. Schölkopf, J. Kohlmorgen, V.N. Vapnik, Predicting time series with support vector machines, in: Proceedings of International Conference on Artificial Neural Networks, Lecture Notes in Computer Science, vol. 1327, Springer, Berlin, 1997, pp. 999-1004.
49. Williams C., and Seeger M. Using the Nyström method to speed up kernel machines. Advances in Neural Information Processing Systems vol. 13 (2001), MIT Press, Cambridge 682-688
50. Y. Bengio, P. Vincent, J.-F. Paiement, Spectral clustering and kernel PCA are learning eigenfunctions, Technical Report, Département d'informatique et recherche opérationelle, université de Montréal, Montréal, Canada, 2003.
51. Jenssen R., Erdogmus D., Principe J.C., and Eltoft T. The laplacian spectral classifier. Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing. Philadelphia, USA (26-23 March, 2005) 325-328
52. Jenssen R., Erdogmus D., Principe J.C., and Eltoft T. Towards a unification of information theoretic learning and kernel methods. Proceedings of IEEE Workshop on Machine Learning for Signal Processing. Sao Luis, Brazil (29 September-1 October, 2004) 93-102
53. Jenssen R., Erdogmus D., Principe J.C., and Eltoft T. The laplacian PDF distance: a cost function for clustering in a kernel feature space. Advances in Neural Information Processing Systems 17 (2005), MIT Press, Cambridge 625-632
54. Jenssen R., Principe J.C., and Eltoft T. Cauchy-Schwartz pdf divergence measure for non-parametric clustering. Proceedings of IEEE Norway Section Signal Processing Symposium (cd-rom). Bergen, Norway (2-4 October, 2003)
55. Jenssen R., Erdogmus D., Hild K.E., Principe J.C., and Eltoft T. Optimizing the Cauchy-Schwarz PDF distance for information theoretic, non-parametric clustering. Proceedings of International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition. St. Augustine, USA (9-11 November, 2005) 34-45
56. Jenssen R., Eltoft T., and Principe J.C. Information theoretic spectral clustering. Proceedings of International Joint Conference on Neural Networks. Budapest, Hungary (July 25-29, 2004) 111-116