Nonextensive information theoretic kernels on measures

Journal of Machine Learning Research

Volumn 10, Issue , 2009, Pages 935-975

  Martins, André F T ^a   Smith, Noah A ^a   Xing, Eric P ^a   Aguiar, Pedro M Q ^b   Figueiredo, Mário A T ^b  

^a Carnegie Mellon University   (United States)

^b INSTITUTO SUPERIOR TÉCNICO   (Portugal)

Author keywords

Jensen Shannon divergence; Nonextensive information theory; Positive definite kernels; String kernels; Tsallis entropy

Indexed keywords

JENSEN-SHANNON DIVERGENCE; NONEXTENSIVE INFORMATION THEORY; POSITIVE DEFINITE KERNELS; STRING KERNELS; TSALLIS ENTROPY;

ENTROPY; TEXT PROCESSING;

INFORMATION THEORY;

EID: 66549127714     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (56)

1. S. Abe. Foundations of nonextensive statistical mechanics. In Chaos, Nonlinearity, Complexity. Springer, 2006.
2. S. Amari and H. Nagaoka. Methods of Information Geometry. Oxford University Press, 2001.
3. R. Baeza-Yates, and B. Ribeiro-Neto. Modern information retrieval. ACM Press New York, 1999.
4. A. Banerjee, S. Merugu, I. Dhillon, and J. Ghosh. Clustering with Bregman divergences. J. of Mach. Learn. Res., 6:1705-1749, 2005.
5. A. Ben-Hamza. A nonextensive information-theoretic measure for image edge detection. J. of Electronic Imaging, 15-1:13011.1-13011.8, 2006.
6. A. Ben-Hamza and H. Krim. Image registration and segmentation by maximizing the jensen-rényi divergence. In Proc. of Int. Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition, pages 147-163. Springer, 2003.
7. C. Berg, J. Christensen, and P. Ressel. Harmonic Analysis on Semigroups. Springer, Berlin, 1984.
8. J. Burbea and C. Rao. On the convexity of some divergence measures based on entropy functions. IEEE Trans. on Information Theory, 28(3):489-495, 1982.
9. C. Cortes, P. Haffner and M. Mohri. Rational Kernels: Theory and Algorithms. J. of Mach. Learn. Res., 5:1035-1062, 2004.
10. T. Cover and J. Thomas. Elements of Information Theory. Wiley, 1991.
11. I. Csiszar. I-divergence geometry of probability distributions and minimization problems. The Annals of Probability, 3(1):146-158, 1975.
12. M. Cuturi and J.-P. Vert. Semigroup kernels on finite sets. In L. Saul, Y. Weiss, and L. Bottou, editors, Advances in Neural Information Processing Systems 17, pages 329-336. MIT Press, Cambridge, MA, 2005.
13. M. Cuturi, K. Fukumizu, and J.-P. Vert. Semigroup kernels on measures. J. of Mach. Learn. Res., 6:1169-1198, 2005.
14. Z. Daróczy. Generalized information functions. Information and Control, 16(1):36-51, 1970.
15. F. Desobry, M. Davy, and W. Fitzgerald. Density kernels on unordered sets for kernel-based signal processing. In Proc. of the IEEE Int. Conf. on Acoustics, Speech, and Signal Proc., 2007.
16. R. El-Yaniv, S. Fine, and N. Tishby. Agnostic classification of markovian sequences. In M. Jordan, M. Kearns, and S. Solla, editors, Advances in Neural Information Processing Systems 10, pages 465-471. MIT Press, Cambridge, MA, 1998.
17. D. Endres and J. Schindelin. A new metric for probability distributions. IEEE Trans. on Information Theory, 49(7):1858-1860, 2003.
18. B. Fuglede. Spirals in Hilbert space, with an application in information theory. Expositiones Math-ematicae, 25(1):23-46, 2005.
19. S. Furuichi. Information theoretical properties of Tsallis entropies. J. of Mathematical Physics, 47 (2), 2006.
20. M. Gell-Mann and C. Tsallis. Nonextensive Entropy: Interdisciplinary Applications. Oxford University Press, 2004.
21. I. Grosse, P. Bernaola-Galvan, P. Carpena, R. Roman-Roldan J. Oliver, and H. E. Stanley. Analysis of symbolic sequences using the Jensen-Shannon divergence. Phys. Review E, 65, 2002.
22. D. Gusfield. Algorithms on Strings, Trees, and Sequences. Cambridge University Press, 1997.
23. D. Haussler. Convolution kernels on discrete structures. In Technical Report UCS-CRL-99-10, 1999.
24. M. Havrda and F. Charvát. Quantification method of classification processes: concept of structural α-entropy. Kybernetika, 3:30-35, 1967.
25. Y. He, A. Ben-Hamza, and H. Krim. A generalized divergence measure for robust image registration. IEEE Trans. on Signal Proc., 51(5):1211-1220, 2003.
26. M. Hein and O. Bousquet. Hilbertian metrics and positive definite kernels on probability measures. In Z. Ghahramani and R. Cowell, editors, Proc. of the 10th Int. Workshop on Artificial Intell. and Stat., 2005.
27. M. Hein, T. Lal, and O. Bousquet. Hilbertian metrics on probability measures and their application in SVMs. In Proc. of the 26th DAGM Symposium, pages 270-277. Springer, 2004.
28. T. Jebara, R. Kondor, and A. Howard. Probability product kernels. J. of Mach. Learn. Res., 5: 819-844, 2004.
29. J. Jensen. Sur les fonctions convexes et les inegalites entre les valeurs moyennes. Acta Mathematica, 30:175-193, 1906.
30. T. Joachims. Learning to Classify Text Using Support Vector Machines: Methods, Theory and Algorithms. Kluwer Academic Publishers, 2002.
31. D. Karakos, S. Khudanpur, J. Eisner, and C. Priebe. Iterative denoising using Jensen-Renyi divergences with an application to unsupervised document categorization. In Proc. ofIEEE Int. Conf. on Acoustics, Speech and Signal Proc., Baltimore, MD, 2007.
32. A. Khinchin. Mathematical Foundations of Information Theory. Dover, New York, 1957.
33. S. Kullback and R.A. Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 79-86, 1951.
34. J. Lafferty and G. Lebanon. Diffusion kernels on statistical manifolds. J. of Mach. Lear. Res., 6: 129-163, 2005.
35. P. Lamberti and A. Majtey. Non-logarithmic Jensen-Shannon divergence. Physica A: Statistical Mechanics and its Applications, 329:81-90, 2003.
36. C. Leslie, E. Eskin, and W. Noble. The spectrum kernel: A string kernel for svm protein classification. In Proc. of the Pacific Symposium on Biocomputing, pages 564-575, 2002.
37. Y. Li, X. Fan, and G. Li. Image segmentation based on Tsallis-entropy and Renyi-entropy and their comparison. In IEEE Int. Conf. on Industrial Informatics, pages 943-948, 2006.
38. J. Lin. Divergence measures based on the Shannon entropy. IEEE Trans. on Information Theory, 37(1):145-151, 1991.
39. J. Lin and S. Wong. A new directed divergence measure and its characterization. Int. J. of General Systems, 17:73-81, 1990.
40. J. Lindhard. On the Theory of Measurement and Its Consequences in Statistical Dynamics. Munks-gaard, Copenhagen, 1974.
41. J. Lindhard and V. Nielsen. Studies in Statistical Dynamics. Munksgaard, Copenhagen, 1971.
42. H. Lodhi, C. Saunders, J. Shawe-Taylor, N. Cristianini, and C. Watkins. Text classification using string kernels. J. of Mach. Learn. Res., 2:419-444, 2002.
43. A. F. T. Martins, P. M. Q. Aguiar, and M. A. T. Figueiredo. Tsallis kernels on measures. In Proc. of the IEEE Information Theory Workshop, Porto, Portugal, 2008a.
44. A. F. T. Martins, M. A. T. Figueiredo, P. M. Q. Aguiar, N. A. Smith, and E. P. Xing. Nonextensive entropic kernels. In Proc. of the Int. Conf. on Machine Learning - ICML '08, Helsinki, Finland, 2008b.
45. P. Moreno, P. Ho, and N. Vasconcelos. A Kullback-Leibler divergence based kernel for SVM classification in multimedia applications. In S. Thrun, L. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems 16. MIT Press, 2004.
46. A. Rényi. On measures of entropy and information. In Proc. of the 4th Berkeley Symposium on Mathematics, Statistics, and Probability, volume 1, pages 547-561, Berkeley, 1961. University of California Press.
47. B. Schölkopf and A. J. Smola. Learning with Kernels. The MIT Press, Cambridge, MA, 2002.
48. C. Shannon and W. Weaver. The Mathematical Theory of Communication. University of Illinois Press, Urbana, Ill., 1949.
49. C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27 (3):379-423, 1948.
50. J. Shawe-Taylor and N. Cristianini. Kernel Methods for Pattern Analysis. Cambridge University Press, 2004.
51. J. Steele. The Cauchy-Schwarz Master Class. Cambridge University Press, Cambridge, 2006.
52. H. Suyari. Generalization of Shannon-Khinchin axioms to nonextensive systems and the uniqueness theorem for the nonextensive entropy. IEEE Trans. on Information Theory, 50(8):1783-1787, 2004.
53. F. Topsøe. Some inequalities for information divergence and related measures of discrimination. IEEE Trans. on Information Theory, 46(4):1602-1609, 2000.
54. C. Tsallis. Possible generalization of Boltzmann-Gibbs statistics. J. of Statistical Physics, 52: 479-487, 1988.
55. S. Vishwanathan and A. Smola. Fast kernels for string and tree matching. In K. Tsuda, B. Scholkopf, and J.P. Vert, editors, Kernels and Bioinformatics, Cambridge, MA, 2003. MIT Press.
56. D. Zhang, X. Chen, and W. Lee. Text classification with kernels on the multinomial manifold. In Proc. of the 28th Annual Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 266-273, New York, NY, 2005.