Sided and symmetrized Bregman centroids

IEEE Transactions on Information Theory

Volumn 55, Issue 6, 2009, Pages 2882-2904

  Nielsen, Frank ^a,b   Nock, Richard ^c  

^a SONY COMPUTER SCIENCE LABORATORIES INC   (Japan)

^b ÉCOLE POLYTECHNIQUE   (France)

^c UNIVERSITÉ DES ANTILLES ET DE LA GUYANE   (France)

Author keywords

Bregman divergence; Bregman information; Bregman power divergence; Burbea Rao divergence; Centroid; Csisz r f divergences; Information geometry; Information radius; Kullback Leibler divergence; Legendre duality

Indexed keywords

GEOMETRY; INFORMATION THEORY; NORMAL DISTRIBUTION;

BREGMAN; BREGMAN DIVERGENCES; BREGMAN INFORMATION; BURBEA-RAO DIVERGENCE; CENTROID; INFORMATION GEOMETRY; INFORMATION RADIUS; KULLBACK LEIBLER DIVERGENCE; LEGENDRE;

APPROXIMATION ALGORITHMS;

EID: 66949153253     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2009.2018176     Document Type: Article

References (73)

1. H. H. Bauschke and J. M. Borwein, "Legendre functions and the method of random Bregman projections," J. Convex Anal., vol. 4, no. 1, pp. 27-67, 1997.
2. F. Nielsen and R. Nock, "On the smallest enclosing information disk," Inf. Process. Lett., vol. 105, no. 3, pp. 93-97, 2008.
3. "Quantum voronoi diagrams and Holevo channel capacity for 1-qubit quantum states," in Proc. IEEE Int. Symp. Inf. Theory, E.-H. Y. F. Kschischang, Ed., 2008, pp. 96-100.
4. T. R. C. Read and N. A. C. Cressie, Goodness-of-Fit Statistics for Discrete Multivariate Data, ser. Statistics. New York: Springer-Verlag, 1988.
5. F. Liese and I. Vajda, Convex Statistical Distances. Leipzig, Germany: B. G. Teubner Verlagsgesellschaft, 1987 [Online]. Available: http://openlibrary.org/b/OL20652743M/Convex-stattistical-distances, ISBN: 3322004287
6. F. Liese and I. Vajda, "On divergences and informations in statistics and information theory," IEEE Trans. Inf. Theory, vol. 52, no. 10, pp. 4394-4412, Oct. 2006.
7. S.-I. Amari, "Natural gradient works efficiently in learning," Neural Comput., vol. 10, no. 2, pp. 251-276, 1998.
8. D. Petz and R. Temesi, "Means of positive numbers and matrices," SIAM J. Matrix Anal. Appl., vol. 27, no. 3, pp. 712-720, 2005.
9. M.-M. Deza and E. Deza, Dictionary of Distances. New York: Elsevier, 2006.
10. S.-I. Amari, "Information geometry and its applications: Convex function and dually flat manifold," in Emerging Trends in Visual Computing (ETVC), ser. Lecture Notes in Statistics, F. Nielsen, Ed. Berlin, Germany: Springer-Verlag, 2008.
11. T. Minka, "Divergence measures and message passing," Microsoft Research, Cambridge, U.K., Tech. Rep. MSR-TR-2005-173, 2005 [Online]. Available: http://research.microsoft.com/en-us/um/people/minka/papers/ message-passing/
12. S.-I. Amari, "Integration of stochastic models by minimizing α-divergence," Neural Comput., vol. 19, no. 10, pp. 2780-2796, 2007.
13. M. N. Do and M. Vetterli, "Wavelet-based texture retrieval using generalized Gaussian density and Kullback-Leibler distance," IEEE Trans. Image Process., vol. 11, no. 2, pp. 146-158, Feb. 2002.
14. T. M. Cover and J. A. Thomas, Elements of Information Theory, ser. Telecommunications and Signal Processing. New York: Wiley-Interscience, 2006.
15. S. P. Lloyd, "Least squares quantization in PCM," IEEE Trans. Inf. Theory, vol. IT-28, no. 2, pp. 129-136, Mar. 1982.
16. E. Forgy, "Cluster analysis of multivariate data: Efficiency versus interpretability of classifications," Biometrics, vol. 21, pp. 768-780, 1965.
17. D. Arthur and S.Vassilvitskii, "k-means++: The advantages of careful seeding," in Proc. 18th Annu. ACM-SIAM Symp. Discrete Algorithms, Philadelphia, PA, 2007, pp. 1027-1035.
18. P. Carmi, S. Har-Peled, and M. J. Katz, "On the Fermat-Weber center of a convex object," Comput. Geom., vol. 32, no. 3, pp. 188-195, 2005.
19. A. Banerjee, S. Merugu, I. S. Dhillon, and J. Ghosh, "Clustering with Bregman divergences," J. Mach. Learn. Res., vol. 6, pp. 1705-1749, 2005.
20. A. Banerjee, X. Guo, and H. Wang, "On the optimality of conditional expectation as a Bregman predictor," IEEE Trans. Inf. Theory, vol. 51, no. 7, pp. 2664-2669, Jul. 2005.
21. L. M. Bregman, "The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming," USSR Comput. Math. Math. Phys., vol. 7, pp. 200-217, 1967.
22. Y. A. Censor and S. A. Zenios, Parallel Optimization: Theory, Algorithms and Applications. Oxford, U.K.: Oxford Univ. Press, 1997.
23. I. S. Dhillon and J. A. Tropp, "Matrix nearness problems with Bregman divergences," SIAM J. Matrix Anal. Appl., vol. 29, no. 4, pp. 1120-1146, 2007.
24. B. A. Frigyik, S. Srivastava, and M. R. Gupta, "Functional Bregman divergence," in Proc. IEEE Int. Symp. Inf. Theory, E.-H. Y. F. Kschischang, Ed., 2008, pp. 1681-1684.
25. F. Nielsen, J.-D. Boissonnat, and R. Nock, "Bregman Voronoi diagrams: Properties, algorithms and applications," INRIA, Sophia-Antipolis, France, Tech. Rep. RR-6154, Sep. 2007.
26. Y. Stylianou and A. K. Syrdal, "Perceptual and objective detection of discontinuities in concatenative speech synthesis," in Proc. IEEE Acoust. Speech Signal Process, Washington, DC, 2001, pp. 837-840.
27. T. A. Myrvoll and F. K. Soong, "On divergence-based clustering of normal distributions and its application to HMM adaptation," in Proc. EuroSpeech, Geneva, Switzerland, 2003, pp. 1517-1520.
28. B. A. Carlson and M. A. Clements, "A computationally compact divergence measure for speech processing," IEEE Trans. Pattern Anal. Mach. Intell., vol. 13, no. 12, pp. 1255-1260, Dec. 1991.
29. B. Wei and J. D. Gibson, "Comparison of distance measures in discrete spectral modeling," in Proc. 9th DSP Workshop/1st Signal Process. Education Workshop, Hunt, TX, Oct. 15-18, 2000 Online. Available: http://spib.rice.edu/DSP2000/program.html
30. C. Févotte, N. Bertin, and J.-L. Durrieu, "Nonnegative matrix factorization with the Itakura-Saito divergence: With application to music analysis," Neural Comput., pp. 159-195, 2008.
31. J. M. Winn and C. M. Bishop, "Variational message passing," J. Mach. Learn. Res., vol. 6, pp. 661-694, 2005.
32. A. Ben-Tal, A. Charnes, and M. Teboulle, "Entropic means," J. Math. Anal. Appl., pp. 537-551, 1989.
33. M. Basseville and J.-F. Cardoso, "On entropies, divergences and mean values," in Proc. IEEE Int. Symp. Inf. Theory, Whistler, BC, Canada, Sep. 1995, pp. 330-330.
34. I. Csiszár, "Information type measures of difference of probability distributions and indirect observations," Studia Scientiarum Mathematicarum Hungarica, vol. 2, pp. 299-318, 1967.
35. I. Csiszár, "Why least squares and maximum entropy? An axiomatic approach to inference for linear inverse problems," Ann. Statist. vol. 19, no. 4, pp. 2032-2066, 1991 [Online]. Available: http://projecteuclid.org/
36. R. N. J. Veldhuis, "The centroid of the symmetrical Kullback-Leibler distance," IEEE Signal Process. Lett., vol. 9, no. 3, pp. 96-99, Mar. 2002.
37. R. Veldhuis and E. Klabbers, "On the computation of the Kullback-Leibler measure for spectral distances," IEEE Trans. Speech Audio Process., vol. 11, no. 1, pp. 100-103, Jan. 2003.
38. R. Sibson, "Information radius," Probab. Theory Relat. Fields, vol. 14, no. 2, pp. 149-160, 1969.
39. J. Lin, "Divergence measures based on the Shannon entropy," IEEE Trans. Inf. Theory, vol. 37, no. 1, pp. 145-151, Jan. 1991.
40. J. Burbea and C. R. Rao, "On the convexity of some divergence measures based on entropy functions," IEEE Trans. Inf. Theory, vol. IT-28, no. 3, pp. 489-495, May 1982.
41. G. Aradilla, J. Vepa, and H. Bourlard, "An acoustic model based on Kullback-Leibler divergence for posterior features," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., 2007, vol. 4, pp. 657-660.
42. K. Shinoda and C. H. Lee, "A structural Bayes approach to speaker adaptation," IEEE Trans. Speech Audio Process., vol. 9, no. 3, pp. 276-287, Mar. 2001.
43. S.-I. Amari and N. Nagaoka, Methods of Information Geometry. Oxford, U.K.: Oxford Univ. Press, 2000, ISBN-10:0821805312.
44. S. Yoshizawa and K. Tanabe, "Dual differential geometry associated with Kullback-Leibler information on the Gaussian distributions and its 2-parameter deformations," SUT J. Math., vol. 35, no. 1, pp. 113-137, 1999.
45. J. Zhang, "Divergence function, duality, and convex analysis," Neural Comput., vol. 16, no. 1, pp. 159-195, 2004.
46. E. Cartan, Geometry of Riemannian Spaces. Brookline, MA: Math Sci. Press, 1983 [Online]. Available: http://openlibrary.org/b/OL3169847M/ Geometry-of-Riemannian-spaces
47. S. M. Ali and S. D. Silvey, "A general class of coefficients of divergence of one distribution from another," J. Roy. Statist. Soc. B, vol. 28, pp. 131-142, 1966.
48. P. Vos, "Geometry of f-divergence," Ann. Inst. Statist. Math. vol. 43, no. 3, pp. 515-537, 1991.
49. A. F. T. Martins, M. A. T. Figueiredo, P. M. Q. Aguiar, N. A. Smith, and E. P. Xing, "Nonextensive entropic kernels," in Proc. Int. Conf. Mach. Learn., 2008, pp. 640-647.
50. S. Arimoto, "Information-theoretical considerations on estimation problems," Inf. Control, vol. 19, no. 3, pp. 181-194, 1971.
51. F. Osterreicher and I. Vajda, "A new class of metric divergences on probability spaces and its applicability in statistics," Ann. Inst. Statist. Math. vol. 55, no. 3, pp. 639-653, Sep. 2003 [Online]. Available: http://ideas.repec.org/a/spr/aistmt/v55y2003i3p639-653.html
52. E. Porcu, J. Mateu, and G. Christakos, "Quasi-arithmetic means of covariance functions with potential applications to space-time data," 2A06 [Online]. Available: ArXiv:math/0611275
53. G. H. Hardy, J. E. Littlewood, and G. Polya, Inequalities. Cambridge, U.K.: Cambridge Univ. Press, 1967.
54. A. Rényi, "On measures of entropy and information," in Proc. 4th Berkeley Symp. Math. Statist. Probab., Berkeley, CA, 1961, vol. 1, pp. 547-561.
55. R. Nock and F. Nielsen, "Fitting the smallest enclosing Bregman ball," in Proc. 16th Eur. Conf. Mach. Learn., 2005, vol. 3720/2005, pp. 649-656.
56. B. Pelletier, "Informative barycentres in statistics," Ann. Inst. Statist. Math., vol. 57, no. 4, pp. 767-780, 2005.
57. T. Hertz, A. Bar-Hillel, and D. Weinshall, "Learning distance functions for image retrieval," in Proc. IEEE Comput. Vis. Pattern Recognit. 2004, vol. 2, pp. II-570-II-577.
58. J. Lafferty, "Additive models, boosting, and inference for generalized divergences," in Proc. 12th Conf. Comput. Learn. Theory, New York, 1999, pp. 125-133 [Online]. Available: http://dx.doi.org/10.1145/307400.307422
59. O. E. Barndorff-Nielsen, Parametric Statistical Models and Likelihood ser. Lecture Notes in Statistics. New York: Springer-Verlag, 1988, vol. 50.
60. A. Boratynska, "Stability of Bayesian inference in exponential families," Statist. Probab. Lett., no. 2, pp. 173-178, Dec. 1997.
61. R. Nock and F. Nielsen, "Bregman divergences and surrogates for learning," IEEE Trans. Pattern Anal. Mach. Intell., 2009 [Online]. Available: http://ieeexplore.ieee.org/xpl/ preabsprintf.jsp?ar-number=4626960, to be published
62. P. L. Bartlett, M. I. Jordan, and J. D. McAuliffe, "Convexity, classification, and risk bounds," J. Amer. Statist. Assoc., vol. 101, no. 473, pp. 138-156, 2006.
63. Y. Freund and R. E. Schapire, "A decision-theoretic generalization of on-line learning and an application to boosting," J. Comput. Syst. Sci., vol. 55, no. 1, pp. 119-139, 1997.
64. J. V. Davis and I. S. Dhillon, "Differential entropic clustering of multivariate Gaussians," in Neural Information Processing Systems (NIPS), B. Scholkopf, J. Platt, and T. Hoffman, Eds. Cambridge, MA: MIT Press, 2006, pp. 337-344.
65. M. Teboulle, "A unified continuous optimization framework for center-based clustering methods," J. Mach. Learn. Res., vol. 8, pp. 65-102, 2007.
66. R. Nock, P. Luosto, and J. Kivinen, "Mixed Bregman clustering with approximation guarantees," in Proc. Eur. Conf. Mach. Learn. Knowl. Disc. Databases, 2008, pp. 154-169.
67. F. Nielsen, P. Piro, and M. Barlaud, "Tailored Bregman ball trees for effective nearest neighbors," presented at the Proc. Eur. Workshop Comput. Geometry, Brussels, Belgium, Mar. 16-18, 2009 [Online]. Available: http://2009.eurocg.org/
68. A. Cont, "Modeling musical anticipation: From the time of music to the music of time," Ph.D. dissertation, IRCAM, Univ. Paris 6/Univ. California in San Diego, Centre Pompidou, Paris, France, Oct. 2008.
69. E. Spellman, B. C. Vemuri, and M. Rao, "Using the KL-center for efficient and accurate retrieval of distributions arising from texture images," in Proc. IEEE Comput. Vis. Pattern Recognit., Washington, DC, 2005, pp. 111-116.
70. J. Goldberger, S. Gordon, and H. Greenspan, "An efficient image similarity measure based on approximations of KL-divergence between two Gaussian mixtures," in Proc. IEEE Int. Conf. Comput. Vis., 2003, vol. 1, pp. 487-493.
71. L. Rigazio, B. Tsakam, and J.-C. Junqua, "An optimal Bhattacharyya centroid algorithm for Gaussian clustering with applications in automatic speech recognition," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., Apr. 2000, vol. 3, pp. 1599-1602.
72. D. Johnson and S. Sinanovic, "Symmetrizing the Kullback-Leibler distance," Rice Univ., Houston, TX, Tech. Rep., 2001.
73. Z. Wang and B. C. Vemuri, "DTI segmentation using an information theoretic tensor dissimilarity measure," IEEE Trans. Med. Imag., vol. 24, no. 10, pp. 1267-1277, Oct. 2005.