A Bregman extension of quasi-Newton updates I: An information geometrical framework

Optimization Methods and Software

Volumn 28, Issue 1, 2013, Pages 96-123

  Kanamori, Takafumi ^a   Ohara, Atsumi ^b  

^a NAGOYA UNIVERSITY   (Japan)

^b UNIVERSITY OF FUKUI   (Japan)

Author keywords

Bregman divergence; Information geometry; Quasi Newton method

Indexed keywords

APPROXIMATE HESSIANS; BREGMAN DIVERGENCES; EM ALGORITHMS; GEOMETRICAL STRUCTURE; INFORMATION GEOMETRY; INTUITIVE UNDERSTANDING; KL-DIVERGENCE; KULLBACK-LEIBLER DIVERGENCE; NUMERICAL ALGORITHMS; POSITIVE-DEFINITE MATRICES; QUASI-NEWTON METHODS; QUASI-NEWTON UPDATE; STATISTICAL ALGORITHM; VARIATIONAL PROBLEMS;

ALGORITHMS; NEWTON-RAPHSON METHOD; OPTIMIZATION; PROBABILITY DISTRIBUTIONS; VARIATIONAL TECHNIQUES;

GEOMETRY;

EID: 84867477968     PISSN: 10556788     EISSN: 10294937     Source Type: Journal    
DOI: 10.1080/10556788.2011.613073     Document Type: Article

References (41)

1. S. Amari, Information geometry of the EM and em algorithms for neural networks, Neural Networks 8(9) (1995), pp. 1379-1408. (Pubitemid 26072898)
2. S. Amari and H. Nagaoka, Methods of Information Geometry, Translations of Mathematical Monographs Vol. 191. Oxford University Press, Oxford, 2000.
3. A. Banerjee, S. Merugu, I. Dhillon, and J. Ghosh, Clustering with Bregman divergences, Journal of Machine Learning Research 6 (2005), pp. 1705-1749.
4. A. Basu, I.R. Harris, N.L. Hjort, and M.C. Jones, Robust and efficient estimation by minimising a density power divergence, Biometrika 85(3) (1998), pp. 549-559.
5. H.H. Bauschke and J.M. Borwein, Legendre functions and the method of random Bregman projections, Journal of Convex Analysis 4(1) (1997), pp. 27-67. (Pubitemid 127714611)
6. H.H. Bauschke and P.L. Combettes, Iterating Bregman retractions, SIAM Journal of Optimization 13(4) (2002), pp. 1159-1173. (Pubitemid 37395691)
7. L.M. Bregman, The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming, USSR Computational Mathematics and Mathematical Physics 7 (1967), pp. 200-217.
8. C.G. Broyden, Quasi-newton methods and their application to function minimisation,Mathematics of Computation 21(99) (1967), pp. 368-381.
9. R.G. Cowell, A.P. Dawid, S.L. Lauritzen, and D.J. Spiegelhalter, Probabilistic Networks and Expert Systems: Exact Computational Methods for Bayesian Networks, Springer Publishing Company, NewYork, 2007.
10. W.C. Davidon, Variable metric method for minimization, A.E.C. Research and Development Report, ANL-5990, 1959.
11. A.P. Dempster, Covariance selection, Biometrics 28 (1972), pp. 157-175.
12. A.P. Dempster, N.M. Laird, and D.B. Rubin, Maximum likelihood from incomplete data via the em algorithm, Journal of the Royal Statistical Society B 39 (1977), pp. 1-38.
13. I.S. Dhillon and J.A. Tropp, Matrix nearness problems with Bregman divergences, SIAM Journal on Matrix Analysis and Applications 29(4) (2007), pp. 1120-1146.
14. R. Fletcher, A new variational result for quasi-Newton formulae, SIAM Journal on Optimization 1 (1991), pp. 18-21.
15. R. Fletcher, An optimal positive definite update for sparse hessian matrices, SIAM Journal on Optimization 5 (1995), pp. 192-218.
16. Y. Freund and R.E. Schapire, A decision-theoretic generalization of on-line learning and an application to boosting, Journal of Computer and System Sciences 55(1) (1997), pp. 119-139. (Pubitemid 127433398)
17. M. Fukuda, M. Kojima, K. Murota, and K. Nakata, Exploiting sparsity in semidefinite programming via matrix completion I: General framework, SIAM Journal on Optimization 11(3) (2000), pp. 647-674.
18. P.E. Gill and W. Murray, Quasi-Newton methods for unconstrained optimization, Journal of the Institute of Mathematics and its Applications 9 (1972), pp. 91-108.
19. G.H. Golub and C.F. Van Loan, Matrix Computations, Johns Hopkins University Press, Baltimore, MD, 1996.
20. M.C. Golumbic, Algorithmic Graph Theory and Perfect Graphs (Annals of Discrete Mathematics, Vol. 57), North- Holland Publishing Co., Amsterdam, The Netherlands, 2004.
21. R. Grone, C.R. Johnson, E.M. Sá, and H. Wolkowicz, Positive definite completions of partial hermitian matrices, Linear Algebra and its Applications 58 (1984), pp. 109-124.
22. O. Güler, F. Gürtuna, and O. Shevchenko, Duality in quasi-Newton methods and new variational characterizations of the DFP and BFGS updates, Optimization Methods and Software 24(1) (2009), pp. 45-62.
23. S. Kakihara, A. Ohara, and T. Tsuchiya, Information geometry and primal-dual interior-point algorithms, Optimization Online, 2010, submitted for publication.
24. T. Kanamori andA. Ohara,A Bregman extension of quasi-Newton updates II: Convergence and robustness properties, arXiv:1010.2847v1, 2010.
25. S. Kobayashi and K. Nomizu, Foundations of Differential Geometry,Wiley-Interscience, NewYork, 1996.
26. S. Kullback and R.A. Leibler, On information and sufficiency, The Annals of Mathematical Statistics 22(1) (1951), pp. 79-86.
27. D. Luenberger andY.Ye, Linear and Nonlinear Programming, Springer, NewYork, 2008.
28. G. J. McLachlan and T. Krishnam, The EM Algorithm and Extensions, 2nd ed.,Wiley, NewYork, 2008.
29. M. Minami and S. Eguchi, Robust blind source separation by beta-divergence, Neural Computation 14(8) (2002), pp. 1859-1886.
30. N. Murata, T. Takenouchi, T. Kanamori, and S. Eguchi, Information geometry of U-boost and Bregman divergence, Neural Computation 16(7) (2004), pp. 1437-1481. (Pubitemid 38740017)
31. J. Nocedal, Updating quasi-newton matrices with limited storage, Mathematics of Computation 35(151) (1980), pp. 773-782.
32. J. Nocedal, Theory of algorithms for unconstrained optimization, Acta Numerica 1 (1992), pp. 199-242.
33. J. Nocedal and S. Wright, Numerical Optimization, Springer, NewYork, 1999.
34. J. Nocedal andY.-X.Yuan, Analysis of a self-scaling quasi-Newton method, Mathematical Programming 61 (1993), pp. 19-37.
35. A. Ohara, Information geometric analysis of an interior point method for semidefinite programming, in Geometry in Present Day Science, O.E. Barndorff-Nielsen and E.B. Vedel Jensen, eds., World Scientific, Singapore, 1999, pp. 49-74.
36. A. Ohara and S. Eguchi, Geometry on positive definite matrices and V-potential function, Technical report, ISM Research Memo, 2005.
37. S.S. Oren and D.G. Luenberger, Self-scaling variable metric (ssvm) algorithms, part i. Criteria and sufficient conditions for scaling a class of algorithms, Management Science 20 (1974), pp. 845-862.
38. R.T. Rockafellar, Convex Analysis, Princeton University Press, Princeton, 1970.
39. D.C. Sorensen, Collinear scaling and sequential estimation in sparse optimization algorithm, Mathematical Programming Studies 18 (1982), pp. 135-159.
40. P.L. Toint, On sparse and symmetric matrix updating subject to a linear equation, Mathematics of Computation 31 (1977), pp. 954-961.
41. N. Yamashita, Sparse quasi-Newton updates with positive definite matrix completion, Mathematical Programming 115(1) (2008), pp. 1-30. (Pubitemid 351702150)