On the use of stochastic hessian information in optimization methods for machine learning

SIAM Journal on Optimization

Volumn 21, Issue 3, 2011, Pages 977-995

  Byrd, Richard H ^a   Chin, Gillian M ^b   Neveitt, Will ^c   Jorge, Nocedal ^b  

^a University of Colorado at Boulder   (United States)

^b Northwestern University   (United States)

^c GOOGLE INC   (United States)

Author keywords

Machine learning; Stochastic optimization; Unconstrained optimization

Indexed keywords

CURVATURE INFORMATION; LIMITED MEMORY; MACHINE LEARNING APPLICATIONS; NEWTON ITERATIONS; OPTIMIZATION METHOD; QUASI-NEWTON METHODS; SAMPLE AVERAGE APPROXIMATION; SAMPLE SIZES; STATISTICAL LEARNING; STOCHASTIC OPTIMIZATION; STOCHASTIC OPTIMIZATIONS; SUPERVISED MACHINE LEARNING; UNCONSTRAINED OPTIMIZATION;

LEARNING SYSTEMS; NEWTON-RAPHSON METHOD; OPTIMIZATION; SPEECH RECOGNITION; STOCHASTIC SYSTEMS;

CURVE FITTING;

EID: 80054732060     PISSN: 10526234     EISSN: None     Source Type: Journal    
DOI: 10.1137/10079923X     Document Type: Article

References (17)

1. L. BOTTOU, Online learning and stochastic approximations, in Online Learning and Neural Networks, Vol. 17, Cambridge University Press, Cambridge, 1998, pp. 9-42.
2. R. FLETCHER, Practical Methods of Optimization, 2nd ed., J. Wiley and Sons, Chichester, England, 1987.
3. I. GRIVA, S. G. NASH, AND A. SOFER, Linear and Nonlinear Optimization, 2nd ed., SIAM, Philadelphia, 2008.
4. J. YU, S. V. N VISHWANATHAN, S. GÜNTER, AND N. N. SCHRAUDOLPH, A quasi-Newton approach to nonsmooth convex optimization problems in machine learning, J. Mach. Learn. Res., 11 (2010), pp. 1145-1200.
5. C. J. LIN, R. C. WENG, AND S. S. KEERTHI, Trust region Newton method for logistic regression, J. Mach. Learn. Res., 9 (2008), pp. 627-650.
6. D. C. LIU AND J. NOCEDAL, On the limited memory BFGS method for large scale optimization, Math. Program., 45 (1989), pp. 503-528. (Pubitemid 20660315)
7. D. G. LUENBERGER, Linear and Nonlinear Programming, 2nd ed., Addison-Wesley, Reading, MA, 1984.
8. R. MALOUF, A comparison of algorithms for maximum entropy parameter estimation, in Proceedings of the Sixth Conference on Natural Language Learning, Taipei, Taiwan, 2002.
9. J. MARTENS, Deep learning via Hessian-free optimization, in Proceedings of the 27th International Conference on Machine Learning (ICML), Vol. 95, 2010.
10. T. P.MINKA, A Comparison of Numerical Optimizers for Logistic Regression, Technical report, Microsoft Research, 2003.
11. S. G. NASH, Newton-type minimization via the Lanczos method, SIAM J. Numer. Anal., 21 (1984), 770-788.
12. J. NOCEDAL AND S. J.WRIGHT, Numerical Optimization, 2nd ed., Springer Ser. Oper. Res., Springer, New York, 2006.
13. H. ROBBINS AND S. MONRO, A stochastic approximation method, Ann. Math. Statist., 22 (1951), pp. 400-407.
14. A. RUSZCZYNSKI AND A. SHAPIRO, Stochastic Programming, Handbook in Operations Research and Management Science, Elsevier Science, New York, 2003.
15. F. SHA AND F. PEREIRA, Shallow parsing with conditional random fields, in Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, Vol. 1, Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, 2003, pp. 131-141.
16. A. SHAPIRO AND T. HOMEM-DE-MELLO, A simulation-based approach to two-stage stochastic programming with recourse, Math. Program., 81 (1998), pp. 301-325. (Pubitemid 128393283)
17. T. STEIHAUG, The conjugate gradient method and trust regions in large scale optimization, SIAM J. Numer. Anal., 20 (1983), pp. 626-637.