A stochastic quasi-Newton method for online convex optimization

Journal of Machine Learning Research

Volumn 2, Issue , 2007, Pages 436-443

  Schraudolph, Nicol N ^a,b   Yu, Jin ^a,b   Günter, Simon ^a,b  

^a CSIRO   (Australia)

^b AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

CONDITIONAL RANDOM FIELD; CONVEX FUNCTIONS; HIGH-DIMENSIONAL PROBLEMS; NATURAL GRADIENT; NATURAL LANGUAGE PROCESSING; NONCONVEX OPTIMIZATION PROBLEM; ONLINE OPTIMIZATION; QUASI-NEWTON METHODS; QUASI-NEWTON OPTIMIZATION METHOD; STOCHASTIC GRADIENT METHODS;

ARTIFICIAL INTELLIGENCE; COMPUTATIONAL LINGUISTICS; CONVEX OPTIMIZATION; NATURAL LANGUAGE PROCESSING SYSTEMS; NEWTON-RAPHSON METHOD; PARAMETER ESTIMATION; STOCHASTIC SYSTEMS;

OPTIMIZATION;

EID: 84862300219     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (18)

1. S.-i. Amari, H. Park, and K. Fukumizu. Adaptive method of realizing natural gradient learning for multilayer perceptrons. Neural Computation, 12(6):1399-1409, 2000.
2. L. Bottou and Y. LeCun. On-line learning for very large datasets. Applied Stochastic Models in Business and Industry, 21(2):137-151, 2005.
3. K. W. Brodlie. An assessment of two approaches to variable metric methods. Mathematical Programming, 12:344-355, 1977.
4. J.-D. Kim, T. Ohta, Y. Tsuruoka, Y. Tateisi, and N. Collier. Introduction to the bio-entity recognition task at JNLPBA. In Proc. Intl. Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA), pages 70-75, Geneva, Switzerland, 2004.
5. S. Kumar and M. Hebert. Discriminative fields for modeling spatial dependencies in natural images. In S. Thrun, L. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems 16, 2004.
6. J. D. Lafferty, A. McCallum, and F. Pereira. Conditional random fields: Probabilistic modeling for segmenting and labeling sequence data. In Proc. Intl. Conf. Machine Learning, volume 18, pages 282-289, San Francisco, CA, 2001. Morgan Kaufmann.
7. M. F. Møller. A scaled conjugate gradient algorithm for fast supervised learning. Neural Networks, 6(4):525-533, 1993.
8. R. Neuneier and H. G. Zimmermann. How to train neural networks. In G. B. Orr and K.-R. Müller, editors, Neural Networks: Tricks of the Trade, volume 1524 of Lecture Notes in Computer Science, chapter 17, pages 373-423.Springer Verlag, Berlin, 1998.
9. J. Nocedal and S. J. Wright. Numerical Optimization. Springer Series in Operations Research. Springer, 1999.
10. G. V. Puskorius and L. A. Feldkamp. Decoupled extended Kalman filter training of feedforward layered networks. In Proc. Intl. Joint Conf. on Neural Networks, volume I, pages 771-777, Seattle, WA, 1991. IEEE.
11. H. E. Robbins and S. Monro. A stochastic approximation method. Annals Mathem. Statistics, 22:400-407, 1951.
12. H. E. Robbins and D. O. Siegmund. A convergence theorem for non negative almost supermartingales and some applications. In Proc. Sympos. Optimizing Methods in Statistics, pages 233-257, Ohio State Univ., Columbus, Ohio, 1971. Academic Press, New York.
13. E. F. T. K. Sang and S. Buchholz. Introduction to the CoNLL-2000 shared task: Chunking. In Proc. Conf. Computational Natural Language Learning, pages 127-132, Lisbon, Portugal, 2000.
14. N. N. Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. Neural Computation, 14(7):1723-1738, 2002.
15. N. N. Schraudolph. Local gain adaptation in stochastic gradient descent. In Proc. Intl. Conf. Artificial Neural Networks, pages 569-574, Edinburgh, Scotland, 1999. IEE, London.
16. N. N. Schraudolph and T. Graepel. Combining conjugate direction methods with stochastic approximation of gradients. In C. M. Bishop and B. J. Frey, editors, Proc. 9th Intl. Workshop Artificial Intelligence and Statistics, pages 7-13, Key West, 2003. ISBN 0-9727358-0-1.
17. F. Sha and F. Pereira. Shallow parsing with conditional random fields. In Proceedings of HLT-NAACL, pages 213-220, Edmonton, Canada, 2003. Association for Computational Linguistics.
18. S. V. N. Vishwanathan, N. N. Schraudolph, M. Schmidt, and K. Murphy. Accelerated training conditional random fields with stochastic gradient methods. In Proc. Intl. Conf. Machine Learning, pages 969-976, New York, NY, USA, 2006. ACM Press. ISBN 1-59593-383-2.