Stable adaptive momentum for rapid online learning in nonlinear systems

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 2415 LNCS, Issue , 2002, Pages 450-455

  Graepel, Thore ^a   Schraudolph, Nicol N ^a  

^a ETH ZURICH   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

E-LEARNING; GRADIENT METHODS; MOMENTUM; NEURAL NETWORKS; NONLINEAR PROGRAMMING; NONLINEAR SYSTEMS; STOCHASTIC SYSTEMS; ALGORITHMS; ITERATIVE METHODS; ONLINE SYSTEMS; TIME VARYING NETWORKS;

AUTOMATIC DIFFERENTIATION TOOL; CURVATURE MATRIX; FORGETTING FACTORS; HIGHLY-CORRELATED; NON-LINEAR OPTIMISATION; NON-STATIONARY ENVIRONMENT; ONLINE LEARNING; STOCHASTIC GRADIENT DESCENT ALGORITHM;

ONLINE SYSTEMS; OPTIMIZATION;

EID: 34848829803     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/3-540-46084-5_73     Document Type: Conference Paper

References (7)

1. Y. LeCun, P. Y. Simard, and B. Pearlmutter. Automatic learning rate maximization in large adaptive machines. In S. J. Hanson, J. D. Cowan, and C. L. Giles, editors, Advances in Neural Information Processing Systems, volume 5, pages 156-163. Morgan Kaufmann, San Mateo, CA, 1993.
2. T. K. Leen and G. B. Orr. Optimal stochastic search and adaptive momentum. In J. D. Cowan, G. Tesauro, and J. Alspector, editors, Advances in Neural Information Processing Systems, volume 6, pages 477-484. Morgan Kaufmann, San Francisco, CA, 1994.
3. D. Marquardt. An algorithm for least-squares estimation of non-linear Parameters. Journal of the Society of Industrial and Applied Mathematics, 11(2):431-441, 1963.
4. G. B. Orr. Dynamics and Algorithms for Stochastic Learning. PhD thesis, Department of Computer Science and Engineering, Oregon Graduate Institute, Beaverton, OR 97006, 1995. ftp://neural.cse.ogi.edu/pub/neural/papers/orrPhDchi- 5. ps.Z, orrPhDch6-9.ps.Z.
5. B. A. Pearlmutter. Fast exact multiplication by the Hessian. Neural Computation, 6(1):147-160, 1994.
6. N. N. Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. Neural Computation, 14(7), 2002. http://www.inf.ethz.ch/ ~schraudo/pubs/mvp.ps.gz.
7. S. Singhal and L. Wu. Training multilayer perceptrons with the extended Kalman filter. In D. S. Touretzky, editor, Advances in Neural Information Processing Systems. Proceedings of the 1988 Conference, pages 133-140, San Mateo, CA, 1989. Morgan Kaufmann.