Using curvature information for fast stochastic search

Advances in Neural Information Processing Systems

Volumn , Issue , 1997, Pages 606-612

  Orr, Genevieve B ^a   Leen, Todd K ^b  

^a WILLAMETTE UNIVERSITY   (United States)

^b OREGON HEALTH AND SCIENCE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS;

CONVERGENCE RATES; CURVATURE INFORMATION; STOCHASTIC GRADIENT DESCENT; STOCHASTIC SEARCH;

INFORMATION USE;

EID: 84898987060     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (14)

1. Genevieve B. Orr and Todd K. Leen. Weight space probability densities in stochastic learning: II. Transients and basin hopping times. In Giles, Hanson, and Cowan, editors, Advances in Neural Information Processing Systems, vol. 5, San Mateo, CA, 1993. Morgan Kaufmann.
2. William Finnoft Diffusion approximations for the constant learning rate backprop-agation algorithm and resistence to local minima. In Giles, Hanson, and Cowan, editors, Advances in Neural Information Processing Systems, vol. 5, San Mateo, CA, 1993. Morgan Kaufmann.
3. Larry Goldstein. Mean square optimality in the continuous time Robbins Monro procedure. Technical Report DRB-306, Dept. of Mathematics, University of Southern California, LA, 1987.
4. Christian Darken and John Moody. Towards faster stochastic gradient search. In J.E. Moody, S.J. Hanson, and R.P. Lipmann, editors, Advances in Neural Information Processing Systems 4-Morgan Kaufmann Publishers, San Mateo, CA, 1992.
5. Halbert White. Learning in artificial neural networks: A statistical perspective. Neural Computation, 1:425-464,1989.
6. J. H. Venter. An extension of the robbins-monro procedure. Annals of Mathematical Statistics, 38:117-127,1967.
7. Todd K. Leen and Genevieve B. Orr. Optimal stochastic search and adaptive momentum. In J.D. Cowan, G. Tesauro, and J. Alspector, editors, Advances in Neural Information Processing Systems 6, San Francisco, CA., 1994. Morgan Kaufmann Publishers.
8. Genevieve B. Orr. Dynamics and Algorithms for Stochastic Search. PhD thesis, Oregon Graduate Institute, 1996.
9. Mehmet Ali Tugay and Yalcin Tanik. Properties of the momentum LMS algorithm. Signal Processing, 18:117-127, 1989.
10. John J. Shynk and Sumit Roy. Analysis of the momentum LMS algorithm. IEEE Transactions on Acoustics, Speech, and Signal Processing, 38(12):2088-2098, 1990.
11. W. Wiegerinck, A. Komoda, and T. Heskes. Stochastic dynamics of learning with momentum in neural networks. Journal of Physics A, 27:4425-4437,1994.
12. Barak A. Pearlmutter. Fast exact multiplication by the hessian. Neural Computation, 6:147-160, 1994.
13. Yann LeCun, Patrice Y. Simard, and Barak Pearlmutter. Automatic learning rate maximization by on-line estimation of the hessians eigenvectors. In Giles, Hanson, and Cowan, editors, Advances in Neural Information Processing Systems, vol. 5, San Mateo, CA, 1993. Morgan Kaufmann.
14. Christian Darken. Learning Rate Schedules for Stochastic Gradient Algorithms. PhD thesis, Yale University, 1993.