Implementing online natural gradient learning: Problems and solutions

IEEE Transactions on Neural Networks

Volumn 17, Issue 2, 2006, Pages 317-329

  Wan, Weishui ^a  

^a CED System Corporation   (Japan)

Author keywords

Adaptive learning rate; Fisher information matrix; Online natural gradient learning

Indexed keywords

ADAPTIVE SYSTEMS; LEARNING ALGORITHMS; MATRIX ALGEBRA; ONLINE SYSTEMS;

ADAPTIVE LEARNING; FISHER INFORMATION MATRIX; LEARNING RATE; ONLINE NATURAL GRADIENT LEARNING;

LEARNING SYSTEMS;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; AUTOMATED PATTERN RECOGNITION; COMPUTER SIMULATION; DECISION SUPPORT SYSTEM; EVALUATION; MATHEMATICAL COMPUTING; METHODOLOGY; THEORETICAL MODEL;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; DECISION SUPPORT TECHNIQUES; MODELS, THEORETICAL; NEURAL NETWORKS (COMPUTER); NUMERICAL ANALYSIS, COMPUTER-ASSISTED; PATTERN RECOGNITION, AUTOMATED;

EID: 33644928410     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2005.863406     Document Type: Article

References (35)

1. S.-I. Amari, H. Park, and K. Fukumizu, "Adaptive method of realizing natural gradient learning for multilayer perceptrons," Neural Comput., vol. 12, no. 6, pp. 1399-1409, 2000.
2. S. Amari, "Natural gradient works efficiently in learning," Neural Comput., vol. 10, pp. 251-276, 1998.
3. M. Rattray, D. Saad, and S. Amari, "Natural gradient descent for online learning," Phys. Rev. Lett., vol. 81, pp. 5461-5461, 1998.
4. H. Park, S. Amari, and K. Fukumizu, "Adaptive natural gradient learning algorithms for various stochastic models," Neural Networks, vol. 13, no. 7, pp. 755-764, 2000.
5. H. H. Yang and S. Amari, "Complexity issues in natural gradient descent method for training multilayer perceptrons," Neural Comput., vol. 10, pp. 2137-2157, 1998.
6. T. Heskes, "On natural learning and pruning in multilayered perceptron," Neural Comput., vol. 12, pp. 1037-1057, 2000.
7. J. Kivinen and M. Warmuth, "Exponential gradient versus gradient descent for linear predictors," Inform. Comput., vol. 132, pp. 1-63, 1997.
8. N. Murata, K.-R. Muller, A. Ziehe, and S. Amari, "Adaptive online learning in changing environments," in Proc. NIPS, vol. 9, 1997, pp. 599-604.
9. H. J. Kushner and D. S. Clark, Stochastic Approximation Methods for Constrained and Unconstrained Systems. New York: Springer-Verlag, 1978.
10. C. MicDiarmid, "On the method of bounded difference," in Survey in Combinatorics 1989. Cambridge, U.K.: Cambridge Univ. Press, 1989, pp. 148-188.
11. V. N. Vapnik, Statistical Learning Theory. New York: Wiley, 1998.
12. H. Park, S. Amari, and K. Fukumizu, "Adaptive natural gradient learning algorithms for various stochastic models," Neural Networks, vol. 13, no. 7, pp. 755-764, 2000.
13. L. Prechelt, "Proben1 - A set of neural network benchmark problems and benchmarking rules," Fakultaet fuer Informatik, Universitaet Karlsruhe, Karlsruhe, Germany, Tech. Rep. 21/94.
14. H. Braun, Neuronale Netze: Optimierung Durch Lernen und Evolution. New York: Springer-Verlag, 1997.
15. P. Auer and C. Gentile, "Adaptive and self-confident online learning algorithms," in Proc. 13th Annu. Conf. Computational Learning Theory, 2000, pp. 107-117.
16. M. Riedmiler and H. Braun, "A direct adaptive method for faster backpropagation learning: The RPROP algorithm," in Proc. IEEE Int. Conf. Neural Networks, San Francisco, CA, 1993, pp. 586-591.
17. C. Darken and J. Moody, "Learning rate schedules for faster stochastic gradient search," in Proc. 1992 IEEE Workshop Neural Networks for Signal Processing 2, Piscataway, NJ, 1991, pp. 3-12.
18. C. Darken and J. Moody, "Toward faster stochastic gradient search," in Advances in Neural Information Processing Systems 4, Moody, Hanson, and Lippman, Eds. Palo Alto, CA: Morgan Kaufman, 1992.
19. T. K. Leen and G. B. Orr, "Using curvature information for fast stochastic search," in Advances in Neural Information Processing Systems 9, M. I. Jordan, M. C. Mozer, and T. Petsche, Eds. Cambridge, MA: MIT Press, 1996.
20. T. Tollenaere, "SuperSAB: Fast adaptive back propagation with good scaling properties," Neural Networks, vol. 3, pp. 561-573, 1990.
21. L. B. Almeida, T. Langlois, and J. D. Amaral, "Step size adaption,", Portugal, Tech. Rep. INSEC RT07, 1997.
22. N. Murata, K. R. Mueller, A. Ziehe, and S. Amari, "Adaptive online learning in changing environment," in Advances in Neural Information Processing Systems 9, M. I. Jordan, M. C. Mozer, and T. Petsche, Eds. Cambridge, MA: MIT Press, 1996, pp. 599-605.
23. N. N. Schraudolph, "Local gain adaption for multi-layer perceptrons,", Tech. Rep. IDSIA-09-98, 1998.
24. R. S. Sutton, "Gain adaption beats least squares?," in Proc. 7th Yale Workshop on Adaptive and Learning Systems, 1992, pp. 161-166.
25. L. B. Almeida, T. Langlois, J. D. Amaral, and A. Plakhov, "Parameter adaption in stochastic optimization," in Learning in Neural Networks, D. Saad, Ed. Cambridge, U.K.: Cambridge Univ. Press, 1999, ch. 6.
26. R. Neuneier and H. G. Zimmermann, "How to train neural networks," in Neural Networks: Tricks of the Trade. Berlin, Germany: Springer-Verlag, 1998, vol. 1524, Lecture Notes in Computer Science, pp. 373-423.
27. H. G. Zimmermann, "Neuronale Netze als Entscheidungskalkuel," in Neuronal Netze in der Oekonomie: Grundlagen und Finanzwirtschaftliche Anwendungen, H. Rehkugler and H. G. Zimmermann, Eds. Munich, Germany: Vahlen-Verlag, 1994, pp. 1-87.
28. R. Jacobs, "Increased rates of convergence through learning rate adaptation," Neural Networks, vol. 1, pp. 295-307, 1988.
29. R. Battiti, "Accelerated back-propagation learning: Two optimization methods," Complex Syst., vol. 3, pp. 331-342, 1989.
30. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning internal representations bt error propagation," in Parallel Distributed Processing: Exporations in the Microstructure of Cognition, D. E. Rumelhart and J. L. McClell, Eds. Cambridge, MA: MIT Press, 1986, vol. 1.
31. W. H. Press, B. P. Flannery, S. A. Teukolsky, and W. T. Wetterling, Numerical Recipes in C. Cambridge, U.K.: Cambridge Univ. Press, 1988.
32. S. E. Fahlman and C. Lebiere, "The cascade-Correlation learning architecture," in NIPS 2, D. S. Touretzky, Ed. San Mateo, CA: Morgen Kaufmann, 1990, pp. 524-532.
33. P. Simard, Y. LeCun, and Denker, "Efficient pattern recognition using a transformation distance," in Advances in Neural Information Processing Systems. San Mateo, CA: Morgan Kaufmann, 1993, pp. 50-58.
34. P. Y. Simard, LeCun, Denker, and Victorri, "Transformation invariance in pattern recognition tangent distance tangent propagation," in LNCS. New York: Springer-Verlag, 1998, vol. 1524, pp. 239-274.
35. Le Cun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, "Handwritten digit recognition with back-propagation network," in Advances in Neural Information Processing Systems. San Mateo, CA: Morgan Kaufmann, 1990, pp. 396-404.