Improving Generalization Performance of Natural Gradient Learning Using Optimized Regularization by NIC

Neural Computation

Volumn 16, Issue 2, 2004, Pages 355-382

  Park, Hyeyoung ^a   Murata, Noboru ^b   Amari, Shun Ichi ^a  

^a RIKEN BRAIN SCIENCE INSTITUTE   (Japan)

^b WASEDA UNIVERSITY   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; NORMAL DISTRIBUTION; SIGNAL PROCESSING; STATISTICS;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; NEURAL NETWORKS (COMPUTER); NORMAL DISTRIBUTION; SIGNAL PROCESSING, COMPUTER-ASSISTED; STOCHASTIC PROCESSES;

EID: 0347029918     PISSN: 08997667     EISSN: None     Source Type: Journal    
DOI: 10.1162/089976604322742065     Document Type: Article

References (18)

1. Akaike, H. (1974). A new look at the statistical model identification. IEEE trans. on Automatic Control, 16, 716-724.
2. Amari, S. (1998). Natural gradient works efficiently. Neural Computation, 10, 251-276.
3. Amari, S., & Murata, N. (1997). Statistical analysis of regularization constant - From Bayes, MDL, and NIC points of view. Lecture Notes in Computer Science, 1240, 284-293.
4. Amari, S., Park, H., & Fukumizu, K. (2000). Adaptive method of realizing natural gradient learning for multilayer perceptrons. Neural Computation, 12, 1399-1409.
5. Bishop, C. M. (1995). Neural networks for pattern recognition. New York: Oxford University Press.
6. Bottou, L. (1998). Online algorithms and stochastic approximations. In D. Saad (Ed.), Online learning and neural networks. Cambridge: Cambridge University Press.
7. Gorman, R. P., & Sejnowski, T. J. (1988). Analysis of hidden units in a layered network trained to classify sonar targets. Neural Networks, 1, 75-89.
8. Hagiwara, K. (2002). On the problem in model selection of neural network regression in overrealizable scenario. Neural Computation, 14, 1979-2002.
9. Moody, M. E. (1992). The effective number of parameters: An analysis of generalization and regularization in nonlinear learning systems. In J. E. Moody, S. J. Hanson, & R. P. Lippmann (Eds.), Advances in Neural Information Processing Systems, 4 (pp. 847-854). San Mateo, CA: Morgan Kaufmann.
10. Murata, N. (2001). Bias of estimators and regularization terms. Dagstuhl seminar 01301. Available on-line: http://www.dagstuhl.de/Seminars/.
11. Murata, N., Yoshizawa, S., & Amari, S. (1994). Network information criterion - Determining the number of hidden units for an artificial neural network model. IEEE Trans. on Neural Networks, 5(6), 865-872.
12. Park, H. (2001). Practical consideration on generalization property of natural gradient learning. Lecture Notes in Computer Science, 2084, 402-409.
13. Park, H., Amari, S., & Fukumizu, K. (2000). Adaptive natural gradient learning algorithms for various stochastic models. Neural Networks, 13, 755-764.
14. Rattray, M., Saad, D., & Amari, S. (1998). Natural gradient descent for on-line learning. Physical Review Letters, 81, 5461-5464.
15. Rissanen, J. (1978). Modelling by shortest data description. Automatica, 14, 465-471.
16. Sigurdsson, S., Larsen, J., & Hansen, L. K. (2000). On comparison of adaptive regularization methods. Proc. of Neural Network for Signal Processing (X, pp. 221-230). Piscataway, NJ: IEEE Press.
17. Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. Journal of Royal Statistical Society, 36, 111-133.
18. Vapnik, V. (1995). The nature of statistical learning theory. Berlin: Springer-Verlag.