Natural Gradient Works Efficiently in Learning

Neural Computation

Volumn 10, Issue 2, 1998, Pages 251-276

  Amari, Shun Ichi ^a  

^a RIKEN   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0000396062     PISSN: 08997667     EISSN: None     Source Type: Journal    
DOI: 10.1162/089976698300017746     Document Type: Article

References (41)

1. Amari, S. (1967). Theory of adaptive pattern classifiers. IEEE Trans., EC-16(3), 299-307.
2. Amari, S. (1977). Neural theory of association and concept-formation. Biological Cybernetics, 26, 175-185.
3. Amari, S. (1985). Differential-geometrical methods in statistics. Lecture Notes in Statistics 28. New York: Springer-Verlag.
4. Amari, S. (1987). Differential geometry of a parametric family of invertible linear systems - Riemannian metric, dual affine connections and divergence. Mathematical Systems Theory, 20, 53-82.
5. Amari, S. (1993). Universal theorem on learning curves. Neural Networks, 6, 161-166.
6. Amari, S. (1995). Learning and statistical inference. In M. A. Arbib (Ed.), Handbook of brain theory and neural networks (pp. 522-526). Cambridge, MA: MIT Press.
7. Amari, S. (1996). Neural learning in structured parameter spaces - Natural Riemannian gradient. In M. C. Mozer, M. I. Jordan, & Th. Petsche (Eds.), Advances in neural processing systems, 9. Cambridge, MA: MIT Press.
8. Amari, S. (1997a). Information geometry. Contemporary Mathematics, 203, 81-95.
9. Amari, S. (1997b). Superefficiency in blind source separation. Unpublished manuscript.
10. Amari, S., & Cardoso, J. F. (In press). Blind source separation - Semi-parametric statistical approach. IEEE Trans. on Signal Processing.
11. Amari, S., Chen, T.-P., & Cichocki, A. (In press). Stability analysis of learning algorithms for blind source separation. Neural Networks.
12. Amari, S., Cichocki, A., & Yang, H. H. (1996). A new learning algorithm for blind signal separation, in NIPS'95, vol. 8, Cambridge, MA: MIT Press.
13. Amari, S., Douglas, S. C., Cichocki, A., & Yang, H. H. (1997). Multichannel blind deconvolution and equalization using the natural gradient. Signal Processing Advance in Wireless Communication Workshop, Paris.
14. Amari, S., & Kawanabe, M. (1997). Information geometry of estimating functions in semiparametric statistical models, Bernoulli, 3, 29-54.
15. Amari, S., Kurata, K., & Nagaoka, H. (1992). Information geometry of Boltzmann machines. IEEE Trans. on Neural Networks, 3, 260-271.
16. Amari, S., & Murata, N. (1993). Statistical theory of learning curves under entropic loss criterion. Neural Computation, 5, 140-153.
17. Barkai, N., Seung, H. S., & Sompolinsky, H. (1995). Local and global convergence of on-line learning. Phys. Rev. Lett., 75, 1415-1418.
18. Bell, A. J., & Sejnowski, T. J. (1995). An information-maximization approach to blind separation and blind deconvolution. Neural Computation, 7, 1129-1159.
19. Bickel, P. J., Klassen, C. A. J., Ritov, Y., & Wellner, J. A. (1993). Efficient and adaptive estimation for semiparametric models. Baltimore: Johns Hopkins University Press.
20. Campbell, L. L. (1985). The relation between information theory and the differential-geometric approach to statistics. Information Sciences, 35, 199-210.
21. Cardoso, J. F., & Laheld, B. (1996). Equivariant adaptive source separation. IEEE Trans. on Signal Processing, 44, 3017-3030.
22. Chentsov, N. N. (1972). Statistical decision rules and optimal inference (in Russian). Moscow: Nauka [translated in English (1982), Rhode Island: AMS].
23. Comon, P. (1994). Independent component analysis, a new concept? Signal Processing, 36, 287-314.
24. Douglas, S. C., Cichocki, A., & Amari, S. (1996). Fast convergence filtered regressor algorithms for blind equalization. Electronics Letters, 32, 2114-2115.
25. Heskes, T., & Kappen, B. (1991). Learning process in neural networks. Physical Review, A44, 2718-2762.
26. Jutten, C., & Hérault, J. (1991). Blind separation of sources, an adaptive algorithm based on neuromimetic architecture. Signal Processing, 24(1), 1-31.
27. Kushner, H. J., & Clark, D. S. (1978). Stochastic approximation methods for constrained and unconstrained systems. Berlin: Springer-Verlag.
28. Murata, N., & Müller, K. R., Ziehe, A., & Amari, S. (1996). Adaptive on-line learning in changing environments. In M. C. Mozer, M. I. Jordan, & Th. Petsche (Eds.), Advaces in neural processing systems, 9. Cambridge, MA: MIT Press.
29. Murray, M. K., & Rice, J. W. (1993). Differential geometry and statistics. New York: Chapman & Hall.
30. Nadal, J. P. & Parga, N. (1994). Nonlinear neurons in the low noise limit - A factorial code maximizes information transfer. Network, 5, 561-581.
31. Oja, E., & Karhunen, J. (1995). Signal separation by nonlinear Hebbian learning. In M. Palaniswami et al. (Eds.), Computational intelligence - A dynamic systems perspective (pp. 83-97). New York: IEEE Press.
32. Opper, M. (1996). Online versus offline learning from random examples: General results. Phys. Rev. Lett., 77, 4671-4674.
33. Rao, C. R. (1945). Information and accuracy attainable in the estimation of statistical parameters. Bulletin of the Calcutta Mathematical Society, 37, 81-91.
34. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning internal representations by error propagation. Parallel Distributed Processing (Vol. 1, pp. 318-362). Cambridge, MA: MIT Press.
35. Saad, D., & Solla, S. A. (1995). On-line learning in soft committee machines. Phys. Rev. E, 52, 4225-4243.
36. Sompolinsky, H., Barkai, N., & Seung, H. S. (1995). On-line learning of dichotomies: Algorithms and learning curves. In J.-H. Oh et al. (Eds.), Neural networks: The statistical mechanics perspective (pp. 105-130). Proceedings of the CTP-PBSRI Joint Workshop on Theoretical Physics. Singapore: World Scientific.
37. Tsypkin, Ya. Z. (1973). Foundation of the theory of learning systems. New York: Academic Press.
38. Van den Broeck, C., & Reimann, P. (1996). Unsupervised learning by examples: On-line versus off-line. Phys. Rev. Lett., 76, 2188-2191.
39. Widrow, B. (1963). A statistical theory of adaptation. Oxford: Pergamon Press.
40. Yang, H. H., & Amari, S. (1997). Application of natural gradient in training multilayer perceptions. Unpublished manuscript.
41. Yang, H. H., & Amari, S. (In press). Adaptive on-line learning algorithms for blind separation - Maximum entropy and minimal mutual information. Neural Computation.