Statistical dynamics of on-line independent component analysis

Journal of Machine Learning Research

Volumn 4, Issue 7-8, 2004, Pages 1393-1410

  Basalyga, Gleb ^a   Rattray, Magnus ^a  

^a UNIVERSITY OF MANCHESTER   (United Kingdom)

Author keywords

Diffusion; Hebbian learning; Independent component analysis; Natural gradient; Statistical mechanics

Indexed keywords

DATA ACQUISITION; MATRIX ALGEBRA; ONLINE SYSTEMS; PARAMETER ESTIMATION; POLYNOMIALS; STATISTICAL MECHANICS;

DATA SETS; DIFFUSION PROCESS; HEBBIAN LEARNING; NATURAL GRADIENT;

INDEPENDENT COMPONENT ANALYSIS;

EID: 8344270246     PISSN: 15324435     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (25)

1. S.-I. Amari. Natural gradient works efficiently in learning. Neural Computation, 10:251-276, 1998.
2. S.-I. Amari. Natural gradient learning for over- and under-complete bases in ICA. Neural Computation, 11(8):1875-1883, 1999.
3. S.-I. Amari, A. Cichocki, and H. H. Yang. A new learning algorithm for blind source separation. In D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, editors, Neural Information Processing Systems 8, pages 757-763. MIT Press, Cambridge, 1996.
4. M. Biehl. An exactly solvable model of unsupervised learning. Europhysics Letters, 25:391-396, 1994.
5. M. Biehl and E. Schlösser. The dynamics of on-line principle component analysis. Journal of Physics A, 31:L97-L103, 1998.
6. M. Biehl and H. Schwarze. Learning by on-line gradient descent. Journal of Physics A, 28:643-656, 1995.
7. J.-F. Cardoso and B. Laheld. Equivariant adaptive source separation. IEEE Transactions on Signal Processing, 44:3017-3030, 1996.
8. A. Edelman, T. A. Arias, and S. T. Smith. The geometry of algorithms with orthogonality constraints. SIAM Journal on Matrix Analysis and Applications, 20(2):303-353, 1999.
9. A. Engel and C. Van den Broeck. Statistical Mechanics of Learning. Cambridge University Press, 2001.
10. C. W. Gardiner. Handbook of Stochastic Methods. Springer-Verlag, New York, 1985.
11. A. Hyvärinen. Survey on independent component analysis. Neural Computing Surveys, 2:94-128, 1999.
12. A. Hyvärinen and E. Oja. A fast fixed-point algorithm for independent component analysis. Neural Computation, 9:1483-1492, 1997.
13. A. Hyvärinen and E. Oja. Independent component analysis by general non-linear Hebbian-like learning rules. Signal Processing, 64:301-313, 1998.
14. H. J. Kushner. Asymptotic global behaviour for stochastic approximation and diffusions with slowly decreasing noise effects: Global minimization via monte carlo. SIAM Journal of Applied Mathematics, 41:169-185, 1987.
15. H. J. Kushner and D. S. Clark. Stochastic Approximation Methods for Constrained and Unconstrained Systems. Springer-Verlag, New York, 1978.
16. T. K. Moon and J. H. Gunther. Contravariant adaptation on structured matrix spaces. Signal Processing, 82(10):1389-1410, 2002.
17. W. H. Press, S. A. Teukolsky, and W. T. Vetterling. Numerical Recipes in C. Cambridge University Press, 2nd edition, 1992.
18. M. Rattray. Stochastic trapping in a solvable model of independent component analysis. Neural Computation, 14:421-435, 2002.
19. M. Rattray and G. Basalyga. Scaling laws and local minima in Hebbian ICA. In T. G. Dietterich, S. Becker, and Z. Ghahramani, editors, Advances in Neural Information Processing Systems 14, pages 495-501. MIT Press, Cambridge, 2002.
20. M. Rattray and D. Saad. Analysis of natural gradient descent for multilayer neural networks. Physical Review E, 59:4523-4532, 1999.
21. M. Rattray, D. Saad, and S.-I. Amari Natural gradient descent for on-line learning. Physical Review Letters, 81:5461-5464, 1998.
22. D. Saad, editor. On-line learning in neural networks. Cambridge University Press, 1998.
23. D. Saad and S. A. Solla. Exact solution for on-line learning in multilayer neural networks. Physical Review Letters, 74:4337-4340, 1995.
24. N. G. van Kampen. Stochastic Processes in Physics and Chemistry. Elsevier, Amsterdam, 1992.
25. H. White. Learning in artificial neural networks: A statistical perspective. Neural Computation, 1 (4):425-464, 1989.