A Hebbian/anti-Hebbian neural network for linear subspace learning: A derivation frommultidimensional scaling of streaming data

Neural Computation

Volumn 27, Issue 7, 2015, Pages 1461-1495

  Pehlevan, Cengiz ^a,b   Hu, Tao ^c   Chklovskii, Dmitri B ^b  

^a HOWARD HUGHES MEDICAL INSTITUTE   (United States)

^b Simons Center for Data Analysis   (United States)

^c TEXAS A AND M UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHM; ARTIFICIAL NEURAL NETWORK; BIOLOGICAL MODEL; LEARNING; NERVE CELL; PRINCIPAL COMPONENT ANALYSIS; STATISTICAL MODEL;

ALGORITHMS; LEARNING; LINEAR MODELS; MODELS, NEUROLOGICAL; NEURAL NETWORKS (COMPUTER); NEURONS; PRINCIPAL COMPONENT ANALYSIS;

EID: 84930973862     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/NECO_a_00745     Document Type: Letter

References (45)

1. Arora, R., Cotter, A., Livescu, K., & Srebro, N. (2012). Stochastic optimization for PCA and PLS. In Proceedings of the Allerton Conference on Communication, Control, and Computing (pp. 861-868). Piscataway, NJ: IEEE.
2. Balzano, L. K. (2012). Handling missing data in high-dimensional subspace modeling Doctoral dissertation, University ofWisconsin-Madison.
3. Becker, S., & Plumbley, M. (1996). Unsupervised neural network learning procedures for feature extraction and classification. Appl. Intell., 6(3), 185-203.
4. Carroll, J., & Chang, J. (1972). IDIOSCAL (individual differences in orientation scaling): A generalization of INDSCAL allowing idiosyncratic reference systems as well as an analytic approximation to INDSCAL. Paper presented at the Psychometric Meeting, Princeton, NJ.
5. Cichocki, A., & Amari, S.-I. (2002). Adaptive blind signal and image processing. Hoboken, NJ: Wiley.
6. Cox, T., & Cox, M. (2000). Multidimensional scaling. Boca Raton, FL: CRC Press.
7. Crammer, K. (2006). Online tracking of linear subspaces. In G. Lugois & H. U. Simon (Eds.), Learning theory (pp. 438-452). New York: Springer.
8. Diamantaras, K. (2002).Neural networks and principal component analysis. In Y.-H. Hu & J.-N. Hwang (Eds.), Handbook of neural networks in signal processing. Boca Raton, FL: CRC Press.
9. Diamantaras, K., & Kung, S. (1996). Principal component neural networks: Theory and applications. Hoboken, NJ: Wiley.
10. Földiak, P. (1989). Adaptive network for optimal linear feature extraction. In Proceedings of the International Joint Conference on Neural Networks (pp. 401-405). Piscataway, NJ: IEEE.
11. Goes, J., Zhang, T., Arora, R., & Lerman, G. (2014). Robust stochastic principal component analysis. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics (pp. 266-274). http://jmlr.org/proceedings /papers/v33/
12. Hornik, K., & Kuan, C.-M. (1992). Convergence analysis of local feature extraction algorithms. Neural Networks, 5, 229-240.
13. Hu, T., Towfic, Z., Pehlevan, C., Genkin, A., & Chklovskii, D. (2013). A neuron as a signal processing device. In Proceedings of the Asilomar Conference on Signals, Systems and Computers (pp. 362-366). Piscataway, NJ: IEEE.
14. Hubel, D. H. (1995). Eye, brain, and vision. New York: Scientific American Library/ Scientific American Books.
15. Hyvärinen, A., Hurri, J., & Hoyer, P. O. (2009). Natural image statistics: A probabilistic approach to early computational vision. New York: Springer.
16. Karhunen, J., & Oja, E. (1982). New methods for stochastic approximation of truncated Karhunen-Loeve expansions. In Proc. 6th Int. Conf. on Pattern Recognition (pp. 550-553). New York: Springer-Verlag.
17. Kung, S.-Y. (2014). Kernel methods and machine learning. Cambridge: Cambridge University Press.
18. Kung, S., & Diamantaras,K. (1990).Aneural network learning algorithm for adaptive principal component extraction (apex). In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing (pp. 861-864). Piscataway, NJ: IEEE.
19. Kung, S.-Y., Diamantaras, K., & Taur, J.-S. (1994). Adaptive principal component extraction (APEX) and applications. IEEE T Signal Proces., 42, 1202-1217.
20. Kushner H. J., & Clark D. S. (1978). Stochastic approximation methods for constrained and unconstrained systems. New York: Springer.
21. Leen, T. K. (1990). Dynamics of learning in recurrent feature-discovery networks. In D. Touretzky & R. Lippmann (Eds.), Advances in neural information processing systems, 3 (pp. 70-76). San Mateo, CA: Morgan Kaufmann.
22. Leen, T. K. (1991). Dynamics of learning in linear feature-discovery networks. Network, 2(1), 85-105.
23. Linsker, R. (1988). Self-organization in a perceptual network. IEEE Computer, 21, 105-117.
24. Luo, Z. Q., & Tseng, P. (1991). On the convergence of a matrix splitting algorithm for the symmetric monotone linear complementarity problem. SIAM J. Control Optim., 29, 1037-1060.
25. Mardia, K., Kent, J., & Bibby, J. (1980). Multivariate analysis. Orlando, FL: Academic Press.
26. Oja, E. (1982). Simplified neuron model as a principal component analyzer. J. Math. Biol., 15, 267-273.
27. Oja, E. (1992). Principal components, minor components, and linear neural networks. Neural Networks, 5, 927-935.
28. Oja, E., & Karhunen, J. (1985). On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix. J. Math. Anal. Appl., 106(1), 69-84.
29. Ostrowksi A. M., (1954). On the linear iteration procedures for symmetric matrices. Rend. Mat. Appl., 14, 140-163.
30. Pearson, K. (1901). On lines and planes of closest fit to systems of points in space. Philos Mag., 2, 559-572.
31. Plumbley, M. D. (1993). A Hebbian/anti-Hebbian network which optimizes information capacity by orthonormalizing the principal subspace. In Proceedings of the International Conference on Artificial Neural Networks (pp. 86-90). Piscataway, NJ: IEEE.
32. Plumbley,M.D. (1995). Lyapunov functions for convergence of principal component algorithms. Neural Networks, 8(1), 11-23.
33. Preisendorfer, R., & Mobley, C. (1988). Principal component analysis in meteorology and oceanography. New York: Elsevier Science.
34. Reich, E. (1949). On the convergence of the classical iterative procedures for symmetric matrices. Ann. Math. Statistics, 20, 448-451.
35. Rubner, J., & Schulten, K. (1990). Development of feature detectors by selforganization. Biol. Cybern., 62, 193-199.
36. Rubner, J., & Tavan, P. (1989). A self-organizing network for principal-component analysis. Europhysics Letters, 10(7), 693.
37. Sanger, T. (1989).Optimal unsupervised learning in a single-layer linear feedforward neural network. Neural Networks, 2(6), 459-473.
38. Shepherd, G. (2003). The synaptic organization of the brain. New York: Oxford University Press.
39. Strang, G. (2009). Introduction to linear algebra. Wellesley, MA: Wellesley-Cambridge Press.
40. Torgerson,W. (1952).Multidimensional scaling: I. Theory and method. Psychometrika, 17, 401-419.
41. Warmuth, M., & Kuzmin, D. (2008). Randomized online PCA algorithms with regret bounds that are logarithmic in the dimension. J. Mach. Learn. Res., 9(10), 2287- 2320.
42. Williams, C. (2001). On a connection between kernel PCA and metric multidimensional scaling. In T. K. Leen, T. G. Dietterich, & V. Tresp (Eds.), Advances in neural information processing systems, 13 (pp. 675-681). Cambridge, MA: MIT Press.
43. Yang, B. (1995). Projection approximation subspace tracking. IEEE T Signal Proces., 43, 95-107.
44. Young, G., & Householder, A. (1938). Discussion of a set of points in terms of their mutual distances. Psychometrika, 3(1), 19-22.
45. Zufiria, P. J. (2002). On the discrete-time dynamics of the basic Hebbian neural network node. IEEE Trans. Neural Netw., 13, 1342-1352.