Optimization theory of Hebbian/anti-Hebbian networks for PCA and whitening

2015 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015

Volumn , Issue , 2016, Pages 1458-1465

  Pehlevan, Cengiz ^a   Chklovskii, Dmitri B ^a  

^a Simons Center for Data Analysis   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATION THEORY; NEURAL NETWORKS; OPTIMIZATION; SIGNAL PROCESSING; SITE SELECTION;

INFORMATION TRANSMISSION; OBJECTIVE FUNCTIONS; ONLINE SIGNAL PROCESSING; OPTIMIZATION THEORY; POST-SYNAPTIC NEURONS; SIMILARITY-MATCHING; STATISTICAL SIGNAL PROCESSING; UNDERUTILIZED NEURONS;

PRINCIPAL COMPONENT ANALYSIS;

EID: 84969848822     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ALLERTON.2015.7447180     Document Type: Conference Paper

References (30)

1. K. Crammer, "Online tracking of linear subspaces," in Learning Theory. Springer, 2006, pp. 438-452.
2. R. Arora, A. Cotter, K. Livescu, and N. Srebro, "Stochastic optimization for pca and pls," in Allerton Conf. Communication, Control, and Computing. IEEE, 2012, pp. 861-868.
3. J. Goes, T. Zhang, R. Arora, and G. Lerman, "Robust stochastic principal component analysis," in Proc. 17th Int. Conf. Artificial Intelligence and Statistics, 2014, pp. 266-274.
4. E. Oja, "Simplified neuron model as a principal component analyzer," J Math Biol, vol. 15, no. 3, pp. 267-273, 1982.
5. T. Hu, Z. Towfic, C. Pehlevan, A. Genkin, and D. B. Chklovskii, "A neuron as a signal processing device," in Asilomar Conf. Signals, Systems and Computers. IEEE, 2013, pp. 362-366.
6. K. Diamantaras and S. Kung, Principal component neural networks: theory and applications. John Wiley & Sons, Inc., 1996.
7. T. Sanger, "Optimal unsupervised learning in a single-layer linear feedforward neural network," Neural Net, vol. 2, pp. 459-473, 1989.
8. P. Földiak, "Adaptive network for optimal linear feature extraction," in Int. Joint Conf. Neural Networks. IEEE, 1989, pp. 401-405.
9. J. Rubner and P. Tavan, "A self-organizing network for principalcomponent analysis," EPL, vol. 10, no. 7, p. 693, 1989.
10. T. Leen, "Dynamics of learning in recurrent feature-discovery networks," Advances in Neural Information Processing Systems, 1990.
11. S. Kung and K. Diamantaras, "A neural network learning algorithm for adaptive principal component extraction (apex)," in Int. Conf. Acoustics, Speech, and Signal Processing. IEEE, 1990, pp. 861-864.
12. E. Oja, "Principal components, minor components, and linear neural networks," Neural Net, vol. 5, no. 6, pp. 927-935, 1992.
13. C. Pehlevan, T. Hu, and D. B. Chklovskii, "A hebbian/anti-hebbian neural network for linear subspace learning: A derivation from multidimensional scaling of streaming data," Neural Comput, vol. 27, pp. 1461-1495, 2015.
14. C. Pehlevan and D. B. Chklovskii, "A hebbian/anti-hebbian network derived from online non-negative matrix factorization can cluster and discover sparse features," in Asilomar Conf. Signals, Systems and Computers. IEEE, 2014, pp. 769-775.
15. T. Hu, C. Pehlevan, and D. B. Chklovskii, "A hebbian/anti-hebbian network for online sparse dictionary learning derived from symmetric matrix factorization," in Asilomar Conference on Signals, Systems and Computers. IEEE, 2014, pp. 613-619.
16. C. Pehlevan and D. B. Chklovskii, "A normative theory of adaptive dimensionality reduction in neural networks," Advances in Neural Information Processing Systems, 2015.
17. K. Mardia, J. Kent, and J. Bibby, Multivariate analysis. Academic press, 1980.
18. R. Linsker, "Self-organization in a perceptual network," Computer, vol. 21, no. 3, pp. 105-117, 1988.
19. M. Plumbley, "A hebbian/anti-hebbian network which optimizes information capacity by orthonormalizing the principal subspace," in Proc. 3rd Int. Conf. Artificial Neural Networks, 1993, pp. 86-90.
20. -, "Information processing in negative feedback neural networks," Network: Comp Neural, vol. 7, no. 2, pp. 301-305, 1996.
21. J. Atick and A. Redlich, "What does the retina know about natural scenes" Neural comput, vol. 4, no. 2, pp. 196-210, 1992.
22. A. Hyvärinen, J. Hurri, and P. O. Hoyer, Natural Image Statistics: A Probabilistic Approach to Early Computational Vision. Springer, 2009, vol. 39.
23. S. Boyd and L. Vandenberghe, Convex optimization. Cambridge university press, 2004.
24. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: A simple way to prevent neural networks from overfitting," J Mach Learn Res, vol. 15, no. 1, pp. 1929-1958, 2014.
25. M. Plumbley, "A subspace network that determines its own output dimension," Technical Report, August 1994.
26. P. D. King, J. Zylberberg, and M. R. DeWeese, "Inhibitory interneurons decorrelate excitatory cells to drive sparse code formation in a spiking model of v1," J Neurosci, vol. 33, pp. 5475-5485, 2013.
27. M. Zhu and C. Rozell, "Modeling inhibitory interneurons in efficient sensory coding models," PLoS Comput Biol, vol. 11, no. 7, p. e1004353, 2015.
28. B. A. Olshausen and D. J. Field, "Sparse coding with an overcomplete basis set: A strategy employed by v1" Vision Res, vol. 37, no. 23, pp. 3311-3325, 1997.
29. A. A. Koulakov and D. Rinberg, "Sparse incomplete representations: a potential role of olfactory granule cells," Neuron, vol. 72, pp. 124-136, 2011.
30. S. Druckmann, T. Hu, and D. B. Chklovskii, "A mechanistic model of early sensory processing based on subtracting sparse representations," in Advances in Neural Information Processing Systems, 2012, pp. 1979-1987.