Why regularized auto-encoders learn sparse representation?

33rd International Conference on Machine Learning, ICML 2016

Volumn 1, Issue , 2016, Pages 211-223

  Arpit, Devansh ^a   Zhou, Yingbo ^a   Ngo, Hung Q ^a   Govindaraju, Venu ^a  

^a UNIVERSITY AT BUFFALO   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ACTIVATION ANALYSIS; ARTIFICIAL INTELLIGENCE; CHEMICAL ACTIVATION; LEARNING ALGORITHMS;

ACTIVATION FUNCTIONS; AUTO ENCODERS; DATA REPRESENTATIONS; DISTRIBUTED REPRESENTATION; EMPIRICAL ANALYSIS; FORMAL ANALYSIS; GENERATION PROCESS; SPARSE REPRESENTATION;

LEARNING SYSTEMS;

EID: 84997712018     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (25)

1. Bengio, Yoshua, Mesnil, Grégoire, Dauphin, Yann, and Rifai, Salah. Better mixing via deep representations. In ICML, pp. 552-560, 2013.
2. Bourlard, H. and Kamp, Y. Auto-association by multilayer perceptrons and singular value decomposition. Biological Cybernetics, 59(4-5):291-294, 1988. ISSN 0340-1200.
3. Chen, Minmin, Weinberger, Kilian Q., Sha, Fei, and Bengio, Yoshua. Marginalized denoising auto-encoders for nonlinear representations. In ICML, pp. 1476-1484, 2014.
4. Glorot, Xavier and Bengio, Yoshua. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
5. Glorot, Xavier, Bordes, Antoine, and Bengio, Yoshua. Deep sparse rectifier neural networks. In AISTATS, pp. 315-323, 2011.
6. Goodfellow, Ian J., Warde-Farley, David, Mirza, Mehdi, Courville, Aaron C, and Bengio, Yoshua. Maxout networks. In ICML, pp. 1319-1327, 2013.
7. Hinton, Geoffrey, Srivastava, Nitish, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Improving neural networks by preventing co-adaptation of feature detectors. CoRR, abs/1207.0580, 2012.
8. Hinton, Geoffrey E. Distributed representations. 1984.
9. Hubel, D. H. and Wiesel, T. N. Receptive fields of single neurones in the cat's striate cortex. The Journal of physiology, 148:574-591, October 1959.
10. Kavukcuoglu, Koray and Lecun, Yann. Fast inference in sparse coding algorithms with applications to object recognition. Technical report, Courant Institute, NYU, 2008.
11. Krizhevsky, Alex. Learning Multiple Layers of Features from Tiny Images. Technical report, 2009.
12. Lecun, Yann and Cortes, Corinna. The MNIST database of handwritten digits. URL http://yann. lecun. com/exdb/mn i s t/.
13. Lee, Honglak, Ekanadham, Chaitanya, and Ng, Andrew Y. Sparse deep belief net model for visual area v2. In Advances in Neural Information Processing Systems 20. MIT Press, 2008.
14. Li, Jun, Luo, Wei, Yang, Jian, and Yuan, Xiaotong. Why does the unsupervised pretraining encourage moderate-sparseness? CoRR, abs/1312.5813, 2013. URL http://arxiv. org/abs/1312.5813.
15. Memisevic, Roland, Konda, Kishore Reddy, and Krueger, David. Zero-bias autoencoders and the benefits of co-adapting features. In ICLR, 2014.
16. Nair, Vinod and Hinton, Geoffrey E. Rectified linear units improve restricted boltzmann machines. In ICML, pp. 807-814, 2010.
17. Ng, Andrew. Sparse autoencoder. CSE294 Lecture notes, 2011.
18. Olshausen, Bruno A. and Fieldt, David J. Sparse coding with an overcomplete basis set: a strategy employed by vl. Vision Research, 37:3311-3325, 1997.
19. Patterson, Karalyn, Nestor, Peter, and Rogers, Timothy. Where do you know what you know? the representation of semantic knowledge in the human brain. Nature Rev. Neuroscience, 8(12):976-987, 2007.
20. Rifai, Salah, Mesnil, Grégoire, Vincent, Pascal, Muller, Xavier, Bengio, Yoshua, Dauphin, Yann, and Glorot, Xavier. Higher order contractive auto-encoder. In ECMUPKDD, pp. 645-660, 2011a.
21. Rifai, Salah, Vincent, Pascal, Muller, Xavier, Glorot, Xavier, and Bengio, Yoshua. Contractive auto-encoders: Explicit invariance during feature extraction. In ICML, pp. 833-840, 2011b.
22. Rumelhart, David E., Hinton, Geoffrey E., and Williams, Ronald J. Learning representations by back-propagating errors. Nature, pp. 533-536, 1986.
23. Vincent, Pascal, Larochelle, Hugo, Bengio, Yoshua, and Manzagol, Pierre-Antoine. Extracting and composing robust features with denoising autoencoders. In ICML, pp. 1096-1103, 2008.
24. Wright, J., Yang, A. Y., Ganesh, A., Sastry, S. S., and Ma, Yi. Robust face recognition via sparse representation. IEEEE TPAMI, 31(2):210-227, Feb. 2009.
25. Yang, Jianchao, Yu, Kai, Gong, Yihong, and Huang, Thomas. Linear spatial pyramid matching using sparse coding for image classification. In CVPR, pp. 1794-1801, 2009.