What regularized auto-encoders learn from the data-generating distribution

Journal of Machine Learning Research

Volumn 15, Issue , 2015, Pages 3563-3593

  Alain, Guillaume ^a   Bengio, Yoshua ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

Auto encoders; Denoising auto encoders; Generative models; Manifold learning; Markov chains; Score matching; Unsupervised representation learning

Indexed keywords

MARKOV PROCESSES; MAXIMUM LIKELIHOOD;

AUTO-ENCODERS; DE-NOISING; GENERATIVE MODEL; MANIFOLD LEARNING; SCORE MATCHING; UNSUPERVISED REPRESENTATION LEARNING;

LEARNING SYSTEMS;

EID: 84919754597     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (28)

1. Y. Bengio. Learning deep architectures for AI. Foundations & Trends in Mach. Learn., 2 (1):1-127, 2009.
2. Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle. Greedy layer-wise training of deep networks. In NIPS'2006, 2007.
3. Yoshua Bengio and Olivier Delalleau. On the expressive power of deep architectures. In ALT'2011, 2011.
4. Yoshua Bengio, Guillaume Alain, and Salah Rifai. Implicit density estimation by local moment matching to sample from auto-encoders. Technical report, arXiv:1207.0057, 2012a.
5. Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. Technical report, arXiv:1206.5538, 2012b.
6. Yoshua Bengio, Yao Li, Guillaume Alain, and Pascal Vincent. Generalized denoising autoencoders as generative models. Technical Report arXiv:1305.6663, Universite de Montreal, 2013a.
7. Yoshua Bengio, Grégoire Mesnil, Yann Dauphin, and Salah Rifai. Better mixing via deep representations. In ICML'13, 2013b.
8. Lawrence Cayton. Algorithms for manifold learning. Technical Report CS2008-0923, UCSD, 2005.
9. B. Dacorogna. Introduction to the Calculus of Variations. World Scientific Publishing Company, 2004.
10. Karol Gregor, Arthur Szlam, and Yann LeCun. Structured sparse coding via lateral inhibition. In NIPS'2011, 2011.
11. G. E. Hinton, S. Osindero, and Y.-W. Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527-1554, 2006.
12. Aapo Hyvärinen. Estimation of non-normalized statistical models using score matching. J. Machine Learning Res., 6, 2005.
13. Aapo Hyvärinen. Some extensions of score matching. Computational Statistics and Data Analysis, 51:2499-2512, 2007.
14. Viren Jain and Sebastian H. Seung. Natural image denoising with convolutional networks. In NIPS'2008, 2008.
15. K. Kavukcuoglu, M-A. Ranzato, R. Fergus, and Y. LeCun. Learning invariant features through topographic filter maps. In CVPR'2009, 2009.
16. Diederik Kingma and Yann LeCun. Regularized estimation of image statistics by score matching. In NIPS'2010, 2010.
17. Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In ICML'2009. 2009.
18. Hariharan Narayanan and Sanjoy Mitter. Sample complexity of testing the manifold hypothesis. In NIPS'2010. 2010.
19. B. A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: a strategy employed by V1? Vision Research, 37:3311-3325, 1997.
20. M. Ranzato, C. Poultney, S. Chopra, and Y. LeCun. Efficient learning of sparse representations with an energy-based model. In NIPS'2006, 2007.
21. M. Ranzato, Y. Boureau, and Y. LeCun. Sparse feature learning for deep belief networks. In NIPS'2007, 2008.
22. Salah Rifai, Yann Dauphin, Pascal Vincent, Yoshua Bengio, and Xavier Muller. The manifold tangent classifier. In NIPS'2011, 2011a.
23. Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto-encoders: Explicit invariance during feature extraction. In ICML'2011, 2011b.
24. Salah Rifai, Yoshua Bengio, Yann Dauphin, and Pascal Vincent. A generative process for sampling contractive auto-encoders. In ICML'2012, 2012.
25. R. Salakhutdinov and G.E. Hinton. Deep Boltzmann machines. In AISTATS'2009, 2009.
26. Kevin Swersky, Marc'Aurelio Ranzato, David Buchman, Benjamin Marlin, and Nando de Freitas. On autoencoders and score matching for energy based models. In ICML'2011. 2011.
27. Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7), 2011.
28. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML 2008, 2008.