What regularized auto-encoders learn from the data generating distribution

1st International Conference on Learning Representations, ICLR 2013 - Conference Track Proceedings

Volumn , Issue , 2013, Pages

  Alain, Guillaume ^a   Bengio, Yoshua ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

LEARNING SYSTEMS; MAXIMUM LIKELIHOOD;

CONVENIENT ALTERNATIVES; ENERGY FUNCTIONS; LOCAL MANIFOLD STRUCTURE; METROPOLIS HASTINGS; PARAMETRIZATIONS; PARTITION FUNCTIONS; RECONSTRUCTION ERROR; RECONSTRUCTION FUNCTIONS;

SIGNAL ENCODING;

EID: 85083953791     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (28)

1. Yoshua Bengio. Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2 (1):1–127, 2009. Also published as a book. Now Publishers, 2009.
2. Yoshua Bengio and Olivier Delalleau. On the expressive power of deep architectures. In ALT’2011, 2011.
3. Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In NIPS 19, pages 153–160. MIT Press, 2007.
4. Yoshua Bengio, Guillaume Alain, and Salah Rifai. Implicit density estimation by local moment matching to sample from auto-encoders. Technical report, arXiv:1207.0057, 2012a.
5. Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. Technical report, arXiv:1206.5538, 2012b.
6. Yoshua Bengio, Yao Li, Guillaume Alain, and Pascal Vincent. Generalized denoising auto-encoders as generative models. Technical Report arXiv:1305.6663, Universite de Montreal, 2013a.
7. Yoshua Bengio, Grégoire Mesnil, Yann Dauphin, and Salah Rifai. Better mixing via deep representations. In ICML’13, 2013b.
8. Lawrence Cayton. Algorithms for manifold learning. Technical Report CS2008-0923, UCSD, 2005. B. Dacorogna. Introduction to the Calculus of Variations. World Scientific Publishing Company, 2004.
9. Karol Gregor, Arthur Szlam, and Yann LeCun. Structured sparse coding via lateral inhibition. In Advances in Neural Information Processing Systems (NIPS 2011), volume 24, 2011.
10. Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527–1554, 2006.
11. Aapo Hyvärinen. Estimation of non-normalized statistical models using score matching. Journal of Machine Learning Research, 6:695–709, 2005.
12. Aapo Hyvärinen. Some extensions of score matching. Computational Statistics and Data Analysis, 51: 2499–2512, 2007.
13. Viren Jain and Sebastian H. Seung. Natural image denoising with convolutional networks. In NIPS’08, pages 769–776, 2008.
14. Koray Kavukcuoglu, MarcʹAurelio Ranzato, Rob Fergus, and Yann LeCun. Learning invariant features through topographic filter maps. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR’09), pages 1605–1612. IEEE, 2009.
15. Diederik Kingma and Yann LeCun. Regularized estimation of image statistics by score matching. In J. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R.S. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems 23, pages 1126–1134, 2010.
16. Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Léon Bottou and Michael Littman, editors, ICML 2009. ACM, Montreal (Qc), Canada, 2009.
17. Hariharan Narayanan and Sanjoy Mitter. Sample complexity of testing the manifold hypothesis. In NIPS’2010. 2010.
18. B. A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: a strategy employed by V1? Vision Research, 37:3311–3325, December 1997. URL http://view.ncbi.nlm.nih.gov/pubmed/9425546.
19. MarcʹAurelio Ranzato, Christopher Poultney, Sumit Chopra, and Yann LeCun. Efficient learning of sparse representations with an energy-based model. In NIPS’06, pages 1137–1144. MIT Press, 2007.
20. MarcʹAurelio Ranzato, Y-Lan Boureau, and Yann LeCun. Sparse feature learning for deep belief networks. In NIPS’07, pages 1185–1192, Cambridge, MA, 2008. MIT Press.
21. Salah Rifai, Yann Dauphin, Pascal Vincent, Yoshua Bengio, and Xavier Muller. The manifold tangent classifier. In NIPS’2011, 2011a. Student paper award.
22. Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive autoencoders: Explicit invariance during feature extraction. In ICML’11, 2011b.
23. Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive autoencoders: Explicit invariance during feature extraction. In ICML’2011, 2011c.
24. Salah Rifai, Yoshua Bengio, Yann Dauphin, and Pascal Vincent. A generative process for sampling contractive auto-encoders. In ICML’12, 2012.
25. R. Salakhutdinov and G.E. Hinton. Deep Boltzmann machines. In Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics (AISTATS 2009), volume 8, 2009.
26. Kevin Swersky, MarcʹAurelio Ranzato, David Buchman, Benjamin Marlin, and Nando de Freitas. On autoencoders and score matching for energy based models. In ICML’2011. ACM, 2011.
27. Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661–1674, 2011.
28. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML’08, pages 1096–1103. ACM, 2008.