On autoencoders and score matching for energy based models

Proceedings of the 28th International Conference on Machine Learning, ICML 2011

Volumn , Issue , 2011, Pages 1201-1208

  Swersky, Kevin ^a   Ranzato, Marc'Aurelio ^b   Buchman, David ^a   Marlin, Benjamin M ^a   De Freitas, Nando ^a  

^a UNIVERSITY OF BRITISH COLUMBIA   (Canada)

^b UNIVERSITY OF TORONTO   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

AUTOENCODERS; CLASSIFICATION PERFORMANCE; CLASSIFICATION RESULTS; CONDITIONAL DISTRIBUTION; ENERGY-BASED MODELS; ESTIMATION METHODS; GAUSSIANS; IMAGE DE-NOISING; MAXIMUM LIKELIHOOD ESTIMATOR; REGULARIZATION FUNCTION; STOCHASTIC APPROXIMATIONS; TRAINING METHODS;

APPROXIMATION THEORY; FEATURE EXTRACTION; GAUSSIAN DISTRIBUTION; IMAGE PROCESSING; LEARNING SYSTEMS; MAXIMUM LIKELIHOOD ESTIMATION;

ESTIMATION;

EID: 80053450846     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (25)

1. Bengio, Y. Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2(1):1-127, 2009.
2. Hinton, G.E. Training products of experts by minimizing contrastive divergence. Neural Computation, 14:1771-1800, 2002.
3. Hinton, G.E. and Zemel, R.S. Autoencoders, minimum description length and Helmholtz free energy. In Advances in Neural Information Processing Systems, pp. 3-10, 1994.
4. Hinton, G.E., Osindero, S., and Teh, Y.W. A fast learning algorithm for deep belief nets. Neural Computation, 18(7):1527-1554, 2006. (Pubitemid 44024729)
5. Hyvärinen, A. Estimation of non-normalized statistical models using score matching. Journal of Machine Learning Research, 6:695-709, 2005.
6. Kingma, D. and LeCun, Y. Regularized estimation of image statistics by score matching. In Advances in Neural Information Processing Systems, 2010.
7. Krizhevsky, A. Learning multiple layers of features from tiny images, 2009. MSc Thesis, Dept. of Comp. Science, Univ. of Toronto.
8. LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., and Huang, F.J. A tutorial on energy-based learning. In Predicting Structured Data. MIT Press, 2006.
9. Marlin, B.M., Swersky, K., Chen, B., and De Freitas, N. Inductive principles for restricted Boltzmann machine learning. In Artificial Intelligence and Statistics, pp. 509-516, 2010.
10. Memisevic, R. and Hinton, G.E. Learning to represent spatial transformations with factored higher-order Boltzmann machines. Neural Computation, 22:1473-1492, 2009.
11. Ranzato, M. and Hinton, G.E. Modeling pixel means and covariances using factorized third-order Boltzmann machines. In IEEE Computer Vision and Pattern Recognition, pp. 2551-2558, 2010.
12. Ranzato, M., Boureau, Y.L., Chopra, S., and LeCun, Y. A unified energy-based framework for unsupervised learning. In Artificial Intelligence and Statistics, 2007.
13. Ranzato, M., Krizhevsky, A., and Hinton, G.E. Factored 3-way restricted Boltzmann machines for modeling natural images. In Artificial Intelligence and Statistics, pp. 621-628, 2010a.
14. Ranzato, M., Mnih, V., and Hinton, G.E. How to generate realistic images using gated MRF's. In Advances in Neural Information Processing Systems, pp. 2002-2010, 2010b.
15. Schmidt, U., Gao, Q., and Roth, S. A generative perspective on MRFs in low-level vision. In IEEE Computer Vision and Pattern Recognition, 2010.
16. Swersky, K., Chen, B., Marlin, B.M., and De Freitas, N. A tutorial on stochastic approximation algorithms for training restricted Boltzmann machines and deep belief nets. In Information Theory and Applications Workshop, pp. 1-10, 2010.
17. Teh, Y.W., Welling, M., Osindero, S., and Hinton, G.E. Energy-based models for sparse overcomplete representations. Journal of Machine Learning Research, 4:1235-1260, 2003.
18. Tieleman, T. Training restricted Boltzmann machines using approximations to the likelihood gradient. In International Conference on Machine Learning, pp. 1064-1071, 2008.
19. Tieleman, T. and Hinton, G.E. Using fast weights to improve persistent contrastive divergence. In International Conference on Machine Learning, 2009.
20. Torralba, A., Fergus, R., and Freeman, W.T. 80 million tiny images: A large dataset for non-parametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30:1958-1970, 2008.
21. Vincent, P. A connection between score matching and denoising autoencoders. Neural Computation, To appear, 2011.
22. Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.A. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning, pp. 1096-1103, 2008.
23. Welling, M., Hinton, G.E., and Osindero, S. Learning sparse topographic representations with products of student-t distributions. In Advances in Neural Information Processing Systems, 2003.
24. Welling, M., Rosen-Zvi, M., and Hinton, G.E. Exponential family harmoniums with an application to information retrieval. In Advances in Neural Information Processing Systems, 2005.
25. Younes, L. Parametric inference for imperfectly observed Gibbsian fields. Probability Theory and Related Fields, 82(4):625-645, 1989.