A connection between scorematching and denoising autoencoders

Neural Computation

Volumn 23, Issue 7, 2011, Pages 1661-1674

  Vincent, Pascal ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; COMPUTER SIMULATION; STATISTICAL MODEL;

COMPUTER SIMULATION; MODELS, STATISTICAL;

EID: 79959575293     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/NECO_a_00142     Document Type: Article

References (22)

1. Bengio, Y., Lamblin, P., Popovici,D.,&Larochelle,H. (2007).Greedy layer-wise training of deep networks. In B. Schölkopf, J. Platt, & T. Hoffman (Eds.), Advances in neural information processing systems, 19 (pp. 153-160). Cambridge,MA: MIT Press.
2. Duane, S., Kennedy, A., Pendleton, B., & Roweth, D. (1987). Hybrid Monte Carlo. Phys. Lett. B, 195, 216-222.
3. Erhan, D., Bengio, Y., Courville, A., Manzagol, P.-A., Vincent, P., & Bengio, S. (2010). Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 11, 625-660.
4. Gallinari, P., LeCun, Y., Thiria, S., & Fogelman-Soulie, F. (1987). Memoires associatives distribués. In Proceedings of COGNITIVA 87. Paris: Cesta-Afcet.
5. Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation, 14, 1771-1800.
6. Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527-1554.
7. Hyvärinen, A. (2005). Estimation of non-normalized statistical models using score matching. Journal of Machine Learning Research, 6, 695-709.
8. Hyvärinen, A. (2007a). Connections between score matching, contrastive divergence, and pseudolikelihood for continuous-valued variables. IEEE Transactions on Neural Networks, 18, 1529-1531.
9. Hyvärinen, A. (2007b). Some extensions of score matching. Computational Statistics and Data Analysis, 51, 2499-2512.
10. Hyvärinen, A. (2008). Optimal approximation of signal priors. Neural Computation, 20(12), 3087-3110.
11. Kingma, D., & LeCun, Y. (2010). Regularized estimation of image statistics by score matching. In J. Lafferty, C.K.I. Williams, J. Shawe-Taylor, R. Zemel, & A. Culotta (Eds.), Advances in neural information processing systems, 23 (pp. 1126-1134). Cambridge, MA: MIT Press.
12. LeCun, Y. (1987). Mod'eles connexionistes de l'apprentissage. Unpublished doctoral dissertation, Universit́e de Paris VI.
13. Lyu, S. (2010). Interpretation and generalization of score matching. In Proceedings of the 25th Conference in Uncertainty in Artificial Intelligence (UAI'09). Corvallis, OR: AUAI Press.
14. Marlin, B., Swersky, K., Chen, B., & de Freitas, N. (2010). Inductive principles for restricted Boltzmann machine learning. In Proceedings of The Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS'10) (Vol. 9, pp. 509-516). N.p.: JMLR.
15. Movellan, J. R. (2008). Contrastive divergence in gaussian diffusions. Neural Computation, 20(9), 2238-2252.
16. Seung, S. H. (1998). Learning continuous attractors in recurrent networks. In M. Jordan, M. Kearns, & S. Solla (Eds.), Advances in neural information processing systems, 10 (pp. 654-660). Cambridge, MA: MIT Press.
17. Sohl-Dickstein, J., Battaglino, P., & DeWeese, M. R. (2009). Minimum probability flow learning (Tech. Rep.). arXiv:0906.4779.
18. Swersky, K. (2010). Inductive principles for learning restricted Boltzmann machines. Unpublished master's thesis, University of British Columbia.
19. Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008). Extracting and composing robust features with denoising autoencoders. In W. W. Cohen, A. McCallum, & S. T. Roweis (Eds.), Proceedings of the 25th Annual International Conference on Machine Learning (pp. 1096-1103). New York: ACM.
20. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., & Manzagol, P.-A. (2010). Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 11, 3371-3408.
21. Welling, M., & Hinton, G. E. (2002). A new learning algorithm for mean field Boltzmann machines. In ICANN '02: Proceedings of the International Conference on Artificial Neural Networks (pp. 351-357). Berlin: Springer-Verlag.
22. Welling, M., Rosen-Zvi,M., & Hinton, G. E. (2005). Exponential family harmoniums with an application to information retrieval. In L. Saul, Y. Weiss, & L. Bottou (Eds.), Advances in neural information processing systems, 17 (pp. 1481-1488). Cambridge, MA: MIT Press.