Justifying and generalizing contrastive divergence

Neural Computation

Volumn 21, Issue 6, 2009, Pages 1601-1621

  Bengio, Yoshua ^a   Delalleau, Olivier ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; EPIDEMIOLOGY; HUMAN; LEARNING; PHYSIOLOGY; STATISTICAL MODEL;

BIAS (EPIDEMIOLOGY); HUMANS; LEARNING; LIKELIHOOD FUNCTIONS; MODELS, STATISTICAL;

EID: 67651049775     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/neco.2008.11-07-647     Document Type: Letter

References (26)

1. Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007). Greedy layer-wise training of deep networks. In B. Schölkopf, J. Platt, & T. Hoffman (Eds.), Advances in neural information processing systems, 19 (pp. 153-160). Cambridge, MA: MIT Press.
2. Bengio, Y., & Le Cun, Y. (2007). Scaling learning algorithms towards AI. In L. Bottou, O. Chapelle, D. DeCoste, & J. Weston (Eds.), Large scale kernel machines. Cambridge, MA: MIT Press.
3. Bourlard, H., & Kamp, Y. (1988). Auto-association by multilayer perceptrons and singular value decomposition. Biological Cybernetics, 59, 291-294.
4. Carreira-Perpiñan, M. A., & Hinton, G. E. (2005). On contrastive divergence learning. In R. G. Cowell, & Z. Ghahramani (Eds.), Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics (pp. 33-40). N. P.: Society for Artificial Intelligence and Statistics.
5. Freund, Y., & Haussler, D. (1994). Unsupervised learning of distributions on binary vectors using two layer networks (Tech. Rep. UCSC-CRL-94-25). Santa Cruz: University of California, Santa Cruz.
6. Hernández-Lerma, O., & Lasserre, J. B. (2003). Markov chains and invariant probabilities. Basel: Birkhäuser Verlag.
7. Hinton, G. E. (1999). Products of experts. In Proceedings of the Ninth International Conference on Artificial Neural Networks (ICANN) (Vol. 1, pp. 1-6). New York: IEEE.
8. Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation, 14, 1771-1800.
9. Hinton, G. E., Osindero, S., & Teh, Y. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527-1554.
10. Hinton, G. E., & Salakhutdinov, R. (2006). Reducing the dimensionality of data with neural networks. Science, 313, 504-507.
11. Hinton, G. E., & Sejnowski, T. J. (1986). Learning and relearning in Boltzmann machines. In D. E. Rumelhart & J. L. McClelland (Eds.), Parallel distributed processing: Explorations in the microstructure of cognition. Vol. 1: Foundations. Cambridge, MA: MIT Press.
12. Hinton, G. E., Sejnowski, T. J., & Ackley, D. H. (1984). Boltzmann machines: Constraint satisfaction networks that learn (Tech. Rep. TR-CMU-CS-84-119). Pittsburgh, PA: Carnegie Mellon University, Department of Computer Science.
13. Hinton, G. E., & Zemel, R. S. (1994). Autoencoders, minimum description length, and Helmholtz free energy. In D. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems, 6 (pp. 3-10). San Francisco: Morgan Kaufmann.
14. Japkowicz, N., Hanson, S. J., & Gluck, M. A. (2000). Nonlinear autoassociation is not equivalent to PCA. Neural Computation, 12(3), 531-545.
15. Larochelle, H., Erhan, D., Courville, A., Bergstra, J., & Bengio, Y. (2007). An empirical evaluation of deep architectures on problems with many factors of variation. In Z. Ghahramani (Ed.), Twenty-Fourth International Conference on Machine Learning (ICML'2007) (pp. 473-480). Madison, WI: Omnipress.
16. MacKay, D. (2001). Failures of the one-step learning algorithm. Unpublished manuscript.
17. Ranzato, M., Poultney, C., Chopra, S., & Le Cun, Y. (2007). Efficient learning of sparse representations with an energy-based model. In B. Schölkopf, J. Platt, & T. Hoffman (Eds.), Advances in neural information processing systems, 19.Cambridge, MA: MIT Press.
18. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533-536.
19. Salakhutdinov, R., & Hinton, G. (2007). Semantic hashing. In A. McCallum & S. Roweis (Eds.), Proceedings of the 2007 Workshop on Information Retrieval and Applications of Graphical Models (SIGIR 2007). Amsterdam: Elsevier.
20. Schmidt, V. (2006). Markov chains and Monte-Carlo simulation. In Lecture Notes, Summer 2006. Ulm: Ulm University, Department of Stochastics.
21. Schwenk, H., & Milgram, M. (1995). Transformation invariant autoassociation with application to handwritten character recognition. In G. Tesauro, D. Touretzky, & T. Leen (Eds.), Advances in neural information processing systems, 7 (pp. 991-998). Cambridge, MA: MIT Press.
22. Smolensky, P. (1986). Information processing in dynamical systems: Foundations of harmony theory. In D. E. Rumelhart & J. L. McClelland (Eds.), Parallel distributed processing (Vol. 1, pp. 194-281). Cambridge, MA: MIT Press.
23. Taylor, G., Hinton, G., & Roweis, S. (2007). Modeling human motion using binary latent variables. In B. Schölkopf, J. Platt, & T. Hoffman (Eds.), Advances in neural information processing systems, 19 (pp. 1345-1352). Cambridge, MA: MIT Press.
24. Tieleman, T. (2008). Training restricted Boltzmann machines using approximations to the likelihood gradient. In A. McCallum & S. Roweis (Eds.), Proceedings of the International Conference on Machine Learning (Vol. 25, pp. 1064-1071). Madison, WI: Omnipress.
25. Welling, M., Rosen-Zvi, M., & Hinton, G. E. (2005). Exponential family harmoniums with an application to information retrieval. In L. Saul, Y. Weiss, & L. Bottou (Eds.), Advances in neural information processing systems, 17. Cambridge, MA: MIT Press.
26. Yuille, A. L. (2005). The convergence of contrastive divergences. In L. Saul, Y. Weiss, & L. Bottou (Eds.), Advances in neural information processing systems, 17 (pp. 1593-1600). Cambridge, MA: MIT Press.