Deep belief networks are compact universal approximators

Neural Computation

Volumn 22, Issue 8, 2010, Pages 2192-2207

  Le Roux, Nicolas ^a   Bengio, Yoshua ^b  

^a MICROSOFT RESEARCH   (United Kingdom)

^b UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77955997114     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/neco.2010.08-09-1081     Document Type: Letter

References (15)

1. Bengio, Y. (2009). Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2, 1-127.
2. Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007). Greedy layer-wise training of deep networks. In B. Schölkopf, J. Platt, & T. Hoffman (Eds.), Advances in neural information processing systems, 19 (pp. 153-160). Cambridge, MA: MIT Press.
3. Freund, Y., & Haussler, D. (1991). Unsupervised learning of distributions of binary vectors using 2-layer networks. In J. Moody, S. J. Hanson, & R. Lippmann (Eds.), Advances in neural information processing systems, 4. San Francisco: Morgan Kaufmann.
4. Gray, F. (1953). Pulse code communication. U.S. Patent 2,632,058.
5. Hastad, J., & Goldmann, M. (1991). On the power of small-depth threshold circuits. Computational Complexity, 1, 113-129.
6. Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation, 14, 1771-1800.
7. Hinton, G. E., Osindero, S., & Teh, Y. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527-1554.
8. Le Roux, N., & Bengio, Y. (2008). Representational power of restricted Boltzmann machines and deep belief networks. Neural Computation, 20(6), 1631-1649.
9. Neal, R. (1992). Connectionist learning of belief networks. Artificial Intelligence, 56, 71-113.
10. Ranzato, M., Poultney, C., Chopra, S., & LeCun, Y. (2007). Efficient learning of sparse representations with an energy-based model. In B. Schölkopf, J. Platt, & T. Hoffman (Eds.), Advances in neural information processing systems, 19. Cambridge, MA: MIT Press.
11. Rojas, R. (2003). Networks of width one are universal classifiers. In International Joint Conference on Neural Networks (Vol. 4, pp. 3124-3127). New York: Elsevier.
12. Smolensky, P. (1986). Information processing in dynamical systems: Foundations of harmony theory. In D. E. Rumelhart & J. L. McClelland (Eds), Parallel distributed processing (Vol. 1, pp. 194-281). Cambridge, MA: MIT Press.
13. Sutskever, I., & Hinton, G. E. (2008). Deep, narrow sigmoid belief networks are universal approximators. Neural Computation, 20(11), 2629-2636.
14. Tieleman, T. (2008). Training restricted Boltzmann machines using approximations to the likelihood gradient. In Proceedings of the International Conference on Machine Learning (Vol. 25). Madison, WI: Omnipress.
15. Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008). Extracting and composing robust features with denoising autoencoders. In Proceedings of the Twenty-Fifth International Conference on Machine Learning (ICML'2008). Madison, WI: Omnipress.