Expressive power and approximation errors of Restricted Boltzmann Machines

Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011, NIPS 2011

Volumn , Issue , 2011, Pages

  Montúfar, Guido F ^a   Rauh, Johannes ^a   Ay, Nihat ^a,b  

^a MAX PLANCK INSTITUTE FOR MATHEMATICS IN THE SCIENCES   (Germany)

^b SANTA FE INSTITUTE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

APPROXIMATION ERRORS; CLASS OF DISTRIBUTIONS; DIFFERENT CLASS; EXPRESSIVE POWER; HIDDEN UNITS; KULLBACK LEIBLER DIVERGENCE; MACHINE MODELLING; POWER ERRORS; PROBABILITY: DISTRIBUTIONS; RESTRICTED BOLTZMANN MACHINE;

PROBABILITY DISTRIBUTIONS;

EID: 85162368944     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (26)

1. N. Ay and A. Knauf. Maximizing multi-information. Kybernetika, 42:517-538, 2006.
2. N. Ay, G. Montúfar, and J. Rauh. Selection criteria for neuromanifolds of stochastic dynamics. International Conference on Cognitive Neurodynamics, 2011.
3. N. Ay and T. Wennekers. Dynamical properties of strongly interacting Markov chains. Neural Networks, 16:1483-1497, 2003.
4. Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle. Greedy layer-wise training of deep networks. NIPS, 2007.
5. L. Brown. Fundamentals of Statistical Exponential Families: With Applications in Statistical Decision Theory. Inst. Math. Statist., Hayworth, CA, USA, 1986.
6. M. A. Carreira-Perpiñan and G. E. Hinton. On contrastive divergence learning. In Proceedings of the 10-th International Workshop on Artificial Intelligence and Statistics, 2005.
7. T. M. Cover and J. A. Thomas. Elements of Information Theory. John Wiley & Sons, 2006.
8. M. A. Cueto, J. Morton, and B. Sturmfels. Geometry of the Restricted Boltzmann Machine. In M. A. G. Viana and H. P. Wynn, editors, Algebraic methods in statistics and probability II, AMS Special Session. AMS, 2010.
9. P. Diaconis and D. Freedman. Finite exchangeable sequences. Ann. Probab., 8:745-764, 1980.
10. Y. Freund and D. Haussler. Unsupervised learning of distributions on binary vectors using 2-layer networks. NIPS, pages 912-919, 1992.
11. G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Comput., 14:1771-1800, 2002.
12. G. E. Hinton. A practical guide to training Restricted Boltzmann Machines, version 1. Technical report, UTML2010-003, University of Toronto, 2010.
13. G. E. Hinton, S. Osindero, and Y. Teh. A fast learning algorithm for Deep Belief Nets. Neural Comput., 18:1527-1554, 2006.
14. S. Kullback and R. Leibler. On information and sufficiency. Ann. Math. Stat., 22:79-86, 1951.
15. N. Le Roux and Y. Bengio. Representational power of Restricted Boltzmann Machines and Deep Belief Networks. Neural Comput., 20(6):1631-1649, 2008.
16. N. Le Roux and Y. Bengio. Deep Belief Networks are compact universal approximators. Neural Comput., 22:2192-2207, 2010.
17. B. Lindsay. Mixture models: theory, geometry, and applications. Inst. Math. Statist., 1995.
18. P. M. Long and R. A. Servedio. Restricted Boltzmann Machines are hard to approximately evaluate or simulate. In Proceedings of the 27-th ICML, pages 703-710, 2010.
19. G. Montúfar. Mixture decompositions using a decomposition of the sample space. ArXiv 1008.0204, 2010.
20. G. Montúfar. Mixture models and representational power of RBMs, DBNs and DBMs. NIPS Deep Learning and Unsupervised Feature Learning Workshop, 2010.
21. G. Montúfar and N. Ay. Refinements of universal approximation results for Deep Belief Networks and Restricted Boltzmann Machines. Neural Comput., 23(5):1306-1319, 2011.
22. J. Rauh. Finding the maximizers of the information divergence from an exponential family. PhD thesis, Universität Leipzig, 2011.
23. J. Rauh, T. Kahle, and N. Ay. Support sets of exponential families and oriented matroids. Int. J. Approx. Reason., 52(5):613-626, 2011.
24. P. Smolensky. Information processing in dynamical systems: foundations of harmony theory. In Symposium on Parallel and Distributed Processing, 1986.
25. R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning). MIT Press, March 1998.
26. K. G. Zahedi, N. Ay, and R. Der. Higher coordination with less control - a result of infromation maximization in the sensori-motor loop. Adaptive Behavior, 18(3-4):338-355, 2010.