Improving mixing rate with tempered transition for learning restricted Boltzmann machines

Neurocomputing

Volumn 139, Issue , 2014, Pages 328-335

  Xu, Jungang ^a   Li, Hui ^a   Zhou, Shilong ^a  

^a UNIVERSITY OF CHINESE ACADEMY OF SCIENCES   (China)

Author keywords

Deep learning; Mixing rate; Restricted Boltzmann machines; Tempered transition

Indexed keywords

MARKOV PROCESSES; TEMPERING;

CONTRASTIVE DIVERGENCE; DEEP BELIEF NETWORKS; DEEP LEARNING; MARKOV CHAIN MONTE CARLO METHOD; MARKOV RANDOM FIELDS; MIXING RATES; RESTRICTED BOLTZMANN MACHINE; TEMPERED TRANSITION;

MIXING;

ARTICLE; ARTIFICIAL NEURAL NETWORK; CONTRASTIVE DIVERGENCE LEARNING ALGORITHM; EXPERIMENTAL STUDY; INTERMETHOD COMPARISON; LEARNING ALGORITHM; MACHINE LEARNING; MATHEMATICAL COMPUTING; MONTE CARLO METHOD; PRIORITY JOURNAL; PROBABILITY; RESTRICTED BOLTZMANN MACHINE; SAMPLING; STATISTICAL MODEL;

EID: 84901064273     PISSN: 09252312     EISSN: 18728286     Source Type: Journal    
DOI: 10.1016/j.neucom.2014.02.024     Document Type: Article

References (32)

1. Hinton G.E., Salakhutdinov R.R. Reducing the dimensionality of data with neural networks. Science 2006, 313(5786):504-507.
2. Hinton G.E., Osindero S., Teh Y.W. A fast learning algorithm for deep belief nets. Neural Comput. 2006, 18(7):1527-1554.
3. R. Salakhutdinov, G. Hinton, Deep Boltzmann machines, in: Proceedings of the 12th International Conference on Artificial Intelligence and Statistics (AISTATS'09), 2009, pp. 448-455.
4. A. Krizhevsky, G.E. Hinton, Using very deep autoencoders for content-based image retrieval, in: Proceedings of the 19th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN'11), 2011.
5. A. Krizhevsky, Learning Multiple Layers of Features from Tiny Images, Master[U+05F3]s Thesis of Department of Computer Science, University of Toronto, 2009.
6. Hinton G.E., Deng L., Yu D. Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 2012, 29(6):82-97.
7. Mohamed A., Dahl G., Hinton G. Acoustic modeling using deep belief networks. IEEE Trans. Audio Speech Lang. Process. 2011, 20(1):14-22.
8. Salakhutdinov R., Hinton G.E. Semantic hashing. Int. J. Approximate Reasoning 2009, 50(7):969-978.
9. Smolensky P. Information processing in dynamical systems: foundations of harmony theory. Parallel Distrib. Process.: Explorations Microstruct. Cognit. 1986, 1:194-281.
10. Y. Freund, D. Haussler, Unsupervised Learning of Distributions on Binary Vectors Using Two Layer Networks, Technical Report of University of California Santa Cruz, 1994.
11. Hinton G.E. Training products of experts by minimizing contrastive divergence. Neural Comput. 2002, 14(8):1771-1800.
12. M. Welling, M. Rosen-Zvi, G.E. Hinton, Exponential family harmoniums with an application to information retrieval, in: Proceedings of the 2005 Conference on Neural Information Processing Systems (NIPS'05), 2005, pp. 1481-1488.
13. Erhan D., Bengio Y., Courville A., Manzagol P.A., Vincent P., Bengio S. Why does unsupervised pre-training help deep learning?. J. Mach. Learn. Res. 2010, 11:625-660.
14. Y. Bengio, O. Delalleau, On the expressive power of deep architectures, in: Proceedings of 22nd International Conference on Algorithmic Learning Theory (ALT'11), 2011, pp. 18-36.
15. Y. Bengio, G. Mesnil, Y. Dauphin, S. Rifai, Better Mixing Via Deep Representations, Technical Report arXiv:1207.4404, 2012.
16. Le Roux N., Bengio Y. Representational power of restricted Boltzmann machines and deep belief networks. Neural Comput. 2008, 20(6):1631-1649.
17. A. Fischer, C. Igel, Empirical analysis of the divergence of Gibbs sampling based learning algorithms for restricted Boltzmann machines, in: Proceedings of the 20th International Conference on Artificial Neural Networks (ICANN'10): Part III, 2010, pp. 208-217.
18. Neal R.M. Sampling from multimodal distributions using tempered transitions. Stat. Comput. 1996, 6(4):353-366.
19. Iba Y. Extended ensemble Monte Carlo. Int. J. Mod. Phys. C 2001, 12(5):623-656.
20. G.E. Hinton, Products of experts, in: Proceedings of the Ninth International Conference on Artificial Neural Networks (ICANN'99), 1999, pp. 1-6.
21. Hinton G.E. Learning multiple layers of representation. Trends Cogn. Sci. 2007, 11(10):428-434.
22. Bengio Y., Delalleau O. Justifying and generalizing contrastive divergence. Neural Comput. 2009, 21(6):1601-1621.
23. T. Tieleman, Training restricted Boltzmann machines using approximations to the likelihood gradient, in: Proceedings of the 25th International Conference on Machine Learning (ICML'08), 2008, pp. 1064-1071.
24. T. Tieleman, G. Hinton, Using fast weights to improve persistent contrastive divergence, in: Proceedings of the 26th Annual International Conference on Machine Learning (ICML'09), 2009, pp. 1033-1040.
25. A. Fischer, C. Igel, An introduction to restricted Boltzmann machines, in: Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, LNCS 7441, 2012, pp. 14-36.
26. Earl D.J., Deem M.W. Parallel tempering: theory, applications, and new perspectives. Phys. Chem. Chem. Phys. 2005, 7:3910-3916.
27. G. Desjardins, A. Courville, Y. Bengio, Parallel tempering for training of restricted Boltzmann machines, in: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS'10), 2010, pp. 145-152.
28. K.H. Cho, T. Raiko, A. Ilin, Parallel tempering is efficient for learning restricted Boltzmann machines, in: Proceedings of the 2010 International Joint Conference on Neural Networks (IJCNN'10), 2010, pp. 1-8.
29. R. Salakhutdinov, Learning in Markov random fields using tempered transitions, in: Proceedings of the 2009 Conference on Neural Information Processing Systems (NIPS'09), 2009, pp. 1598-1606.
30. Y. LeCun, C. Cortes, C. Burges, The MNIST Database of Handwritten Digits,. http://yann.lecun.com/exdb/mnist/.
31. Rumelhart D.E., Hinton G.E., Williams R.J. Learning internal representations by error propagation. Parallel Distrib. Process.: Explorations Microstruct. Cognit. 1986, 1:318-362.
32. G. Hinton, A Practical Guide to Training Restricted Boltzmann Machines, Neural Networks: Tricks of the Trade, LNCS 7700, 2012. pp. 599-619.