Training restricted Boltzmann machines: An introduction

Pattern Recognition

Volumn 47, Issue 1, 2014, Pages 25-39

  Fischer, Asja ^a,b   Igel, Christian ^b  

^a RUHR UNIVERSITY BOCHUM   (Germany)

^b UNIVERSITY OF COPENHAGEN   (Denmark)

Author keywords

Contrastive divergence learning; Gibbs sampling; Markov chains; Markov random fields; Neural networks; Parallel tempering; Restricted Boltzmann machines

Indexed keywords

CONTRASTIVE DIVERGENCE; GIBBS SAMPLING; MARKOV RANDOM FIELDS; PARALLEL TEMPERING; RESTRICTED BOLTZMANN MACHINE;

LEARNING ALGORITHMS; NEURAL NETWORKS; PATTERN RECOGNITION; TEMPERING;

MARKOV PROCESSES;

EID: 84885023116     PISSN: 00313203     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patcog.2013.05.025     Document Type: Article

References (62)

1. D.H. Ackley, G.E. Hinton, and T.J. Sejnowski A learning algorithm for Boltzmann machines Cognitive Science 9 1985 147 169
2. Y. Bengio Learning deep architectures for AI Foundations and Trends in Machine Learning 21 6 2009 1601 1621
3. Y. Bengio, and O. Delalleau Justifying and generalizing contrastive divergence Neural Computation 21 6 2009 1601 1621
4. Y. Bengio, P. Lamblin, D. Popovici, H. Larochelle, and U. Montreal Greedy layer-wise training of deep networks B. Schölkopf, J. Platt, T. Hoffman, Advances in Neural Information Processing (NIPS 19) 2007 MIT Press 153 160
5. C.M. Bishop Pattern Recognition and Machine Learning 2006 Springer
6. P. Brakel, S. Dieleman, and B. Schrauwen Training restricted Boltzmann machines with multi-tempering: harnessing parallelization M. Verleysen, European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN) 2012 D-side publications Evere, Belgium 287 292
7. P. Brémaud Markov Chains Gibbs Fields, Monte Carlo Simulation, and Queues 1999 Springer-Verlag
8. K. Brügge, A. Fischer, C. Igel, The flip-the-state transition operator for restricted Boltzmann machines. Machine Learning, 2013. 10.1007/s10994-013-5390-3
9. M.Á. Carreira-Perpi nán, and G.E. Hinton On contrastive divergence learning R.G. Cowell, Z. Ghahramani, 10th International Workshop on Artificial Intelligence and Statistics (AISTATS) 2005 The Society for Artificial Intelligence and Statistics 59 66
10. K. Cho, T. Raiko, A. Ilin, Parallel tempering is efficient for learning restricted Boltzmann machines, in: Proceedings of the International Joint Conference on Neural Networks (IJCNN), IEEE Press, 2010, pp. 3246-3253.
11. G. Desjardins, A. Courville, and Y. Bengio Adaptive parallel tempering for stochastic maximum likelihood learning of RBMs H. Lee, M. Ranzato, Y. Bengio, G.E. Hinton, Y. LeCun, A.Y. Ng, NIPS 2010 Workshop on Deep Learning and Unsupervised Feature Learning 2010
12. G. Desjardins, A. Courville, Y. Bengio, P. Vincent, and O. Dellaleau Parallel tempering for training of restricted Boltzmann machines JMLR W &CP AISTATS 2010 9 2010 145 152
13. A. Fischer, and C. Igel Empirical analysis of the divergence of Gibbs sampling based learning algorithms for restricted Boltzmann machines K. Diamantaras, W. Duch, L.S. Iliadis, International Conference on Artificial Neural Networks ICANN, LNCS vol. 6354 2010 Springer-Verlag 208 217
14. A. Fischer, and C. Igel Bounding the bias of contrastive divergence learning Neural Computation 23 2011 664 673
15. A. Fischer, C. Igel, Parallel tempering, importance sampling, and restricted Boltzmann machines, in: 5th Workshop on Theory of Randomized Search Heuristics (ThRaSH 2011), 2011 (Online abstract).
16. A. Fischer, and C. Igel An introduction to restricted Boltzmann machines L. Alvarez, M. Mejaill, L. Gomez, J. Jacobo, Proceedings of the 17th Iberoamerican Congress on Pattern Recognition CIARP LNCS vol. 7441 2012 Springer 14 36
17. P.V. Gehler, A.D. Holub, and M. Welling The rate adapting poisson model for information retrieval and object recognition W. Cohen, A. Moore, Proceedings of 23rd International Conference on Machine Learning ICML 2006 ACM 337 344
18. D. Geman, and S. Geman Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images IEEE Transactions on Pattern Analysis and Machine Intelligence 6 6 1984 721 741 (Pubitemid 15453722)
19. W.K. Hastings Monte Carlo sampling methods using Markov chains and their applications Biometrika 57 1 1970 97 109
20. G. Hinton, A Practical Guide to Training Restricted Boltzmann Machines, Technical Report UTML TR 2010003, Department of Computer Science, University of Toronto, 2010.
21. G.E. Hinton Training products of experts by minimizing contrastive divergence Neural Computation 14 2002 1771 1800
22. G.E. Hinton Boltzmann machine Scholarpedia 2 5 2007 1668
23. G.E. Hinton Learning multiple layers of representation Trends in Cognitive Sciences 11 10 2007 428 434 (Pubitemid 47588994)
24. G.E. Hinton, S. Osindero, and Y.-W. Teh A fast learning algorithm for deep belief nets Neural Computation 18 7 2006 1527 1554 (Pubitemid 44024729)
25. G.E. Hinton, and R.R. Salakhutdinov Reducing the dimensionality of data with neural networks Science 313 5786 2006 504 507 (Pubitemid 44148451)
26. G.E. Hinton, and T.J. Sejnowski Learning and relearning in Boltzmann machines D.E. Rumelhart, J.L. McClelland, Parallel Distributed Processing Explorations in the Microstructure of Cognition, Foundations vol. 1 1986 MIT Press 282 317
27. C. Igel, T. Glasmachers, and V. Heidrich-Meisner Shark Journal of Machine Learning Research 9 2008 993 996
28. J. Kivinen, and C. Williams Multiple texture Boltzmann machines JMLR W&CP AISTATS 22 2012 638 646
29. D. Koller, and N. Friedman Probabilistic Graphical Models Principles and Techniques 2009 MIT Press
30. O. Krause, A. Fischer, T. Glasmachers, and C. Igel Approximation properties of DBNs with binary hidden units and real-valued visible units JMLR W&CP ICML 2013 28 1 2013 419 426
31. H. Larochelle, and Y. Bengio Classification using discriminative restricted Boltzmann machines W.W. Cohen, A. McCallum, S.T. Roweis, International Conference on Machine Learning (ICML) 2008 ACM 536 543
32. S.L. Lauritzen Graphical Models 1996 Oxford University Press
33. N. Le Roux, and Y. Bengio Representational power of restricted Boltzmann machines and deep belief networks Neural Computation 20 6 2008 1631 1649
34. N. Le Roux, N. Heess, J. Shotton, and J.M. Winn Learning a generative model of images by factoring appearance and shape Neural Computation 23 3 2011 593 650
35. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner Gradient-based learning applied to document recognition Proceedings of the IEEE 86 11 1998 2278 2324
36. N.M. Lingenheil, R. Denschlag, G. Mathias, and P. Tavan Efficiency of exchange schemes in replica exchange Chemical Physics Letters 478 2009 80 84
37. D.J.C. MacKay, Failures of the One-step Learning Algorithm, Cavendish Laboratory, Madingley Road, Cambridge CB3 0HE, UK, 2001. http://www.cs.toronto. edu/~mackay/gbm.pdf .
38. D.J.C. MacKay Information Theory, Inference & Learning Algorithms 2002 Cambridge University Press
39. V. Mnih, H. Larochelle, and G.E. Hinton Conditional restricted Boltzmann machines for structured output prediction F.G. Cozman, A. Pfeffer, Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence UAI 2011 AUAI Press 514
40. A. Mohamed, G.E. Hinton, Phone recognition using restricted Boltzmann machines, in: IEEE International Conference on Acoustics Speech and Signal Processing ICASSP, IEEE Press, 2010, pp. 4354-4357.
41. G. Montufar, and N. Ay Refinements of universal approximation results for deep belief networks and restricted Boltzmann machines Neural Computation 23 5 2011 1306 1319
42. R.M. Neal, Probabilistic Inference using Markov Chain Monte Carlo Methods. Technical Report CRG-TR-93-1, Department of Computer Science, University of Toronto, 1993.
43. D.E. Rumelhart, G.E. Hinton, and R.J. Williams Learning internal representations by error propagation D.E. Rumelhart, J.L. McClelland, Parallel Distributed Processing Explorations in the Microstructure of Cognition, Foundations vol. 1 1986 MIT Press 318 362
44. R. Salakhutdinov Learning in Markov random fields using tempered transitions Y. Bengio, D. Schuurmans, J. Lafferty, C.K.I. Williams, A. Culotta, Advances in Neural Information Processing Systems, NIPS 22 2009 1598 1606
45. R. Salakhutdinov, and G.E. Hinton Deep Boltzmann machines JMLR W&CP AISTATS 2009 5 2009 448 455
46. R. Salakhutdinov, and G.E. Hinton Replicated softmax an undirected topic model Y. Bengio, D. Schuurmans, J. Lafferty, C.K.I. Williams, A. Culotta, Advances in Neural Information Processing Systems, NIPS 22 2009 1607 1614
47. R. Salakhutdinov, A. Mnih, and G.E. Hinton Restricted Boltzmann machines for collaborative filtering Z. Ghahramani, Proceedings of the 24th International Conference on Machine Learning (ICML) 2007 ACM 791 798
48. T. Schmah, G.E. Hinton, R.S. Zemel, S.L. Small, and S.C. Strother Generative versus discriminative training of RBMs for classification of fMRI images D. Koller, D. Schuurmans, Y. Bengio, L. Bottou, Advances in Neural Information Processing Systems, NIPS 21 2009 1409 1416
49. P. Smolensky Information processing in dynamical systems foundations of harmony theory D.E. Rumelhart, J.L. McClelland, Parallel Distributed Processing Explorations in the Microstructure of Cognition, Foundations vol. 1 1986 MIT Press 194 281
50. K. Swersky, B. Chen, B. Marlin, N. de Freitas, A tutorial on stochastic approximation algorithms for training restricted Boltzmann machines and deep belief nets, in: Information Theory and Applications Workshop (ITA), 2010, IEEE, 2010, pp. 1-10.
51. Y. Tang, R. Salakhutdinov, G.E. Hinton, Robust Boltzmann machines for recognition and denoising, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2012, pp. 2264-2271.
52. G.W. Taylor, E.H. G, and S. Roweis Modeling human motion using binary latent variables B. Schölkopf, J. Platt, T. Hoffman, Advances in Neural Information Processing Systems, NIPS 19 2007 MIT Press 1345 1352
53. G.W. Taylor, and G.E. Hinton Factored conditional restricted Boltzmann machines for modeling motion style A.P. Danyluk, L. Bottou, M.L. Littman, Proceedings of the 26th Annual International Conference on Machine Learning (ICML) 2009 ACM 1025 1032
54. G.W. Taylor, G.E. Hinton, and S.T. Roweis Modeling human motion using binary latent variables B. Schölkopf, J. Platt, T. Hoffman, Advances in Neural Information Processing Systems, NIPS 19 2007 MIT Press 1345 1352
55. T. Tieleman Training restricted Boltzmann machines using approximations to the likelihood gradient W.W. Cohen, A. McCallum, S.T. Roweis, International Conference on Machine learning (ICML) 2008 ACM 1064 1071
56. T. Tieleman, and G.E. Hinton Using fast weights to improve persistent contrastive divergence A. Pohoreckyj Danyluk, L. Bottou, M.L. Littman, International Conference on Machine Learning (ICML) 2009 ACM 1033 1040
57. N. Wang, J. Melchior, and L. Wiskott An analysis of Gaussian-binary restricted Boltzmann machines for natural images M. Verleysen, European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN) 2012 D-side publications Evere, Belgium 287 292
58. M. Welling Product of experts Scholarpedia 2 10 2007 3879
59. M. Welling, M. Rosen-Zvi, and G. Hinton Exponential family harmoniums with an application to information retrieval L.K. Saul, Y. Weiss, L. Bottou, Advances in Neural Information Processing Systems, NIPS 17 2005 MIT Press 1481 1488
60. E.P. Xing, R. Yan, A.G. Hauptmann, Mining associated text and images with dual-wing harmoniums, in: Proceedings of the 21st Conference on Uncertainty in Artificial Intelligence (UAI), AUAI Press, 2005.
61. L. Younes Maximum likelihood estimation of Gibbs fields A. Possolo, Proceedings of an AMS-IMS-SIAM Joint Conference on Spacial Statistics and Imaging, Lecture Notes Monograph Series 1991 Institute of Mathematical Statistics Hayward, California
62. A.L. Yuille The convergence of contrastive divergence L. Saul, Y. Weiss, L. Bottou, Advances in Neural Processing Systems, NIPS 17 2005 MIT Press 1593 1600