Research and development on Boltzmann machine

Jisuanji Yanjiu yu Fazhan/Computer Research and Development

Volumn 51, Issue 1, 2014, Pages 1-16

  Liu, Jianwei ^a   Liu, Yuan ^a   Luo, Xionglin ^a  

^a CHINA UNIVERSITY OF PETROLEUM   (China)

Author keywords

Boltzmann machine; Expected value; Gibbs sampling; Hidden unit; Markov chain; Probability distribution; Simulated annealing; Visible unit

Indexed keywords

ALGORITHMS; COMPLEX NETWORKS; MARKOV PROCESSES; NETWORK LAYERS; PROBABILITY DISTRIBUTIONS; SIMULATED ANNEALING;

BOLTZMANN MACHINES; EXPECTED VALUES; GIBBS SAMPLING; HIDDEN UNITS; VISIBLE UNIT;

RESEARCH;

EID: 84896028793     PISSN: 10001239     EISSN: None     Source Type: Journal    
DOI: 10.7544/issn1000-1239.2014.20121044     Document Type: Review

References (46)

1. Hopfield J J. Neural networks and physical systems with emergent collective computational abilities [J]. Proceedings of the National Academy of Sciences of the United States of America, 1982, 79(8): 2554-2558
2. Hinton G E, Sejnowski T J. Optimal perceptual inference [C]//Proc of the 1983 IEEE Conf on Computer Vision and Pattern Recognition. Los Alamitos, CA: IEEE Computer Society, 1983: 448-453
3. Hinton G E, Sejnowski T J. Analyzing cooperative computation [C]//Proc of the 5th Annual Congress of the Cognitive Science Society. New York: ACM, 1983: 2554-2558
4. Hofstadter D R. The copycat project: An experiment in nondeterminism and creative analogies [DB/OL]. MIT Artificial Intelligence Laboratory Memo 755. (1984-01-01) [2004-10-01]. http://hdl.handle.net/1721.1/5648
5. Hofstadter D R. A Non-Deterministic Approach to Analogy, Involving the Ising Model of Ferromagnetism [M]//The Physics of Cognitive Processes. Hackensack: World Scientific, 1987
6. Smolensky P. Information Processing in Dynamical Systems: Foundations of Harmony Theory [M]//Parallel Distributed Processing, Vol 1: Foundations. Cambridge: MIT Press, 1986: 194-281
7. Ackley D H, Hinton G E, Sejnowski T J. A learning algorithm for Boltzmann machines [J]. Cognitive Science, 1985, 9(1): 147-169
8. Hinton G E. Training products of experts by minimizing contrastive divergence [J]. Neural Computation, 2002, 14(8): 1771-1800
9. Kirkpatrick S, Gelatt C D, Vecchi M P. Optimization by simulated annealing [J]. Science, 1983, 220(4598): 671-680
10. Hinton G E. To recognize shapes, first learn to generate images [J]. Computational Neuroscience: Theoretical Insights into Brain Function, 2007, 165(1): 535-547
11. Hinton G E. A practical guide to training restricted Boltzmann machines [G]//LNCS 7700: Neural Networks: Tricks of the Trade. Berlin: Springer, 2012: 599-619
12. Desjardins G, Courville A, Bengio Y, et al. Parallel tempering for training of restricted Boltzmann machines [C]//Proc of the 13th Int Conf on Artificial Intelligence and Statistics. New York: ACM, 2010: 145-152
13. Jordan M I, Ghahramani Z, Jaakkola T S, et al. An introduction to variational methods for graphical models [J]. Machine Learning, 1999, 37(2): 183-233
14. Neal R M. Connectionist learning of belief networks [J]. Artificial Intelligence, 1992, 56(1): 71-113
15. Tieleman T. Training restricted Boltzmann machines using approximations to the likelihood gradient [C]//Proc of the 25th Int Conf on Machine Learning. New York: ACM, 2008: 1064-1071
16. Hinton G E. Training products of experts by minimizing contrastive divergence [J]. Neural Computation, 2002, 14(8): 1711-1800
17. Tieleman T, Hinton G E. Using fast weights to improve persistent contrastive divergence [C]//Proc of the 26th Annual Int Conf on Machine Learning. New York: ACM, 2009: 1033-1040
18. Salakhutdinov R, Hinton G E. Deep Boltzmann machines [J]. Journal of Machine Learning Research-Proceedings Track, 2009, 9(1): 448-455
19. Hinton G E, Salakhutdinov R. Reducing the dimensionality of data with neural networks [J]. Science, 2006, 313(5786): 504-507
20. Montufar G F, Rauh J, Ay N. Expressive power and approximation errors of restricted Boltzmann machines [C]//Proc of the 2011 Neural Information Processing Systems. New York: ACM, 2011: 415-423
21. Sutskever I, Hinton G E, Taylor G. The recurrent temporal restricted Boltzmann machine [C]//Proc of the 2008 Neural Information Processing Systems. New York: ACM, 2008: 1601-1608
22. Mohamed A, Sainath T N, Dahl G E. Deep belief networks using discriminative features for phone recognition [C]//Proc of the 2011 IEEE Int Conf on Acoustics, Speech and Signal Processing. Piscataway, NJ: IEEE, 2011: 5060-5063
23. Dahl G, Ranzato M A, Mohamed A, et al. Phone recognition with the mean-covariance restricted Boltzmann machine [C]//Proc of the 2010 Neural Information Processing Systems. New York: ACM, 2010: 469-477
24. Pham P, Lee H, Largman Y, et al. Unsupervised feature learning for audio classification using convolutional deep belief networks [C]//Proc of the 2009 Neural Information Processing Systems. New York: ACM, 2009: 1096-1104
25. Salakhutdinov R, Hinton G E. An efficient learning procedure for deep Boltzmann machines [J]. Neural Computation, 2012, 24(8): 1967-2006
26. Salakhutdinov R. Learning deep Boltzmann machines using adaptive MCMC [C]//Proc of the 27th Int Conf on Machine Learning. New York: ACM, 2010: 943-950
27. Salakhutdinov R. Learning in Markov random fields using tempered transitions [C]//Proc of the 2009 Neural Information Processing Systems. New York: ACM, 2009: 1598-1606
28. Cho K, Raiko T, Ilin A. Parallel tempering is efficient for learning restricted Boltzmann machines [C]//Proc of the 2010 Int Joint Conf on Neural Networks. New York: ACM, 2010: 1-8
29. Desjardins G, Courville A, Bengio Y, et al. Tempered Markov chain Monte Carlo for training of restricted Boltzmann machines [J]. Journal of Machine Learning Research-Proceedings Track, 2010, 9(1): 145-152
30. Cho K, Raiko T, Ilin A. Enhanced gradient and adaptive learning rate for training restricted Boltzmann machines [C]//Proc of the 28th Int Conf on Machine Learning. New York: ACM, 2011: 105-112
31. Nair V, Hinton G E. Restified linear units improve restricted Boltzmann machines [C]//Proc of the 27th Int Conf on Machine Learning. New York: ACM, 2010: 807-814
32. Ranzato M A, Krizhevsky A, Hinton G E. Factored 3-way restricted Boltzmann machines for modeling natural images [J]. Journal of Machine Learning Research-Proceedings Track, 2010, 9(1): 621-628
33. Courville A, Bergstra J, Bengio Y. A spike and slab restricted Boltzmann machine [J]. Journal of Machine Learning Research: Proceedings Track, 2011, 15(1): 233-241
34. Eslami S M A, Heess N, Winn J. The shape Boltzmann machine: a strong model of object shape [C]//Proc of the 2012 IEEE Conf on Computer Vision and Pattern Recognition. Los Alamitos, CA: IEEE Computer Society, 2012: 406-413
35. Cai X, Hu S, Lin X. Feature extraction using restricted Boltzmann machine for stock price prediction [C]//Proc of the 2012 IEEE Int Conf on Computer Science and Automation Engineering. Los Alamitos, CA: IEEE Computer Society, 2012, 3: 80-83
36. Ranzato M A, Hinton G E. Modeling pixel means and convariances using factorized third-order Boltzmann machines [C]//Proc of the 2010 IEEE Conf on Computer Vision and Pattern Recognition. Los Alamitos, CA: IEEE Computer Society, 2010: 2551-2558
37. Yu D, Deng L. Accelerated parallelizable neural network learning algorithm for speech recognition [C]//Proc of Interspeech 2011. Piscataway, NJ: IEEE, 2011: 2281-2284
38. Mohamed A, Yu D, Deng L. Investigation of full-sequence training of deep belief networks for speech recognition [C]//Proc of Interspeech 2010. Piscataway, NJ: IEEE, 2010: 2846-2849
39. Dahl G E, Yu D, Deng L, et al. Large vocabulary continuous speech recognition with context-dependent DBN-HMMS [C]//Proc of the 2011 IEEE Int Conf on Acoustics, Speech and Signal Processing. Piscataway, NJ: IEEE, 2011: 4688-4691
40. Yu D, Deng L, Dahl G E. Roles of pre-training and fine-tuning in context-dependent DBN-HMMs for real-world speech recognition [C]//Proc of NIPS 2010 Workshop on Deep Learning and Unsupervised Feature Learning. New York: Academy Press, 2010
41. Deng L, Seltzer M, Yu D, et al. Binary coding of speech spectrograms using a deep auto-encoder [C]//Proc of Interspeech 2010. Piscataway, NJ: IEEE, 2010: 1692-1695
42. Stafylakis T, Kenny P, Senoussaoui M, et al. Preliminary investigation of Boltzmann machine classifiers for speaker recognition [C]//Proc of Odyssey 2012: The Speaker and Language Recognition Workshop. 2012: 109-116
43. Jaitly N, Hinton G E. Learning a better representation of speech sound waves using restricted Boltzmann machines [C]//Proc of the 2011 IEEE Int Conf on Acoustics, Speech and Signal Processing. Piscataway, NJ: IEEE, 2011: 5884-5887
44. Salakhutdinov R, Larochelle H. Efficient learning of deep Boltzmann machines [J]. Journal of Machine Learning Research : Proceedings Track, 2010, 9(1): 693-700
45. Yang Ying, Wu Chengwei, Hu Su. Chinese documents classification based on restricted Boltzmann machines [J]. Science and Technology Innovation Herald, 2012, 26(16): 35-36 (in Chinese)
46. Zhou S K, Chellappa R. From sample similarity to ensemble similarity: Probabilistic distance measures in reproducing kernel hilbert space [J]. IEEE Trans on Pattern Analysis and Machine Intelligence, 2006, 28(6): 917-929