Enhanced gradient and adaptive learning rate for training restricted Boltzmann machines

Proceedings of the 28th International Conference on Machine Learning, ICML 2011

Volumn , Issue , 2011, Pages 105-112

  Cho, KyungHyun ^a   Raiko, Tapani ^a   Ilin, Alexander ^a  

^a AALTO UNIVERSITY   (Finland)

Author keywords

[No Author keywords available]

Indexed keywords

ADAPTIVE LEARNING RATES; BIT FLIPPING; BOLTZMANN MACHINES; BUILDING BLOCKES; DATA REPRESENTATIONS; GREEDY LEARNING; INITIAL WEIGHTS; LEARNING RATES; LOCAL LIKELIHOOD; RESTRICTED BOLTZMANN MACHINE; SIMPLIFIED MODELS; TRADITIONAL LEARNING;

LEARNING SYSTEMS; SCALES (WEIGHING INSTRUMENTS);

LEARNING ALGORITHMS;

EID: 80053444761     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (18)

1. Bengio, Y. Learning Deep Architectures for AI. Found. Trends Mach. Learn., 2:1-127, January 2009.
2. Carreira-Perpiñán, M. A. and Hinton, G. E. On Contrastive Divergence Learning. In Proceedings of the 10th International Workshop on Artificial Intelligence and Statistics, pp. 33-40, 2005.
3. Cho, K., Raiko, T., and Ilin, A. Parallel Tempering is Efficient for Learning Restricted Boltzmann Machines. In Proceedings of the International Joint Conference on Neural Networks (IJCNN 2010), July 2010.
4. Desjardins, G., Courville, A., Bengio, Y., Vincent, P., and Delalleau, O. Tempered Markov Chain Monte Carlo for training of Restricted Boltzmann Machines. Technical Report 1345, Universitée de Montréal, 2009.
5. Desjardins, G., Courville, A., Bengio, Y., Vincent, P., and Delalleau, O. Parallel Tempering for Training of Restricted Boltzmann Machines. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, pp. 145-152, 2010.
6. Erhan, D., Bengio, Y., Courville, A., Manzagol, P. A., Vincent, P., and Bengio, S. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 11:625-660, 2010.
7. Fischer, A. and Igel, C. Empirical analysis of the divergence of Gibbs sampling based learning algorithms for restricted Boltzmann machines. In Proceedings of the 20th international conference on Artificial neural networks: Part III, pp. 208-217, 2010.
8. Hinton, G. E. Training products of experts by minimizing contrastive divergence. Neural Computation, 14:1771-1800, August 2002.
9. Hinton, G. E. A Practical Guide to Training Restricted Boltzmann Machines. Technical Report 2010-003, Department of Computer Science, University of Toronto, 2010.
10. Hinton, G. E. and Salakhutdinov, R. R. Reducing the Dimensionality of Data with Neural Networks. Science, 313(5786):504-507, July 2006. (Pubitemid 44148451)
11. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-Based Learning Applied to Document Recognition. In Proceedings of the IEEE, volume 86, pp. 2278-2324, 1998.
12. Lee, H., Ekanadham, C., and Ng, A. Y. Sparse deep belief net model for visual area V2. In Advances in Neural Information Processing Systems 20, 2008.
13. Marlin, B. M., Swersky, K., Chen, B., and De Freitas, N. Inductive Principles for Restricted Boltzmann Machine Learning. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, pp. 509-516, 2010.
14. Salakhutdinov, R. Learning Deep Generative Models. PhD thesis, University of Toronto, 2009a.
15. Salakhutdinov, R. Learning in Markov Random Fields using Tempered Transitions. In Advances in Neural Information Processing Systems, pp. 1598-1606, 2009b.
16. Salakhutdinov, R. and Hinton, G. E. Deep Boltzmann machines. In Proceedings of the 12th International Conference on Artificial Intelligence and Statistics, pp. 448-455, 2009.
17. Schulz, H., Müller, A., and Behnke, S. Investigating Convergence of Restricted Boltzmann Machine Learning. In NIPS 2010 Workshop on Deep Learning and Unsupervised Feature Learning, 2010.
18. Smolensky, P. Information processing in dynamical systems: foundations of harmony theory. In Parallel distributed processing: explorations in the micro structure of cognition, vol. 1: foundations, pp. 194-281. MIT Press, Cambridge, MA, USA, 1986.