Deep Boltzmann machines and the centering trick

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 7700 LECTURE NO, Issue , 2012, Pages 621-637

  Montavon, Grégoire ^a   Müller, Klaus Robert ^a,b  

^a TECHNISCHE UNIVERSITÄT BERLIN   (Germany)

^b Korea University   (South Korea)

Author keywords

centering; Deep Boltzmann machine; optimization; reparameterization; representations; unsupervised learning

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPUTER SCIENCE; COMPUTERS; OPTIMIZATION; UNSUPERVISED LEARNING;

CENTERING; COMPLEX DATA; DATA REPRESENTATIONS; DEEP BOLTZMANN MACHINES; OPTIMIZATION PROBLEMS; REPARAMETERIZATION; REPRESENTATIONS;

LEARNING SYSTEMS;

EID: 84872571941     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-642-35289-8_33     Document Type: Article

References (19)

1. Arnold, L., Auger, A., Hansen, N., Ollivier, Y.: Information-geometric optimization algorithms: A unifying picture via invariance principles, arXiv:1106.3708 (2011)
2. Braun, M.L., Buhmann, J., Muller, K.-R.: On relevant dimensions in kernel feature spaces. Journal of Machine Learning Research 9, 1875-1908 (2008)
3. Cho, K., Raiko, T., Ilin, A.: Enhanced gradient and adaptive learning rate for training restricted Boltzmann machines. In: Proceedings of the 28th International Conference on Machine Learning, pp. 105-112 (2011)
4. Hinton, G.E.: Training products of experts by minimizing contrastive divergence. Neural Computation 14(8), 1771-1800 (2002)
5. Hinton, G.E.: A Practical Guide to Training Restricted Boltzmann Machines. In: Montavon, G., Orr, G.B., Muller, K.-R. (eds.) NN: Tricks of the Trade, 2nd edn. LNCS, vol. 7700, pp. 599-619. Springer, Heidelberg (2012)
6. Hinton, G.E., Sejnowski, T.J.: Learning and relearning in Boltzmann machines. In: Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1, pp. 282-317. MIT Press (1986)
7. LeCun, Y., Bottou, L., Orr, G.B., Muller, K.-R.: Efficient BackProp. In: Orr, G.B., Muller, K.-R. (eds.) NIPS-WS 1996. LNCS, vol. 1524, pp. 9-50. Springer, Heidelberg (1998)
8. Montavon, G., Braun, M.L., Muller, K.-R.: Kernel analysis of deep networks. Journal of Machine Learning Research 12, 2563-2581 (2011)
9. Montavon, G., Braun, M.L., Muller, K.-R.: Deep Boltzmann machines as feedforward hierarchies. Journal of Machine Learning Research - Proceedings Track 22, 789-804 (2012)
10. Neal, R.M.: Annealed importance sampling. Statistics and Computing 11(2), 125-139 (2001) (Pubitemid 33638185)
11. Pearlmutter, B.A.: Fast exact multiplication by the Hessian. Neural Computation 6(1), 147-160 (1994)
12. Salakhutdinov, R., Hinton, G.E.: Deep Boltzmann machines. In: Proceedings of the International Conference on Artificial Intelligence and Statistics, vol. 5, pp. 448-455 (2009)
13. Salakhutdinov, R.: Learning and Evaluating Boltzmann Machines. Technical Report UTML TR 2008-002, Dept. of Computer Science, University of Toronto (2008)
14. Salakhutdinov, R., Murray, I.: On the quantitative analysis of deep belief networks. In: Proceedings of the 25th International Conference on Machine Learning, ICML 2008, pp. 872-879 (2008)
15. Scholkopf, B., Smola, A., Muller, K.-R.: Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation 10(5), 1299-1319 (1998) (Pubitemid 128463674)
16. Scholkopf, B., Mika, S., Burges, C.J.C., Knirsch, P., Muller, K.-R., Ratsch, G., Smola, A.J.: Input space versus feature space in kernel-based methods. IEEE Transactions on Neural Networks 10(5), 1000-1017 (1999)
17. Schraudolph, N.N.: Centering Neural Network Gradient Factors. In: Orr, G.B., Muller, K.-R. (eds.) NIPS-WS 1996. LNCS, vol. 1524, pp. 207-226. Springer, Heidelberg (1998)
18. Tang, Y., Sutskever, I.: Data normalization in the learning of restricted Boltzmann machines. Technical Report UTML-TR-11-2, Department of Computer Science, University of Toronto (2011)
19. Tieleman, T.: Training restricted Boltzmann machines using approximations to the likelihood gradient. In: Proceedings of the 25th International Conference on Machine Learning, pp. 1064-1071 (2008)