A tutorial on stochastic approximation algorithms for training restricted Boltzmann machines and deep belief nets

2010 Information Theory and Applications Workshop, ITA 2010 - Conference Proceedings

Volumn , Issue , 2010, Pages 80-89

  Swersky, Kevin ^a   Chen, Bo ^a   Marlin, Ben ^a   De Freitas, Nando ^a  

^a UNIVERSITY OF BRITISH COLUMBIA   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

BELIEF NETWORKS; CONTRASTIVE DIVERGENCE; DATA SETS; FINE TUNING; OPTIMAL PARAMETER; OPTIMAL RESULTS; PARAMETER CHANGES; RESTRICTED BOLTZMANN MACHINE; STOCHASTIC APPROXIMATION ALGORITHMS; STOCHASTIC APPROXIMATIONS; STOCHASTIC MAXIMUM LIKELIHOOD ALGORITHMS;

APPROXIMATION THEORY; INFORMATION THEORY; MAXIMUM LIKELIHOOD; OPTIMIZATION; STOCHASTIC SYSTEMS;

APPROXIMATION ALGORITHMS;

EID: 77952681438     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ITA.2010.5454138     Document Type: Conference Paper

References (44)

1. G. Hinton and R. Salakhutdinov, "Reducing the dimensionality of data with neural networks," Science, vol. 313, no. 5786, pp. 504-507, 2006.
2. R. Salakhutdinov, A. Mnih, and G. Hinton, "Restricted Boltzmann machines for collaborative filtering," in International Conference on Machine learning, 2007, pp. 791-798.
3. R. Salakhutdinov and G. Hinton, "Semantic hashing," International Journal of Approximate Reasoning, 2008.
4. Y. Bengio, P. Lamblin, D. Popovici, H. Larochelle, and U. Montreal, "Greedy layer-wise training of deep networks," in Advances in Neural Information Processing Systems. MIT Press, 2007.
5. Y. Bengio and Y. Le Cun, "Scaling learning algorithms towards AI," Large-Scale Kernel Machines, 2007.
6. H. Lee, R. Grosse, R. Ranganath, and A. Ng, "Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations," in International Conference on Machine Learning, 2009.
7. L. Wiskott and T. Sejnowski, "Slow feature analysis: Unsupervised learning of invariances," Neural Computation, vol. 14, no. 4, pp. 715-770, 2002.
8. J. Hawkins and D. George, "Hierarchical temporal memory: Concepts, theory and terminology," Numenta, Tech. Rep., 2006.
9. I. J. Goodfellow, Q. V. Le, A. M. Saxe, H. Lee, and A. Y. Ng., "Measuring invariances in deep networks," Advances in neural information processing systems, 2009.
10. G. Hinton, S. Osindero, and Y. Teh, "A fast learning algorithm for deep belief nets," Neural Computation, vol. 18, no. 7, pp. 1527-1554, 2006.
11. L. Younes, "Parametric inference for imperfectly observed Gibbsian fields," Probability Theory and Related Fields, vol. 82, no. 4, pp. 625-645, 1989.
12. T. Tieleman, "Training restricted Boltzmann machines using approximations to the likelihood gradient," in International conference on Machine Learning, 2008, pp. 1064-1071.
13. M. Welling, M. Rosen-Zvi, and G. Hinton, "Exponential family harmoniums with an application to information retrieval," Advances in neural information processing systems, vol. 17, pp. 1481-1488, 2005.
14. G. E. Hinton, "Training products of experts by minimizing contrastive divergence," Neural Computation, vol. 14, p. 2002, 2002.
15. M. Carreira-Perpinan and G. Hinton, "On contrastive divergence learning," in Artificial Intelligence and Statistics, vol. 2005, 2005.
16. B. Marlin, "A direct proof that the true RBM parameters are a fixed point of both ML and CD1 in the asymptotic setting," 2008.
17. A. Yuille, "The convergence of contrastive divergences," in Advances in Neural Information Processing Systems, 2004.
18. A. Benveniste, M. Métivier, and P. Priouret, Adaptive algorithms and stochastic approximations. Springer-Verlag, 1990.
19. D. Ruppert, "A Newton-Raphson version of the multivariate Robbins-Monro procedure," Ann. Statist., vol. 13, no. 1, pp. 236-245, 1985.
20. J. Spall, "Adaptive stochastic approximation by the simultaneous perturbation method," IEEE Conference on Decision and Control, pp. 3872-3879, 1998.
21. B. T. Polyak, "A new method of stochastic approximation type," Avtomat. i Telemekh., no. 7, pp. 98-107, 1990.
22. B. Polyak and A. Juditsky, "Acceleration of stochastic approximation by averaging," SIAM Journal on Control and Optimization, vol. 30, p. 838, 1992.
23. H. J. Kushner and H. Huang, "Averaging methods for the asymptotic analysis of learning and adaptive systems, with small adjustment rate," SIAM J. Control Optim., vol. 19, no. 5, pp. 635-650, 1981.
24. -, "Asymptotic properties of stochastic approximations with constant coefficients," SIAM J. Control Optim., vol. 19, no. 1, pp. 87-105, 1981.
25. H. J. Kushner and G. G. Yin, Stochastic Approximation Algorithms and Applications. Springer-Verlag, 1997.
26. L. Ljung and T. Söderström, Theory and practice of recursive identification. MIT Press, 1983.
27. V. Borkar, Stochastic Approximation: A Dynamical Systems Viewpoint. Cambridge University Press, 2008.
28. A. Bhaya and E. Kaszkurewicz, "Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method," Neural Networks, vol. 17, no. 1, pp. 65-71, 2004.
29. R. Sharma, W. Sethares, and J. Bucklew, "Analysis of momentum adaptive filtering algorithms," IEEE Transactions on Signal Processing, vol. 46, no. 5, pp. 1430-1434, May 1998.
30. J. J. Hopfield, "Hopfield network," Scholarpedia, 2007.
31. M. Welling, "Herding Dynamic Weights for Partially Observed Random Field Models," in UAI, 2009.
32. A. Juditsky, A. Nazin, A. Tsybakov, and N. Vayatis, "Generalization error bounds for aggregation by mirror descent with averaging," Advances in neural information processing systems, 2005.
33. B. Delyon, "General results on the convergence of stochastic algorithms," IEEE Transactions on Automatic Control, vol. 41, no. 9, pp. 1245-1255, 1996.
34. C. Andrieu and J. Thoms, "A tutorial on adaptive MCMC," Statistics and Computing, vol. 18, no. 4, pp. 343-373, 2008.
35. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Athena Scientific, 1996.
36. Y. LeCun and C. Cortes, "The MNIST database of handwritten digits," NEC Research Institute, http://yann. lecun. com/exdb/mnist/index. html.
37. B. Polyak, "Some methods of speeding up the convergence of iterative methods," USSR Computational Mathematics and Mathematical Physics., vol. 4, pp. 1-17, 1964.
38. L. Bottou and O. Bousquet, "The tradeoffs of large scale learning," Advances in neural information processing systems, vol. 20, 2007.
39. R. Salakhutdinov and G. Hinton, "Training a deep autoencoder or a classifier on MNIST digits - source code," 2006.
40. H. Larochelle, Y. Bengio, J. Louradour, and P. Lamblin, "Exploring Strategies for Training Deep Neural Networks," Journal of Machine Learning Research, vol. 1, pp. 1-40, 2009.
41. H. Lee, C. Ekanadham, and A. Ng, "Sparse deep belief net model for visual area V2," Advances in neural information processing systems, vol. 20, 2008.
42. P. Garrigues and B. Olshausen, "Learning horizontal connections in a sparse coding model of natural images," Advances in Neural Information Processing Systems, vol. 20, pp. 505-512, 2008.
43. H. Larochelle and Y. Bengio, "Classification using discriminative restricted Boltzmann machines," in International Conference on Machine learning, 2008, pp. 536-543.
44. D. Erhan, P. Manzagol, Y. Bengio, S. Bengio, and P. Vincent, "The difficulty of training deep architectures and the effect of unsupervised pre-training," AISTATS, 2009.