GSNs: Generative stochastic networks

Information and Inference

Volumn 5, Issue 2, 2016, Pages 210-249

  Alain, Guillaume ^a   Bengio, Yoshua ^a   Yao, Li ^a   Yosinski, Jason ^b   Thibodeau Laufer, Éric ^a   Zhang, Saizheng ^a   Vincent, Pascal ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

^b School of Operations Research and Information Engineering   (United States)

Author keywords

Auto encoders; Deep learning; Generative models

Indexed keywords

EID: 85033452370     PISSN: None     EISSN: 20498772     Source Type: Journal    
DOI: 10.1093/imaiai/iaw003     Document Type: Article

References (65)

1. Alain, G. & Bengio, Y. (2013) What regularized auto-encoders learn from the data generating distribution. International Conference on Learning Representations (ICLR'2013).
2. Anselmi, F., Rosasco, L. & Poggio, T. (2016) On invariance and selectivity in representation learning. Inf. Inference, to appear.
3. Behnke, S. (2001) Learning iterative image reconstruction in the neural abstraction pyramid. Int. J. Comput. Intell. Appl., 1, 427-438.
4. Bengio, Y. (2009) Learning deep architectures for AI. Hanover, MA, USA: Now Publishers.
5. Bengio, Y. (2013) Estimating or propagating gradients through stochastic neurons. Technical Report, Universite de Montreal, arXiv:1305.2982.
6. Bengio, Y., Courville, A. & Vincent, P. (2013a) Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 35, 1798-1828.
7. Bengio, Y., Lamblin, P., Popovici, D. & Larochelle, H. (2007) Greedy layer-wise training of deep networks. NIPS'2006.
8. Bengio, Y., Mesnil, G., Dauphin, Y. & Rifai, S. (2013b) Better mixing via deep representations. ICML'13.
9. Bengio, Y., Thibodeau-Laufer, E., Alain, G. & Yosinski, J. (2014) Deep generative stochastic networks trainable by backprop. Technical Report, arXiv:1306.1091.
10. Bengio, Y., Yao, L., Alain, G. & Vincent, P. (2013c) Generalized denoising auto-encoders as generative models. NIPS'2013.
11. Bengio, Y., Yao, L. & Cho, K. (2013d) Bounding the test log-likelihood of generative models. Technical Report, U. Montreal, arXiv:1311.6184.
12. Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D. & Bengio, Y. (2010) Theano: A CPU and GPU math expression compiler. Proceedings of the Python for Scientific Computing Conference (SciPy).
13. Bordes, A., Glorot, X., Weston, J. & Bengio, Y. (2013) A semantic matching energy function for learning with multi-relational data. Machine Learning: Special Issue on Learning Semantics.
14. Bornschein, J. & Bengio, Y. (2014) Reweighted wake-sleep. Technical Report, arXiv:1406.2751.
15. Breuleux, O., Bengio, Y. & Vincent, P. (2011) Quickly generating representative samples from an RBM-derived process. Neural Comput., 23, 2053-2073.
16. Cheng, X., Chen, X. & Mallat, S. (2016) Deep haar scattering networks. Inf. Inference, to appear.
17. Cho, G. E., Meyer, C. D. & Meyer, C. D. (2000) Comparison of perturbation bounds for the stationary distribution of a Markov chain. Linear Algebra Appl., 335, 137-150.
18. Cho, K. H., Raiko, T. & Ilin, A. (2013) Enhanced gradient for training restricted Boltzmann machines. Neural Comput., 25, 805-831.
19. Collobert, R. & Weston, J. (2008) A unified architecture for natural language processing: Deep neural networks with multitask learning. ICML'2008.
20. Dahl, G. E., Ranzato, M., Mohamed, A. & Hinton., G. E. (2010) Phone recognition with the mean-covariance restricted Boltzmann machine. NIPS'2010.
21. Dayan, P., Hinton, G. E., Neal, R. M. & Zemel, R. S. (1995) The Helmholtz machine. Neural Comput., 7, 889-904.
22. Deng, L., Seltzer, M., Yu, D., Acero, A., Mohamed, A. & Hinton, G. (2010) Binary coding of speech spectrograms using a deep auto-encoder. Interspeech 2010. Chiba, Japan: Makuhari.
23. Dinh, L., Krueger, D. & Bengio, Y. (2015) Nice: non-linear independent components estimation. arXiv:1410.8516.
24. Galanti, T., Wolf, L. & Hazan, T. (2016) A PAC theory of the transferable. Inf. Inference, to appear.
25. Goodfellow, I. J., Courville, A. & Bengio, Y. (2015) Deep Learning. Book in preparation for MIT Press. http://www.deeplearningbook.org/. Last Accessed March 1, 2015.
26. Goodfellow, I. J., Mirza, M., Courville, A. & Bengio, Y. (2013) Multi-prediction deep Boltzmann machines. NIPS'2013.
27. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. & Bengio, Y. (2014) Generative adversarial nets. Advances in Neural Information Processing Systems. pp. 2672-2680.
28. Gregor, K., Danihelka, I., Mnih, A., Blundell, C. & Wierstra, D. (2014) Deep autoregressive networks. International Conference on Machine Learning (ICML'2014).
29. Gutmann, M. & Hyvarinen, A. (2010) Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. AISTATS'2010.
30. Heckerman, D., Chickering, D. M., Meek, C., Rounthwaite, R. & Kadie, C. (2000) Dependency networks for inference, collaborative filtering, and data visualization. J. Mach. Learn. Res., 1, 49-75.
31. Hinton, G. E. (1999) Products of experts. ICANN'1999.
32. Hinton, G. E. (2000) Training products of experts by minimizing contrastive divergence. Technical Report GCNU TR 2000-004, Gatsby Unit, University College London.
33. Hinton, G. E., Dayan, P., Frey, B. J. & Neal, R. M. (1995) The wake-sleep algorithm for unsupervised neural networks. Science, 268, 1558-1561.
34. Hinton, G. E., Osindero, S. & Teh, Y. W. (2006) A fast learning algorithm for deep belief nets. Neural Comput., 18, 1527-1554.
35. Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. (2012) Improving neural networks by preventing co-adaptation of feature detectors. Technical Report, arXiv:1207.0580.
36. Hyvärinen, A. (2006) Consistency of pseudolikelihood estimation of fully visible Boltzmann machines. Neural Comput, 18, 2283-2292.
37. Kingma, D. P. (2013) Fast gradient-based inference with continuous latent variable models in auxiliary form. Technical Report, arxiv:1306.0733.
38. Kingma, D. P. & Welling, M. (2014) Auto-encoding variational bayes. Proceedings of the International Conference on Learning Representations (ICLR).
39. Krizhevsky, A., Sutskever, I. & Hinton, G. (2012) ImageNet classification with deep convolutional neural networks. NIPS'2012.
40. Larochelle, H. & Murray, I. (2011) The Neural Autoregressive Distribution Estimator. AISTATS'2011.
41. Lee, H., Battle, A., Raina, R. & Ng, A. (2007) Efficient sparse coding algorithms. NIPS'06. Cambridge, MA: MIT Press, pp. 801-808.
42. Li, Y., Tarlow, D. & Zemel, R. (2013) Exploring compositional high-order pattern potentials for structured output learning. CVPR'2013.
43. Luo, H., Carrier, P. L., Courville, A. & Bengio, Y. (2013) Texture modeling with convolutional spike-and-slab RBMs and deep extensions. AISTATS'2013.
44. Mnih, A. & Gregor, K. (2014) Neural variational inference and learning in belief networks. ICML'2014.
45. Montavon, G. & Muller, K.-R. (2012) Deep Boltzmann machines and the centering trick. Neural Networks: Tricks of the Trade (G. Montavon, G. Orr & K.-R. Müller eds). Lecture Notes in Computer Science, vol. 7700. Berlin: Springer, pp. 621-637.
46. Ozair, S. & Bengio, Y. (2014) Deep directed generative autoencoders. Technical Report, U. Montreal, arXiv:1410.0630.
47. Ozair, S., Yao, L. & Bengio, Y. (2014) Multimodal transitions for generative stochastic networks. Technical Report, U. Montreal, arXiv:1312.5578.
48. Poon, H. & Domingos, P. (2011) Sum-product networks: A new deep architecture. UAI'2011. Barcelona, Spain.
49. Ranzato, M., Poultney, C., Chopra, S. & LeCun, Y. (2007) Efficient learning of sparse representations with an energy-based model. NIPS'2006.
50. Rezende, D. J., Mohamed, S. & Wierstra, D. (2014) Stochastic backpropagation and approximate inference in deep generative models. Technical Report, arXiv:1401.4082.
51. Rifai, S., Bengio, Y., Dauphin, Y. & Vincent, P. (2012) A generative process for sampling contractive autoencoders. ICML'12.
52. Salakhutdinov, R. & Hinton, G. E. (2009) Deep Boltzmann machines. AISTATS'2009, pp. 448-455.
53. Savard, F. (2011) Réseaux de neurones à relaxation entraînés par critère d'autoencodeur débruitant. Master's Thesis, U. Montréal.
54. Schweitzer, P. J. (1968) Perturbation theory and finite Markov chains. J. Appl. Probab., 5, 401-413.
55. Seide, F., Li, G. & Yu, D. (2011) Conversational speech transcription using context-dependent deep neural networks. Interspeech 2011, pp. 437-440.
56. Seung, S. H. (1998) Learning continuous attractors in recurrent networks. NIPS'97. Cambridge, MA: MIT Press, pp. 654-660.
57. Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N. & Ganguli, S. (2015) Deep unsupervised learning using nonequilibrium thermodynamics. arXiv:1503.03585.
58. Tieleman, T. (2008) Training restricted Boltzmann machines using approximations to the likelihood gradient. ICML 2008 (W. W. Cohen, A. McCallum & S. T. Roweis eds). New York, NY, USA: ACM, pp. 1064-1071.
59. Vincent, P., Larochelle, H., Bengio, Y. & Manzagol, P.-A. (2008) Extracting and composing robust features with denoising autoencoders. ICML 2008.
60. Williams, R. J. (1992) Simple statistical gradient-following algorithms connectionist reinforcement learning. Mach. Learn., 8, 229-256.
61. Yao, L., Ozair, S., Cho, K. & Bengio, Y. (2014) On the equivalence between deep nade and generative stochastic networks. Technical Report, U. Montreal, arXiv:1409.0585.
62. Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. (2014) How transferable are features in deep neural networks? NIPS'2014.
63. Younes, L. (1998) On the convergence of Markovian stochastic algorithms with rapidly decreasing ergodicity rates. Stochastics and Stochastics Models, pp. 177-228.
64. Zhou, J. & Troyanskaya, O. G. (2014) Deep supervised and convolutional generative stochastic network for protein secondary structure prediction. ICML'2014.
65. Zöhrer, M. & Pernkopf, F. (2014) General stochastic networks for classification. NIPS'2014.