An Overview of Deep Generative Models

IETE Technical Review (Institution of Electronics and Telecommunication Engineers, India)

Volumn 32, Issue 2, 2015, Pages 131-139

  Xu, Jungang ^a   Li, Hui ^a   Zhou, Shilong ^a  

^a UNIVERSITY OF CHINESE ACADEMY OF SCIENCES   (China)

Author keywords

Deep autoencoder; Deep belief networks; Deep boltzmann machine; Deep generative model; Restricted boltzmann machine

Indexed keywords

IMAGE PROCESSING; LEARNING SYSTEMS;

AUTO ENCODERS; DEEP AUTOENCODER; DEEP BELIEF NETWORKS; DEEP BOLTZMANN MACHINES; DEEP GENERATIVE MODEL; GENERATIVE MODEL; IMAGES PROCESSING; RESTRICTED BOLTZMANN MACHINE; TRAINING ALGORITHMS;

SPEECH RECOGNITION;

EID: 85085272778     PISSN: 02564602     EISSN: 09745971     Source Type: Journal    
DOI: 10.1080/02564602.2014.987328     Document Type: Review

References (75)

1. T. S. Lee, and D. Mumford, “Hierarchical Bayesian inference in the visual cortex,” The Journal of the Optical Society of America A, Vol. 20, no. 7, pp. 1434–48, Jul. 2003.
2. T. Serre, L. Wolf, and S. Bileschi, “Robust object recognition with cortex-like mechanisms,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 29, no. 3, pp. 411–26, Mar. 2007.
3. T. S. Lee, D. Mumford, and R. Romero, “The role of the primary visual cortex in higher level vision,” Vision Research, Vol. 38, no. 15, pp. 2429–54, Aug. 1998.
4. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” Nature, Vol. 323, no. 7, pp. 533–6, Oct. 1986.
5. Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, “Greedy layer-wise training of deep networks,” in Advances in Neural Information Processing Systems, Vol. 19, B. Schölkopf, J. C. Platt and T. Hoffman, Eds. Cambridge, MA: MIT Press, 2006, pp. 153–60.
6. H. Larochelle, Y. Bengio, J. Louradour, and P. Lamblin, “Exploring strategies for training deep neural networks,” Journal of Machine Learning Research, Vol. 1, pp. 1–40, Jan. 2009.
7. P. J. Werbos, Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. Boston: Harvard University, 1974.
8. R. Hecht-Nielsen, “Replicator neural networks for universal optimal source coding,” Science, Vol. 269, pp. 1860–3, Sept. 1995.
9. G. Tesauro, “Practical issues in temporal difference learning,” Machine Learning, Vol. 8, no. 3–4, pp. 257–77, May 1992.
10. Y. Bengio, “Learning deep architectures for AI,” Foundations and trends® in Machine Learning, Vol. 2, no. 1, pp. 1–127, Jan. 2009.
11. Y. Bengio, and Y. LeCun, “Scaling learning algorithms towards AI,” Large-Scale Kernel Machines, Vol. 34, pp. 1–41, Sept. 2007.
12. G. E. Hinton, S. Osindero, and Y. W. The, “A learning algorithm for deep belief nets,” Neural Computation, Vol. 18, no. 7, pp. 1527–54, Jul. 2006.
13. B. Taskar, P. Abbeel, and D. Koller, “Discriminative probabilistic models for relational data,” in Proceedings of Conference on Uncertainty in Artificial Intelligence, Alberta, 2002, pp. 485–92.
14. J. A. Lasserre, C. M. Bishop, and T. P. Minka, “Principled hybrids of generative and discriminative models,” in Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, New York, 2006, pp. 87–94.
15. M. I. Jordan, Learning in Graphical Models. Dordrecht: Kluwer, 1998.
16. P. Dayan, G. E. Hinton, R. Neal, and R. Zemel. “The Helmholtz machine,”Neural Computation, Vol. 7, no. 5, pp. 889–904, Sept. 1995.
17. G. E. Hinton, P. Dayan, B. J. Frey, and R. M. Neal, “The “wake-sleep” algorithm for unsupervised neural network,” Science, Vol. 268, no. 5214, pp. 1558–161, May 1995.
18. L. K. Saul, T. Jaakkola, and M. I. Jordan, “Mean field theory for sigmoid belief networks,” Journal of Artificial Intelligence Research, Vol. 4, no. 1, pp. 61–76, Jan. 1996.
19. I. Titov, and J. Henderson, “Constituent parsing with incremental sigmoid belief networks,” in Proceedings of Meeting of Association for Computational Linguistics, Prague, 2007, pp. 632–9.
20. P. Smolensky, “Information processing in dynamical systems: foundations of harmony theory,” Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1, pp. 194–281, Feb. 1986.
21. Y. Freund, and D. Haussler, “Unsupervised learning of distributions on binary vectors using two layer networks,” in Advances in Neural Information Processing Systems, Vol. 4, J. E. Moody, S. J. Hanson, and R.P. Lippmann, Eds. Denver, CO: Morgan Kaufmann, 1991, pp. 912–9.
22. G. E. Hinton, “Training products of experts by minimizing contrastive divergence,”Neural Computation, Vol. 14, no. 8, pp. 1771–800, Aug. 2002.
23. M. Welling, M. Rosen-Zvi, and G. E. Hinton, “Exponential family harmoniums with an application to information retrieval,” in Advances in Neural Information Processing Systems, Vol. 17, L. K. Saul, Y. Weiss and L. Bottou, Eds. Cambridge, MA: MIT Press, 2004, pp. 1481–8.
24. R. Salakhutdinov, and G. E. Hinton, “Deep Boltzmann machines,” in Proceedings of International Conference on Artificial Intelligence and Statistics, Florida, 2009, pp. 448–55.
25. G. E. Hinton, and R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” Science, Vol. 313, no. 5786, pp. 504–7, May 2006.
26. R. Collobert, and J. Weston, “A unified architecture for natural language processing: Deep neural networks with multitask learning,” in Proceedings of International Conference on Machine learning, Helsinki, 2008, pp. 160–7.
27. M. Ranzato, C. Poultney, S. Chopra, and Y. LeCun, “Efficient learning of sparse representations with an energy-based model,” in Advances in Neural Information Processing Systems, Vol. 19, B. Schölkopf, J. C. Platt and T. Hoffman, Eds. Cambridge: MIT Press, 2006, pp. 1137–44.
28. P. Y. Simard, D. Steinkraus, and J. C. Platt, “Best practices for convolutional neural networks applied to visual document analysis,” in Proceedings of the 7th International Conference on Document Analysis and Recognition, Washington DC, 2003, pp. 958–63.
29. L. Deng, and D. Yu, “Deep learning for signal and information processing,” Microsoft Research Report, Redmond, 2013.
30. K. H. Cho, T. Raiko, and A. Ilin, “Parallel tempering is efficient for learning restricted Boltzmann machines,” in Proceedings of the 2010 International Joint Conference on Neural Networks, Thessaloniki, 2010, pp. 1–8.
31. N. Le Roux, and Y. Bengio, “Representational power of restricted Boltzmann machines and deep belief networks,” Neural Computation, Vol. 20, no. 6, pp. 1631–49, Jun. 2008.
32. A. Fischer, and C. Igel, “Empirical analysis of the divergence of Gibbs sampling based learning algorithms for restricted Boltzmann machines,” in Proceedings of the 20th International Conference on Artificial Neural Networks, Thessaloniki, 2010, pp. 208–17.
33. G. E. Hinton, “Products of experts,” in Proceedings of the 9th International Conference on Artificial Neural Networks, London, 1999, pp. 1–6.
34. G. E. Hinton, “Learning multiple layers of representation,” Trends in Cognitive Sciences, Vol. 11, no. 10, pp. 428–34, Oct. 2007.
35. T. Tieleman, “Training restricted Boltzmann machines using approximations to the likelihood gradient,” in Proceedings of the 25th International Conference on Machine Learning, New York, 2008, pp. 1064–71.
36. T. Tieleman, and G. Hinton, “Using fast weights to improve persistent contrastive divergence,” in Proceedings of the 26th Annual International Conference on Machine Learning, New York, 2009, pp. 1033–40.
37. Y. Bengio, and O. Delalleau, “Justifying and generalizing contrastive divergence,” Neural Computation, Vol. 21, no. 6, pp. 1601–21, Jun. 2009.
38. A. Fischer, and C. Igel, “Empirical analysis of the divergence of Gibbs sampling based learning algorithms for restricted Boltzmann machines,” in Proceedings of the 20th International Conference on Artificial Neural Networks, Thessaloniki, 2010, pp. 208–17.
39. D. J. Earl, and M. W. Deem, “Parallel tempering: theory, applications, and new perspectives,” Physical Chemistry Chemical Physics, Vol. 7, pp. 3910–6, Aug. 2005.
40. G. Desjardins, A. Courville, and Y. Bengio, “Parallel tempering for training of restricted Boltzmann machines,” in Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, New York, 2010, pp. 145–52.
41. R. M. Neal, “Sampling from multimodal distributions using tempered transitions,” Statistics and Computing, Vol. 6, no. 4, pp. 353–66, Dec. 1996.
42. Y. Iba, “Extended ensemble monte carlo,” International Journal of Modern Physics, Vol. 12, no. 5, pp. 623–56, Jun. 2001.
43. J. Xu, H. Li, and S. Zhou, “Improving Mixing Rate with Tempered Transition for Learning Restricted Boltzmann Machines,” Neurocomputing, Vol. 139, pp. 328–35, Sept. 2014.
44. D. C. Plaut, and G. E. Hinton, “Learning sets of filters using back-propagation,” Computer, Speech and Language, Vol. 2, no. 1, pp. 35–61, Mar. 1987.
45. D. DeMers, and G. Cottrell, “Non-linear dimension reduction,” in Advances in Neural Information Processing Systems, Vol. 5, S. J. Hanson, J. D. Cowan and C. L. Giles, Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 580–7.
46. R. Hecht-Nielsen, “Replicator neural networks for universal optimal source coding,” Science, Vol. 269, no. 5232, pp. 1860–3, Sept. 1995.
47. N. Kambhatla, and T. K. Leen, “Dimension reduction by local principal component analysis,” Neural Computation, Vol. 9, no. 7, pp. 1493–516, Oct. 1997.
48. R. Salakhutdinov, and G. E. Hinton, “Deep Boltzmann machines,” in Proceedings of the 12th International Conference on Artificial Intelligence and Statistics, Clearwater Beach, 2009, pp. 448–55.
49. R. Salakhutdinov, and G. Hinton, “A better way to pretrain deep Boltzmann machines,” Advances in Neural Information Processing Systems, Vol. 25, F. Pereira, C. J. C. Burges, L. Bottou and K. Q. Weinberger, Eds. Cambridge, MA: MIT Press, 2012, pp. 1–9.
50. R. Salakhutdinov, “Learning deep generative models,” Ph.D. Dissertation, Graduate Department of Computer Science, Univ. Toronto, Toronto, 2009.
51. A. Tanveer, J. Taskeed, and C. Ui-Pil, “Facial expression recognition using local transitional pattern on gabor filtered facial images”, IETE Technical Review, Vol. 30, no. 1, pp. 47–52, Jan. 2013.
52. G. E. Hinton, S. Osindero, and Y. W. Teh, “A fast learning algorithm for deep belief nets,” Neural Computation, Vol. 18, no. 7, pp. 1527–54, Jul. 2006.
53. J. Luo, and A. Brodsky, “An EM-based multi-step piecewise surface regression learning algorithm,” in Proceedings of the 7th International Conference on Data Mining, Las Vegas, 2011, pp. 286–92.
54. J. Luo, A. Brodsky, and Y. Li, “An EM-based ensemble learning algorithm on piecewise surface regression problem,” International Journal of Applied Mathematics and Statistics, Vol. 28, no. 4, pp. 59–74, Aug. 2012.
55. V. Nair, and G.Hinton, “3-d object recognition with deep belief nets,” in Advances in Neural Information Processing Systems, Vol. 22, Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams and A. Culotta, Eds. Cambridge, MA: MIT Press, 2009, pp. 1339–47.
56. Y. Tang, and C. Eliasmith, “Deep networks for robust visual recognition,” in Proceedings of the 27th International Conference on Machine Learning, Haifa, 2010, pp. 1055–62.
57. A. Taralba, R. Fergus, and Y. Weiss, “Small codes and large image databases for recognition,” in Proceedings of Computer Vision and Pattern Recognition, Anchorage, 2008, pp. 1–8.
58. J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Ng, “Multimodal deep learning,” in Proceedings of the 28th International Conference on Machine Learning, Bellevue, 2011, pp. 689–96.
59. N. Srivastava, and R. Salakhutdinov, “Multimodal learning with deep Boltzmann machines,”in Advances in Neural Information Processing Systems, Vol. 25, F. Pereira, C. J. C. Burges, L. Bottou and K. Q. Weinberger, Eds. Montreal, Canada: NIPS, 2012, pp. 2222–30.
60. A. Mohamed, G. Dahl, and G. Hinton, “Deep belief networks for phone recognition,” in Proceedings of Neural Information Processing Systems 2009 Workshop on Deep Learning for Speech Recognition and Related Applications, Vancouver, 2009.
61. G. Sivaram, and H. Hermansky, “Sparse multilayer perceptron for phoneme recognition,”IEEE Trans. Audio, Speech, & Language Processing, Vol. 20, no. 1, pp. 23–9, Jan. 2012.
62. A. Mohamed, G. Dahl, and G. Hinton, “Acoustic Modeling Using Deep Belief Networks,” IEEE Trans. Audio, Speech, & Language Processing, Vol. 20, no. 1, pp. 14–22, Jan. 2012.
63. A. Mohamed, G. Hinton, and G. Penn, “Understanding how deep belief networks perform acoustic modelling,” in Proceedings of the 37th International Conference on Acoustics, Speech, and Signal Processing, Kyoto, 2012, pp. 4273–76.
64. A. Mohamed, D. Yu, and L. Deng, “Investigation of full-sequence training of deep belief networks for speech recognition,” in Proceedings of the 11th Annual Conference of the International Speech Communication Association, Makuhari, 2010, pp. 2846–9.
65. D. Yu, F. Seide, G. Li, and L. Deng, “Exploiting sparseness in deep neural networks for large vocabulary speech recognition,” in Proceedings of the 37th International Conference on Acoustics, Speech, and Signal Processing, Kyoto, 2012, pp. 4409–12.
66. D. Yu, S. Wang, Z. Karam, and L. Deng, “Language recognition using deep-structured conditional random fields,” in Proceedings of the 35th International Conference on Acoustics, Speech and Signal Processing, 2010, pp. 5030–3.
67. F. Seide, G. Li, X. Chen, and D. Yu, “Feature engineering in context-dependent deep neural networks for conversational speech transcription,” in Proceedings of the 2011 IEEE Workshop on Automatic Speech Recognition and Understanding, Hawaii, 2011, pp. 24–9.
68. G. Dahl, D. Yu, L. Deng, and A. Acero, “Context-dependent DBN-HMMs in large vocabulary continuous speech recognition,” in Proceedings of the 36th International Conference on Acoustics, Speech, and Signal Processing, Prague, 2011, pp. 4688–91.
69. G. Dahl, D. Yu, L. Deng, and A. Acero, “Context-dependent, pre-trained deep neural networks for large vocabulary speech recognition,” IEEE Trans. Audio, Speech, & Language Proc., Vol. 20, no. 1, pp. 30–42, Jan. 2012.
70. Y. Kubo, T. Hori, and A. Nakamura, “Integrating deep neural networks into structural classification approach based on weighted finite-state transducers,” in Proceedings of the 13th Annual Conference of the International Speech Communication Association, Portland, 2012.
71. L. Deng, J. Li, K. Huang, D. Yao, F. Yu, M. Seide, G. Seltzer, X. Zweig, J. He, Y. Williams, and A. Acero. “Recent advances in deep learning for speech research at Microsoft,” in Proceedings of International Conference on Acoustics, Speech and Signal Processing, Vancouver, 2013, pp. 8604–8.
72. G. Hinton, and R. Salakhutdinov, “Discovering binary codes for documents by learning deep generative models,” Topics in Cognitive Science, Vol. 3, no. 1, pp. 74–91, Jan. 2011.
73. R. Salakhutdinov, and G. Hinton, “Semantic hashing,” International Journal of Approximate Reasoning, Vol. 50, no. 7, pp. 969–78, Jul. 2009.
74. N. Srivastava, R. Salakhutdinov, and G. E. Hinton, “Modeling documents with deep Boltzmann machines,” in Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence, Bellevue, 2013, pp. 616–24.
75. W. Fang, W. Pan, and Z. Cui, “View of MapReduce: Programming model, methods, and its applications”, IETE Technical Review, Vol. 29, no. 5, pp. 380–7, Sept. 2012.