Unsupervised learning of hierarchical representations with convolutional deep belief networks

Communications of the ACM

Volumn 54, Issue 10, 2011, Pages 95-103

  Lee, Honglak ^a   Grosse, Roger ^b   Ranganath, Rajesh ^c   Ng, Andrew Y ^c  

^a UNIVERSITY OF MICHIGAN   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^c Stanford University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BOTTOM-UP AND TOP-DOWN; EXCELLENT PERFORMANCE; GENERATIVE MODEL; HIERARCHICAL REPRESENTATION; HIGH-DIMENSIONAL IMAGES; NATURAL SCENES; NOVEL TECHNIQUES; PROBABILISTIC INFERENCE; REALISTIC IMAGES; TRANSLATION INVARIANTS; VISUAL FEATURE; VISUAL RECOGNITION;

CONVOLUTION; UNSUPERVISED LEARNING;

BAYESIAN NETWORKS;

EID: 80053540444     PISSN: 00010782     EISSN: 15577317     Source Type: Journal    
DOI: 10.1145/2001269.2001295     Document Type: Article

References (38)

1. Bell, A.J., Sejnowski, T.J. The 'independent components' of natural scenes are edge filters. Vis. Res. 37, 23 (1997), 3327-3338. (Pubitemid 27493806)
2. Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H. Greedy layerwise training of deep networks. In Advances in Neural Information Processing Systems, 2007.
3. Berg, A.C., Berg, T.L., Malik, J. Shape matching and object recognition using low distortion correspondence. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition, 2005.
4. Desjardins, G., Bengio, Y. Empirical evaluation of convolutional RBMs for vision. Technical report, University of Montreal, Monreal, Quebec, Canada, 2008.
5. Fei-Fei, L., Fergus, R., Perona, P. Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories. In CVPRWorkshop on Generative Model Based Vision, 2004.
6. Gehler, P., Nowozin, S. On feature combination for multiclass object classification. In Proceedings of the International Conference on Computer Vision, 2009.
7. Grosse, R., Raina, R., Kwong, H., Ng, A.Y. Shift-invariant sparse coding for audio classification. In Proceedings of the Conference on Uncertainty in Artificial Intelligence, 2007.
8. Hinton, G.E. Training products of experts by minimizing contrastive divergence. Neural Comput. 14, 8 (2002), 1771-1800.
9. Hinton, G.E., Osindero, S., Bao, K. Learning causally linked MRFs. In Proceedings of the International Conference on Artificial Intelligence and Statistics, 2005.
10. Hinton, G.E., Osindero, S., Teh, Y.-W. A fast learning algorithm for deep belief nets. Neural Comput. 18, 7 (2006), 1527-1554. (Pubitemid 44024729)
11. Hinton, G.E., Salakhutdinov, R. Reducing the dimensionality of data with neural networks. Science 313, 5786 (2006), 504-507. (Pubitemid 44148451)
12. Hyvarinen, A., Gutmann, M., Hoyer, P.O. Statistical model of natural stimuli predicts edge-like pooling of spatial frequency channels in V2. BMC Neurosci. 6 (2005), 12.
13. I to, M., Komatsu, H. Representation of angles embedded within contour stimuli in area V2 of macaque monkeys. J. Neurosci. 24, 13 (2004), 3313-3324. (Pubitemid 38451897)
14. Koller, D., Friedman, N. Probabilistic Graphical Models: Principles and Techniques. The MITPress, Cambridge, MA, 2009.
15. Larochelle, H., Erhan, D., Courville, A., Bergstra, J., Bengio, Y. An empirical evaluation of deep architectures on problems with many factors of variation. In Proceedings of the International Conference on Machine Learning, 2007.
16. Lazebnik, S., Schmid, C., Ponce, J. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition, 2006.
17. LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D. Backpropagation applied to handwritten zip code recognition. Neural Comput. 1 (1989), 541-551.
18. Lee, H., Ekanadham, C., Ng, A.Y. Sparse deep belief network model for visual area V2. In Advances in Neural Information Processing Systems 20, 2008.
19. Lee, T.S., Mumford, D. Hierarchical Bayesian inference in the visual cortex. J. Opt. Soc. Am. A 20, 7 (2003), 1434-1448.
20. Mutch, J., Lowe, D.G. Multiclass object recognition with sparse, localized features. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition, 2006.
21. Norouzi, M., Ranjbar, M., Mori, G. Stacks of convolutional restricted Boltzmann machines for shiftinvariant feature learning. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition, 2009.
22. Olshausen, B.A., Field, D.J. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381 (1993), 607-609. (Pubitemid 26177476)
23. Osindero, S., Welling, M., Hinton, G.E. Topographic product models applied to natural scene statistics. Neural Comput. 18, 2 (2006), 381-344.
24. Raina, R., Battle, A., Lee, H., Packer, B., Ng, A.Y. Self-taught learning: Transfer learning from unlabeled data. In Proceedings of the International Conference on Machine Learning, 2007.
25. Raina, R., Madhavan, A., Ng, A.Y. Large-scale deep unsupervised learning using graphics processors. In Proceedings of the International Conference on Machine Learning, 2009.
26. Ranzato, M., Boureau, Y., LeCun, Y. Sparse feature learning for deep belief networks. In Advances in Neural Information Processing Systems, 2007.
27. Ranzato, M., Huang, F.-J., Boureau, Y.-L., LeCun, Y. Unsupervised learning of invariant feature hierarchies with applications to object recognition. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition, 2007.
28. Ranzato, M., Poultney, C., Chopra, S., LeCun, Y. Efficient learning of sparse representations with an energybased model. In Advances in Neural Information Processing Systems (2006), 1137-1144, 2006.
29. Salakhutdinov, R., Hinton, G.E. Deep Boltzmann machines. In Proceedings of the International Conference on Artificial Intelligence and Statistics, 2009.
30. Salakhutdinov, R., Mnih, A., Hinton, G. Restricted Boltzmann machines for collaborative filtering. In Proceedings of the International Conference on Machine learning, 2007.
31. Taylor, G.W., Hinton, G.E., Roweis, S.T. Modeling human motion using binary latent variables. In Advances in Neural Information Processing Systems 19, 2007.
32. Tieleman, T. Training restricted Boltzmann machines using approximations to the likelihood gradient. In Proceedings of the International Conference on Machine Learning, 2008.
33. van Hateren, J.H., van der Schaaf, A. Independent component filters of natural images compared with simple cells in primary visual cortex. Proc. R. Soc. B265 (1998), 359-366. (Pubitemid 28111919)
34. Weston, J., Ratle, F., Collobert, R. Deep learning via semi-supervised embedding. In Proceedings of the International Conference on Machine Learning, 2008.
35. Younes, L. Maximum of likelihood estimation for Gibbsian fields. Probab. Theory Relat. Fields 82 (1989), 625-645.
36. Yu, K., Xu, W., Gong, Y. Deep learning with kernel regularization for visual recognition. In Advances in Neural Information Processing Systems, 2009.
37. Yuille, A.L. The convergence of contrastive divergences. In Advances in Neural Information Processing Systems 17, 2005.
38. Zhang, H., Berg, A.C., Maire, M., Malik, J. SVM-KNN: Discriminative nearest neighbor classification for visual category recognition. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition, 2006.