Learning Contextual Dependence With Convolutional Hierarchical Recurrent Neural Networks

IEEE Transactions on Image Processing

Volumn 25, Issue 7, 2016, Pages 2983-2996

  Zuo, Zhen ^a,b   Shuai, Bing ^a,b   Wang, Gang ^a,b   Liu, Xiao ^a,b   Wang, Xingxing ^a,b   Wang, Bing ^a,b   Chen, Yushi ^a,b  

^a NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^b HARBIN INSTITUTE OF TECHNOLOGY   (China)

Author keywords

Convolutional Neural Networks; Deep Learning; Image Classification; Recurrent Neural Networks

Indexed keywords

C (PROGRAMMING LANGUAGE); CONVOLUTION; COSTS; ENCODING (SYMBOLS); NEURAL NETWORKS; RECURRENT NEURAL NETWORKS;

CONTEXTUAL INFORMATION; CONVOLUTIONAL NEURAL NETWORK; DEEP LEARNING; IMAGE REPRESENTATIONS; LONG SHORT TERM MEMORY; RECURRENT NEURAL NETWORK (RNNS); REPRESENTATION POWER; SIMPLE RECURRENT NETWORKS;

IMAGE CLASSIFICATION;

EID: 84971671959     PISSN: 10577149     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIP.2016.2548241     Document Type: Article

References (72)

1. Y. Le Cun et al., "Handwritten digit recognition with a back-propagation network," in Proc. NIPS, 1990, pp. 396-404.
2. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks," in Proc. NIPS, 2012, pp. 1097-1105.
3. B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva, "Learning deep features for scene recognition using places database," in Proc. NIPS, 2014, pp. 487-495.
4. Y. Gong, L. Wang, R. Guo, and S. Lazebnik, "Multi-scale orderless pooling of deep convolutional activation features," in Proc. ECCV, 2014, pp. 392-407.
5. K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman, "Return of the devil in the details: Delving deep into convolutional nets," in Proc. British Mach. Vis. Conf., 2014.
6. C. Szegedy et al. (2014). "Going deeper with convolutions." [Online]. Available: http://arxiv.org/abs/1409.4842
7. K. Simonyan and A. Zisserman. (2014). "Very deep convolutional networks for large-scale image recognition." [Online]. Available: http://arxiv.org/abs/1409.1556
8. K. He, X. Zhang, S. Ren, and J. Sun, "Spatial pyramid pooling in deep convolutional networks for visual recognition," in Proc. ECCV, 2014, pp. 1-14.
9. R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation," in Proc. CVPR, 2014, pp. 580-587.
10. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. (2013). "OverFeat: Integrated recognition, localization and detection using convolutional networks." [Online]. Available: http://arxiv.org/abs/1312.6229
11. W. Ouyang et al. (2014). "DeepID-Net: Multi-stage and deformable deep convolutional neural networks for object detection." [Online]. Available: http://arxiv.org/abs/1409.3505
12. Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, "DeepFace: Closing the gap to human-level performance in face verification," in Proc. CVPR, Jun. 2014, pp. 1701-1708.
13. Y. Sun, X. Wang, and X. Tang, "Deep learning face representation from predicting 10,000 classes," in Proc. CVPR, 2014, pp. 1891-1898.
14. A. Toshev and C. Szegedy, "DeepPose: Human pose estimation via deep neural networks," in Proc. CVPR, 2014, pp. 1653-1660.
15. T. Mikolov, "Statistical language models based on neural networks," Ph.D. dissertation, Dept. Comput. Graph. Multimedia, Brno Univ. Technol., Brno, Czech Republic, 2012.
16. I. Sutskever, "Training recurrent neural networks," Ph.D. dissertation, Dept. Comput. Sci., Univ. Toronto, Toronto, ON, Canada, 2013.
17. J. Koutník, K. Greff, F. Gomez, and J. Schmidhuber, "A clockwork RNN," in Proc. Int. Conf. Mach. Learn. (ICML), vol. 32. Beijing, China, 2014, pp. 1863-1871.
18. A. Graves and N. Jaitly, "Towards end-to-end speech recognition with recurrent neural networks," in Proc. ICML, 2014, pp. 1764-1772.
19. I. Sutskever, G. E. Hinton, and G. W. Taylor, "The recurrent temporal restricted Boltzmann machine," in Proc. NIPS, 2009, pp. 1601-1608.
20. N. Boulanger-Lewandowski, Y. Bengio, and P. Vincent, "Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription," in Proc. ICML, 2012, pp. 1159-1166.
21. J. L. Elman, "Finding structure in time," Cognit. Sci., vol. 14, no. 2, pp. 179-211, Mar. 1990.
22. H. Jaeger, Tutorial on Training Recurrent Neural Networks, Covering BPPT, RTRL, EKF and the 'Echo State Network' Approach. German Nat. Res. Center Inf. Technol., GMD Rep. 159, 2002, pp. 1-48.
23. A. Graves and J. Schmidhuber, "Offline handwriting recognition with multidimensional recurrent neural networks," in Proc. NIPS, 2009, pp. 545-552.
24. Z. Zuo et al., "Convolutional recurrent neural networks: Learning spatial dependencies for image representation," in Proc. CVPRW, Jun. 2015, pp. 18-26.
25. M. Oquab, L. Bottou, I. Laptev, and J. Sivic, "Learning and transferring mid-level image representations using convolutional neural networks," in Proc. CVPR, 2014, pp. 1717-1724.
26. J. Donahue et al., "DeCAF: A deep convolutional activation feature for generic visual recognition," in Proc. Int. Conf. Mach. Learn. (ICML), 2014, pp. 647-655.
27. G. E. Hinton, S. Osindero, and Y.-W. Teh, "A fast learning algorithm for deep belief nets," Neural Comput., vol. 18, no. 7, pp. 1527-1554, Jul. 2006.
28. H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng, "Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations," in Proc. ICML, 2009, pp. 609-616.
29. V. Nair and G. E. Hinton, "3D object recognition with deep belief nets," in Proc. NIPS, 2009, pp. 1339-1347.
30. G. E. Hinton and R. R. Salakhutdinov, "Reducing the dimensionality of data with neural networks," Science, vol. 313, no. 5786, pp. 504-507, 2006.
31. X. Yan, H. Chang, S. Shan, and X. Chen, "Modeling video dynamics with deep dynencoder," in Proc. ECCV, 2014, pp. 215-230.
32. J. Zhang, S. Shan, M. Kan, and X. Chen, "Coarse-to-fine auto-encoder networks (CFAN) for real-time face alignment," in Proc. ECCV, 2014, pp. 1-16.
33. Y. Bengio, L. Yao, G. Alain, and P. Vincent, "Generalized denoising auto-encoders as generative models," in Proc. NIPS, 2013, pp. 899-907.
34. Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," Nature, vol. 521, no. 7553, pp. 436-444, May 2015.
35. I. Sutskever, J. Martens, and G. E. Hinton, "Generating text with recurrent neural networks," in Proc. ICML, 2011, pp. 1017-1024.
36. A. Graves. (2013). "Generating sequences with recurrent neural networks." [Online]. Available: http://arxiv.org/abs/1308.0850
37. P. O. Pinheiro and R. Collobert, "Recurrent convolutional neural networks for scene labeling," in Proc. ICML, 2014, pp. 1-9.
38. J. T. Rolfe and Y. LeCun. (2013). "Discriminative recurrent sparse autoencoders." [Online]. Available: http://arxiv.org/abs/1301.3775
39. D. Eigen, J. Rolfe, R. Fergus, and Y. LeCun. (2013). "Understanding deep architectures using a recursive convolutional network." [Online]. Available: http://arxiv.org/abs/1312.1847
40. K. Gregor, I. Danihelka, A. Graves, and D. Wierstra. (2015). "DRAW: A recurrent neural network for image generation." [Online]. Available: http://arxiv.org/abs/1502.04623
41. V. Mnih, N. Heess, A. Graves, and K. Kavukcuoglu, "Recurrent models of visual attention," in Proc. NIPS, 2014, pp. 2204-2212.
42. J. Donahue et al. (2014). "Long-term recurrent convolutional networks for visual recognition and description." [Online]. Available: http://arxiv.org/abs/1411.4389
43. J. Y.-H. Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici. (2015). "Beyond short snippets: Deep networks for video classification." [Online]. Available: http://arxiv.org/abs/1503.08909
44. X. Chen and C. L. Zitnick. (2014). "Learning a recurrent visual representation for image caption generation." [Online]. Available: http://arxiv.org/abs/1411.5654
45. A. Karpathy and L. Fei-Fei. (2014). "Deep visual-semantic alignments for generating image descriptions." [Online]. Available: http://arxiv.org/abs/1412.2306
46. S. Venugopalan, H. Xu, J. Donahue, M. Rohrbach, R. Mooney, and K. Saenko. (2014). "Translating videos to natural language using deep recurrent neural networks." [Online]. Available: http://arxiv.org/abs/1412.4729
47. J. Mao, W. Xu, Y. Yang, J. Wang, Z. Huang, and A. Yuille. (2014). "Deep captioning with multimodal recurrent neural networks (m-RNN)." [Online]. Available: http://arxiv.org/abs/1412.6632
48. F. Visin, K. Kastner, K. Cho, M. Matteucci, A. Courville, and Y. Bengio. (2015). "ReNet: A recurrent neural network based alternative to convolutional networks." [Online]. Available: http://arxiv.org/abs/1505.00393
49. Y. Du, W. Wang, and L. Wang, "Hierarchical recurrent neural network for skeleton based action recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2015, pp. 1110-1118.
50. R. Lin, S. Liu, M. Yang, M. Li, M. Zhou, and S. Li, "Hierarchical recurrent neural network for document modeling," in Proc. Conf. Empirical Methods Natural Lang. Process., 2015, pp. 899-907.
51. S. El Hihi and Y. Bengio, "Hierarchical recurrent neural networks for long-term dependencies," in Proc. NIPS, 1995, pp. 493-499.
52. J. Yu, Y. Rui, Y. Y. Tang, and D. Tao, "High-order distance-based multiview stochastic learning in image classification," IEEE Trans. Cybern., vol. 44, no. 12, pp. 2431-2442, Dec. 2014.
53. J. Yu, D. Tao, and M. Wang, "Adaptive hypergraph learning and its application in image classification," IEEE Trans. Image Process., vol. 21, no. 7, pp. 3262-3272, Jul. 2012.
54. J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong, "Localityconstrained Linear Coding for image classification," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2010, pp. 3360-3367.
55. M. Yang, L. Zhang, X. Feng, and D. Zhang, "Fisher discrimination dictionary learning for sparse representation," in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Nov. 2011, pp. 543-550.
56. M. D. Zeiler and R. Fergus. (2013). "Visualizing and understanding convolutional networks." [Online]. Available: http://arxiv.org/abs/1311.2901
57. D. G. Lowe, "Distinctive image features from scale-invariant keypoints," Int. J. Comput. Vis., vol. 60, no. 2, pp. 91-110, 2004.
58. N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection," in Proc. CVPR, Jun. 2005, pp. 886-893.
59. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Comput., vol. 9, no. 8, pp. 1735-1780, 1997.
60. Y. Jia et al. (2014). "Caffe: Convolutional architecture for fast feature embedding." [Online]. Available: http://arxiv.org/abs/1408.5093
61. J. Deng, W. Dong, R. Socher, L. J. Li, K. Li, and L. Fei-Fei, "ImageNet: A large-scale hierarchical image database," in Proc. CVPR, Jun. 2009, pp. 248-255.
62. J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, "SUN database: Large-scale scene recognition from abbey to zoo," in Proc. CVPR, Jun. 2010, pp. 3485-3492.
63. A. Quattoni and A. Torralba, "Recognizing indoor scenes," in Proc. CVPR, Jun. 2009, pp. 413-420.
64. J. Sánchez, F. Perronnin, T. Mensink, and J. Verbeek, "Image classification with the Fisher vector: Theory and practice," Int. J. Comput. Vis., vol. 105, no. 3, pp. 222-245, Dec. 2013.
65. J. Sánchez, F. Perronnin, T. Mensink, and J. Verbeek, "Image classification with the Fisher vector: Theory and practice," Int. J. Comput. Vis., vol. 105, no. 3, pp. 222-245, Dec. 2013.
66. M. Lapin, B. Schiele, and M. Hein, "Scalable multitask representation learning for scene classification," in Proc. CVPR, Jun. 2014, pp. 1434-1441.
67. L.-J. Li, H. Su, L. Fei-Fei, and E. P. Xing, "Object bank: A highlevel image representation for scene classification & semantic feature sparsification," in Proc. NIPS, 2010, pp. 1378-1386.
68. Q. Li, J. Wu, and Z. Tu, "Harvesting mid-level visual concepts from large-scale Internet images," in Proc. CVPR, Jun. 2013, pp. 851-858.
69. X. Wang, B. Wang, X. Bai, W. Liu, and Z. Tu, "Max-margin multipleinstance dictionary learning," in Proc. ICML, 2013, pp. 846-854.
70. M. Juneja, A. Vedaldi, C. V. Jawahar, and A. Zisserman, "Blocks that shout: Distinctive parts for scene classification," in Proc. CVPR, Jun. 2013, pp. 923-930.
71. C. Doersch, A. Gupta, and A. A. Efros, "Mid-level visual element discovery as discriminative mode seeking," in Proc. NIPS, 2013, pp. 494-502.
72. D. Lin, C. Lu, R. Liao, and J. Jia, "Learning important spatial pooling regions for scene classification," in Proc. CVPR, Jun. 2014, pp. 3726-3733.