DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 40, Issue 4, 2018, Pages 834-848

  Chen, Liang Chieh ^a   Papandreou, George ^a   Kokkinos, Iasonas ^b   Murphy, Kevin ^a   Yuille, Alan L ^c  

^a GOOGLE INC   (United States)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

^c JOHNS HOPKINS UNIVERSITY   (United States)

Author keywords

atrous convolution; conditional random fields; Convolutional neural networks; semantic segmentation

Indexed keywords

CONVOLUTION; DEEP LEARNING; DEEP NEURAL NETWORKS; NEURAL NETWORKS; RANDOM PROCESSES; SEMANTICS;

CONDITIONAL RANDOM FIELD; CONVOLUTIONAL NEURAL NETWORK; DEEP CONVOLUTIONAL NEURAL NETWORKS; LOCALIZATION ACCURACY; LOCALIZATION PERFORMANCE; PROBABILISTIC GRAPHICAL MODELS; SEMANTIC IMAGE SEGMENTATIONS; SEMANTIC SEGMENTATION;

IMAGE SEGMENTATION;

EID: 85042712042     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2017.2699184     Document Type: Article

References (102)

1. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition, " in Proc. IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998.
2. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classi-fication with deep convolutional neural networks, " in Proc. 25th Int. Conf. Neural Inf. Process. Syst., 2013, pp. 1097-1105.
3. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, "OverFeat: Integrated recognition, localization and detection using convolutional networks, " arXiv:1312.6229, 2013.
4. K. Simonyan and A. Zisserman, "Very deep convolutional net-works for large-scale image recognition, " in Proc. Int. Conf. Learn. Representations, 2015.
5. C. Szegedy, et al., "Going deeper with convolutions, " arXiv:1409.4842, 2014.
6. G. Papandreou, I. Kokkinos, and P.-A. Savalle, "Modeling local and global deformations in deep learning: Epitomic convolution, multiple instance learning, and sliding window detection, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 390-399.
7. R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 580-587.
8. D. Erhan, C. Szegedy, A. Toshev, and D. Anguelov, "Scalable object detection using deep neural networks, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 2155-2162.
9. R. Girshick, "Fast R-CNN, " in Proc. IEEE Int. Conf. Comput. Vis., 2015.
10. S. Ren, K. He, R. Girshick, and J. Sun, "Faster R-CNN: Towards real-time object detection with region proposal networks, " in Proc. 28th Int. Conf. Neural Inf. Process. Syst., 2015, pp. 91-99.
11. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition, " arXiv:1512.03385, 2015.
12. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, and S. Reed, "SSD: Single shot multibox detector, " arXiv:1512.02325, 2015.
13. M. D. Zeiler and R. Fergus, "Visualizing and understanding convolutional networks, " in Proc. Eur. Conf. Comput. Vis., 2014, pp. 818-833.
14. J. Long, E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation, " in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2015.
15. M. Holschneider, R. Kronland-Martinet, J. Morlet, and P. Tcha-mitchian, "A real-time algorithm for signal analysis with the help of the wavelet transform, " in Proc. Wavelets: Time-Frequency Methods Phase Space, 1989, pp. 289-297.
16. A. Giusti, D. Ciresan, J. Masci, L. Gambardella, and J. Schmid-huber, "Fast image scanning with deep max-pooling convolutional neural networks, " in Proc. IEEE Int. Conf. Image Process., 2013, pp. 4034-4038.
17. L.-C. Chen, Y. Yang, J. Wang, W. Xu, and A. L. Yuille, "Attention to scale: Scale-aware semantic image segmentation, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2016.
18. I. Kokkinos, "Pushing the boundaries of boundary detection using deep learning, " in Proc. Int. Conf. Learn. Representations, 2016.
19. S. Lazebnik, C. Schmid, and J. Ponce, "Beyond bags of features: Spatial pyramid matching for recognizing natural scene catego-ries, " in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recog., 2006, pp. 2169-2178.
20. K. He, X. Zhang, S. Ren, and J. Sun, "Spatial pyramid pooling in deep convolutional networks for visual recognition, " in Proc. Eur. Conf. Comput. Vis., 2014, pp. 346-361.
21. B. Hariharan, P. Arbelaez, R. Girshick, and J. Malik, "Hypercolumns for object segmentation and fine-grained local-ization, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 447-456.
22. P. Krahenbuhl and V. Koltun, "Efficient inference in fully con-nected CRFs with Gaussian edge potentials, " in Proc. Advances Neural Inf. Process. Syst., 2011, pp. 109-117
23. C. Rother, V. Kolmogorov, and A. Blake, "GrabCut: Interactive foreground extraction using iterated graph cuts, " in Proc. ACM SIGGRAPH, 2004, pp. 309-314.
24. J. Shotton, J. Winn, C. Rother, and A. Criminisi, "Textonboost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context, " Int. J. Comput. Vis., vol. 81, pp. 2-23, 2009.
25. A. Lucchi, Y. Li, X. Boix, K. Smith, and P. Fua, "Are spatial and global constraints really necessary for segmentation?" in Proc. Int. Conf. Comput. Vis., 2011, pp. 9-16.
26. X. He, R. S. Zemel, and M. Carreira-Perpindn, "Multiscale conditional random fields for image labeling, " in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recog., 2004, pp. 695-703.
27. L. Ladicky, C. Russell, P. Kohli, and P. H. Torr, "Associative hier-archical CRFs for object class image segmentation, " in Proc. Int. Conf. Comput. Vis., 2009, pp. 739-746.
28. V. Lempitsky, A. Vedaldi, and A. Zisserman, "Pylon model for semantic segmentation, " in Proc. Advances Neural Inf. Process. Syst., 2011, pp. 1485-1493.
29. A. Delong, A. Osokin, H. N. Isack, and Y. Boykov, "Fast approxi-mate energy minimization with label costs, " Int. J. Comput. Vis., vol. 96, pp. 1-27, 2012.
30. J. M. Gonfaus, X. Boix, J. Van de Weijer, A. D. Bagdanov, J. Serrat, and J. Gonzalez, "Harmony potentials for joint classification and segmentation, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2010.
31. P. Kohli, P. H. Torr, and L. Ladick, "Robust higher order potentials for enforcing label consistency, " Int. J. Comput. Vis., vol. 82, no. 3, pp. 302-324, 2009.
32. L.-C. Chen, G Papandreou, and A. Yuille, "Learning a dictionary of shape epitomes with applications to image labeling, " in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 337-344.
33. P. Wang, X. Shen, Z. Lin, S. Cohen, B. Price, and A. Yuille, "Towards unified depth and semantic prediction from a single image, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 2800-2809.
34. M. Everingham, S. M. A. Eslami, L. V. Gool, C. K. I. Williams, J. Winn, and A. Zisserma, "The PASCAL visual object classes challenge a retrospective, " Int. J. Comput. Vis., 2014.
35. R. Mottaghi, et al., "The role of context for object detection and semantic segmentation in the wild, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 891-898.
36. X. Chen, R. Mottaghi, X. Liu, S. Fidler, R. Urtasun, and A. Yuille, "Detect what you can: Detecting and representing objects using holistic models and body parts, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014.
37. M. Cordts, et al., "The cityscapes dataset for semantic urban scene understanding, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2016.
38. L.-C. Chen, G Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, "Semantic image segmentation with deep convolutional nets and fully connected CRFs, " in Proc. Int. Conf. Learn. Represen-tations, 2015.
39. C. Farabet, C. Couprie, L. Najman, and Y. LeCun, "Learning hierarchical features for scene labeling, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 8, pp. 1915-1929, Aug. 2013.
40. G Lin, C. Shen, I. Reid et al., "Efficient piecewise training of deep structured models for semantic segmentation, " arXiv:1504.01013, 2015.
41. Y. Jia, et al., "Caffe: Convolutional architecture for fast feature embedding, " arXiv:1408.5093, 2014.
42. Z. Tu and X. Bai, "Auto-context and its application to high-level vision tasks and 3D brain image segmentation, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 10, pp. 1744-1757, Oct. 2010.
43. J. Shotton, M. Johnson, and R. Cipolla, "Semantic texton forests for image categorization and segmentation, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2008, pp. 1-8.
44. B. Fulkerson, A. Vedaldi, and S. Soatto, "Class segmentation and object localization with superpixel neighborhoods, " in Proc. IEEE 12th Int. Conf. Comput. Vis., 2009, pp. 670-677.
45. J. Carreira, R. Caseiro, J. Batista, and C. Sminchisescu, "Semantic segmentation with second-order pooling, " in Proc. Eur. Conf. Comput. Vis., 2012, pp. 430-443.
46. J. Carreira and C. Sminchisescu, "CPMC: Automatic object seg-mentation using constrained parametric min-cuts, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 7, pp. 1312-1328, Jul. 2012.
47. P. Arbelaez, J. Pont-Tuset, J. T. Barron, F. Marques, and J. Malik, "Multiscale combinatorial grouping, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 328-335.
48. J. Uijlings, K. van de Sande, T. Gevers, and A. Smeulders, "Selective search for object recognition, " Int. J. Comput. Vis., vol. 104, pp. 154-171, 2013.
49. B. Hariharan, P. Arbelaez, R. Girshick, and J. Malik, "Simultaneous detection and segmentation, " in Proc. Eur. Conf. Comput. Vis., 2014, pp. 297-312.
50. M. Mostajabi, P. Yadollahpour, and G Shakhnarovich, "Feedforward semantic segmentation with zoom-out features, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 3376-3385.
51. J. Dai, K. He, and J. Sun, "Convolutional feature masking for joint object and stuff segmentation, " arXiv:1412.1283, 2014.
52. D. Eigen and R. Fergus, "Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture, " arXiv:1411.4734, 2014.
53. M. Cogswell, X. Lin, S. Purushwalkam, and D. Batra, "Combining the best of graphical models and convnets for semantic segmentation, " arXiv:1412.4313, 2014.
54. D. Geiger and F. Girosi, "Parallel and deterministic algorithms from MRFs: Surface reconstruction, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 13, no. 5, pp. 401-412, May 1991.
55. D. Geiger and A. Yuille, "A common framework for image segmentation, " Int. J. Comput. Vis., vol. 6, no. 3, pp. 227-243, 1991.
56. I. Kokkinos, R. Deriche, O. Faugeras, and P. Maragos, "Computational analysis and learning for a biologically motivated model of boundary detection, " Neurocomputing, vol. 71, no. 10, pp. 1798-1812, 2008.
57. S. Bell, P. Upchurch, N. Snavely, and K. Bala, "Material recogni-tion in the wild with the materials in context database, " arXiv:1412.0623, 2014.
58. G Papandreou, L.-C. Chen, K. Murphy, and A. L. Yuille, "Weakly-and semi-supervised learning of a DCNN for semantic image segmentation, " in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 1742-1750.
59. S. Zheng, et al., "Conditional random fields as recurrent neural networks, " in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 1529-1537.
60. J. Dai, K. He, and J. Sun, "Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation, " in Proc. Int. Conf. Comput. Vis., 2015.
61. H. Noh, S. Hong, and B. Han, "Learning deconvolution network for semantic segmentation, " in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 1520-1528.
62. Z. Liu, X. Li, P. Luo, C. C. Loy, and X. Tang, "Semantic image segmentation via deep parsing network, " in Proc. Int. Conf. Comput. Vis., 2015, pp. 1377-1385.
63. L.-C. Chen, J. T. Barron, G. Papandreou, K. Murphy, and A. L. Yuille, "Semantic image segmentation with task-specific edge detection using cnns and a discriminatively trained domain transform, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2016, pp. 4545-4554.
64. L.-C. Chen, A. Schwing, A. Yuille, and R. Urtasun, "Learning deep structured models, " in Proc. 32nd Int. Conf. Int. Conf. Mach. Learn., 2015, pp. 1785-1794.
65. A. G. Schwing and R. Urtasun, "Fully connected deep structured networks, " arXiv:1503.02351, 2015.
66. S. Chandra and I. Kokkinos, "Fast, exact and multi-scale inference for semantic image segmentation with deep Gaussian CRFs, " arXiv:1603.08358, 2016.
67. E. S. L. Gastal and M. M. Oliveira, "Domain transform for edge-aware image and video processing, " in Proc. ACM SIGGRAPH, 2011, Art. no. 69.
68. G Bertasius, J. Shi, and L. Torresani, "High-for-low and low-for-high: Efficient boundary detection from deep object features and its applications to high-level vision, " in Proc. IEEE Int. Conf. Comput. Vis., 2015.
69. P. O. Pinheiro and R. Collobert, "Weakly supervised semantic segmentation with convolutional networks, " arXiv:1411.6228, 2014.
70. D. Pathak, P. Krahenbuhl, and T. Darrell, "Constrained convolu-tional neural networks for weakly supervised segmentation, " Proc. IEEE Int. Conf. Comput. Vis., 2015.
71. S. Hong, H. Noh, and B. Han, "Decoupled deep neural network for semi-supervised semantic segmentation, " in Proc. 28th Int. Conf. Neural Inf. Process. Syst., 2015, pp. 1495-1503.
72. A. Vezhnevets, V. Ferrari, and J. M. Buhmann, "Weakly super-vised semantic segmentation with a multi-image model, " in Proc. Int. Conf. Comput. Vis., 2011, pp. 643-650.
73. X. Liang, Y. Wei, X. Shen, J. Yang, L. Lin, and S. Yan, "Proposal-free network for instance-level object segmentation, " arXiv:1509.02636, 2015.
74. J. E. Fowler, "The redundant discrete wavelet transform and additive noise, " IEEE Signal Process. Lett., vol. 12, no. 9, pp. 629-632, Sep. 2005.
75. P. P. Vaidyanathan, "Multirate digital filters, filter banks, polyphase networks, and applications: a tutorial, " Proc. IEEE, vol. 78, no. 1, pp. 56-93, Jan. 1990.
76. F. Yu and V. Koltun, "Multi-scale context aggregation by dilated convolutions, " in Proc. Int. Conf. Learn. Representations, 2016.
77. J. Dai, Y. Li, K. He, and J. Sun, "R-FCN: Object detection via region-based fully convolutional networks, " arXiv:1605.06409, 2016.
78. J. Dai, K. He, Y. Li, S. Ren, and J. Sun, "Instance-sensitive fully convolutional networks, " arXiv:1603.08678, 2016.
79. K. Chen, J. Wang, L.-C. Chen, H. Gao, W. Xu, and R. Nevatia, "ABC-CNN: An attention based convolutional neural network for visual question answering, " arXiv:1511.05960, 2015.
80. L. Sevilla-Lara, D. Sun, V. Jampani, and M. J. Black, "Optical flow with semantic segmentation and localized layers, " arXiv:1603.03911, 2016.
81. Z. Wu, C. Shen, and A. van den Hengel, "High-performance semantic segmentation using very deep fully convolutional networks, " arXiv:1604.04339, 2016.
82. M. J. Shensa, "The discrete wavelet transform: Wedding the a trous and Mallat algorithms, " IEEE Trans. Signal Process., vol. 40, no. 10, pp. 2464-2482, Oct. 1992.
83. M. Abadi, A. Agarwal et al., "Tensorflow: Large-scale machine learning on heterogeneous distributed systems, " arXiv:1603.04467, 2016.
84. A. Adams, J. Baek, and M. A. Davis, "Fast high-dimensional filtering using the permutohedral lattice, " in Eurographics, vol. 29, pp. 753-762, 2010.
85. B. Hariharan, P. Arbelaez, L. Bourdev, S. Maji, and J. Malik, "Semantic contours from inverse detectors, " in Proc. IEEE Int. Conf. Comput. Vis., 2011, pp. 991-998.
86. W. Liu, A. Rabinovich, and A. C. Berg, "Parsenet: Looking wider to see better, " arXiv:1506.04579, 2015.
87. T.-Y. Lin, et al., "Microsoft COCO: Common objects in context, " in Proc. Eur. Conf. Comput. Vis., 2014, pp. 740-755.
88. R. Vemulapalli, O. Tuzel, M.-Y. Liu, and R. Chellappa, "Gaussian conditional random field network for semantic segmentation, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2016, pp. 3224-3233.
89. Z. Yan, H. Zhang, Y. Jia, T. Breuel, and Y. Yu, "Combining the best of convolutional layers and recurrent layers: A hybrid network for semantic segmentation, " arXiv:1603.04871, 2016.
90. G. Ghiasi and C. C. Fowlkes, "Laplacian reconstruction and refinement for semantic segmentation, " arXiv:1605.02264, 2016.
91. A. Arnab, S. Jayasumana, S. Zheng, and P. Torr, "Higher order potentials in end-to-end trainable conditional random fields, " arXiv:1511.08119, 2015.
92. F. Shen and G. Zeng, "Fast semantic image segmentation with high order context and guided filtering, " arXiv:1605.04068, 2016.
93. Z. Wu, C. Shen, and A. van den Hengel, "Bridging category-level and instance-level semantic image segmentation, " arXiv:1605.06885, 2016.
94. K. He, X. Zhang, S. Ren, and J. Sun, "Identity mappings in deep residual networks, " arXiv:1603.05027, 2016.
95. F. Xia, P. Wang, L.-C. Chen, and A. L. Yuille, "Zoom better to see clearer: Huamn part segmentation with auto zoom net, " arXiv:1511.06881, 2015.
96. X. Liang, X. Shen, D. Xiang, J. Feng, L. Lin, and S. Yan, "Semantic object parsing with local-global long short-term memory, " arXiv:1511.04510, 2015.
97. X. Liang, X. Shen, J. Feng, L. Lin, and S. Yan, "Semantic object parsing with graph lstm, " arXiv:1603.07063, 2016.
98. J. Wang and A. Yuille, "Semantic part segmentation using compositional model combining shape and appearance, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015.
99. P. Wang, X. Shen, Z. Lin, S. Cohen, B. Price, and A. Yuille, "Joint object and part segmentation using deep learned potentials, " in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 1573-1581.
100. V. Badrinarayanan, A. Kendall, and R. Cipolla, "Segnet: A deep convolutional encoder-decoder architecture for image seg-mentation, " arXiv:1511.00561, 2015.
101. J. Uhrig, M. Cordts, U. Franke, and T. Brox, "Pixel-level encoding and depth layering for instance-level semantic labeling, " arXiv:1604.05096, 2016.
102. O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation, " in Proc. Int. Conf. Medical Image Comput. Comput.-Assisted Intervention, 2015, pp. 234-241.