Semantic labeling in very high resolution images via a self-cascaded convolutional neural network

ISPRS Journal of Photogrammetry and Remote Sensing

Volumn 145, Issue , 2018, Pages 78-95

  Liu, Yongcheng ^a,b   Fan, Bin ^a   Wang, Lingfeng ^a   Bai, Jun ^a   Xiang, Shiming ^a   Pan, Chunhong ^a  

^a INSTITUTE OF AUTOMATION   (China)

^b UNIVERSITY OF CHINESE ACADEMY OF SCIENCES   (China)

Author keywords

Convolutional neural networks (CNNs); End to end; Multi scale contexts; Semantic labeling

Indexed keywords

BENCHMARKING; CONVOLUTION; NEURAL NETWORKS; REMOTE SENSING; SEMANTIC WEB; SEMANTICS;

CONVOLUTIONAL NEURAL NETWORK; END TO END; MULTI-FEATURE FUSION; MULTI-SCALE; REMOTE SENSING APPLICATIONS; SEMANTIC LABELING; STATE-OF-THE-ART PERFORMANCE; VERY HIGH RESOLUTION (VHR) IMAGE;

IMAGE PROCESSING;

ARTIFICIAL NEURAL NETWORK; CORRECTION; IMAGE RESOLUTION; REMOTE SENSING; URBAN AREA;

EID: 85038939811     PISSN: 09242716     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.isprsjprs.2017.12.007     Document Type: Article

References (79)

1. Alshehhi, R., Marpu, P.R., Woon, W.L., Mura, M.D., Simultaneous extraction of roads and buildings in remote sensing imagery with convolutional neural networks. ISPRS J. Photogramm. Remote Sens. 130 (2017), 139–149.
2. Audebert, N., Saux, B.L., Lefèvre, S., 2016. Semantic Segmentation of Earth Observation Data Using Multimodal and Multi-scale Deep Networks. Available from: < 1609.06846>.
3. Badrinarayanan, V., Kendall, A., Cipolla, R., 2015. SegNet: A Deep Convolutional Encoder-decoder Architecture for Image Segmentation. Available from: < 1511.00561>.
4. Bell, S., Lawrence Zitnick, C., Bala, K., Girshick, R., 2016. Inside-outside net: detecting objects in context with skip pooling and recurrent neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 2874–2883.
5. Bertasius, G., Shi, J., Torresani, L., 2016. Semantic segmentation with boundary neural fields. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 3602–3610.
6. Bridle, J.S., Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. Neurocomputing: Algorithms, Architectures and Applications, 1989, Springer.
7. Chen, L., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L., 2015. Semantic image segmentation with deep convolutional nets and fully connected crfs. In: International Conference on Learning Representations.
8. Chen, L.-C., Yang, Y., Wang, J., Xu, W., Yuille, A.L., 2016a. Attention to scale: scale-aware semantic image segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 3640–3649.
9. Chen, S., Wang, H., Xu, F., Jin, Y.-Q., Target classification using the deep convolutional networks for SAR images. IEEE Trans. Geosci. Remote Sens. 54:8 (2016), 4806–4817.
10. Cheng, G., Han, J., A survey on object detection in optical remote sensing images. ISPRS J. Photogramm. Remote Sens. 117 (2016), 11–28.
11. Cheng, G., Han, J., Lu, X., 2017a. Remote Sensing Image Scene Classification: Benchmark and State of the Art. Available from: < 1703.00121>.
12. Cheng, G., Wang, Y., Xu, S., Wang, H., Xiang, S., Pan, C., Automatic road detection and centerline extraction via cascaded end-to-end convolutional neural network. IEEE Trans. Geosci. Remote Sens. 55:6 (2017), 3322–3337.
13. Cheng, G., Zhu, F., Xiang, S., Wang, Y., Pan, C., Accurate urban road centerline extraction from VHR imagery via multiscale segmentation and tensor voting. Neurocomputing. 205 (2016), 407–420.
14. Dalal, N., Triggs, B., 2005. Histograms of oriented gradients for human detection. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 886–893.
15. Everingham, M., Eslami, S.M.A., Gool, L.J.V., Williams, C.K.I., Winn, J.M., Zisserman, A., The pascal visual object classes challenge: a retrospective. Int. J. Comput. Vision 111:1 (2015), 98–136.
16. Farabet, C., Couprie, C., Najman, L., LeCun, Y., Learning hierarchical features for scene labeling. IEEE Trans. Pattern Anal. Mach. Intell. 35:8 (2013), 1915–1929.
17. Gerke, M., 2015. Use of the Stair Vision Library Within the ISPRS 2d Semantic Labeling Benchmark (Vaihingen). Technical Report.
18. Gidaris, S., Komodakis, N., 2015. Object detection via a multi-region and semantic segmentation-aware cnn model. In: IEEE International Conference on Computer Vision. pp. 1134–1142.
19. Glorot, X., Bordes, A., Bengio, Y., 2011. Deep sparse rectifier neural networks. In: International Conference on Artificial Intelligence and Statistics, vol. 15. pp. 275.
20. Gong, M., Yang, H., Zhang, P., Feature learning and change feature classification based on deep learning for ternary change detection in SAR images. ISPRS J. Photogramm. Remote Sens. 129 (2017), 212–225.
21. Gould, S., Russakovsky, O., Goodfellow, I., Baumstarck, P., 2011. The Stair Vision Library (v2.5). Stanford University.
22. Hariharan, B., Arbeláez, P., Girshick, R., Malik, J., 2015. Hypercolumns for object segmentation and fine-grained localization. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 447–456.
23. He, K., Zhang, X., Ren, S., Sun, J., 2015a. Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: IEEE International Conference on Computer Vision. pp. 1026–1034.
24. He, K., Zhang, X., Ren, S., Sun, J., Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37:9 (2015), 1904–1916.
25. He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778.
26. Hu, F., Xia, G.-S., Hu, J., Zhang, L., Transferring deep convolutional neural networks for the scene classification of high-resolution remote sensing imagery. Remote Sens. 7:11 (2015), 14680–14707.
27. Ioffe, S., Szegedy, C., 2015. Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning. pp. 448–456.
28. ISPRS, 2016. International Society for Photogrammetry and Remote Sensing. 2D Semantic Labeling Challenge. < http://www2.isprs.org/commissions/comm3/wg4/semantic-labeling.html>.
29. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R.B., Guadarrama, S., Darrell, T., 2014. Caffe: convolutional architecture for fast feature embedding. In: ACM International Conference on Multimedia. pp. 675–678.
30. Krizhevsky, A., Sutskever, I., Hinton, G.E., 2012. Imagenet classification with deep convolutional neural networks. In: Neural Information Processing Systems. pp. 1106–1114.
31. Lecun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D., 1990. Handwritten digit recognition with a back-propagation network. In: Neural Information Processing Systems. pp. 396–404.
32. Lecun, Y., Bottou, L., Bengio, Y., Haffner, P., Gradient-based learning applied to document recognition. Proc. IEEE 86:11 (1998), 2278–2324.
33. Li, E., Femiani, J., Xu, S., Zhang, X., Wonka, P., Robust rooftop extraction from visible band images using higher order CRF. IEEE Trans. Geosci. Remote Sens. 53:8 (2015), 4483–4495.
34. Li, J., Huang, X., Gamba, P., Bioucas-Dias, J.M., Zhang, L., Benediktsson, J.A., Plaza, A., Multiple feature learning for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 53:3 (2015), 1592–1606.
35. Lin, G., Milan, A., Shen, C., Reid, I.D., 2016. RefineNet: Multi-path Refinement Networks for High-resolution Semantic Segmentation. Available from: < 1611.06612>.
36. Liu, C., Yuen, J., Torralba, A., Sivic, J., Freeman, W., 2008. Sift flow: dense correspondence across different scenes. In: European Conference on Computer Vision. pp. 28–42.
37. Liu, W., Rabinovich, A., Berg, A.C., 2016a. Parsenet: looking wider to see better. In: International Conference on Learning Representations Workshop.
38. Liu, Y., Fan, B., Wang, L., Bai, J., Xiang, S., Pan, C., 2017. Context-aware cascade network for semantic labeling in VHR image. In: IEEE International Conference on Image Processing.
39. Liu, Y., Zhong, Y., Fei, F., Zhang, L., 2016b. Scene semantic classification based on random-scale stretched convolutional neural network for high-spatial resolution remote sensing imagery. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). pp. 763–766.
40. Long, J., Shelhamer, E., Darrell, T., 2015. Fully convolutional networks for semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 3431–3440.
41. Lowe, D.G., Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60:2 (2004), 91–110.
42. Lu, X., Yuan, Y., Zheng, X., Joint dictionary learning for multispectral change detection. IEEE Trans. Cybernet. 47:4 (2017), 884–897.
43. Lu, X., Zheng, X., Yuan, Y., Remote sensing scene classification by unsupervised representation learning. IEEE Trans. Geosci. Remote Sens.(99), 2017, 1–10.
44. Luus, F.P., Salmon, B.P., van den Bergh, F., Maharaj, B., Multiview deep learning for land-use classification. IEEE Geosci. Remote Sens. Lett. 12:12 (2015), 2448–2452.
45. Maggiori, E., Tarabalka, Y., Charpiat, G., Alliez, P., Convolutional neural networks for large-scale remote-sensing image classification. IEEE Trans. Geosci. Remote Sens. 55:2 (2017), 645–657.
46. Marmanis, D., Schindler, K., Wegner, J.D., Galliani, S., Datcu, M., Stilla, U., 2016. Classification With an Edge: Improving Semantic Image Segmentation with Boundary Detection. Available from: < 1612.01337>.
47. Mas, J.F., Flores, J.J., The application of artificial neural networks to the analysis of remotely sensed data. Int. J. Remote Sens. 29:3 (2008), 617–663.
48. Matikainen, L., Karila, K., Segment-based land cover mapping of a suburban area-comparison of high-resolution remotely sensed datasets using classification trees and test field points. Remote Sens. 3:8 (2011), 1777–1804.
49. Mnih, V., 2013. Machine Learning for Aerial Image Labeling [Ph.D. thesis]. University of Toronto.
50. Mostajabi, M., Yadollahpour, P., Shakhnarovich, G., 2015. Feedforward semantic segmentation with zoom-out features. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 3376–3385.
51. Nair, V., Hinton, G.E., 2010. Rectified linear units improve restricted boltzmann machines. In: International Conference on Machine Learning. pp. 807–814.
52. Nogueira, K., Mura, M.D., Chanussot, J., Schwartz, W.R., dos Santos, J.A., 2016. Learning to semantically segment high-resolution remote sensing images. In: IEEE International Conference on Pattern Recognition. pp. 3566–3571.
53. Noh, H., Hong, S., Han, B., 2015. Learning deconvolution network for semantic segmentation. In: IEEE International Conference on Computer Vision. pp. 1520–1528.
54. Paisitkriangkrai, S., Sherrah, J., Janney, P., van den Hengel, A., Semantic labeling of aerial and satellite imagery. IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens. 9:7 (2016), 2868–2881.
55. Penatti, O.A., Nogueira, K., dos Santos, J.A., 2015. Do deep features generalize from everyday objects to remote sensing and aerial scenes domains. In: IEEE Conference on Computer Vision and Pattern Recognition Workshop. pp. 44–51.
56. Pinheiro, P.O., Lin, T.-Y., Collobert, R., Dollár, P., 2016. Learning to Refine Object Segments. Available from: < 1603.08695>.
57. Ronneberger, O., Fischer, P., Brox, T., 2015. U-net: convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention – MICCAI. pp. 234–241.
58. Rumelhart, D.E., Hinton, G.E., Williams, R.J., Learning representations by back-propagating errors. Nature 323:6088 (1986), 533–536.
59. Sherrah, J., 2016. Fully Convolutional Networks for Dense Semantic Labelling of High-resolution Aerial Imagery. Available from: < 1606.02585>.
60. Silberman, N., Hoiem, D., Kohli, P., Fergus, R., 2012. Indoor segmentation and support inference from rgbd images. In: European Conference on Computer Vision. pp. 746–760.
61. Simonyan, K., Zisserman, A., 2015. Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations. pp. 1–14.
62. Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R., Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15:1 (2014), 1929–1958.
63. Volpi, M., Tuia, D., Dense semantic labeling of subdecimeter resolution images with convolutional neural networks. IEEE Trans. Geosci. Remote Sens. 55:2 (2017), 881–893.
64. Wen, D., Huang, X., Liu, H., Liao, W., Zhang, L., Semantic classification of urban trees using very high resolution satellite imagery. IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens. 10:4 (2017), 1413–1424.
65. Xie, M., Jean, N., Burke, M., Lobell, D., Ermon, S., 2015. Transfer Learning from Deep Features for Remote Sensing and Poverty Mapping. Available from: < 1510.00098>.
66. Xu, X., Li, J., Huang, X., Mura, M.D., Plaza, A., Multiple morphological component analysis based decomposition for remote sensing image classification. IEEE Trans. Geosci. Remote Sens. 54:5 (2016), 3083–3102.
67. Xue, Z., Li, J., Cheng, L., Du, P., Spectral-spatial classification of hyperspectral data via morphological component analysis-based image separation. IEEE Trans. Geosci. Remote Sens. 53:1 (2015), 70–84.
68. Yosinski, J., Clune, J., Bengio, Y., Lipson, H., 2014. How transferable are features in deep neural networks. In: Neural Information Processing Systems. pp. 3320–3328.
69. Yu, F., Koltun, V., 2016. Multi-scale context aggregation by dilated convolutions. In: International Conference on Learning Representations.
70. Yuan, Y., Mou, L., Lu, X., Scene recognition by manifold regularized deep learning architecture. IEEE Trans. Neural Netw. Learn. Syst. 26:10 (2015), 2222–2233.
71. Zeiler, M.D., Fergus, R., 2014. Visualizing and understanding convolutional networks. In: European Conference on Computer Vision. pp. 818–833.
72. Zeiler, M.D., Krishnan, D., Taylor, G.W., Fergus, R., 2010. Deconvolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 2528–2535.
73. Zhang, C., Pan, X., Li, H., Gardiner, A., Sargent, I., Hare, J., Atkinson, P.M., A hybrid MLP-CNN classifier for very fine resolution remotely sensed image classification. ISPRS J. Photogramm. Remote Sens., 2017, 1–12.
74. Zhang, L., Zhang, L., Tao, D., Huang, X., On combining multiple features for hyperspectral remote sensing image classification. IEEE Trans. Geosci. Remote Sens. 50:3 (2012), 879–893.
75. Zhang, P., Gong, M., Su, L., Liu, J., Li, Z., Change detection based on deep feature representation and mapping transformation for multi-spatial-resolution remote sensing images. ISPRS J. Photogramm. Remote Sens. 116 (2016), 24–41.
76. Zhang, Q., Seto, K.C., Mapping urbanization dynamics at regional and global scales using multi-temporal DMSP/OLS nighttime light data. Remote Sens. Environ. 115:9 (2011), 2320–2329.
77. Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J., 2016. Pyramid Scene Parsing Network. Available from: < 1612.01105>.
78. Zhao, W., Du, S., Learning multiscale and deep representations for classifying remotely sensed imagery. ISPRS J. Photogramm. Remote Sens. 113 (2016), 155–165.
79. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A., 2015. Object detectors emerge in deep scene cnns. In: International Conference on Learning Representations.