Fusion of images and point clouds for the semantic segmentation of large-scale 3D scenes based on deep learning

ISPRS Journal of Photogrammetry and Remote Sensing

Volumn 143, Issue , 2018, Pages 85-96

  Zhang, Rui ^a,b   Li, Guangyun ^a   Li, Minglei ^a   Wang, Li ^a  

^a INFORMATION ENGINEERING UNIVERSITY   (China)

^b NORTH CHINA UNIVERSITY OF WATER RESOURCES AND ELECTRIC POWER   (China)

Author keywords

2D image; 3D point cloud; 3D scene segmentation; High resolution; Large scale

Indexed keywords

DEEP NEURAL NETWORKS; IMAGE RESOLUTION; LARGE DATASET; SEMANTIC SEGMENTATION; SEMANTICS; THREE DIMENSIONAL COMPUTER GRAPHICS;

2D IMAGES; 3D POINT CLOUD; 3D SCENE SEGMENTATION; 3D SCENES; HIGH RESOLUTION; LARGE-SCALES; POINT-CLOUDS; SCALE RESOLUTION; SCENE SEGMENTATION; SEMANTIC SEGMENTATION;

PIXELS;

ACCURACY ASSESSMENT; ALGORITHM; DATA SET; IMAGE ANALYSIS; IMAGE RESOLUTION; MAPPING METHOD; NUMERICAL METHOD; PIXEL; SEGMENTATION; THREE-DIMENSIONAL MODELING; TWO-DIMENSIONAL MODELING;

EID: 85046796612     PISSN: 09242716     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.isprsjprs.2018.04.022     Document Type: Article

References (57)

1. Abdel-Aziz, Y.I., Karara, H.M., Direct linear transformation from comparator coordinates into object space coordinates in close-range photogrammetry. Photogram. Eng. Rem. Sens. 81:1 (2015), 103–107.
2. Badrinarayanan, Kendall, V.A., Cipolla, R., 2016. Segnet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation, arXiv preprint arXiv:1511.00561v3.
3. Berthold, K.P.H., Extended Gaussian images. Proc. IEEE 72:12 (1984), 1671–1686.
4. Bu, S., Liu, Z., Han, J., Wu, J., Ji, R., Learning high-level feature by deep belief networks for 3-D model retrieval and recognition. IEEE TMM 16:8 (2014), 2154–2167.
5. Burochin, J.P., Vallet, B., Bredif, M., Paparoditis, N., Detecting blind building facades from highly overlapping wide angle aerial imagery. ISPRS J. Photogram. Rem. Sens., 2014, 96:193–209, 10.1016/j.isprsjprs.2014.07.011.
6. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L., 2014. Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFS, arXiv preprint arXiv:1412.7062.
7. Chen, L.C. Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L., 2017. Deeplab: Semantic Image Segmentation with Deep Convolutional Nets, ATROUS Convolution, and Fully Connected CRFS, arXiv preprint arXiv:1606.00915v2.
8. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B., The cityscapes data set for semantic urban scene understanding. Comp. Vis. Pattern Recog. IEEE, 2016, 3213–3223.
9. Deng, F., Zhang, Z.X., Zhang, J.Q., A method of registration between laser scanning data and digital images. Geomat. Inf. Sci. Wuhan Univ. 32:4 (2007), 290–292.
10. Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A., Int. J. Comp. Vis. 111:1 (2015), 98–136.
11. Fang, W., Research on Automatic Texture Mapping of Terrestrial Laser Scanning Data Combining Photogrammetry Technique. 2014, Wuhan University, Wuhan.
12. Garcia-Garcia, A., Orts-Escolano, S., Oprea, S.O., Villena-Martinez, V., Garcia-Rodriguez, J., 2017. A Review on Deep Learning Techniques Applied to Semantic Segmentation. arXiv:1704.06857v1.
13. Guo, Y.L, 2017. Deep Feature Expression of 3D Data. Classroom of Deep Learning < http://mp.weixin.qq.com/s/g9ANliOMLJalJtpt4YCZVw>.
14. Hackel, T., Wegner, J.D., Schindler, K., Fast semantic segmentation of 3d point clouds with strongly varying density. ISPRS Ann. Photogram., Rem. Sens. Spat. Inf. Sci. III-3 (2016), 177–184, 10.5194/isprsannals-III-3-177-2016.
15. He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778.
16. Josep, M.B., Jose, L.L., Unsupervised robust planar segmentation of terrestrial laser scanner point clouds based on fuzzy clustering method. ISPRS J. Photogram. Rem. Sens. 63 (2008), 84–98.
17. Kalogerakis, E., Averkiou, M., Maji, S., Chaudhuri, S., 2016. 3D Shape Segmentation with Projective Convolutional Networks. Computer Vision and Pattern Recognition (CVPR 2015). Boston, MA, 8–10 June, arXiv preprint arXiv:1612.02808.
18. Kendall, A., Badrinarayanan, V., Cipolla, R., 2016. Bayesian SEGNET: Model Uncertainty in Deep Convolutional Encoderdecoder Architectures for Scene Understanding, arXiv preprint arXiv:1511.02680v2.
19. Krizhevsky, A., Sutskever, I., Hinton, G.E., 2012. Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105.
20. Lecun, Y., Bottou, L., Bengio, Y., Haffner, P., 1998. Gradient-based learning applied to document recognition. In: Proceedings of the IEEE, (on CD-ROM).
21. Lecun, Y., Bengio, Y., Hinton, G., Deep learning. Nature. 521 (2015), 436–444, 10.1038/Nature14539.
22. Li, M.D., Jiang, S.P., Wang, H.P., A RANSCA-based stable plane fitting method of point clouds. Sci. Survey. Map. 40 (2015), 102–106.
23. Li, M.L., 2014. Technology of Preprocessing on 3D Laser Scanned Point Clouds. Master D. Dissertation of PLA Information Engineering University.
24. Li, M.L., Li, G.Y., Wang, L., Li, H.B., Fan, Z.R., Automatic feature detecting from point clouds using 3D hough transform. Bull. Survey. Map.(2), 2015, 29–33.
25. Li, M.L., Gao, X.Y., Li, G.Y., Wang, L., Liu, S.L., High accuracy calibration of installation parameters between 3D terrestrial laser scanner and external-installed digital camera. Optical Prec. Eng. 24 (2016), 2158–2166.
26. Li, N., Ma, Y.W., Tang, Y., Gao, S.L., Segmentation of building facade point cloud using RANSAC. Sci. Survey. Map. 36:5 (2011), 144–146.
27. Li, Y., Pirk, S., Su, H., Qi, C.R., Guibas, L.J., FPNN: field probing neural networks for 3D data. NIPS, 2016, 307–315.
28. Liu, Y., Wang, F., Dobaie, A.M., 2017. Comparison of 2D image models in segmentation performance for 3D laser point clouds. Neurocomputing, doi: https://doi.org/10.1016/j.neucom.2017.04.030 (in press).
29. Long, J., Shelhamer, E., Darrell, T., 2015. Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440.
30. Murase, H., Nayar, S.K., Visual learning and recognition of 3-D objects from appearance. Int. J. Comp. Vis. 14:1 (1995), 5–24.
31. Nalani, H.A., Maas, H.G., 2012. Automatic Building Facade detection in Mobile Laser Scanner point clouds. In: The German Society for Photogrammetry, Remote Sensing and Geoinformation (DGPF), Potsdam, Germany (on CD-ROM).
32. Nalani, H.A., Sanka, Nirodha, P., Maas, H.G., 2012. Automatic processing of mobile laser scanner point clouds for building facade detection. In: International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. XXXIX-B5, 2012 XXII ISPRS Congress, 25 August–01 September, Melbourne, Australia.
33. Petar, V., Igor, K., Marko, R.S., Obtaining structural descriptions of building facades. Comp. Sci. Inf. Syst. 13:1 (2015), 23–43.
34. Qi, C.R., Su, H., Mo, K., Guibas, L.J., 2016a. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. arXiv preprint arXiv:1612.00593.
35. Qi, C.R., Su, H., Niessner, M., Dai, A., Yan, M., Guibas, L.J., 2016b. Volumetric and multi-view CNNs for object classification on 3D data. In: Computer Vision and Pattern Recognition (CVPR 2016). Las Vegas, Nevada, 26 June–1 July, pp. 5648–5656, doi: https://doi.org/10.1109/CVPR.2016.609.
36. Shi, B., Bai, S., Zhou, Z., Bai, X., DeepPano: deep panoramic representation for 3-D shape recognition. IEEE Sig. Process. Lett. 22:12 (2015), 2339–2343, 10.1109/LSP.2015. 2480802.
37. Simonyan, K., Zisserman, A., 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition, arXiv preprint arXiv:1409.1556.
38. Sinha, A., Bai, J., Ramani, K., Deep learning 3D shape surfaces using geometry images. Computer Vision-ECCV, 2016, Springer International Publishing, 223–240.
39. Song, X.Y., Yuan, S., Guo, H.B., Liu, J.F., Pattern identification algorithm with adaptive threshold interval based extended Hough transform[J]. Chin. J. Scient. Instrum. 35:5 (2014), 1109–1117.
40. Su, H., Maji, S., Kalogerakis, E., Learned-Miller, E., 2015. Multi-view convolutional neural networks for 3D shape recognition. In: Proceedings of ICCV, pp. 945–953, doi: https://doi.org/10.1109/ICCV.2015.114.
41. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A., 2015. Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9.
42. Vinyals, O., Bengio, S., Kudlur, M., 2015. Order matters: Sequence to sequence for sets. Comp. Sci. arXiv preprint arXiv:1511.06393.
43. Wang, Y.M., Hu, C.M., A robust registration method for terrestrial LiDAR point clouds and texture image. Acta Geodaet. Cartograph. Sin. 41:2 (2012), 266–272.
44. Wei, Z., Yang, B.S., Li, Q.Q., Automated extraction of building footprints from mobile LIDAR point clouds. J. Rem. Sens. 16:2 (2012), 286–296.
45. Wu, J., Zhang, C., Xue, T., Freemanand, W.T., Tenenbaum, J.B., 2016. Learning a probabilistic latent space of object shapes via 3D generative-adversarial modeling. In: Advances in Neural Information Processing Systems.
46. Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., 2015. 3D ShapeNets: a deep representation for volumetric shapes. In: Computer Vision and Pattern Recognition (CVPR 2015). Boston, MA, 8–10 June, pp. 1912–1920.
47. Xu, J.Z., Kou, Y., Yuan, F., Zhang, W., Auto-registration of aerial imagery and airborne LiDAR data based on structure feature. Infrared Laser Eng. 42:12 (2013), 3502–3508.
48. Xu, Q., Shou, H., Zhu, S.L., Modern Photogrammetry. 2000, The Peoples Liberation Army Press, Beijing.
49. Xu, X., Corrigan, D., Dehghani, A., Caulfield, S., Moloney, D., 3D object recognition based on volumetric representation using convolutional neural networks. International Conference on Articulated Motion and Deformable Objects, 2016, Springer International Publishing, 147–156, 10.1007/978-3-319-41778-3_15.
50. Yan, J.F., 2014. Research on Terrestrial LiDAR Point Cloud Data Registration and Fusion Method of Point Cloud and Images. China University of Mining & Technology, Xuzhou.
51. Yang, B.S., Wei, Z., Li, Q.Q., Li, J., Automated extraction of street-scene objects from mobile lidar point clouds. Int. J. Rem. Sens. 33:18 (2012), 5839–5861.
52. Yang, B.S., Dong, Z., Wei, Z., Fang, L.N., Li, H.W., Extracting complex building facades from mobile laser scanning data. Acta Geodaet. Cartograph. Sin. 42:3 (2013), 411–417.
53. Yang, B.S., Wei, Z., Li, Q.Q., Li, J., Semiautomated building facade footprint extraction from mobile LiDAR point clouds. IEEE Geosci. Rem. Sens. Lett. 10:4 (2013), 766–770.
54. Yao, J.L., Zhang, D.F., Improvement of the direct-solution of spatial-resection. J. Shandong Univ. Technol. (Sci & Tech) 19:2 (2005), 6–9.
55. Yao, J.L., Sun, Y.T., Wang, S.G., The direct-solution of spatial-resection based on Luodigues matrix. J. Shandong Univ. Technol. (Sci & Tech) 20:2 (2006), 36–39.
56. Zhang, R., Li, G.Y., Wang, L., Zhou, Y.L., New method of hybrid index for mobile LiDAR point cloud data. Geomat. Inf. Sci. Wuhan Univ.(11), 2017, 1–7, 10.13203/j.whugis20160441 http://kns.cnki.net/kcms/detail/42.1676.TN.20171109.1531.002.html.
57. Zhou, F., Yang, C., Wang, C.G., Wang, B.Q., Liu, J., Circle detection and its number identification in complex condition based on random Hough transform. Chin. J. Scient. Instrum. 34:3 (2013), 622–628.