Multi-modal feature fusion for geographic image annotation

Pattern Recognition

Volumn 73, Issue , 2018, Pages 1-14

  Li, Ke ^a   Zou, Changqing ^b,c   Bu, Shuhui ^d   Liang, Yun ^c   Zhang, Jian ^e   Gong, Minglun ^f  

^a INFORMATION ENGINEERING UNIVERSITY   (China)

^b HENGYANG NORMAL UNIVERSITY   (China)

^c SIMON FRASER UNIVERSITY   (Canada)

^d NORTHWESTERN POLYTECHNICAL UNIVERSITY   (China)

^e ZHEJIANG INTERNATIONAL STUDIES UNIVERSITY   (China)

^f MEMORIAL UNIVERSITY OF NEWFOUNDLAND   (Canada)

Author keywords

Convolutional neural networks (CNNs); Deep learning; Geographic image annotation; Multi modal feature fusion

Indexed keywords

DEEP LEARNING; IMAGE ANALYSIS; NEURAL NETWORKS; PIXELS; VIDEO STREAMING;

CONVOLUTIONAL NEURAL NETWORK; DEEP BELIEF NETWORK (DBN); FEATURE FUSION; GEOGRAPHIC IMAGES; HIGH-LEVEL FEATURES; LOW-LEVEL FEATURES; MID-LEVEL FEATURES; RESTRICTED BOLTZMANN MACHINE;

IMAGE FUSION;

EID: 85028468404     PISSN: 00313203     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patcog.2017.06.036     Document Type: Article

References (61)

1. Lu, D., Weng, Q., A survey of image classification methods and techniques for improving classification performance. Int. J. Remote Sens. 28:5 (2007), 823–870.
2. Lowe, D.G., Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60:2 (2004), 91–110.
3. Daugman, J.G., Two-dimensional spectral analysis of cortical receptive field profiles. Vis. Res. 20:10 (1980), 847–856.
4. Dalal, N., Triggs, B., Histograms of oriented gradients for human detection. Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1, 2005, 886–893.
5. Bengio, Y., Courville, A., Vincent, P., Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35:8 (2013), 1798–1828.
6. J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, T. Darrell, Decaf: a deep convolutional activation feature for generic visual recognition, arXiv preprint arXiv:1310.1531 (2013).
7. Razavian, A., Azizpour, H., Sullivan, J., Carlsson, S., CNN features off-the-shelf: an astounding baseline for recognition. Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops, 2014, 806–813.
8. Aytekin, H. Zongur, Texture-based airport runway detection. Geosci. Remote Sens. Lett. 10:3 (2013), 471–475.
9. Yang, Y., Newsam, S., Geographic image retrieval using local invariant features. IEEE Trans. Geosci. Remote Sens. 51:2 (2013), 818–832.
10. Chen, L., Yang, W., Xu, K., Xu, T., Evaluation of local features for scene classification using vhr satellite images. Proc. Urban Remote Sensing Event, 2011, 385–388.
11. Ahonen, T., Hadid, A., Pietikainen, M., Face description with local binary patterns: application to face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 28:12 (2006), 2037–2041.
12. Tao, C., Tan, Y., Cai, H., Tian, J., Airport detection from large ikonos images using clustered sift keypoints and region information. Geosci. Remote Sens. Lett. 8:1 (2011), 128–132.
13. Li, Y., Sun, X., Wang, H., Sun, H., Li, X., Automatic target detection in high-resolution remote sensing images using a contour-based spatial model. Geosci. Remote Sens. Lett. 9:5 (2012), 886–890.
14. Li, J., Zhang, H., Zhang, L., Huang, X., Zhang, L., Joint collaborative representation with multitask learning for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 52:9 (2014), 5923–5936.
15. Yang, Y., Newsam, S., Semi-supervised learning of geospatial objects through multi-modal data integration. Proc. 22nd International Conference on Pattern Recognition, 2014, 4062–4067.
16. Zhang, L., Zhang, L., Tao, D., Huang, X., On combining multiple features for hyperspectral remote sensing image classification. IEEE Trans. Geosci. Remote Sens. 50:3 (2012), 879–893.
17. Tuia, D., Camps-Valls, G., Urban image classification with semisupervised multiscale cluster kernels. Sel. Top. Appl. Earth Obs. Remote Sens. 4:1 (2011), 65–74.
18. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L., Imagenet: A large-scale hierarchical image database. Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2009, 248–255.
19. Krizhevsky, A., Sutskever, I., Hinton, G.E., Imagenet classification with deep convolutional neural networks. Proc. Advances in neural information processing systems, 2012, 1097–1105.
20. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L., Semantic image segmentation with deep convolutional nets and fully connected CRFs. Proc. International Conference on Learning Representations, 2015.
21. Luo, W., Li, H., Liu, G., Automatic annotation of multispectral satellite images using author–topic model. Geosci. Remote Sens. Lett. 9:4 (2012), 634–638.
22. Sirmacek, B., Ünsalan, C., Urban-area and building detection using sift keypoints and graph theory. IEEE Trans. Geosci. Remote Sens. 47:4 (2009), 1156–1167.
23. Grabner, H., Nguyen, T.T., Gruber, B., Bischof, H., On-line boosting-based car detection from aerial images. ISPRS J. Photogramm. Remote Sens. 63:3 (2008), 382–396.
24. Wang, L., Hao, S., Wang, Q., Wang, Y., Semi-supervised classification for hyperspectral imagery based on spatial-spectral label propagation. ISPRS J. Photogramm. Remote Sens. 97 (2014), 123–137.
25. Farabet, C., Couprie, C., Najman, L., LeCun, Y., Learning hierarchical features for scene labeling. IEEE Trans. Pattern Anal. Mach. Intell. 35:8 (2013), 1915–1929.
26. Yue, J., Zhao, W., Mao, S., Liu, H., Spectral spatial classification of hyperspectral images using deep convolutional neural networks. Remote Sens. Lett. 6:6 (2015), 468–477.
27. Zhao, W., Guo, Z., Yue, J., Zhang, X., Luo, L., On combining multiscale deep learning features for the classification of hyperspectral remote sensing imagery. Int. J. Remote Sens. 36:13 (2015), 3368–3379.
28. Chen, X., Xiang, S., Liu, C.-L., Pan, C.-H., Vehicle detection in satellite images by hybrid deep convolutional neural networks. Geosci. Remote Sens. Lett. 11:10 (2014), 1797–1801.
29. Huang, X., Zhang, L., Comparison of vector stacking, multi-svms fuzzy output, and multi-svm+s voting methods for multiscale vhr urban mapping. Geosci. Remote Sens. Lett. 7:2 (2010), 261–265.
30. Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y., Multimodal deep learning. Proc. the 28th International Conference on Machine Learning, 2011, 689–696.
31. Srivastava, N., Salakhutdinov, R.R., Multimodal learning with deep boltzmann machines. Proc. Advances in neural information processing systems, 2012, 2222–2230.
32. Li, J., Zhang, H., Zhang, L., Efficient superpixel-level multitask joint sparse representation for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 53:10 (2015), 5338–5351.
33. Zheng, X., Sun, X., Fu, K., Wang, H., Automatic annotation of satellite images via multifeature joint sparse coding with spatial relation constraint. Geosci. Remote Sens. Lett. 10:4 (2013), 652–656.
34. A. Shahroudy, T.-T. Ng, Y. Gong, G. Wang, Deep multimodal feature analysis for action recognition in rgb+d videos, arXiv preprint arXiv:1603.07120 (2016).
35. Kan, M., Shan, S., Chen, X., Multi-view deep network for cross-view classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, 4847–4855.
36. Wang, W., Arora, R., Livescu, K., Bilmes, J.A., On deep multi-view representation learning. ICML, 2015, 1083–1092.
37. Porway, J., Wang, K., Yao, B., Zhu, S.C., A hierarchical and contextual model for aerial image understanding. Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, 2008, IEEE, 1–8.
38. I. Markus Gerke, Use of the stair vision library within the isprs 2d semantic labeling benchmark (vaihingen).
39. Cheriyadat, A.M., Unsupervised feature learning for aerial scene classification. IEEE Trans. Geosci. Remote Sens. 52:1 (2014), 439–451.
40. Kembhavi, A., Harwood, D., Davis, L.S., Vehicle detection using partial least squares. IEEE Trans. Pattern Anal. Mach. Intell. 33:6 (2011), 1250–1265.
41. Chen, Y., Lin, Z., Zhao, X., Wang, G., Gu, Y., Deep learning-based classification of hyperspectral data. Sel. Top. Appl. Earth Obs. Remote Sens. 7:6 (2014), 2094–2107.
42. M. Castelluccio, G. Poggi, C. Sansone, L. Verdoliva, Land use classification in remote sensing images by convolutional neural networks, arXiv preprint arXiv:1508.00092 (2015).
43. Marmanis, D., Datcu, M., Esch, T., Stilla, U., Deep learning earth observation classification using imagenet pretrained networks. IEEE Geosci. Remote Sens. Lett. 13:1 (2016), 105–109.
44. Paisitkriangkrai, S., Sherrah, J., Janney, P., Hengel, V.-D., et al. Effective semantic pixel labelling with convolutional networks and conditional random fields. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2015, 36–43.
45. J. Sherrah, Fully convolutional networks for dense semantic labelling of high-resolution aerial imagery, arXiv preprint arXiv:1606.02585 (2016).
46. Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Susstrunk, S., Slic superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34:11 (2012), 2274–2282.
47. Vedaldi, A., Fulkerson, B., Vlfeat: An open and portable library of computer vision algorithms. Proc. 18th ACM International Conference on Multimedia, 2010, 1469–1472.
48. Ojala, T., Pietikainen, M., Harwood, D., A comparative study of texture measures with classification based on featured distributions. Pattern Recognit. 29:1 (1996), 51–59.
49. Wang, X., Han, T.X., Yan, S., An HOG-LBP human detector with partial occlusion handling. Proc. 12th International Conference on Computer Vision, 2009, 32–39.
50. Jolliffe, I.T., Principal component analysis. 2005.
51. Larochelle, H., Erhan, D., Courville, A., Bergstra, J., Bengio, Y., An empirical evaluation of deep architectures on problems with many factors of variation. Proc. 24th International Conference on Machine Learning, 2007, 473–480.
52. Lee, H., Grosse, R., Ranganath, R., Ng, A.Y., Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Proc. 26th International Conference on Machine Learning, 2009, 609–616.
53. Hinton, G., A practical guide to training restricted boltzmann machines. Neural Networks: Tricks of the Trade, 2012, 599–619.
54. Bengio, Y., Learning deep architectures for AI. Found. Trends Mach. Learn. 2:1 (2009), 1–127.
55. Hinton, G.E., Osindero, S., Teh, Y.-W., A fast learning algorithm for deep belief nets. Neural Comput. 18:7 (2006), 1527–1554.
56. He, K., Zhang, X., Ren, S., Sun, J., Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37:9 (2015), 1904–1916.
57. Wilcoxon, F., Individual comparisons by ranking methods. Biom. Bull. 1:6 (1945), 80–83.
58. ( http://www2.isprs.org/commissions/comm3/wg4/semantic-labeling.html).
59. ( http://ftp.ipi.uni-hannover.de/ISPRS_WGIII_website/ISPRSIII_4_Test_results/papers/Description_UCalgary.pdf).
60. Byeon, W., Breuel, T.M., Raue, F., Liwicki, M., Scene labeling with lstm recurrent neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, 3547–3555.
61. Liang, X., Shen, X., Feng, J., Lin, L., Yan, S., Semantic object parsing with graph lstm. European Conference on Computer Vision, 2016, Springer, 125–143.