Integrating Multilayer Features of Convolutional Neural Networks for Remote Sensing Scene Classification

IEEE Transactions on Geoscience and Remote Sensing

Volumn 55, Issue 10, 2017, Pages 5653-5665

  Li, Erzhu ^a,b   Xia, Junshi ^c   Du, Peijun ^a,b   Lin, Cong ^a,b   Samat, Alim ^d  

^a NANJING UNIVERSITY   (China)

^b NANJING NORMAL UNIVERSITY   (China)

^c UNIVERSITY OF TOKYO   (Japan)

^d XINJIANG INSTITUTE OF ECOLOGY AND GEOGRAPHY   (China)

Author keywords

Convolutional neural networks (CNN); feature fusion; improved Fisher kernel; scene classification; spectral regression kernel discriminant analysis (SRKDA)

Indexed keywords

CONVOLUTION; CONVOLUTIONAL CODES; DATA MINING; DATA VISUALIZATION; DISCRIMINANT ANALYSIS; FEATURE EXTRACTION; FLOW VISUALIZATION; IMAGE CLASSIFICATION; IMAGE RECONSTRUCTION; IMAGE RESOLUTION; MULTILAYER NEURAL NETWORKS; MULTILAYERS; NEURAL NETWORKS; PERSONNEL TRAINING; PRINCIPAL COMPONENT ANALYSIS; REMOTE SENSING; SEMANTICS; VIDEO STREAMING;

CONVOLUTIONAL NEURAL NETWORK; FEATURE FUSION; FISHER KERNELS; KERNEL DISCRIMINANT ANALYSIS; SCENE CLASSIFICATION;

CLASSIFICATION (OF INFORMATION);

ALGORITHM; ARTIFICIAL NEURAL NETWORK; DATA PROCESSING; DATA SET; DISCRIMINANT ANALYSIS; IMAGE CLASSIFICATION; PRINCIPAL COMPONENT ANALYSIS; REMOTE SENSING; SPATIAL RESOLUTION;

EID: 85023777322     PISSN: 01962892     EISSN: None     Source Type: Journal    
DOI: 10.1109/TGRS.2017.2711275     Document Type: Article

References (60)

1. M. Voltersen, C. Berger, S. Hese, and C. Schmullius, "Expanding an urban structure type mapping approach from a subarea to the entire city of Berlin," in Proc. Joint Urban Remote Sens. Event (JURSE), Lausanne, Switzerland, Mar. 2015, pp. 1-4.
2. X. Zhang and S. Du, "A linear Dirichlet mixture model for decomposing scenes: Application to analyzing urban functional zonings," Remote Sens. Environ., vol. 169, pp. 37-49, Nov. 2015.
3. L. Zhang, L. Zhang, D. Tao, and X. Huang, "On combining multiple features for hyperspectral remote sensing image classification," IEEE Trans. Geosci. Remote Sens., vol. 50, no. 3, pp. 879-893, Mar. 2012.
4. L. Zhang, Q. Zhang, L. Zhang, D. Tao, X. Huang, and B. Du, "Ensemble manifold regularized sparse low-rank approximation for multiview feature embedding," Pattern Recognit., vol. 48, no. 10, pp. 3102-3112, Oct. 2015.
5. K. van de Sande, T. Gevers, and C. Snoek, "Evaluating color descriptors for object and scene recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 9, pp. 1582-1596, Sep. 2010.
6. S. Chen and Y. Tian, "Pyramid of spatial relatons for scene-level land use classification," IEEE Trans. Geosci. Remote Sens., vol. 53, no. 4, pp. 1947-1957, Apr. 2015.
7. R. Raja, S. M. M. Roomi, and D. Dharmalakshmi, "Outdoor Scene classification using invariant features," in Proc. 4th Nat. Conf. Comput. Vis., Pattern Recognit., Image Process. Graph. (NCVPRIPG), New York, NY, USA, Dec. 2013, pp. 1-4.
8. Y. Jiang, R. Wang, and P. Zhang, "Texture description based on multiresolution moments of image histograms," Opt. Eng., vol. 47, no. 3, p. 037005, Mar. 2008.
9. Y. Yang and S. Newsam, "Bag-of-visual-words and spatial extensions for land-use classification," in Proc. 18th ACM SIGSPATIAL Int. Conf. Adv. Geogr. Inf. Syst. (GIS), San Jose, CA, USA, Nov. 2010, pp. 270-279.
10. G. Cheng, L. Guo, T. Zhao, J. Han, H. Li, and J. Fang, "Automatic landslide detection from remote-sensing imagery using a scene classification method based on BoVW and pLSA," Int. J. Remote Sens., vol. 34, no. 1, pp. 45-59, Jan. 2013.
11. B. Zhao, Y. Zhong, G.-S. Xia, and L. Zhang, "Dirichlet-derived multiple topic scene classification model for high spatial resolution remote sensing imagery," IEEE Trans. Geosci. Remote Sens., vol. 54, no. 4, pp. 2108-2123, Apr. 2016.
12. F. Perronnin and C. Dance, "Fisher kernels on visual vocabularies for image categorization," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Minneapolis, MN, USA, Jun. 2007, pp. 1-8.
13. A. M. Cheriyadat, "Unsupervised feature learning for aerial scene classification," IEEE Trans. Geosci. Remote Sens., vol. 52, no. 1, pp. 439-451, Jan. 2014.
14. F. Hu, G.-S. Xia, Z. Wang, X. Huang, L. Zhang, and H. Sun, "Unsupervised feature learning via spectral clustering of multidimensional patches for remotely sensed scene classification," IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 8, no. 5, pp. 2015-2030, May 2015.
15. J. Schmidhuber, "Deep learning in neural networks: An overview," Neural Netw., vol. 61, pp. 85-117, Jan. 2015.
16. L. Zhang, L. Zhang, and B. Du, "Deep learning for remote sensing data: A technical tutorial on the state of the art," IEEE Geosci. Remote Sens. Mag., vol. 4, no. 2, pp. 22-40, Jun. 2016.
17. R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Region-based convolutional networks for accurate object detection and segmentation," IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 1, pp. 142-158, Jan. 2016.
18. S. Balaban, "Deep learning and face recognition: The state of the art," Proc. SPIE, vol. 9457, May 2015, Art. no. 94570B.
19. O. Abdel-Hamid, A.-R. Mohamed, H. Jiang, and G. Penn, "Applying convolutional neural networks concepts to hybrid NN-HMM model for speech recognition," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), Kyoto, Japan, Mar. 2012, pp. 4277-4280.
20. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and F.-F. Li, "Large-scale video classification with convolutional neural networks," in Proc. 27th IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Columbus, OH, USA, Jun. 2014, pp. 1725-1732.
21. S. Zheng et al., "Conditional random fields as recurrent neural networks," in Proc. 15th IEEE Int. Conf. Comput. Vis. (ICCV), Santiago, Chile, Dec. 2016, pp. 1529-1537.
22. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proc. 29th IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Las Vegas, NV, USA, Jun. 2016, pp. 770-778.
23. O. A. B. Penatti, K. Nogueira, and J. A. dos Santos, "Do deep features generalize from everyday objects to remote sensing and aerial scenes domains?" in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), Boston, MA, USA, Jun. 2015, pp. 44-51.
24. M. Vakalopoulou, K. Karantzalos, N. Komodakis, and N. Paragios, "Building detection in very high resolution multispectral data with deep learning features," in Proc. IEEE Int. Geosci. Remote Sens. Symp. (IGARSS), Milan, Italy, Jul. 2015, pp. 1873-1876.
25. C. Szegedy et al., "Going deeper with convolutions," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Boston, MA, USA, Jun. 2015, pp. 1-9.
26. K. Nogueira, O. A. B. Penatti, and J. A. dos Santos, "Towards better exploiting convolutional neural networks for remote sensing scene classification," Pattern Recognit., vol. 61, pp. 539-556, Jan. 2017.
27. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "ImageNet: A large-scale hierarchical image database," in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (CVPR), Miami, FL, USA, Jun. 2009, pp. 248-255.
28. F. Hu, G.-S. Xia, J. Hu, and L. Zhang, "Transferring deep convolutional neural networks for the scene classification of high-resolution remote sensing imagery," Remote Sens., vol. 7, no. 11, pp. 14680-14707, Nov. 2015.
29. F. Zhang, B. Du, and L. Zhang, "Scene classification via a gradient boosting random convolutional network framework," IEEE Trans. Geosci. Remote Sens., vol. 54, no. 3, pp. 1793-1802, Mar. 2016.
30. W. Zhao and S. Du, "Scene classification using multi-scale deeply described visual words," Int. J. Remote Sens., vol. 37, no. 17, pp. 4119-4131, Sep. 2016.
31. E. Othman, Y. Bazi, N. Alajlan, H. Alhichri, and F. Melgani, "Using convolutional features and a sparse autoencoder for land-use scene classification," Int. J. Remote Sens., vol. 37, no. 10, pp. 1977-1995, May 2016.
32. J. Yue, W. Zhao, S. Mao, and H. Liu, "Spectral-spatial classification of hyperspectral images using deep convolutional neural networks," Remote Sens. Lett., vol. 6, no. 6, pp. 468-477, Jun. 2015.
33. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks," in Proc. 26th Annu. Conf. Neural Inf. Process. Syst. (NIPS), Lake Tahoe, NV, USA, 2012, pp. 1097-1105.
34. J. Long, E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation," in Proc. IEEE Conf. Comput. Soc. Vis. Pattern Recognit. (CVPR), Boston, MA, USA, Jun. 2015, pp. 3431-3440.
35. W. Zhao and S. Du, "Learning multiscale and deep representations for classifying remotely sensed imagery," ISPRS J. Photogramm. Remote Sens., vol. 113, pp. 155-165, Mar. 2016.
36. L. Bottou, "Large-scale machine learning with stochastic gradient descent," in Proc. 19th Int. Conf. Comput. Stat. (COMPSTAT), Paris, France, 2010, pp. 177-186.
37. Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," Nature, vol. 521, pp. 436-444, May 2015.
38. K. Simonyan and A. Zisserman. (2014). "Very deep convolutional networks for large-scale image recognition." [Online]. Available: http://arxiv.org/abs/1409.1556
39. X. Chen, S. Xiang, C.-L. Liu, and C.-H. Pan, "Vehicle detection in satellite images by hybrid deep convolutional neural networks," IEEE Geosci. Remote Sens. Lett., vol. 11, no. 10, pp. 1797-1801, Oct. 2014.
40. J. Tang, C. Deng, G. B. Huang, and B. Zhao, "Compressed-domain ship detection on spaceborne optical image using deep neural network and extreme learning machine," IEEE Trans. Geosci. Remote Sens., vol. 53, no. 3, pp. 1174-1185, Mar. 2015.
41. K. Makantasis, K. Karantzalos, A. Doulamis, and N. Doulamis, "Deep supervised learning for hyperspectral data classification through convolutional neural networks," in Proc. Int Geosci Remote Sens. Symp. (IGARSS), Milan, Italy, Jul. 2015, pp. 4959-4962.
42. W. Zhao and S. Du, "Spectral-spatial feature extraction for hyperspectral image classification: A dimension reduction and deep learning approach," IEEE Trans. Geosci. Remote Sens., vol. 54, no. 8, pp. 4544-4554, Aug. 2016.
43. G. Gando, T. Yamada, H. Sato, S. Oyama, and M. Kurihara, "Fine-tuning deep convolutional neural networks for distinguishing illustrations from photographs," Expert Sys. Appl., vol. 66, pp. 295-301, Dec. 2016.
44. F. Perronnin, J. Sánchez, and T. Mensink, "Improving the fisher kernel for large-scale image classification," in Proc. 11th Eur. Conf. Comput. Vis. (ECCV), Heraklion, Greece, 2010, pp. 143-156.
45. S. Baker and T. Kanade, "Hallucinating faces," in Proc. 4th IEEE Int. Conf. Autom. Face Gesture Recognit. (FG), Grenoble, France, Mar. 2000, pp. 83-88.
46. D. Cai, X. He, and J. Han, "SRDA: An efficient algorithm for largescale discriminant analysis," IEEE Trans. Knowl. Data Eng., vol. 20, no. 1, pp. 1-12, Jan. 2008.
47. D. Cai, X. He, and J. Han, "Speed up kernel discriminant analysis," VLDB J., vol. 20, no. 1, pp. 21-33, 2011.
48. G. Baudat and F. Anouar, "Generalized discriminant analysis using a kernel approach," Neural Comput., vol. 12, no. 10, pp. 2385-2404, 2000.
49. S. Mika, G. Ratsch, J. Weston, B. Scholkopf, and K. R. Müllers, "Fisher discriminant analysis with kernels," in Proc. 9th IEEE Workshop Neural Netw. Signal Process. (NNSP), Madison, WI, USA, Aug. 1999, pp. 41-48.
50. G.-S. Xia, W. Yang, J. Delon, Y. Gousseau, H. Sun, and H. Maître, "Structural high-resolution satellite image indexing," in Proc. ISPRS Int. Arch. Photogramm., Remote Sens. Spat. Inf. Sci., Vienna, Austria, Jul. 2010, pp. 298-303.
51. Y. Jia et al., "Caffe: Convolutional architecture for fast feature embedding," in Proc. ACM Conf. Multimedia (MM), Orlando, FL, USA, Nov. 2014, pp. 675-678.
52. K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. (2014). "Return of the devil in the details: Delving deep into convolutional nets." [Online]. Available: http://arxiv.org/abs/1405.3531
53. R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin, "LIBLINEAR: A library for large linear classification," J. Mach. Learn. Res., vol. 9, pp. 1871-1874, Jun. 2008.
54. A. Vedaldi and B. Fulkerson, "Vlfeat: An open and portable library of computer vision algorithms," in Proc. 18th ACM Int. Conf. Multimedia, Firenze, Italy, Oct. 2010, pp. 1469-1472.
55. A. Vedaldi and K. Lenc, "MatConvNet: Convolutional neural networks for MATLAB," in Proc. 23rd ACM Int. Conf. Multimedia (MM), Brisbane, QLD, Australia, Oct. 2015, pp. 689-692.
56. J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong, "Locality-constrained linear coding for image classification," in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (CVPR), San Francisco, CA, USA, Jun. 2010, pp. 3360-3367.
57. H. Jégou, M. Douze, C. Schmid, and P. Pérez, "Aggregating local descriptors into a compact image representation," in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (CVPR), San Francisco, CA, USA, Jun. 2010, pp. 3304-3311.
58. G.-S. Xia et al., "AID: A benchmark data set for performance evaluation of aerial scene classification," IEEE Trans. Geosci. Remote Sens., vol. 55, no. 7, pp. 3965-3981, Jul. 2017.
59. M. Castelluccio, G. Poggi, C. Sansone, and L. Verdoliva. (2015). "Land use classification in remote sensing images by convolutional neural networks." [Online]. Available: http://arxiv.org/abs/1508.00092
60. J. Zou, W. Li, C. Chen, and Q. Du, "Scene classification using local and global features with collaborative representation fusion," Inf. Sci., vol. 348, pp. 209-226, Jun. 2016.