Hyperspectral and LiDAR data fusion using extinction profiles and deep convolutional neural network

IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing

Volumn 10, Issue 6, 2017, Pages 3011-3024

  Ghamisi, Pedram ^a   Höfle, Bernhard ^b   Zhu, Xiao Xiang ^a  

^a TECHNICAL UNIVERSITY OF MUNICH   (Germany)

^b UNIVERSITY OF HEIDELBERG   (Germany)

Author keywords

Convolutional neural network (CNN); Deep learning; Extinction profile (EP); Graph based feature fusion (GBFF); Hyperspectral; Light detection and ranging (LiDAR); Random forest (RF); Support vector machines (SVMs)

Indexed keywords

CONVOLUTION; DATA FUSION; DECISION TREES; DEEP LEARNING; GRAPHIC METHODS; NEURAL NETWORKS; OPTICAL RADAR; SUPPORT VECTOR MACHINES;

CONVOLUTIONAL NEURAL NETWORKS (CNN); EXTINCTION PROFILES; GRAPH-BASED FEATURES; HYPERSPECTRAL; LIGHT DETECTION AND RANGING; RANDOM FORESTS; SUPPORT VECTOR MACHINE (SVMS);

DEEP NEURAL NETWORKS;

ACCURACY ASSESSMENT; ARTIFICIAL NEURAL NETWORK; DATA SET; IMAGE CLASSIFICATION; LIDAR; RESEARCH WORK; SPECTRAL RESOLUTION; SUPPORT VECTOR MACHINE;

HOUSTON; ITALY; TEXAS; TRENTINO-ALTO ADIGE; TRENTO; UNITED STATES;

EID: 85007305421     PISSN: 19391404     EISSN: 21511535     Source Type: Journal    
DOI: 10.1109/JSTARS.2016.2634863     Document Type: Article

References (63)

1. C. Debes et al., "Hyperspectral and LiDAR data fusion: Outcome of the 2013 GRSS data fusion contest", IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 7, no. 6, pp. 2405-2418, Jun. 2014.
2. P. Ghamisi, J. A. Benediktsson, and M. O. Ulfarsson, "Spectral-spatial classification of hyperspectral images based on hidden Markov random fields", IEEE Trans. Geosci. Remote Sens., vol. 52, no. 5, pp. 2565-2574, May 2014.
3. P. Ghamisi, M. S. Couceiro, F. M. L. Martins, and J. A. Benediktsson, "Multilevel image segmentation approach for remote sensing images based on fractional-order Darwinian particle swarm optimization", IEEE Trans. Geosci. Remote Sens., vol. 52, no. 5, pp. 2382-2394, May 2014.
4. M. Fauvel, J. A. Benediktsson, J. Chanussot, and J. R. Sveinsson, "Spectral and spatial classification of hyperspectral data using SVMs and morphological profiles", IEEE Trans. Geosci. Remote Sens., vol. 46, no. 11, pp. 3804-3814, Nov. 2008.
5. J. Li, J. Bioucas-Dias, and A. Plaza, "Semi-supervised hyperspectral image segmentation using multinomial logistic regression with active learning", IEEE Trans. Geosci. Remote Sens., vol. 48, no. 11, p. 4085-4098, Nov. 2010.
6. Y. Tarabalka, J. A. Benediktsson, J. Chanussot, and J. C. Tilton, "Multiple spectral-spatial classification approach for hyperspectral data", IEEE Trans. Geosci. Remote Sens., vol. 48, no. 11, pp. 4122-4132, Nov. 2010.
7. Y. Tarabalka, J. Chanussot, and J. A. Benediktsson, "Segmentation and classification of hyperspectral images using minimum spanning forest grown from automatically selected markers", IEEE Trans. Syst. Man, Cybern., Part B, vol. 40, no. 5, pp. 1267-1279, Oct. 2010.
8. S. Bernabé, P. R. Marpu, A. Plaza, M. D. Mura, and J. A. Benediktsson, "Spectral-spatial classification of multispectral images using kernel feature space representation", IEEE Geosci. Remote Sens. Lett., vol. 11, no. 1, pp. 288-292, Jan. 2014.
9. J. A. Benediktsson and P. Ghamisi, Spectral-Spatial Classification of Hyperspectral Remote Sensing Images. Boston, MA, USA: Artech House, 2015.
10. M. Belgiu, I. Tomljenovic, T. J. Lampoltshammer, T. Blaschke, and B. Höfle, "Ontology-based classification of building types detected from airborne laser scanning data", Remote Sens., vol. 6, no. 2, pp. 1347-1366, 2014.
11. A. Jochem, B. Höfle, M. Rutzinger, and N. Pfeifer, "Automatic roof plane detection and analysis in airborne Lidar point clouds for solar potential assessment", Sensors, vol. 9, no. 7, pp. 5241-5262, 2009.
12. M. Rutzinger, B. Höfle, M. Hollaus, and N. Pfeifer, "Object-based point cloud analysis of full-waveform airborne laser scanning data for urban vegetation classification", Sensors, vol. 8, no. 8, pp. 4505-4528, 2008.
13. R. K. Hall et al., "Quantifying structural physical habitat attributes using Lidar and hyperspectral imagery", Int. J. Image Data Fusion, vol. 59, no. 1, pp. 63-83, 2009.
14. M. Dalponte, L. Bruzzone, and D. Gianelle, "Fusion of hyperspectral and Lidar remote sensing data for classification of complex forest areas", IEEE Trans. Geosci. Remote Sens., vol. 46, no. 5, pp. 1416-1427, May 2008.
15. I. Tomljenovic, B. Hofle, D. Tiede, and T. Blaschke, "Building extraction from airborne laser scanning data: An analysis of the state of the art", Remote Sens., vol. 7, no. 4, pp. 3826-3862, 2015.
16. B. Hofle, M. Hollaus, and J. Hagenauer, "Urban vegetation detection using radiometrically calibrated small-footprint full-waveform airborne Lidar data", ISPRS J. Photogrammetry Remote Sens., vol. 67, pp. 134-147, 2012.
17. P. Gamba, F. Dell'Acqua, and B. V. Dasarathy, "Urban remote sensing using multiple data sets: Past, present, and future", Inf. Fusion, vol. 6, pp. 319-326, 2005.
18. Q. Chen, "Airborne Lidar data processing and information extraction", Photogrammetric Eng. Remote Sens., vol. 73, no. 2, pp. 109-112, 2007.
19. P. Ghamisi, J. A. Benediktsson, and S. Phinn, "Land-cover classification using both hyperspectral and Lidar data", Int. J. Image Data Fusion, vol. 6, no. 3, pp. 189-215, 2015.
20. G. Hughes, "On the mean accuracy of statistical pattern recognizers", IEEE Trans. Inf. Theory, vol. IT-14, no. 1, pp. 55-63, Jan. 1968.
21. P. Ghamisi, "Spectral and spatial classification of hyperspectral data", Ph. D. dissertation, Univ. Iceland, Reykjavk, Iceland, 2015.
22. M. Fauvel, J. Chanussot, and J. A. Benediktsson, "Kernel principal component analysis for the classification of hyperspectral remote-sensing data over urban areas", EURASIP J. Adv. Signal Process., vol. 2009, 2009, Art.no. 783194.
23. P. Ghamisi and J. A. Benediktsson, "Feature selection based on hybridization of genetic algorithm and particle swarm optimization", IEEE Geosci. Remote Sens. Lett, vol. 12, no. 2, pp. 309-313, Feb. 2015.
24. P. Ghamisi, M. S. Couceiro, and J. A. Benediktsson, "A novel feature selection approach based on FODPSO and SVM", IEEE Trans. Geosci. Remote Sens., vol. 53, no. 5, pp. 2935-2947, May 2015.
25. Y. Bazi and F. Melgani, "Toward an optimal SVM classification system for hyperspectral remote sensing images", IEEE Trans. Geosci. Remote Sens., vol. 44, no. 11, pp. 3374-3385, Nov. 2006.
26. G. P. Asner, D. E. Knapp, T. Kennedy-Bowdoin, M. O. Jones, R. E. Martin, and J. Boardman, "Invasive species detection in Hawaiian rainforests using airborne imaging spectroscopy and LiDAR", Remote Sens. Environ., vol. 112, pp. 1942-1955, 2008.
27. G. A. Blackburn, "Remote sensing of forest pigments using airborne imaging spectrometer and LiDAR imagery", Remote Sens. Environ., vol. 82, pp. 311-321, 2002.
28. M. Voss and R. Sugumaran, "Seasonal effect on tree species classification in an urban environment using hyperspectral data, Lidar, and an objectoriented approach", Sensors, vol. 8, pp. 3020-3036, 2008.
29. R. M. Lucas, A. C. Lee, and P. J. Bunting, "Retrieving forest biomass through integration of CASI and LiDAR data", Int. J. Remote Sens., vol. 29, pp. 1553-1577, 2008.
30. U. Heiden, W. Heldens, S. Roessner, K. Segl, T. Esch, and A. Mueller, "Urban structure type characterization using hyperspectral remote sensing and height information", Landscape Urban Planning, vol. 105, no. 6, pp. 361-375, Jun. 2012.
31. R. K. Hall, R. L. Watkins, D. T. Heggem, K. B. Jones, P. R. Kaufmann, and S. B. Moore, "Quantifying structural physical habitat attributes using Lidar and hyperspectral imagery", Environ. Monit. Assessment, vol. 159, pp. 63-83, 2009.
32. B. Koetz, F. Morsdorf, S. Linder, T. Curt, and B. Allgower, "Multi-source land cover classification for forest fire management based on imaging spectrometry and LiDAR data", Forest Ecology Manage., vol. 256, pp. 263-271, 2008.
33. J. T. Mundt, D. R. Streutker, and N. F. Glenn, "Mapping sagebrush distribution using fusion of hyperspectral and Lidar classifications", Photogrammetric Eng. Remote Sens., vol. 72, no. 1, pp. 47-54, Sep. 2006.
34. R. Sugumaran and M. Voss, "Object-oriented classification of Lidar fused hyperspectral imagery for tree species identification in an urban environment", in Proc. Urban Remote Sens. Joint Event, Apr. 2007, pp. 1-6.
35. W. Liao, R. Bellens, A. Pizurica, S. Gautama, and W. Philips, "Generalized graph-based fusion of hyperspectral and LiDAR data using morphological features", IEEE Geosci. Remote Sens. Lett, vol. 12, no. 3, pp. 552-556, Mar. 2015.
36. M. D. Mura, J. A. Benediktsson, B. Waske, and L. Bruzzone, "Morphological attribute profiles for the analysis of very high resolution images", IEEE Trans. Geosci. Remote Sens., vol. 48, no. 10, pp. 3747-3762, Oct. 2010.
37. J. A. Benediktsson, J. A. Palmason, and J. R. Sveinsson, "Classification of hyperspectral data from urban areas based on extended morphological profiles", IEEE Trans. Geosci. Remote Sens., vol. 43, no. 3, pp. 480-491, Mar. 2005.
38. P. Ghamisi, M. D. Mura, and J. A. Benediktsson, "A survey on spectralspatial classification techniques based on attribute profiles", IEEE Trans. Geosci. Remote Sens., vol. 53, no. 5, pp. 2335-2353, May 2015.
39. P. Ghamisi, R. Souza, J. A. Beneiktsson, X. X. Zhu, L. Rittner, and R. A. Lotufo, "Extinction profiles for the classification of remote sensing data", IEEE Trans. Geosci. Remote Sens., vol. 54, no. 10, pp. 5631-5645, Oct. 2016.
40. P. Ghamisi, R. Souza, J. A. Benediktsson, L. Rittner, R. Lotufo, and X. X. Zhu, "Hyperspectral data classification using extended extinction profiles", IEEE Geosci. Remote Sens. Lett, vol. 13, no. 11, pp. 1641-1645, Nov. 2016.
41. G. E. Hinton, S. Osindero, and Y. Teh, "A fast learning algorithm for deep belief nets", Neural Comput., vol. 18, no. 7, pp. 1527-1554, 2006.
42. R. Salakhutdinov and G. E. Hinton, "Deep Boltzmann machines", in Proc. Int. Conf. Artif Intell. Statist, Clearwater Beach, FL, USA, 2009, pp. 448-455.
43. Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, "Greedy layer-wise training of deep networks", in Advances in Neural Information Processing Systems 19. Cambridge, MA, USA: MIT Press, 2007, pp. 153-160.
44. Y. Chen, Z. Lin, X. Zhao, G. Wang, and Y. Gu, "Deep learning-based classification of hyperspectral data", IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 7, no. 6, pp. 2094-2107, Jun. 2014.
45. Y. Chen, X. Zhao, and X. Jia, "Spectra-spatial classification of hyperspectral data based on deep belief network", IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 8, no. 6, pp. 2381-2292, Jun. 2015.
46. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks", in Proc. Adv. Neural Inf. Process. Syst, 2012, pp. 1097-1105.
47. P. Ghamisi, Y. Chen, and X. X. Zhu, "A self-improving convolution neural network for the classification of hyperspectral data", IEEE Geosci. Remote Sens. Lett, vol. 13, no. 10, pp. 1537-1541, Oct. 2016.
48. W. Liao, R. Bellens, S. Gautama, and W. Philips, "Feature fusion of hyperspectral and Lidar data for classification of remote sensing data from urban area", in Proc. 5th Workshop EARSeL Special Interest Group Land Use Land Cover, 2014, pp. 34-34.
49. L. Breiman, "Random forests", Mach. Learn. J., vol. 45, no. 1, pp. 5-32, 2001.
50. C. Vachier, "Extinction value: A new measurement of persistence", in Proc. IEEE Workshop Nonlinear Signal Image Process., 1995, vol. I, pp. 254-257.
51. P. Soille, Morphological Image Analysis: Principles and Applications, 2nd ed. Secaucus, NJ, USA: Springer-Verlag, 2003.
52. G. Matheron, Elements pour une theprie des milieux poreus. Paris, France: Masson, 1967.
53. R. Souza, L. Rittner, R. Machado, and R. Lotufo, "A comparison between extinction filters and attribute filters", in Proc. Int. Symp. Math. Morphology, 2015, pp. 63-74.
54. E. Carlinet and T. Geraud, "A comparative review of component tree computation algorithms", IEEE Trans. Image Process., vol. 23, no. 9, pp. 3885-3895, Sep. 2014.
55. R. Souza, L. Rittner, R. Lotufo, and R. Machado, "An array-based nodeoriented max-tree representation", in Proc. Int. Conf. Image Process., Sep. 2015, pp. 3620-3624.
56. Y. LeCun and Y. Bengio, "Convolutional networks for images, speech, and time series", in The Handbook of Brain Theory and Neural Networks, M. A. Arbib, Ed. Cambridge, MA, USA: MIT Press, 1998, pp. 255-258. [Online]. Available: http://dl.acm.org/citation. cfm?id=303568.303704
57. D. Scherer, A. Müller, and S. Behnke, "Evaluation of pooling operations in convolutional architectures for object recognition", in Proc. 20th Int. Conf. Artif. Neural Netw., 2010, pp. 92-101.
58. Y. Chen, H. Jiang, C. Li, X. Jia, and P. Ghamisi, "Deep feature extraction and classification of hyperspectral images based on convolutional neural networks", IEEE Trans. Geosci. Remote Sens., vol. 54, no. 10, pp. 6232-6251, Oct. 2016.
59. P. Ghamisi, J. A. Benediktsson, G. Cavallaro, and A. Plaza, "Automatic framework for spectral-spatial classification based on supervised feature extraction and morphological attribute profiles", IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 7, no. 6, pp. 2147-2160, Jun. 2014.
60. P. Ghamisi, J. A. Benediktsson, and J. R. Sveinsson, "Automatic spectralspatial classification framework based on attribute profiles and supervised feature extraction", IEEE Trans. Geosci. Remote Sens., vol. 52, no. 5, pp. 5771-5782, Dec. 2014.
61. B. Schölkopf, A. Smola, and K. Müller, "Nonlinear component analysis as a kernel eigenvalue problem", Neural Comput., vol. 10, no. 5, pp. 1299-1319, 1998.
62. M. Pedergnana, P. R. Marpu, M. D. Mura, J. A. Benediktsson, and L. Bruzzone, "Classification of remote sensing optical and LiDAR data using extended attribute profiles", IEEE J. Sel. Topics Signal Process., vol. 6, no. 7, pp. 856-865, Nov. 2012.
63. M. Khodadadzadeh, J. Li, S. Prasad, and A. Plaza, "Fusion of hyperspectral and Lidar remote sensing data using multiple feature learning", IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 8, no. 6, pp. 2971-2983, Jun. 2015.