Target-Adaptive CNN-based pansharpening

IEEE Transactions on Geoscience and Remote Sensing

Volumn 56, Issue 9, 2018, Pages 5443-5457

  Scarpa, Giuseppe ^a   Vitale, Sergio ^b   Cozzolino, Davide ^a  

^a UNIVERSITY OF NAPLES FEDERICO II   (Italy)

^b UNIVERSITY OF NAPLES PARTHENOPE   (Italy)

Author keywords

Convolutional neural networks (CNN); Image classification; Image fusion; Image resolution; Object analysis; Urban areas

Indexed keywords

CONVOLUTION; IMAGE FUSION; IMAGE RESOLUTION; NEURAL NETWORKS; REMOTE SENSING;

CONVOLUTIONAL NEURAL NETWORKS (CNN); OBJECT ANALYSIS; PAN-SHARPENING; PERFORMANCE GAIN; REMOTE SENSING IMAGES; STATE OF THE ART; TRAINING SETS; URBAN AREAS;

IMAGE CLASSIFICATION;

ARTIFICIAL NEURAL NETWORK; IMAGE ANALYSIS; IMAGE CLASSIFICATION; IMAGE PROCESSING; IMAGE RESOLUTION; METHODOLOGY; PERFORMANCE ASSESSMENT; REMOTE SENSING; URBAN AREA;

EID: 85045214848     PISSN: 01962892     EISSN: None     Source Type: Journal    
DOI: 10.1109/TGRS.2018.2817393     Document Type: Article

References (75)

1. V. K. Shettigara, "A generalized component substitution technique for spatial enhancement of multispectral images using a higher resolution data set, " Photogram. Eng. Remote Sens., vol. 58, no. 5, pp. 561-567, 1992.
2. T.-M. Tu, S.-C. Su, H.-C. Shyu, and P. S. Huang, "A new look at IHS-like image fusion methods, " Inf. Fusion, vol. 2, no. 3, pp. 177-186, Sep. 2001.
3. T.-M. Tu, P. S. Huang, C.-L. Hung, and C.-P. Chang, "A fast intensityhue- saturation fusion technique with spectral adjustment for IKONOS imagery, " IEEE Geosci. Remote Sens. Lett., vol. 1, no. 4, pp. 309-312, Oct. 2004.
4. P. S. Chavez Jr., and A. Y. Kwarteng, "Extracting spectral contrast in landsat thematic mapper image data using selective principal component analysis, " Photogramm. Eng. Remote Sens., vol. 55, no. 3, pp. 339-348, 1989.
5. A. R. Gillespie, A. B. Kahle, and R. E. Walker, "Color enhancement of highly correlated images. II. Channel ratio and 'chromaticity' transformation techniques, " Remote Sens. Environ., vol. 22, no. 3, pp. 343-365, Aug. 1987.
6. C. A. Laben and B. V. Brower, "Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening, " U.S. Patent 6 011 875, Apr. 29, 2000.
7. B. Aiazzi, S. Baronti, and M. Selva, "Improving component substitution pansharpening through multivariate regression of MS+Pan data, " IEEE Trans. Geosci. Remote Sens., vol. 45, no. 10, pp. 3230-3239, Oct. 2007.
8. J. Choi, K. Yu, and Y. Kim, "A new adaptive component-substitutionbased satellite image fusion by using partial replacement, " IEEE Trans. Geosci. Remote Sens., vol. 49, no. 1, pp. 295-309, Jan. 2011.
9. A. Garzelli, F. Nencini, and L. Capobianco, "Optimal MMSE pan sharpening of very high resolution multispectral images, " IEEE Trans. Geosci. Remote Sens., vol. 46, no. 1, pp. 228-236, Jan. 2008.
10. T. Ranchin and L. Wald, "Fusion of high spatial and spectral resolution images: The ARSIS concept and its implementation, " Photogramm. Eng. Remote Sens., vol. 66, no. 1, pp. 49-61, Jan. 2000.
11. J. Nunez, X. Otazu, O. Fors, A. Prades, V. Pala, and R. Arbiol, "Multiresolution-based image fusion with additive wavelet decomposition, " IEEE Trans. Geosci. Remote Sens., vol. 37, no. 3, pp. 1204-1211, May 1999.
12. X. Otazu, M. González-Audícana, O. Fors, and J. Núñez, "Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods, " IEEE Trans. Geosci. Remote Sens., vol. 43, no. 10, pp. 2376-2385, Oct. 2005.
13. M. M. Khan, J. Chanussot, L. Condat, and A. Montanvert, "Indusion: Fusion of multispectral and panchromatic images using the induction scaling technique, " IEEE Geosci. Remote Sens. Lett., vol. 5, no. 1, pp. 98-102, Jan. 2008.
14. B. Aiazzi, L. Alparone, S. Baronti, and A. Garzelli, "Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis, " IEEE Trans. Geosci. Remote Sens., vol. 40, no. 10, pp. 2300-2312, Oct. 2002.
15. B. Aiazzi, L. Alparone, S. Baronti, A. Garzelli, and M. Selva, "An MTFbased spectral distortion minimizing model for pan-sharpening of very high resolution multispectral images of urban areas, " in Proc. GRSS/ISPRS Joint Workshop Remote Sens. Data Fusion Urban Areas, May 2003, pp. 90-94.
16. B. Aiazzi, L. Alparone, S. Baronti, A. Garzelli, and M. Selva, "MTFtailored multiscale fusion of high-resolution MS and PAN imagery, " Photogramm. Eng. Remote Sens., vol. 72, no. 5, pp. 591-596, May 2006.
17. J. Lee and C. Lee, "Fast and efficient panchromatic sharpening, " IEEE Trans. Geosci. Remote Sens., vol. 48, no. 1, pp. 155-163, Jan. 2010.
18. R. Restaino, M. D. Mura, G. Vivone, and J. Chanussot, "Contextadaptive pansharpening based on image segmentation, " IEEE Trans. Geosci. Remote Sens., vol. 55, no. 2, pp. 753-766, Feb. 2017.
19. V. P. Shah, N. H. Younan, and R. L. King, "An efficient pan-sharpening method via a combined adaptive PCA approach and contourlets, " IEEE Trans. Geosci. Remote Sens., vol. 46, no. 5, pp. 1323-1335, May 2008.
20. D. Fasbender, J. Radoux, and P. Bogaert, "Bayesian data fusion for adaptable image pansharpening, " IEEE Trans. Geosci. Remote Sens., vol. 46, no. 6, pp. 1847-1857, Jun. 2008.
21. L. Zhang, H. Shen, W. Gong, and H. Zhang, "Adjustable model-based fusion method for multispectral and panchromatic images, " IEEE Trans. Syst. Man, Cybern. B, Cybern., vol. 42, no. 6, pp. 1693-1704, Dec. 2012.
22. A. Garzelli, "Pansharpening of multispectral images based on nonlocal parameter optimization, " IEEE Trans. Geosci. Remote Sens., vol. 53, no. 4, pp. 2096-2107, Apr. 2015.
23. X. Meng, H. Shen, H. Li, Q. Yuan, H. Zhang, and L. Zhang, "Improving the spatial resolution of hyperspectral image using panchromatic and multispectral images: An integrated method, " in Proc. WHISPERS, Jun. 2015, pp. 1-4.
24. H. Shen, X. Meng, and L. Zhang, "An integrated framework for the spatio-Temporal-spectral fusion of remote sensing images, " IEEE Trans. Geosci. Remote Sens., vol. 54, no. 12, pp. 7135-7148, Dec. 2016.
25. S. Zhong, Y. Zhang, Y. Chen, and D. Wu, "Combining component substitution and multiresolution analysis: A novel generalized BDSD pansharpening algorithm, " IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 10, no. 6, pp. 2867-2875, Jun. 2017.
26. F. Palsson, J. R. Sveinsson, and M. O. Ulfarsson, "A new pansharpening algorithm based on total variation, " IEEE Geosci. Remote Sens. Lett., vol. 11, no. 1, pp. 318-322, Jan. 2014.
27. J. Duran, A. Buades, B. Coll, C. Sbert, and G. Blanchet, "A survey of pansharpening methods with a new band-decoupled variational model, " ISPRS J. Photogramm. Remote Sens., vol. 125, pp. 78-105, Mar. 2017.
28. G. Vivone et al., "A critical comparison among pansharpening algorithms, " IEEE Trans. Geosci. Remote Sens., vol. 53, no. 5, pp. 2565-2586, May 2015.
29. B. Aiazzi, L. Alparone, S. Baronti, R. Carlà, A. Garzelli, and L. Santurri, "Sensitivity of pansharpening methods to temporal and instrumental changes between multispectral and panchromatic data sets, " IEEE Trans. Geosci. Remote Sens., vol. 55, no. 1, pp. 308-319, Jan. 2017.
30. L. Alparone, S. Baronti, B. Aiazzi, and A. Garzelli, "Spatial methods for multispectral pansharpening: Multiresolution analysis demystified, " IEEE Trans. Geosci. Remote Sens., vol. 54, no. 5, pp. 2563-2576, May 2016.
31. S. Li and B. Yang, "A new pan-sharpening method using a compressed sensing technique, " IEEE Trans. Geosci. Remote Sens., vol. 49, no. 2, pp. 738-746, Feb. 2011.
32. S. Li, H. Yin, and L. Fang, "Remote sensing image fusion via sparse representations over learned dictionaries, " IEEE Trans. Geosci. Remote Sens., vol. 51, no. 9, pp. 4779-4789, Sep. 2013.
33. X. X. Zhu and R. Bamler, "A sparse image fusion algorithm with application to pan-sharpening, " IEEE Trans. Geosci. Remote Sens., vol. 51, no. 5, pp. 2827-2836, May 2013.
34. M. Cheng, C. Wang, and J. Li, "Sparse representation based pansharpening using trained dictionary, " IEEE Geosci. Remote Sens. Lett., vol. 11, no. 1, pp. 293-297, Jan. 2014.
35. M. R. Vicinanza, R. Restaino, G. Vivone, M. Dalla Mura, and J. Chanussot, "A pansharpening method based on the sparse representation of injected details, " IEEE Geosci. Remote Sens. Lett., vol. 12, no. 1, pp. 180-184, Jan. 2015.
36. X. X. Zhu, C. Grohnfeld, and R. Bamler, "Exploiting joint sparsity for pansharpening: The J-SparseFI algorithm, " IEEE Trans. Geosci. Remote Sens., vol. 54, no. 5, pp. 2664-2681, May 2016.
37. M. Castelluccio, G. Poggi, C. Sansone, and L. Verdoliva. (2015). "Land use classification in remote sensing images by convolutional neural networks." [Online]. Available: https://arxiv.org/abs/1508.00092.
38. D. Ienco, R. Gaetano, C. Dupaquier, and P. Maurel, "Land cover classification via multitemporal spatial data by deep recurrent neural networks, " IEEE Geosci. Remote Sens. Lett., vol. 14, no. 10, pp. 1685-1689, Oct. 2017.
39. X. X. Zhu et al. (2017). "Deep learning in remote sensing: A review." [Online]. Available: https://arxiv.org/abs/1710.03959.
40. W. Huang, L. Xiao, Z. Wei, H. Liu, and S. Tang, "A new pan-sharpening method with deep neural networks, " IEEE Geosci. Remote Sens. Lett., vol. 12, no. 5, pp. 1037-1041, May 2015.
41. G. Masi, D. Cozzolino, L. Verdoliva, and G. Scarpa, "Pansharpening by convolutional neural networks, " Remote Sens., vol. 8, no. 7, p. 594, 2016.
42. Y. Wei and Q. Yuan, "Deep residual learning for remote sensed imagery pansharpening, " in Proc. RSIP, May 2017, pp. 1-4.
43. Y. Wei, Q. Yuan, H. Shen, and L. Zhang, "Boosting the accuracy of multispectral image pansharpening by learning a deep residual network, " IEEE Geosci. Remote Sens. Lett., vol. 14, no. 10, pp. 1795-1799, Oct. 2017.
44. Y. Rao, L. He, and J. Zhu, "A residualconvolutionalneural network for pan-shaprening, " in Proc. Int. Workshop Remote Sens. Intell. Process. (RSIP), May 2017, pp. 1-4, doi:10.1109/RSIP.2017.7958807.
45. A. Azarang and H. Ghassemian, "A new pansharpeningmethodusing multi resolutionanalysisframework and deepneural networks, " in Proc. 3rd Int. Conf. Pattern Recognit. Image Anal. (IPRIA), Apr. 2017, pp. 1-6.
46. C. Dong, C. C. Loy, K. He, and X. Tang, "Image super-resolution using deep convolutional networks, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 2, pp. 295-307, Feb. 2016.
47. Open Remote Sensing. Accessed: May 5, 2017. [Online]. Available: http://openremotesensing.net.
48. G. Masi, D. Cozzolino, L. Verdoliva, and G. Scarpa, "CNN-based pansharpening of multi-resolution remote-sensing images, " in Proc. Joint Urban Remote Sens. Event (JURSE), Dubai, United Arab Emirates, Mar. 2017, pp. 1-4.
49. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks, " in Proc. Adv. Neural Inf. Process. Syst., 2012, pp. 1106-1114.
50. L. Wald, T. Ranchin, and M. Mangolini, "Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images, " Photogramm. Eng. Remote Sens., vol. 63, no. 6, pp. 691-699, 1997.
51. I. Sutskever, J. Martens, G. Dahl, and G. Hinton, "On the importance of initialization and momentum in deep learning, " in Proc. 30th Int. Conf. Mach. Learn., Feb. 2013, pp. 1139-1147.
52. J. Yang, J. Wright, T. S. Huang, and Y. Ma, "Image super-resolution via sparse representation, " IEEE Trans. Image Process., vol. 19, no. 11, pp. 2861-2873, Nov. 2010.
53. R. H. Yuhas, A. F. H. Goetz, and J. W. Boardman, "Discrimination among semi-Arid landscape endmembers using the Spectral Angle Mapper (SAM) algorithm, " in Proc. Summaries 3rd Annu. JPL Airborne Geosci. Workshop, 1992, pp. 147-149.
54. C.-I. Chang, "An information-Theoretic approach to spectral variability, similarity, and discrimination for hyperspectral image analysis, " IEEE Trans. Inf. Theory, vol. 46, no. 5, pp. 1927-1932, Aug. 2000.
55. R. Girshick, "Fast R-CNN, " in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Sep. 2015, pp. 1440-1448.
56. R. Zeyde, M. Elad, and M. Protter, "On single image scale-up using sparse-representations, " in Curves Surfaces. Springer, 2012, pp. 711-730.
57. R. Timofte, V. De Smet, and L. Van Gool, "Anchored neighborhood regression for fast example-based super-resolution, " in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Dec. 2013, pp. 1920-1927.
58. J. Kim, J. K. Lee, and K. M. Lee, "Accurate image super-resolution using very deep convolutional networks, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 1646-1654.
59. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2016, pp. 770-778.
60. K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, "Beyond a Gaussian Denoiser: Residual learning of deep CNN for image denoising, " IEEE Trans. Image Process., vol. 26, no. 7, pp. 3142-3155, Jul. 2017.
61. Q. Wang, W. Shi, and P. M. Atkinson, "Area-To-point regression Kriging for pan-sharpening, " ISPRS J. Photogramm. Remote Sens., vol. 114, pp. 151-165, Apr. 2016.
62. I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. Cambridge, MA, USA: MIT Press, 2016.
63. K. Simonyan and A. Zisserman. (2014). "Very deep convolutional networks for large-scale image recognition." [Online]. Available: https://arxiv.org/abs/1409.1556.
64. S. Ioffe and C. Szegedy. (2015). "Batch normalization: Accelerating deep network training by reducing internal covariate shift." [Online]. Available: https://arxiv.org/abs/1502.03167.
65. M. M. Khan, L. Alparone, and J. Chanussot, "Pansharpening quality assessment using the modulation transfer functions of instruments, " IEEE Trans. Geosci. Remote Sens., vol. 47, no. 11, pp. 3880-3891, Nov. 2009.
66. B. Aiazzi, L. Alparone, S. Baronti, R. Carlà, A. Garzelli, and L. Santurri, "Full scale assessment of pansharpening methods and data products, " Proc. SPIE, vol. 9244, p. 924402, Oct. 2014.
67. F. Palsson, J. R. Sveinsson, M. O. Ulfarsson, and J. A. Benediktsson, "Quantitative quality evaluation of pansharpened imagery: Consistency versus synthesis, " IEEE Trans. Geosci. Remote Sens., vol. 54, no. 3, pp. 1247-1259, Mar. 2016.
68. Theano Documentation Webpage. Accessed: May 5, 2017. [Online]. Available: http://deeplearning.net/software/theano.
69. Y. Jia et al. (2014). "Caffe: Convolutional architecture for fast feature embedding." [Online]. Available: https://arxiv.org/abs/1408.5093.
70. GRIP Home Page. Accessed: May 5, 2017. [Online]. Available: http://grip.unina.it.
71. L. Wald, Data Fusion. Definitions and Architectures-Fusion of Images of Different Spatial Resolutions. Paris, France: Presses de l'Ecole, EcoledesMines de Paris, 2002, p. 200. [Online]. Available: https://halmines-paristech.archives-ouvertes.fr/hal-00464703.
72. Z. Wang and A. C. Bovik, "A universal image quality index, " IEEE Signal Process. Lett., vol. 9, no. 3, pp. 81-84, Mar. 2002.
73. L. Alparone, S. Baronti, A. Garzelli, and F. Nencini, "A global quality measurement of pan-sharpened multispectral imagery, " IEEE Geosci. Remote Sens. Lett., vol. 1, no. 4, pp. 313-317, Oct. 2004.
74. L. Alparone, B. Aiazzi, S. Baronti, A. Garzelli, F. Nencini, and M. Selva, "Multispectral and panchromatic data fusion assessment without reference, " Photogramm. Eng. Remote Sens., vol. 74, no. 2, pp. 193-200, Feb. 2008.
75. S. Lolli, L. Alparone, A. Garzelli, and G. Vivone, "Haze correction for contrast-based multispectral pansharpening, " IEEE Geosci. Remote Sens. Lett., vol. 14, no. 12, pp. 2255-2259, Dec. 2017.