A Multiscale and Multidepth Convolutional Neural Network for Remote Sensing Imagery Pan-Sharpening

IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing

Volumn 11, Issue 3, 2018, Pages 978-989

  Yuan, Qiangqiang ^a   Wei, Yancong ^a   Meng, Xiangchao ^b   Shen, Huanfeng ^a   Zhang, Liangpei ^c  

^a WUHAN UNIVERSITY   (China)

^b NINGBO UNIVERSITY   (China)

^c Wuhan University   (China)

Author keywords

Convolutional neural network (CNN); multiscale feature learning; pan sharpening; remote sensing

Indexed keywords

CONVOLUTION; DEEP LEARNING; DEEP NEURAL NETWORKS; EXTRACTION; FEATURE EXTRACTION; IMAGE ENHANCEMENT; IMAGE PROCESSING; IMAGE RESOLUTION; JOB ANALYSIS; LEARNING SYSTEMS; MATHEMATICAL TRANSFORMATIONS; NEURAL NETWORKS;

CONVOLUTIONAL NEURAL NETWORK; MULTI-SCALE FEATURES; PAN-SHARPENING; SPATIAL RESOLUTION; TASK ANALYSIS;

REMOTE SENSING;

ARTIFICIAL NEURAL NETWORK; IMAGE PROCESSING; IMAGERY; MULTISPECTRAL IMAGE; PANCHROMATIC IMAGE; REMOTE SENSING; SPATIAL RESOLUTION;

EID: 85041545449     PISSN: 19391404     EISSN: 21511535     Source Type: Journal    
DOI: 10.1109/JSTARS.2018.2794888     Document Type: Article

References (55)

1. T. M. Tu, P. S. Huang, C. L. Hung, and C. P. Chang, "A fast intensityhue-saturation fusion technique with spectral adjustment for IKONOS imagery, " IEEE Geosci. Remote Sens. Lett., vol. 1, no. 4, pp. 309-312, Oct. 2004.
2. A. R. Gillespie, A. B. Kahle, and R. E. Walker, "Color enhancement of highly correlated images. 2. Channel ratio and chromaticity transformation techniques, " Remote Sens. Environ., vol. 22, pp. 343-365, Aug. 1987.
3. P. S. Chavez and A. Y. Kwarteng, "Extracting spectral contrast in landsat thematic mapper image data using selective principal component analysis, " Photogramm. Eng. Remote Sens., vol. 55, pp. 339-348, Mar. 1989.
4. C. A. Laben and B. V. Brower, "Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening, " Google Patents US 6011875 A, 2000.
5. J. Choi, K. Yu, and Y. Kim, "A new adaptive component-substitutionbased satellite image fusion by using partial replacement, " IEEE Trans. Geosci. Remote Sens., vol. 49, no. 1, pp. 295-309, Jan. 2011.
6. B. Aiazzi, L. Alparone, S. Baronti, and A. Garzelli, "Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis, " IEEE Trans. Geosci. Remote Sens., vol. 40, no. 10, pp. 2300-2312, Oct. 2002.
7. F. Nencini, A. Garzelli, S. Baronti, and L. Alparone, "Remote sensing image fusion using the curvelet transform, " Inf. Fusion, vol. 8, pp. 143-156, Apr. 2007.
8. B. Aiazzi, L. Alparone, S. Baronti, A. Garzelli, and M. Selva, "MTFtailored multiscale fusion of high-resolution MS and pan imagery, " Photogramm. Eng. Remote Sens., vol. 72, pp. 591-596, May 2006.
9. F. Palsson, J. R. Sveinsson, M. O. Ulfarsson, and J. A. Benediktsson, "MTF-based deblurring using a wiener filter for CS and MRA pansharpening methods, " IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens., vol. 9, pp. 2255-2269, Jun. 2016.
10. J. G. Liu, "Smoothing filter-based intensity modulation: A spectral preserve image fusion technique for improving spatial details, " Int. J. Remote Sens., vol. 21, pp. 3461-3472, Dec. 2000.
11. S. W. Zhong, Y. Zhang, Y. S. Chen, and D. Wu, "Combining component substitution and multiresolution analysis: A novel generalized BDSD pansharpening algorithm, " IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens., vol. 10, no. 6, pp. 2867-2875, Jun. 2017.
12. A. Garzelli, F. Nencini, and L. Capobianco, "Optimal MMSE pan sharpening of very high resolution multispectral images, " IEEE Trans. Geosci. Remote Sens., vol. 46, no. 1, pp. 228-236, Jan. 2008.
13. A. Garzelli, "Pansharpening of multispectral images based on nonlocal parameter optimization, " IEEE Trans. Geosci. Remote Sens., vol. 53, no. 4, pp. 2096-2107, Apr. 2015.
14. D. Fasbender, J. Radoux, and P. Bogaert, "Bayesian data fusion for adaptable image pansharpening, " IEEE Trans. Geosci. Remote Sens., vol. 46, no. 6, pp. 1847-1857, Jun. 2008.
15. L. Zhang, H. Shen, W. Gong, and H. Zhang, "Adjustable model-based fusion method for multispectral and panchromatic images, " IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 42, no. 6, pp. 1693-1704, Dec. 2012.
16. F. Palsson, J. R. Sveinsson, and M. O. Ulfarsson, "A new pansharpening algorithm based on total variation, " IEEE Geosci. Remote Sens. Lett., vol. 11, no. 1, pp. 318-322, Jan. 2014.
17. H. Shen, X. Meng, and L. Zhang, "An integrated framework for the spatio-temporal-spectral fusion of remote sensing images, " IEEE Trans. Geosci. Remote Sens., vol. 54, no. 12, pp. 7135-7148, Dec. 2016.
18. C. Jiang, H. Zhang, H. Shen, and L. Zhang, "A practical compressed sensing-based pan-sharpening method, " IEEE Geosci. Remote Sens. Lett., vol. 9, no. 4, pp. 629-633, Jul. 2012.
19. C. Jiang, H. Zhang, H. Shen, and L. Zhang, "Two-step sparse coding for the pan-sharpening of remote sensing images, " IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens., vol. 7, no. 5, pp. 1792-1805, May 2014.
20. S. Li, H. Yin, and L. Fang, "Remote sensing image fusion via sparse representations over learned dictionaries, " IEEE Trans. Geosci. Remote Sens., vol. 51, no. 9, pp. 4779-4789, Sep. 2013.
21. X. X. Zhu and R. Bamler, "A sparse image fusion algorithm with application to pan-sharpening, " IEEE Trans. Geosci. Remote Sens., vol. 51, no. 5, pp. 2827-2836, May 2013.
22. C. Dong, C. C. Loy, K. He, and X. Tang, "Image super-resolution using deep convolutional networks, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 2, pp. 295-307, Feb. 1, 2016.
23. K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, "Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising, " IEEE Trans. Image Process., vol. 26, no. 7, pp. 3142-3155, Jul. 2017.
24. P. Svoboda, M. Hradis, L. Marsík, and P. Zemcík, "CNN for license plate motion deblurring, " in Proc. IEEE Int. Conf. Image Process., Phoenix, AZ, USA, 2016, pp. 3832-3836.
25. B. Cai, X. Xu, K. Jia, C. Qing, and D. Tao, "DehazeNet: An end-toend system for single image haze removal, " IEEE Trans. Image Process., vol. 25, no. 11, pp. 5187-5198, Nov. 2016.
26. N. Cai, Z. H. Su, Z. N. Lin, H. Wang, Z. J. Yang, and B. W. K. Ling, "Blind inpainting using the fully convolutional neural network, " Vis. Comput., vol. 33, pp. 249-261, Feb. 2017.
27. J. Kim, J. K. Lee, and K. M. Lee, "Accurate image super-resolution using very deep convolutional networks, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Las Vegas, NV, USA, 2016, pp. 1646-1654.
28. J. Kim, J. K. Lee, and K. M. Lee, "Deeply-recursive convolutional network for image super-resolution, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Las Vegas, NV, USA, 2016, pp. 1637-1645.
29. X. J. Mao, C. Shen, and Y. B. Yang, "Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections, " Adv. Neural Inf. Process. Syst., pp. 2802-2810, 2016.
30. Y. Wang, L. Wang, H. Wang, and P. Li, "End-to-end image super-resolution via deep and shallow convolutional networks, " arXiv:1607. 07680, 2016.
31. L. Zhang, L. Zhang, and B. Du, "Deep learning for remote sensing data: A technical tutorial on the state of the art, " IEEE Geosci. Remote Sens. Mag., vol. 4, no. 2, pp. 22-40, Jun. 2016.
32. W. Huang, L. Xiao, Z. Wei, H. Liu, and S. Tang, "A new pan-sharpening method with deep neural networks, " IEEE Geosci. Remote Sens. Lett., vol. 12, no. 5, pp. 1037-1041, May 2015.
33. G. Masi, D. Cozzolino, L. Verdoliva, and G. Scarpa, "Pansharpening by convolutional neural networks, " Remote Sens., vol. 8, Jul. 2016, Art. no. 594.
34. A. Garzelli, "A review of image fusion algorithms based on the superresolution paradigm, " Remote Sens., vol. 8, Oct. 2016, Art. no. 797.
35. C. Szegedy et al., "Going deeper with convolutions, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Boston, MA, USA, 2015, pp. 1-9.
36. D. Erhan, Y. Bengio, A. Courville, and P. Vincent, "Visualizing higherlayer features of a deep network, " Univ. Montreal, Montreal, QC, Canada, Tech. Rep. 1341, 2009.
37. J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson, "Understanding neural networks through deep visualization, " Presented at the Deep Learn. Workshop, Int. Conf. Machine Learn., 2015.
38. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Las Vegas, NV, USA, 2016, pp. 770-778.
39. B. T. Polyak, "Some methods of speeding up the convergence of iteration methods, " USSR Comput. Math. Math. Phys., vol. 4, pp. 1-17, 1964.
40. Y. Jia et al., "Caffe: Convolutional architecture for fast feature embedding, " in Proc. 22nd ACM Int. Conf. Multimedia, 2014, pp. 675-678.
41. A. Vedaldi and K. Lenc, "MatConvNet: Convolutional Neural Networks for MATLAB, " ACM Int. Conf. Multimedia, pp. 689-692, 2015.
42. X. Otazu, M. Gonzalez-Audicana, O. Fors, and J. Nunez, "Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods, " IEEE Trans. Geosci. Remote Sens., vol. 43, no. 10, pp. 2376-2385, Oct. 2005.
43. G. Vivone, L. Alparone, J. Chanussot, M. Dalla Mura, A. Garzelli, G. A. Licciardi et al., "Acritical comparison among pansharpening algorithms, " IEEE Trans. Geosci. Remote Sens., vol. 53, pp. 2565-2586, May 2015.
44. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: From error visibility to structural similarity, " IEEE Trans. Image Process., vol. 13, no. 4, pp. 600-612, Apr. 2004.
45. Z. Wang and A. C. Bovik, "A universal image quality index, " IEEE Signal Process. Lett., vol. 9, no. 3, pp. 81-84, Mar. 2002.
46. L. Wald, Data fusion: definitions and architectures: Fusion of images of different spatial resolutions. Paris, France: Presses des MINES, 2002.
47. R. H. Yuhas, A. F. Goetz, and J. W. Boardman, "Discrimination among semi-arid landscape endmembers using the spectral angle mapper (SAM) algorithm, " in Proc. Annu. JPL Airborne Geosci. Workshop, 1992, pp. 147-149.
48. A. Garzelli and F. Nencini, "Hypercomplex quality assessment of multi-/hyper-spectral images, " IEEE Geosci. Remote Sens. Lett., vol. 6, no. 4, pp. 662-665, Oct. 2009.
49. L. Alparone, B. Alazzi, S. Baronti, A. Garzelli, F. Nencini, and M. Selva, "Multispectral and panchromatic data fusion assessment without reference, " Photogramm. Eng. Remote Sens., vol. 74, pp. 193-200, Feb. 2008.
50. P. Wu, H. Shen, L. Zhang, and F. M. Göttsche, "Integrated fusion of multi-scale polar-orbiting and geostationary satellite observations for the mapping of high spatial and temporal resolution land surface temperature, " Remote Sens. Environ., vol. 156, pp. 169-181, 2015.
51. J. Li, Q. Yuan, H. Shen, and L. Zhang, "Noise removal from hyperspectral image with joint spectral-spatial distributed sparse representation, " IEEE Trans. Geosci. Remote Sens., vol. 54, no. 9, pp. 5425-5439, Sep. 2016.
52. Q. Yuan, L. Zhang, and H. Shen, "Hyperspectral image denoising employing a spectral-spatial adaptive total variation model, " IEEE Trans. Geosci. Remote Sens., vol. 50, no. 10, pp. 3660-3677, Oct. 2012.
53. F. Hu, G. S. Xia, J. W. Hu, and L. P. Zhang, "Transferring deep convolutional neural networks for the scene classification of high-resolution remote sensing imagery, " Remote Sens., vol. 7, pp. 14680-14707, Nov. 2015.
54. G. S. Xia et al., "AID: A benchmark data set for performance evaluation of aerial scene classification, " IEEE Trans. Geosci. Remote Sens., vol. 55, no. 7, pp. 3965-3981, Jul. 2017.
55. F. Zhang, B. Du, L. Zhang, and M. Xu, "Weakly supervised learning based on coupled convolutional neural networks for aircraft detection, " IEEE Trans. Geosci. Remote Sens., vol. 54, no. 9, pp. 5553-5563, Sep. 2016.