A deep learning framework for remote sensing image registration

ISPRS Journal of Photogrammetry and Remote Sensing

Volumn 145, Issue , 2018, Pages 148-164

  Wang, Shuang ^a   Quan, Dou ^a   Liang, Xuefeng ^b   Ning, Mengdan ^a   Guo, Yanhe ^a   Jiao, Licheng ^a  

^a XIDIAN UNIVERSITY   (China)

^b KYOTO UNIVERSITY   (Japan)

Author keywords

Deep neural network; Image registration; Remote sensing image; Self learning; Transfer learning

Indexed keywords

DEEP NEURAL NETWORKS; IMAGE ENHANCEMENT; IMAGE REGISTRATION; MAPPING;

CONVENTIONAL METHODS; INFORMATION FEEDBACK; LEARNING MAPPING FUNCTIONS; REGISTRATION ACCURACY; REGISTRATION PROBLEMS; REMOTE SENSING IMAGES; SELF-LEARNING; TRANSFER LEARNING;

REMOTE SENSING;

ALGORITHM; ARTIFICIAL NEURAL NETWORK; EXPERIMENTAL DESIGN; FOOD PROCESSING; IMAGE ANALYSIS; IMAGE PROCESSING; LANDSAT; MAPPING METHOD; RADARSAT; REMOTE SENSING; SATELLITE IMAGERY; SPOT;

EID: 85039951284     PISSN: 09242716     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.isprsjprs.2017.12.012     Document Type: Article

References (71)

1. Bay, H., Ess, A., Tuytelaars, T., Gool, L.V., Speeded-up robust features (surf). Comput. Vis. Image Understand. 110:3 (2008), 346–359, 10.1016/j.cviu.2007.09.014.
2. Bengio, Y., Learning deep architectures for ai. Found. Trends Mach. Learn. 2:1 (2009), 1–55, 10.1561/2200000006.
3. Cheng, G., Han, J., Guo, L., Liu, Z., Bu, S., Ren, J., Effective and efficient midlevel visual elements-oriented land-use classification using vhr remote sensing images. IEEE Trans. Geosci. Rem. Sens 53:8 (2015), 4238–4249, 10.1109/TGRS.2015.2393857.
4. Cheng, G., Zhou, P., Han, J., Learning rotation-invariant convolutional neural networks for object detection in vhr optical remote sensing images. IEEE Trans. Geosci. Remote Sens 54:12 (2016), 7405–7415, 10.1109/TGRS.2016.2601622.
5. Cheng, G., Zhou, P., Han, J., 2016b. Rifd-cnn: Rotation-invariant and fisher discriminative convolutional neural networks for object detection (June), 2884–2893, doi: https://doi.org/10.1109/CVPR.2016.315.
6. Dalal, N., Triggs, B., Histograms of oriented gradients for human detection. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recog. 1:12 (2005), 886–893, 10.1109/CVPR.2005.177.
7. Dellinger, F., Delon, J., Gousseau, Y., Michel, J., Sar-sift: a sift-like algorithm for SAR images. IEEE Trans. Geosci. Remote Sens. 53:1 (2015), 453–466, 10.1109/TGRS.2014.2323552.
8. Erhan, D., Bengio, Y., Courville, A., Manzagol, P.A., Vincent, P., Bengio, S., Why does unsupervised pre-training help deep learning? J. Mach. Learn. Res. 11:January (2010), 625–660.
9. Fan, B., Huo, C., Pan, C., Kong, Q., Registration of optical and SAR satellite images by exploring the spatial relationship of the improved sift. IEEE Geosci. Remote Sens. Lett 10:4 (2013), 657–661, 10.1109/LGRS.2012.2216500.
10. Fan, J., Wu, Y., Wang, F., Zhang, Q., Liao, G., Li, M., Sar image registration using phase congruency and nonlinear diffusion-based sift. IEEE Geosci. Remote Sens. Lett 12:3 (2015), 562–566, 10.1109/LGRS.2014.2351396.
11. Farabet, C., Couprie, C., Najman, L., Lecun, Y., Learning hierarchical features for scene labeling. IEEE Trans. Pattern Anal. Mach. Intell 35:8 (2013), 1915–1929, 10.1109/TPAMI.2012.231.
12. Fischler, M.A., Bolles, R.C., Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24:6 (1981), 381–395, 10.1145/358669.358692.
13. Goncalves, H., Corte-Real, L., Goncalves, J., Automatic image registration through image segmentation and sift. IEEE Trans. Geosci. Remote Sens 49:7 (2011), 2589–2600, 10.1109/TGRS.2011.2109389.
14. Goncalves, H., Goncalves, J.A., Corte-Real, L., Measures for an objective evaluation of the geometric correction process quality. IEEE Geosci. Remote Sens. Lett 6:2 (2009), 292–296, 10.1109/LGRS.2008.2012441.
15. Gong, M., Zhao, J., Liu, J., Miao, Q., Jiao, L., Change detection in synthetic aperture radar images based on deep neural networks. IEEE Trans. Neural Netw. Learn Syst 27:1 (2016), 125–138, 10.1109/TNNLS.2015.2435783.
16. Gong, M., Zhao, S., Jiao, L., Tian, D., A novel coarse-to-fine scheme for automatic image registration based on sift and mutual information. IEEE Trans. Geosci. Remote Sens 52:7 (2014), 4328–4338, 10.1109/TGRS.2013.2281391.
17. Graves, A., Mohamed, A.R., Hinton, G.E., Speech recognition with deep recurrent neural networks. IEEE Int. Conf. Acoust Speech Signal Process 38:2003 (2013), 6645–6649, 10.1109/ICASSP.2013.6638947.
18. Han, J., Zhang, D., Cheng, G., Guo, L., Ren, J., Object detection in optical remote sensing images based on weakly supervised learning and high-level feature learning. IEEE Trans. Geosci. Remote Sens 53:6 (2015), 3325–3337, 10.1109/TGRS.2014.2374218.
19. Han, X., Leung, T., Jia, Y., Sukthankar, R., Berg, A.C., 2015b. Matchnet: Unifying feature and metric learning for patch-based matching (October), 3279–3286, doi: https://doi.org/10.1109/CVPR.2015.7298948.
20. Hinton, G.E., Training products of experts by minimizing contrastive divergence. Neural Comput 14:8 (2002), 1771–1800, 10.1162/089976602760128018.
21. Hinton, G.E., A practical guide to training restricted boltzmann machines. Momentum 9:1 (2010), 599–619.
22. Hinton, G.E., Deng, L., Yu, D., Dahl, G.E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T.N., Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Sig. Process. Mag. 29:6 (2012), 82–97, 10.1109/MSP.2012.2205597.
23. Hinton, G.E., Osindero, S., Teh, Y.W., A fast learning algorithm for deep belief nets. Neural Comput. 18:7 (2006), 1527–1554, 10.1162/neco.2006.18.7.1527.
24. Jacqueline, L.M., 2017. Introduction to remote sensing image registration. In: IEEE International Geoscience and Remote Sensing Symposium.
25. Kern, J.P., Pattichis, M.S., Robust multispectral image registration using mutual-information models. IEEE Trans. Geosci. Rem. Sens. 45:5 (2007), 1494–1505, 10.1109/TGRS.2007.892599.
26. Krizhevsky, A., Sutskever, I., Hinton, G.E., Dec. 2012. Imagenet classification with deep convolutional neural networks. pp. 1097–1105.
27. Kupfer, B., Netanyahu, N.S., Shimshoni, I., An efficient sift-based mode-seeking algorithm for sub-pixel registration of remotely sensed images. IEEE Geosci. Remote Sens. Lett. 12:2 (2015), 379–383, 10.1109/LGRS.2014.2343471.
28. Lecun, Y., Bengio, Y., Hinton, G.E., Deep learning. Nature 521:7553 (2015), 436–444, 10.1038/nature14539.
29. Liang, J., Liu, X., Huang, K., Li, X., Wang, D., Wang, X., Automatic registration of multisensor images using an integrated spatial and mutual information (SMI) metric. IEEE Trans. Geosci. Rem. Sens 52:1 (2013), 603–615, 10.1109/TGRS.2013.2242895.
30. Liu, J., Gong, M., Qin, K., Zhang, P., A deep convolutional coupling network for change detection based on heterogeneous optical and radar images. IEEE Trans. Neural Netw. Learn Syst. 99:August (2016), 1–15, 10.1109/TNNLS.2016.2636227.
31. Lowe, D.G., Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60:2 (2004), 91–110, 10.1023/B:VISI.0000029664.99615.94.
32. Ma, J., Chan, J., Canters, F., Fully automatic subpixel image registration of multiangle chris/proba data. IEEE Trans. Geosci. Remote Sens 48:7 (2010), 2829–2839, 10.1109/TGRS.2010.2042813.
33. Ma, W., Wen, Z., Wu, Y., Jiao, L., Gong, M., Zheng, Y., Liu, L., Remote sensing image registration with modified sift and enhanced feature matching. IEEE Geosci. Remote Sens. Lett 14:1 (2017), 3–7, 10.1109/LGRS.2016.2600858.
34. Ma, X., Wang, H., Wang, J., Semisupervised classification for hyperspectral image based on multi-decision labeling and deep feature learning. ISPRS J. Photogramm Remote Sens 120 (2016), 99–107, 10.1016/j.isprsjprs.2016.09.001.
35. Maes, F., Collignon, A., Vandermeulen, D., Marchal, G., Suetens, P., Multimodality image registration by maximization of mutual information. IEEE Trans. Med. Imag. 16:2 (1997), 187–198, 10.1109/42.563664.
36. Matas, J., Chum, O., Urban, M., Pajdla, T., Robust wide-baseline stereo from maximally stable extremal regions. Image Vis. Comput. 22:10 (2004), 761–767, 10.1016/j.imavis.2004.02.006.
37. Morel, J.M., Yu, G., Asift: a new framework for fully affine invariant image comparison. SIAM J. Imag. Sci. 2:2 (2009), 438–469, 10.1137/080732730.
38. Parmehr, E.G., Zhang, C., Fraser, C.S., Automatic registration of multi-source data using mutual information. ISPRS J. Photogramm. Rem. Sens. I-7:July (2012), 303–308, 10.5194/isprsannals-I-7-303-2012.
39. Patel, M.I., Thakar, V.K., 2015. Speed improvement in image registration using maximum likelihood based mutual information. In: International Conference on Advanced Computing and Communication Systems. (Jan), https://doi.org/10.1109/ICACCS.2015.7324130.
40. Reddy, B.S., Chatterji, B.N., An fft-based technique for translation, rotation, and scale-invariant image registration. IEEE Trans. Image Process. Publ. IEEE Signal Process. Soc., 5(8), 1996, 1266, 10.1109/83.506761.
41. Ren, S., He, K., Girshick, R., Sun, J., Faster r-cnn: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39:6 (2016), 1137–1149, 10.1109/TPAMI.2016.2577031.
42. Romero, A., Gatta, C., Camps-Valls, G., Unsupervised deep feature extraction for remote sensing image classification. IEEE Trans. Geosci. Remote Sens. 54:3 (2016), 1349–1362, 10.1109/TGRS.2015.2478379.
43. Rumelhart, D.E., Hinton, G.E., Williams, R.J., Learning representations by back-propagating errors. Nature 323:6088 (1986), 533–536, 10.1038/323533a0.
44. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115:3 (2015), 211–252, 10.1007/s11263-015-0816-y.
45. Schmidhuber, J., Deep learning in neural networks: an overview. Neural Netw. 61:October (2014), 85–117, 10.1016/j.neunet.2014.09.003.
46. Schwind, P., Suri, S., Reinartz, P., Siebert, A., Applicability of the sift operator to geometric SAR image registration. Int. J. Remote Sens. 31:8 (2010), 1959–1980, 10.1080/01431160902927622.
47. Scott, G.J., England, M.R., Starms, W.A., Marcum, R.A., Davis, C.H., Training deep convolutional neural networks for land-cover classification of high-resolution imagery. IEEE Geosci. Rem. Sens. Lett. 14:4 (2017), 549–553, 10.1109/LGRS.2017.2657778.
48. Sedaghat, A., Mokhtarzade, M., Ebadi, H., Uniform robust scale-invariant feature matching for optical remote sensing images. IEEE Trans. Geosci. Remote Sens. 49:11 (2011), 4516–4527, 10.1109/TGRS.2011.2144607.
49. Simonyan, K., Zisserman, A., 2014. Very deep convolutional networks for large-scale image recognition. Comp. Sci. (September), ArXiv:1409.1556.
50. Simoserra, E., Trulls, E., Ferraz, L., Kokkinos, I., Fua, P., Morenonoguer, F., Dec. 2015. Discriminative learning of deep convolutional feature point descriptors, pp. 118–126, doi: https://doi.org/10.1109/ICCV.2015.22.
51. Suri, S., Reinartz, P., Mutual-information-based registration of terrasar-x and ikonos imagery in urban areas. IEEE Trans. Geosci. Remote Sens. 48:2 (2009), 939–949, 10.1109/TGRS.2009.2034842.
52. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.A., Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11:December (2010), 3371–3408.
53. Wang, S., You, H., Fu, K., Bfsift: a novel method to find feature matches for SAR image registration. IEEE Geosci. Remote Sens. Lett 9:4 (2012), 649–653, 10.1109/LGRS.2011.2177437.
54. Wu, G., Kim, M.J., Wang, Q., Munsell, B., Shen, D., Scalable high performance image registration framework by unsupervised deep feature representations learning. IEEE Trans. Biomed. Eng. 63:7 (2016), 1505–1516, 10.1109/TBME.2015.2496253.
55. Wu, Y., Ma, W., Gong, M., Su, L., Jiao, L., A novel point-matching algorithm based on fast sample consensus for image registration. IEEE Geosci. Rem. Sens. Lett. 12:1 (2015), 43–47, 10.1109/LGRS.2014.2325970.
56. Xu, C., Sui, H.G., Li, D.R., Sun, K.M., Liu, J.Y., An automatic optical and SAR image registration method using iterative multi-level and refinement model. ISPRS J. Photogramm. Rem. Sens XLI-B7:June (2016), 593–600, 10.5194/isprsarchives-XLI-B7-593-2016.
57. Xu, X., Li, X., Liu, X., Shen, H., Shi, Q., Multimodal registration of remotely sensed images based on Jeffreys divergence. ISPRS J. Photogramm. Remote Sens. 122:December (2016), 97–115, 10.1016/j.isprsjprs.2016.10.005.
58. Yang, K., Pan, A., Yang, Y., Zhang, S., Ong, S., Tang, H., Remote sensing image registration using multiple image features. Rem. Sens., 9(6), 2017, 581, 10.3390/rs9060581.
59. Yao, X., Han, J., Cheng, G., Qian, X., Guo, L., Semantic annotation of high-resolution satellite images via weakly supervised learning. IEEE Trans. Geosci. Remote Sens. 54:6 (2016), 3660–3671, 10.1109/TGRS.2016.2523563.
60. Ye, Y., Shan, J., A local descriptor based registration method for multispectral remote sensing images with non-linear intensity differences. ISPRS J. Photogramm. Remote Sens. 90:3 (2014), 83–95, 10.1016/j.isprsjprs.2014.01.009.
61. Ye, Y., Shan, J., Bruzzone, L., Shen, L., Robust registration of multimodal remote sensing images based on structural similarity. IEEE Trans. Geosci. Remote Sens. 55:5 (2017), 2941–2958, 10.1109/TGRS.2017.2656380.
62. Ye, Y., Shen, L., Hopc: a novel similarity metric based on geometric structural properties for multi-modal remote sensing image matching. ISPRS Ann. Photogram. Rem. Sens. Spatial Inf. III-1:June (2016), 9–16, 10.5194/isprs-annals-III-1-9-2016.
63. Ye, Y., Shen, L., Hao, M., Wang, J., Xu, Z., Robust optical-to-SAR image matching based on shape properties. IEEE Geosci. Remote Sens. Lett. 14:4 (2017), 564–568, 10.1109/LGRS.2017.2660067.
64. Yong, S.H., Lee, K.M., Sang, U.L., 2009. Mutual information-based stereo matching combined with sift descriptor in log-chromaticity color space (August), 445–452, doi: https://doi.org/10.1109/CVPR.2009.5206507.
65. Yosinski, J., Clune, J., Bengio, Y., Lipson, H., 2014. How transferable are features in deep neural networks? In: Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 (NIPS’14) (December), pp. 3320–3328.
66. Zagoruyko, S., Komodakis, N., 2015. Learning to compare image patches via convolutional neural networks (June), 4353–4361, doi: https://doi.org/10.1109/CVPR.2015.7299064.
67. Zhang, P., Gong, M., Su, L., Liu, J., Li, Z., Change detection based on deep feature representation and mapping transformation for multi-spatial-resolution remote sensing images. ISPRS J. Photogramm. Remote Sens. 116:February (2016), 24–41, 10.1016/j.isprsjprs.2016.02.013.
68. Zhao, W., Du, S., Learning multiscale and deep representations for classifying remotely sensed imagery. ISPRS J. Photogramm Remote Sens. 113:March (2016), 155–165, 10.1016/j.isprsjprs.2016.01.004.
69. Zhao, W., Du, S., Spectral-spatial feature extraction for hyperspectral image classification: a dimension reduction and deep learning approach. IEEE Trans. Geosci. Remote Sens. 54:8 (2016), 4544–4554, 10.1109/TGRS.2016.2543748.
70. Zhou, P., Cheng, G., Liu, Z., Bu, S., Hu, X., Weakly supervised target detection in remote sensing images based on transferred deep features and negative bootstrapping. Multidim Syst. Sign. Process. 27:4 (2016), 925–944, 10.1007/s11045-015-0370-3.
71. Zitova, B., Flusser, J., Image registration methods: a survey. Image Vis. Comput. 21:11 (2003), 977–1000, 10.1016/S0262-8856(03)00137-9.