Toward Seamless Multiview Scene Analysis from Satellite to Street Level

Proceedings of the IEEE

Volumn 105, Issue 10, 2017, Pages 1884-1899

  Lefevre, Sebastien ^a   Tuia, Devis ^b,c   Wegner, Jan Dirk ^d   Produit, Timothee ^e   Nassaar, Ahmed Samy ^a  

^a UNIVERSITÉ BRETAGNE SUD   (France)

^b UNIVERSITY OF ZURICH   (Switzerland)

^c WAGENINGEN UNIVERSITY   (Netherlands)

^d ETH ZURICH   (Switzerland)

^e University of Applied Sciences   (Switzerland)

Author keywords

classification; Computer vision; data fusion; ground based imagery; localization; object detection; remote sensing

Indexed keywords

CLASSIFICATION (OF INFORMATION); COMPUTER VISION; DATA FUSION; OBJECT DETECTION; OBJECT RECOGNITION; SATELLITE IMAGERY;

COMPLEMENTARY DATA; CROSS FERTILIZATION; EARTH OBSERVATION SENSORS; GROUND-BASED IMAGERIES; LOCALIZATION; REAL APPLICATIONS; SCENE REGISTRATION; SCENE UNDERSTANDING;

REMOTE SENSING;

EID: 85018876359     PISSN: 00189219     EISSN: 15582256     Source Type: Journal    
DOI: 10.1109/JPROC.2017.2684300     Document Type: Article

References (86)

1. D. Belcher, "The five facets of aerial photography, " Photogramm. Eng., vol. 19, no. 5, pp. 746-752, 1953.
2. J. Luo, D. Joshi, J. Yu, and A. Gallagher, "Geotagging in multimedia and computer vision-A survey, " Multimedia Tools Appl., vol. 51, no. 1, pp. 187-211, 2011. [Online]. Available: http://dx. doi. org/10. 1007/s11042-010-0623-y
3. L. Vincent, "Taking online maps down to street level, " Computer, vol. 40, no. 12, pp. 118-120, Dec. 2007.
4. N. Ghouaiel and S. Lefèvre, "Coupling ground-level panoramas and aerial imagery for change detection, " Geo-Spatial Inf. Sci., vol. 19, no. 3, pp. 222-232, 2016.
5. T. Produit, D. Tuia, and V. Lepetit, "Pose estimation of Web-shared landscape pictures, " in Proc. PCV Worskhop, ISPRS Ann., 2014.
6. J. D. Wegner, S. Branson, D. Hall, K. Schindler, and P. Perona, "Cataloging public objects using aerial and street-level images-Urban trees, " in Proc. CVPR, 2016, pp. 6014-6023.
7. D. Leung and S. Newsam, "Proximate sensing: Inferring what-is-where from georeferenced photo collections, " in Proc. CVPR, 2010, pp. 2955-2962.
8. J. Hays and A. A. Efros, "IM2GPS: Estimating geographic information from a single image, " in Proc. CVPR, 2008, pp. 1-8.
9. P. L. Lai and C. Fyfe, "Kernel and nonlinear canonical correlation nalysis, " Int. J. Neural Syst., vol. 10, pp. 365-377, Oct. 2000.
10. T.-Y. Lin, S. Belongie, and J. Hays, "Crossview image geolocalization, " in Proc. CVPR, 2013, pp. 891-898.
11. S. Workman, R. Souvenir, and N. Jacobs, "Wide-area image geolocalization with aerial reference imagery, " in Proc. ICCV, 2015, pp. 3961-3969.
12. R. Hadsell, S. Chopra, and Y. LeCun, "Dimensionality reduction by learning an invariant mapping, " in Proc. CVPR, 2006, pp. 1735-1742.
13. T.-Y. Lin, Y. Cui, S. Belongie, and J. Hays, "Learning deep representations for groundto-aerial geolocalization, " in Proc. CVPR, 2015, pp. 5007-5015.
14. N. N. Vo and J. Hays, "Localizing and orienting street views using overhead imagery, " in Proc. ECCV, 2016, pp. 494-509.
15. M. Hussein, M. Renner, M. Watanabe, and K. Iagnemma, "Matching of ground-based LiDAR and aerial image data for mobile robot localization in densely forested environments, " in Proc. IROS, 2013, pp. 1432-1437.
16. K. Y. K. Leung, C. M. Clark, and J. P. Huisson, "Localization in urban environments by matching ground level video images with an aerial image, " in Proc. ICRA, 2008, pp. 551-556.
17. A. Viswanathan, B. R. Pires, and D. Huber, "Vision based robot localization by ground to satellite matching in GPS-denied situations, " in Proc. IROS, 2014, pp. 192-198.
18. C. Früh and A. Zakhor, "Constructing 3D city models by merging ground-based and airborne views, " in Proc. CVPR, 2003, pp. II-562-II-569.
19. C. Fruh and A. Zakhor, "Constructing 3D city models by merging aerial and ground views, " IEEE Comput. Graph. Appl., vol. 23, no. 6, pp. 52-61, Nov. 2003.
20. M. Fiocco, G. Bostrom, J. G. M. Goncalves, and V. Sequeira, "Multisensor fusion for volumetric reconstruction of large outdoor areas, " in Proc. Int. Conf. 3-D Digit. Imag. Modeling, 2005, pp. 47-54.
21. R. S. Kaminsky, N. Snavely, S. M. Seitz, and R. Szeliski, "Alignment of 3D point clouds to overhead images, " in Proc. IEEE Workshop Internet Vis., 2009, pp. 63-70.
22. A. Bódis-Szomorú, H. Riemenschneider, and L. van Gool, "Efficient volumetric fusion of airborne and street-side data for urban reconstruction, " in Proc. ICPR, 2016 pp. 127-134.
23. M. Bansal and K. Daniilidis, "Geometric urban geo-localization, " in Proc. CVPR, 2014, pp. 3978-3985.
24. R. I. Hammoud et al., "Overhead-based image and video geo-localization framework, " in Proc. CVPR Workshops, 2013, pp. 320-327.
25. Q. Shan, C. Wu, B. Curless, Y. Furukawa, C. Hernandez, and S. M. Seitz, "Accurate geo-registration by ground-to-aerial image matching, " in Proc. Int. Conf. 3D Vis., 2014, pp. 525-532.
26. D. G. Lowe, "Distinctive image features from scale-invariant keypoints, " Int. J. Comput. Vis., vol. 60, no. 2, pp. 91-110, 2004.
27. M. Bansal, H. S. Sawhney, H. Cheng, and K. Daniilidis, "Geo-localization of street views with aerial image databases, " in Proc. MM, 2011, pp. 1125-1128.
28. M. Bansal, K. Daniilidis, and H. S. Sawhney, "Ultra-wide baseline facade matching for geo-localization, " in Proc. ECCV, 2012, pp. 175-186.
29. T.-J. Cham, A. Ciptadi, W.-C. Tan, M.-T. Pham, and L.-T. Chia, "Estimating camera pose from a single urban ground-view omnidirectional image and a 2D building outline map, " in Proc. CVPR, 2010, pp. 366-373.
30. L. Baboud, M. Cadik, E. Eisemann, and H.-P. Seidel, "Automatic photo-to-terrain alignment for the annotation of mountain pictures, " in Proc. CVPR, 2011, pp. 41-48.
31. G. Baatz, O. Saurer, K. Köser, and M. Pollefeys, "Large scale visual geo-localization of images in mountainous terrain, " in Proc. ECCV, 2012, pp. 517-530.
32. P. Chippendale, M. Zanin, and C. Andreatta, "Spatial and temporal attractiveness analysis through geo-referenced photo alignment, " in Proc. IGARSS, 2008, pp. II-1116-II-1119.
33. L. Gómez-Chova, D. Tuia, G. Moser, and G. Camps-Valls, "Multimodal classification of remote sensing images: A review and future directions, " Proc. IEEE, vol. 103, no. 9, pp. 1560-1584, Sep. 2015.
34. C. C. Fonte, L. Bastin, L. See, G. Foody, and F. Lupia, "Usability of VGI for validation of land cover maps, " Int. J. Geograph. Inf. Sci., vol. 29, no. 7, pp. 1269-1291, 2015.
35. G. M. Foody and D. S. Boyd, "Using volunteered data in land cover map validation: Mapping West African forests, " IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 6, no. 3, pp. 1305-1312, Jun. 2013.
36. D. Leung and S. Newsam, "Land cover classification using geo-referenced photos, " Multimedia Tools Appl., vol. 74, no. 24, pp. 11741-11761, 2015.
37. H. Zhang, M. Korayem, D. J. Crandall, and G. LeBuhn, "Mining photo-sharing websites to study ecological phenomena, " in Proc. WWW, 2012, pp. 749-758.
38. J. Wang, M. Korayem, and D. J. Crandall, "Observing the natural world with Flickr, " in Proc. ICCV Worskhops, 2013, pp. 452-459.
39. J. Estima and M. Painho, "Flickr geotagged and publicly available photos: Preliminary study of its adequacy for helping quality control of Corine Land Cover, " in Proc. ICCSA, 2013, pp. 205-220.
40. J. Estima, C. C. Fonte, and M. Painho, "Comparative study of land use/cover classification using Flickr photos, satellite imagery and Corine Land Cover database, " in Proc. AGILE, 2014 p. 4, ISBN: 978-90-816960-4-3.
41. G. Matthyus, S. Wang, S. Fidler, and R. Urtasun, "HD maps: Fine-grained road segmentation by parsing ground and aerial images, " in Proc. CVPR, 2016, pp. 3611-3619.
42. G. Bakir, T. Hofmann, B. Schölkopf, A. J. Smola, and S. Vishwanathan, Predicting Structured Data. Cambridge, MA, USA: MIT Press, 2007.
43. A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, "Vision meets robotics: The KITTI dataset, " Int. J. Robot. Res., vol. 32, no. 11, pp. 1231-1237, 2013.
44. C. Doersch, S. Singh, A. Gupta, J. Sivic, and A. A. Efros, "What makes Paris look like Paris?" ACM Trans. Graph., vol. 31, no. 3, pp. 101:1-101:9, 2012.
45. B. Zhou, L. Liu, A. Olive, and A. Torralba, "Recognizing city identity via attribute analysis of geo-tagged images, " in Proc. ECCV, 2014, pp. 519-534.
46. S. M. Arietta, A. A. Efros, R. Ramamoorthi, and M. Agrawala, "City forensics: Using visual elements to predict non-visual city attributes, " IEEE Trans. Vis. Comput. Graphics, vol. 20, no. 12, pp. 2624-2633, Dec. 2014.
47. S. Workman and N. Jacobs, "On the location dependence of convolutional neural network features, " in Proc. CVPR Workshops, 2015, pp. 70-78.
48. S. Lee, H. Zhang, and D. J. Crandall, "Predicting geo-informative attributes in large-scale image collections using convolutional neural networks, " in Proc. WACV, 2015, pp. 550-557.
49. T. Produit, D. Tuia, F. de Morsier, and F. Golay, "Do geographic features impact pictures location shared on the Web? Modeling photographic suitability in the Swiss Alps, " in Proc. Int. Workshop Environ. Multimedia Retr., 2014, p. 22.
50. B. Schölkopf, R. Williamson, A. Smola, J. Shawe-Taylor, and J. Platt, "Support vector method for novelty detection, " in Proc. NIPS, 1999, pp. 582-588.
51. J. Luo, J. Yu, D. Joshi, and W. Hao, "Event recognition: Viewing the world with a third eye, " in Proc. MM, 2008, pp. 1071-1080.
52. D. Tuia, C. Persello, and L. Bruzzone, "Domain adaptation for the classification of remote sensing data: An overview of recent advances, " IEEE Geosci. Remote Sens. Mag., vol. 4, no. 2, pp. 41-57, Jun. 2016.
53. Z. Sun, C. Wang, H. Wang, and J. Li, "Learn multiple-kernel SVMs for domain adaptation in hyperspectral data, " IEEE Geosci. Remote Sens. Lett., vol. 10, no. 5, pp. 1224-1228, Sep. 2013.
54. H. Sun, S. Liu, S. Zhou, and H. Zou, "Unsupervised cross-view semantic transfer for remote sensing image classification, " IEEE Geosci. Remote Sens. Lett., vol. 13, no. 1, pp. 13-17, Jan. 2016.
55. J. Xiao, K. A. Ehinger, A. Oliva, and A. Torralba, "Recognizing scene viewpoint using panoramic place representation, " in Proc. CVPR, 2012, pp. 2695-2702.
56. O. Augereau, N. Journet, and J.-P. Domenger, "Semi-structured document image matching and recognition, " in Proc. EI, 2013 p. 12 doi:10. 1117/12. 2003911.
57. M. Muja and D. G. Lowe, "Fast approximate nearest neighbors with automatic algorithm configuration, " in Proc. VISAPP, 2009, pp. 331-340.
58. A. Agarwal, C. V. Jawahar, and P. J. Narayanan, "A survey of planar homography estimation techniques, " Centre Vis. Inf. Technol., Tech. Rep. IIIT/TR/2005/12, 2005.
59. M. A. Fischler and R. Bolles, "Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography, " Commun. ACM, vol. 24, no. 6, pp. 381-395, 1981.
60. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks, " in Proc. NIPS, 2012, pp. 1097-1105.
61. Y. Jia et al., "Caffe: Convolutional architecture for fast feature embedding, " in Proc. MM, 2014, pp. 675-678.
62. R. Shirvany, M. Chabert, F. Chatelain, and J.-Y. Tourneret, "Maximum-likelihood estimation of the polarization degree from two multi-look intensity images, " in Proc. ICASSP, 2010, pp. 1198-1201.
63. Q. Xu and L. J. Karam, "Change detection on SAR images by a parametric estimation of the KL-divergence between Gaussian mixture models, " in Proc. ICASSP, 2013, pp. 2109-2113.
64. W. Liu and F. Yamazaki, "Urban monitoring and change detection of central Tokyo using high-resolution X-band SAR images, " in Proc. IGARSS, 2011, pp. 2133-2136.
65. S. Chopra, R. Hadsell, and Y. LeCun, "Learning a similarity metric discriminatively, with application to face verification, " in Proc. CVPR, 2005, pp. 539-546.
66. T.-Y. Lin, Y. Cui, S. Belongie, and J. Hays, "Learning deep representations for groundto-aerial geolocalization, " in Proc. CVPR, 2015, pp. 5007-5015.
67. R. Hadsell, S. Chopra, and Y. LeCun, "Dimensionality reduction by learning an invariant mapping, " in Proc. CVPR, 2006, pp. 1735-1742.
68. R. Webb, D. Boyer, and R. Turner, Eds., Repeat Photography: Methods and Applications in the Natural Sciences. Washington, DC, USA: Island Press, 2010.
69. D. J. Berndt and J. Clifford, "Using dynamic time warping to find patterns in time series, " in Proc. Int. Conf. Knowl. Disc. Data Mining, 1994 pp. 229-248.
70. K. Kraus and I. A. Harley, Eds., Photogrammetry: Geometry From Images and Laser Scans. Berlin, Germany: Walter de Gruyter, 2007.
71. M. Fischler and R. Bolles, "Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography, " Commun. ACM, vol. 24, no. 6, pp. 381-395, 1981.
72. R. Girshick, "Fast R-CNN, " in Proc. ICCV, 2015, pp. 1440-1448.
73. S. Ren, K. He, R. Girshick, and J. Sun, "Faster R-CNN: Towards real-time object detection with region proposal networks, " in Proc. NIPS, 2015, pp. 91-99.
74. K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition, " in Proc. ICLR, 2015 p. 14, arXiv:1409. 1556v6.
75. C. Szegedy et al., "Going deeper with convolutions, " in Proc. CVPR, 2015, pp. 1-9.
76. E. G. McPherson, N. van Doorn, and J. de Goede, "Structure, function and value of street trees in California, USA, " Urban Forestry Urban Greening, vol. 17, pp. 104-115, Jun. 2016.
77. M. Larsen, M. Eriksson, X. Descombes, G. Perrin, T. Brandtberg, and F. A. Gougeon, "Comparison of six individual tree crown detection algorithms evaluated under varying forest conditions, " Int. J. Remote Sens., vol. 32, no. 20, pp. 5827-5852, 2011.
78. H. Kaartinen et al., "An international comparison of individual tree detection and extraction using airborne laser scanning, " Remote Sens., vol. 4, no. 4, pp. 950-974, 2012.
79. D. G. Leckie, F. A. Gougeon, S. Tinis, T. Nelson, C. N. Burnett, and D. Paradine, "Automated tree recognition in old growth conifer stands with high resolution digital imagery, " Remote Sens. Environ., vol. 94, no. 3, pp. 311-326, 2005.
80. V. Heikkinen, I. Korpela, T. Tokola, E. Honkavaara, and J. Parkkinen, "An SVM classification of tree species radiometric signatures based on the Leica ADS40 sensor, " IEEE Trans. Geosci. Remote Sens., vol. 49, no. 11, pp. 4539-4551, Nov. 2011.
81. I. Korpela, V. Heikkinen, E. Honkavaara, F. Rohrbach, and T. Tokola, "Variation and directional anisotropy of reflectance at the crown scale-Implications for tree species classification in digital aerial images, " Remote Sens. Environ., vol. 115, no. 8, pp. 2062-2074, 2011.
82. J. Heinzel and B. Koch, "Investigating multiple data sources for tree species classification in temperate forest and use for single tree delineation, " Int. J. Appl. Earth Observ. Geoinf., vol. 18, pp. 101-110, Aug. 2012.
83. L. T. Waser, C. Ginzler, M. Kuechler, E. Baltsavias, and L. Hurni, "Semi-automatic classification of tree species in different forest ecosystems by spectral and geometric variables derived from airborne digital sensor (ADS40) and RC30 data, " Remote Sens. Environ., vol. 115, no. 1, pp. 76-85, 2011.
84. R. Pu and S. Landry, "A comparative analysis of high spatial resolution IKONOS and WorldView-2 imagery for mapping urban tree species, " Remote Sens. Environ., vol. 124, pp. 516-533, Sep. 2012.
85. S. Agarwal, N. Snavely, I. Simon, S. M. Seitz, and R. Szeliski, "Building Rome in a day, " in Proc. ICCV, 2009, pp. 72-79.
86. K. Matzen and N. Snavely, "Scene chronology, " in Proc. ECCV, 2014, pp. 615-630.