Conflation of expert and crowd reference data to validate global binary thematic maps

Remote Sensing of Environment

Volumn 221, Issue , 2019, Pages 235-246

  Waldner, François ^a,b   Schucknecht, Anne ^c,d   Lesiv, Myroslava ^e   Gallego, Javier ^c   See, Linda ^e   Pérez Hoyos, Ana ^c   d'Andrimont, Raphaël ^a,c   de Maet, Thomas ^a   Bayas, Juan Carlos Laso ^e   Fritz, Steffen ^e   Leo, Olivier ^c   Kerdiles, Hervé ^c   Díez, Mónica ^f   Van Tricht, Kristof ^g   Gilliams, Sven ^g   Shelestov, Andrii ^h   Lavreniuk, Mykola ^h   Simões, Margareth ^i,q   Ferraz, Rodrigo ⁱ   Bellón, Beatriz ^j   Bégué, Agnès ^j,r   Hazeu, Gerard ^k   Stonacek, Vaclav ^l   Kolomaznik, Jan ^l   Misurec, Jan ^l   Verón, Santiago R ^m,n   de Abelleyra, Diego ^m   Plotnikov, Dmitry ^o   Mingyong, Li ^p   Singha, Mrinal ^p   Patil, Prashant ^p   Zhang, Miao ^p   Defourny, Pierre ^a   more..

^a UNIVERSITÉ CATHOLIQUE DE LOUVAIN   (Belgium)

^b CSIRO   (Australia)

^c JOINT RESEARCH CENTRE   (Italy)

^d KARLSRUHE INSTITUTE OF TECHNOLOGY   (Germany)

^e INTERNATIONAL INSTITUTE FOR APPLIED SYSTEMS ANALYSIS   (Austria)

^f DEIMOS Imaging SLU   (Spain)

^g FLEMISH INSTITUTE FOR TECHNOLOGICAL RESEARCH VITO   (Belgium)

^h NATIONAL TECHNICAL UNIVERSITY OF UKRAINE   (Ukraine)

ⁱ EMBRAPA CERRADOS   (Brazil)

^j UMR TETIS   (France)

^k WAGENINGEN UNIVERSITY   (Netherlands)

^l PC Progress Sro   (Czech Republic)

^m INSTITUTO NACIONAL DE TECNOLOGÍA AGROPECUARIA INTA   (Argentina)

ⁿ UNIVERSIDAD DE BUENOS AIRES   (Argentina)

^o Russian Academy of Sciences   (Russian Federation)

^p INSTITUTE OF GEOLOGY AND GEOPHYSICS   (China)

^q STATE UNIVERSITY OF RIO DE JANEIRO   (Brazil)

^r UNIV MONTPELLIER   (France)

more..

Author keywords

Accuracy assessment; Crowdsourcing; Data quality; Photo interpretation; Stratified systematic sampling; Volunteered geographic information

Indexed keywords

CROWDSOURCING; MAPS; PHOTOINTERPRETATION; RELIABILITY;

ACCURACY ASSESSMENT; ADDITIONAL COSTS; CONSISTENCY CHECKS; DATA COLLECTION; DATA QUALITY; STANDARD DEVIATION; SYSTEMATIC SAMPLING; VOLUNTEERED GEOGRAPHIC INFORMATION;

DATA ACQUISITION;

ACCURACY ASSESSMENT; BENCHMARKING; DATA QUALITY; DATA SET; GIS; LAND COVER; PHOTOGRAPH; SAMPLING; SATELLITE DATA; THEMATIC MAPPING;

EID: 85056826218     PISSN: 00344257     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.rse.2018.10.039     Document Type: Article

References (78)

1. Arino, O., et al. GlobCover: the most detailed picture of earth. ESA Bulletin-European Space Agency, 2008, 136.
2. Bastin, L., Buchanan, G., Beresford, A., Pekel, J.F., Dubois, G., Open-source mapping and Services for web-based land-cover validation. Eco. Inform. 14 (2013), 9–16.
3. Bellhouse, D.R., Systematic sampling methods. Wiley StatsRef: Statistics Reference Online, 2014.
4. Bey, A., et al. Collect earth: land use and land cover assessment through augmented visual interpretation. Remote Sens., 8(10), 2016, 807.
5. Bicheron, P., et al. GLOBCOVER Products Report Description and Products Description and Validation Report. 33(0), 2008, ESA Globcover Project Led by MEDIAS FrancePOSTEL, 140–147.
6. Bogaert, P., Waldner, F., Defourny, P., An information-based criterion to measure pixel-level thematic uncertainty in land cover classifications. Stoch. Env. Res. Risk A., 2016, 1–16.
7. Bontemps, S., Defourny, P., Van Bogaert, E., Kalogirou, V., Perez, J.R., GLOBCOVER 2009 products description and validation report. ESA Bull., 2011, 1–53.
8. Bontemps, S., et al. Consistent global land cover maps for climate modelling communities: current achievements of the ESA' land cover CCI. ESA Living Planet Symposium, 2013 Retrieved http://adsabs.harvard.edu/abs/2013ESASP.722E.62B.
9. Chen, J., et al. Global land cover mapping at 30 m resolution: a pok-based operational approach. ISPRS J. Photogramm. Remote Sens. 103 (2015), 7–27 Retrieved https://doi.org/10.1016/j.isprsjprs.2014.09.002.
10. Chen, J., Li, S., Wu, H., Chen, X., Towards a collaborative global land cover information service. Int. J. Digital Earth 10:4 (2017), 356–370 Retrieved July 12, 2018 https://www.tandfonline.com/doi/full/10.1080/17538947.2016.1267268.
11. Clark, M.L., Aide, T.M., Virtual interpretation of earth web-Interface tool (VIEW-IT) for collecting land-use/land-cover reference data. Remote Sens. 3:3 (2011), 601–620.
12. Comber, A., et al. Using control data to determine the reliability of volunteered geographic information about land cover. Int. J. Appl. Earth Obs. Geoinf. 23:1 (2013), 37–48.
13. Danylo, O., et al. The picture pile tool for rapid image assessment: a demonstration using hurricane matthew. ISPRS Technical Commission IV Symposium, 2018 (Delft).
14. Defourny, P., Mayaux, P., Herold, M., Bontemps, S., Global land cover map validation experiences: toward the characterization of quantitative uncertainty. Remote Sensing of Land Use and Land Cover: Principles and Applications, 2012, 207–223 Retrieved http://dial.uclouvain.be/handle/boreal:114555.
15. Dierckx, W., et al. PROBA-V mission for global vegetation monitoring: standard products and image quality. Int. J. Remote Sens. 35:7 (2014), 2589–2614.
16. Dunn, R., Harrison, A.R., Two-dimensional systematic sampling of land use. J. R. Stat. Soc.: Ser. C: Appl. Stat. 42 (1993), 585–601.
17. European Environment Agency, Copernicus Land Service – Pan-European Component: High Resolution Layers. 2015, 2.
18. FAO, Global Strategy for Improving Agricultural & Rural Statistics. http://gsars.org/en/, 2017.
19. Fonte, C.C., Bastin, L., See, L., Foody, G., Lupia, F., Usability of VGI for validation of land cover maps. Int. J. Geogr. Inf. Sci. 29:7 (2015), 1269–1291.
20. Foody, G.M., Impacts of imperfect reference data on the apparent accuracy of species presence-absence models and their predictions. Glob. Ecol. Biogeogr. 20:3 (2011), 498–508.
21. Foody, G.M., Ground reference data error and the Mis-estimation of the area of land cover change as a function of its abundance. Remote Sens. Lett. 4:8 (2013), 783–792.
22. Foody, G., et al. Assessing the accuracy of volunteered geographic information arising from multiple contributors to an internet based collaborative project. Trans. GIS 17:6 (2013), 847–860.
23. Foody, G., et al. Increasing the accuracy of crowdsourced information on land cover via a voting procedure weighted by information inferred from the contributed data. ISPRS Int. J. Geo-Inf., 7(3), 2018, 80 Retrieved July 6, 2018 http://www.mdpi.com/2220-9964/7/3/80.
24. Fritz, S., et al. Geo-Wiki.Org: the use of crowdsourcing to improve global land cover. Remote Sens. 1:3 (2009), 345–354 Retrieved August 12, 2014 http://www.mdpi.com/2072-4292/1/3/345/.
25. Fritz, S., et al. Highlighting continued uncertainty in global land cover maps for the user community. Environ. Res. Lett., 6(4), 2011, 044005 Retrieved September 6, 2014 http://stacks.iop.org/1748-9326/6/i=4/a=044005?key=crossref.e37ea06a1a56e58be6fbd1f907045b97.
26. Fritz, S., et al. Downgrading recent estimates of land available for biofuel production. Environ. Sci. Technol. 47:3 (2013), 1688–1694.
27. Fritz, S., et al. Mapping global cropland and field size. Glob. Chang. Biol. 21:5 (2015), 1980–1992.
28. Fritz, S., See, L., Brovelli, M.A., Motivating and sustaining participation in VGI. Mapping and the Citizen Sensor, 2017, 93–117.
29. Gallego, J., Delincé, J., The european land use and cover area-frame statistical survey. Agricultural Survey Methods, 2010, 149–168.
30. Gallego, J., Schucknecht, A., Waldner, F., Sampling to validate a global cropland map. Proceedings of Spatial Accuracy 2016, 2016.
31. GOFC-GOLD LC Office, The GOFC-GOLD reference data portal. Retrieved July 3, 2017 http://www.gofcgold.wur.nl/sites/gofcgold_refdataportal.php, 2015.
32. Goodchild, M.F., Citizens as sensors: the world of volunteered geography. GeoJournal 69:4 (2007), 211–221.
33. Gumma, M.K., et al. NASA Making Earth System Data Records for Use in Research Environments (MEaSUREs) Global Food Security-Support Analysis Data (GFSAD) @ 30-M for South Asia, Afghanistan and Iran. 2017, NASA EOSDIS Land Processes DAAC.
34. Howe, J., The wisdom of crowds. Wired Mag., 161(4), 2015, 697.
35. Iwao, K., Nishida, K., Kinoshita, T., Yamagata, Y., Validating land cover maps with degree confluence project information. Geophys. Res. Lett., 33(23), 2006.
36. JECAM, Guidelines for Field Data Collection. Retrieved http://www.jecam.org/JECAM_Guidelines_for_Field_Data_Collection_v1_0.pdf, 2015.
37. Khatami, R., Mountrakis, G., Stehman, S.V., Mapping per-pixel predicted accuracy of classified remote sensing images. Remote Sens. Environ. 191 (2017), 156–167 Retrieved September 2, 2018 https://www.sciencedirect.com/science/article/pii/S0034425717300378.
38. Lamarche, C., et al. Compilation and validation of SAR and optical data products for a complete and global map of inland/ocean water tailored to the climate modeling community. Remote Sens., 9(1), 2017, 36 Retrieved July 4, 2017 http://www.mdpi.com/2072-4292/9/1/36.
39. Laso Bayas, J.C., et al. Crowdsourcing in-situ data on land cover and land use using gamification and Mobile technology. Remote Sens., 8(11), 2016, 905 Retrieved July 13, 2018 http://www.mdpi.com/2072-4292/8/11/905.
40. Laso Bayas, J.C., et al. A global reference database of crowdsourced cropland data collected using the Geo-Wiki platform. Sci. Data, 4, 2017.
41. de Leeuw, J., et al. An assessment of the accuracy of volunteered road map production in Western Kenya. Remote Sens. 3:2 (2011), 247–256.
42. Lesiv, M., et al. Characterizing the Spatial and Temporal Availability of Very High Resolution Satellite Imagery in Google Earth and Microsoft Bing Maps as a Source of Reference Data. Land, 7.4, 2018, 118.
43. Lin, Lawrence I-Kuei, A concordance correlation coefficient to evaluate reproducibility. Biometrics 45 (1989), 255–268.
44. Löw, F., et al. Mapping cropland abandonment in the Aral Sea basin with MODIS time series. Remote Sens., 10(2), 2018.
45. Mann, R.P., Helbing, D., Optimal incentives for collective intelligence. PNAS 114:20 (2017), 5077–5082 Retrieved http://arxiv.org/abs/1611.03899%0A https://doi.org/10.1073/pnas.1618722114.
46. Massey, R., et al. NASA Making Earth System Data Records for Use in Research Environments (MEaSUREs) Global Food Security-Support Analysis Data (GFSAD) @ 30m for North America: Cropland Extent Product (GFSAD30NACE). 2017, NASA EOSDIS Land Processes DAAC, 10.5067/MEaSUREs/GFSAD/GFSAD30NACE.001 Retrieved February 14, 2018.
47. Mayaux, P., et al. Validation of the global land cover 2000 map. IEEE Trans. Geosci. Remote Sens. 44:7 (2006), 1728–1737.
48. Morton, D., et al. Cropland expansion changes deforestation dynamics in the Southern Brazilian amazon. Proceedings of the National Academy of Sciences of the United States of America, 103(39), 2006, 14637–14641 Retrieved http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1600012&tool=pmcentrez&rendertype=abstract.
49. Oliphant, A.J., et al. NASA Making Earth System Data Records for Use in Research Environments (MEaSUREs) Global Food Security-Support Analysis Data (GFSAD) @ 30-m for Southeast & Northeast Asia: Cropland Extent Product (GFSAD30SEACE). 2017, NASA EOSDIS Land Processes DAAC.
50. Olofsson, P., et al. A global land-cover validation data set, part I: fundamental design principles. Int. J. Remote Sens. 33:18 (2012), 5768–5788 Retrieved July 12, 2018 http://www.tandfonline.com/doi/abs/10.1080/01431161.2012.674230.
51. Olofsson, P., et al. Good practices for estimating area and assessing accuracy of land change. Remote Sens. Environ. 148 (2014), 42–57.
52. Pekel, J.F., Cottam, A., Gorelick, N., Belward, A.S., High-resolution mapping of global surface water and its long-term changes. Nature 540:7633 (2016), 418–422 Retrieved July 4, 2017 http://www.nature.com/doifinder/10.1038/nature20584.
53. Pesaresi, M., et al. A global human settlement layer from optical HR/VHR RS data: concept and first results. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 6:5 (2013), 2102–2131 Retrieved July 4, 2017 http://ieeexplore.ieee.org/document/6578177/.
54. Phalke, A., et al. NASA Making Earth System Data Records for Use in Research Environments (MEaSUREs) Global Food Security-Support Analysis Data (GFSAD) @ 30-m for Europe, Middle-East, Russia and Central Asia: Cropland Extent Product (GFSAD30EUCEARUMECE). 2017, NASA EOSDIS Land Processes DAAC, 10.5067/MEaSUREs/GFSAD/GFSAD30EUCEARUMECE.001 Retrieved February 14, 2018.
55. Pontius, R., Thontteh, O., Chen, H., Components of information for multiple resolution comparison between maps that share a real variable. Environ. Ecol. Stat. 15:2 (2008), 111–142.
56. Powell, R.L., et al. Sources of error in accuracy assessment of thematic land-cover maps in the Brazilian Amazon. Remote Sens. Environ. 90:2 (2004), 221–234.
57. Prelec, D., Seung, H.S., McCoy, J., A solution to the single-question crowd wisdom problem. Nature 541:7638 (2017), 532–535.
58. Schepaschenko, D., et al. Development of a global hybrid forest mask through the synergy of remote sensing, crowdsourcing and FAO statistics. Remote Sens. Environ. 162 (2015), 208–220.
59. See, L., et al. Comparing the quality of crowdsourced data contributed by expert and non-experts. PLoS One, 8(7), 2013.
60. See, L., et al. Cropland capture: a gaming approach to improve global land cover. Connecting a Digital Europe Through Location and Place. Proceedings of the AGILE 2014 International Conference on Geographic Information Science, 2014, 3–6.
61. See, L., et al. Harnessing the power of volunteers, the internet and Google earth to collect and validate global spatial information using geo-wiki. Technol. Forecast. Soc. Chang. 98 (2015), 324–335.
62. Stehman, S.V., Sampling designs for accuracy assessment of land cover. Int. J. Remote Sens. 30:20 (2009), 5243–5272.
63. Stehman, S.V., Olofsson, P., Woodcock, C.E., Herold, Martin, Friedl, Mark A., A global land-cover validation data set, II: augmenting a stratified sampling design to estimate accuracy by region and land-cover class. Int. J. Remote Sens. 33:22 (2012), 6975–6993 Retrieved July 12, 2018 http://www.tandfonline.com/doi/abs/10.1080/01431161.2012.695092.
64. Stehman, Stephen V., Fonte, Cidália C., Foody, Giles M., See, Linda, Using volunteered geographic information (VGI) in design-based statistical inference for area estimation and accuracy assessment of land cover. Remote Sens. Environ. 212 (2018), 47–59 Retrieved July 6, 2018 https://www.sciencedirect.com/science/article/pii/S0034425718301627.
65. Strahler, A.H., et al. Global Land Cover Validation: Recommendations for Evaluation and Accuracy Assessment of Global Land Cover Maps. 2006.
66. Strand, G.H., Dramstad, W., Engan, G., The effect of field experience on the accuracy of identifying land cover types in aerial photographs. Int. J. Appl. Earth Obs. Geoinf. 4:2 (2002), 137–146.
67. Taubenböck, H., Wegmann, M., Roth, A., Mehl, H., Dech, S., Urbanization in India – spatiotemporal analysis using remote sensing data. Comput. Environ. Urban. Syst. 33:3 (2009), 179–188 Retrieved July 12, 2018 https://www.sciencedirect.com/science/article/pii/S0198971508000604#fig3.
68. Teluguntla, P., et al. NASA Making Earth System Data Records for Use in Research Environments (MEaSUREs) Global Food Security-Support Analysis Data (GFSAD) @ 30-m for Australia, New Zealand, China, and Mongolia: Cropland Extent Product (GFSAD30AUNZCNMOCE). 2017, NASA EOSDIS Land Processes DAAC, 10.5067/MEaSUREs/GFSAD/GFSAD30AUNZCNMOCE.001 Retrieved February 14, 2018.
69. Van Coillie, F.M.B., et al. Variability of operator performance in remote-sensing image interpretation: the importance of human and external factors. Int. J. Remote Sens. 35:2 (2014), 754–778.
70. Vancutsem, C., Marinho, E., Kayitakire, F., See, L., Fritz, S., Harmonizing and combining existing land cover/land use datasets for cropland area monitoring at the African continental scale. Remote Sens. 5:1 (2012), 19–41 Retrieved December 27, 2012 http://www.mdpi.com/2072-4292/5/1/19/.
71. Waldner, F., Fritz, S., Di Gregorio, A., Defourny, P., Mapping priorities to focus cropland mapping activities: fitness assessment of existing global, regional and National Cropland Maps. Remote Sens. 7:6 (2015), 7959–7986.
72. Waldner, F., et al. Towards a set of agrosystem-specific cropland mapping methods to address the global cropland diversity. Int. J. Remote Sens. 37:14 (2016), 3196–3231.
73. Wolter, K.M., An investigation of some estimators of variance for systematic sampling. J. Am. Stat. Assoc. 79:388 (1984), 781–790.
74. Woodcock, C.E., Gopal, S., Fuzzy set theory and thematic maps: accuracy assessment and area estimation. Int. J. Geogr. Inf. Sci. 14:2 (2000), 153–172.
75. Xing, H., Meng, Y., Wang, Z., Fan, K., Hou, D., Exploring geo-tagged photos for land cover validation with deep learning. ISPRS J. Photogramm. Remote Sens. 141 (2018), 237–251 Retrieved July 13, 2018 https://www.sciencedirect.com/science/article/pii/S0924271618301333.
76. Xiong, J., et al. NASA Making Earth System Data Records for Use in Research Environments (MEaSUREs) Global Food Security-Support Analysis Data (GFSAD) @ 30-m Africa: Cropland Extent Product (GFSAD30AFCE). 2017, NASA EOSDIS Land Processes DAAC, 10.5067/MEaSUREs/GFSAD/GFSAD30AFCE.001 Retrieved February 14, 2018.
77. Yu, L., et al. FROM-GC: 30 m global cropland extent derived through multisource data integration. Int. J. Digital Earth 6:6 (2013), 521–533 Retrieved September 13, 2018 http://www.tandfonline.com/doi/abs/10.1080/17538947.2013.822574.
78. Zhong, Y., et al. NASA Making Earth System Data Records for Use in Research Environments (MEaSUREs) Global Food Security-Support Analysis Data (GFSAD) @ 30-m for South America: Cropland Extent Product (GFSAD30SACE). 2017, NASA EOSDIS Land Processes DAAC, 10.5067/MEaSUREs/GFSAD/GFSAD30SACE.001 Retrieved February 14, 2018.