Key issues in rigorous accuracy assessment of land cover products

Remote Sensing of Environment

Volumn 231, Issue , 2019, Pages

  Stehman, Stephen V ^a   Foody, Giles M ^b  

^a STATE UNIVERSITY OF NEW YORK   (United States)

^b UNIVERSITY OF NOTTINGHAM   (United Kingdom)

Author keywords

Area estimation; Error matrix; Inference; Probability sampling; Remote sensing; Uncertainty

Indexed keywords

ERRORS; MATRIX ALGEBRA; REMOTE SENSING; SPACE OPTICS;

AREA ESTIMATION; ERROR MATRICES; INFERENCE; PROBABILITY SAMPLING; UNCERTAINTY;

IMPORTANCE SAMPLING;

ACCURACY ASSESSMENT; ADVANCED TECHNOLOGY; AREAL INTERPOLATION; ENVIRONMENTAL MONITORING; ERROR ANALYSIS; LAND COVER; PROBABILITY; SAMPLING; UNCERTAINTY ANALYSIS;

EID: 85066970779     PISSN: 00344257     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.rse.2019.05.018     Document Type: Article

References (201)

1. Ahlqvist, O., Using uncertain conceptual spaces to translate between land cover categories. Int. J. Geogr. Inf. Sci. 19 (2005), 831–857.
2. Anderson, J.R., Land use classification schemes. Photogramm. Eng. 37 (1971), 379–387.
3. Antoniou, V., Fonte, C.C., See, L., Estima, J., Arsanjani, J.J., Lupia, F., Minghini, M., Foody, G., Fritz, S., Investigating the feasibility of geo-tagged photographs as sources of land cover input data. ISPRS International Journal of Geo-Information, 5, 2016, 64, 10.3390/ijgi5050064.
4. Anuta, P.E., MacDonald, R.B., Crop surveys from multiband satellite photography using digital techniques. Remote Sens. Environ. 2 (1971), 53–67.
5. Arevalo, P., Woodcock, C.E., Olofsson, P., Continuous monitoring of land change activities and post-disturbance dynamics from Landsat time series: a test methodology for REDD+ reporting. Remote Sens. Environ., 2019, 10.1016/j.rse.2019.01.013.
6. Aronoff, S., Classification accuracy: a user approach. Photogramm. Eng. Remote Sens. 48 (1982), 1299–1307.
7. Aronoff, S., The map accuracy report: a user's view. Photogramm. Eng. Remote Sens. 48 (1982), 1309–1312.
8. Badjana, H.M., Olofsson, P., Woodcock, C.E., Helmschrot, J., Wala, K., Akpagana, K., Mapping and estimating land change between 2001 and 2013 in a heterogeneous landscape in West Africa: loss of forestlands and capacity building opportunities. International Journal of Applied Earth Observation Geoinformation 63 (2017), 15–23.
9. Benedetti, R., Piersimoni, F., Postiglione, P., Spatially balanced sampling: A review and a reappraisal. Int. Stat. Rev. 85 (2017), 439–454.
10. Bey, A., Diaz, A.S.-P., Maniatis, D., Marchi, G., Mollicone, D., et al. Collect earth: land use and land cover assessment through augmented visual interpretation. Remote Sens., 8(10), 2016, 807.
11. Bishop, Y.M.M., Fienberg, S.E., Holland, P.W., Discrete Multivariate Analysis. 1975, MIT Press, Cambridge, MA.
12. Blaschke, T., Hay, G.J., Kelly, M., Lang, S., Hofmann, P., et al. Geographic object-based image analysis – towards a new paradigm. ISPRS J. Photogramm. Remote Sens. 87 (2014), 180–191.
13. Boschetti, L., Stehman, S.V., Roy, D.P., A stratified random sampling design in space and time for regional to global scale burned area product validation. Remote Sens. Environ. 186 (2016), 465–478.
14. Brus, D.J., Kempen, B., Heuvelink, G.B.M., Sampling for validation of digital soil maps. Eur. J. Soil Sci. 62 (2011), 394–407.
15. Bryan, M.L., Interpretation of an urban scene using multi-channel radar imagery. Remote Sens. Environ. 4 (1975), 49–66.
16. Card, D.H., Using known map category marginal frequencies to improve estimates of thematic map accuracy. Photogramm. Eng. Remote Sens. 48 (1982), 431–439.
17. Carlotto, M.J., Effect of errors in ground truth on classification accuracy. Int. J. Remote Sens. 30 (2009), 4831–4849.
18. Castilla, G., We must all pay more attention to rigor in accuracy assessment. Remote Sens., 8, 2016, 288, 10.3390/rs8040288.
19. Cha, S.Y., Park, C.H., The utilization of Google Earth images as reference data for the multitemporal land cover classification with MODIS data of North Korea. Korean Journal of Remote Sensing 23 (2007), 483–491.
20. Chapin, F.S., Zavaleta, E.S., Eviner, V.T., Naylor, R.L., Vitousek, P.M., et al. Consequences of changing biodiversity. Nature 405 (2000), 234–242.
21. Cohen, W.B., Yang, Z., Kennedy, R., Detecting trends in forest disturbance and recovery using yearly Landsat time series: 2. TimeSync - tools for calibration and validation. Remote Sens. Environ. 114 (2010), 2911–2924.
22. Comber, A., Fisher, P., Wadsworth, R., What is land cover?. Environment and Planning B: Planning and Design 32 (2005), 199–209.
23. Comber, A., Fisher, P., Brunsdon, C., Khmag, A., Spatial analysis of remote sensing image classification accuracy. Remote Sens. Environ. 127 (2012), 237–246.
24. Comber, A., See, L., Fritz, S., Van der Velde, M., Perger, C., Foody, G., Using control data to determine the reliability of volunteered geographic information about land cover. Int. J. Appl. Earth Obs. Geoinf. 23 (2013), 37–48.
25. Congalton, R.G., A review of assessing the accuracy of classifications of remotely sensed data. Remote Sens. Environ. 37 (1991), 35–46.
26. Congalton, R.G., Green, K., A practical look at the sources of confusion in error matrix generation. Photogramm. Eng. Remote Sens. 59 (1993), 641–644.
27. Congalton, R.G., Green, K., Assessing the Accuracy of Remotely Sensed Data: Principles and Practices. 1999, CRC Press, Boca Raton, FL (137 pp).
28. Congalton, R.G., Oderwald, R.G., Mead, R.A., Assessing Landsat classification accuracy using discrete multivariate analysis statistical techniques. Photogramm. Eng. Remote Sens. 49 (1983), 1671–1678.
29. Costanza, R., d'Arge, R., de Groot, R., Farber, S., Grasso, M., et al. The value of the world's ecosystem services and natural capital. Nature 387 (1997), 253–260.
30. de Bruin, Bregt, A., van de Ven, M., Assessing fitness for use: the expected value of spatial data sets. Int. J. Geogr. Inf. Sci. 15 (2001), 457–471.
31. De Gruijter, J.J., Ter Braak, C.J.F., Model-free estimation from spatial samples: a reappraisal of classical sampling theory. Math. Geol. 22 (1990), 407–415.
32. Defourny, P., Mayaux, P., Herold, M., Bontemps, S., Global land-cover map validation experiences: toward the characterization of uncertainty. Giri, C.P., (eds.) Remote Sensing of Land Use and Land Cover: Principles and Applications, 2012, Taylor & Francis, Boca Raton, 207–224.
33. DeFries, R.S., Los, S.O., Implications of land-cover misclassification for parameter estimates in global land-surface models: an example from the simple biosphere model (SiB2). Photogramm. Eng. Remote Sens. 65 (1999), 1083–1088.
34. Denham, R., Mengersen, K., Witte, C., Bayesian analysis of thematic map accuracy data. Remote Sens. Environ. 113 (2009), 371–379.
35. Dorn, H., Törnros, T., Zipf, A., Quality evaluation of VGI using authoritative data - A comparison with land use data in Southern Germany. ISPRS International Journal of Geo-Information 4 (2015), 1657–1671.
36. Edwards, T.C. Jr., Moisen, G.G., Cutler, D.R., Assessing map accuracy in an ecoregion-scale cover-map. Remote Sens. Environ. 63 (1998), 73–83.
37. Estes, L., Chen, P., Debats, S., Evans, T., Ferreira, S., Kuemmerle, T., Ragazzo, G., Sheffield, J., Wolf, A., Wood, E., Caylor, K., A large-area, spatially continuous assessment of land cover map error and its impact on downstream analyses. Glob. Chang. Biol. 24 (2017), 322–337.
38. Fattorini, L., Corona, P., Chirici, G., Pagliarella, M.C., Design-based strategies for sampling spatial units from regular grids with applications to forest surveys, land use, and land cover estimation. Environmetrics 26 (2015), 216–228.
39. Fichet, L.-V., Sannier, C., Makaga, E.M.K., Seyler, F., Assessing the accuracy of forest cover map for 1990, 2000 and 2010 at national scale in Gabon. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 7 (2014), 1346–1356.
40. Finn, J.T., Use of the average mutual information index in evaluating classification error and consistency. Int. J. Geogr. Inf. Sci. 7 (1993), 349–366.
41. Fitzpatrick-Lins, K., Comparison of sampling procedures and data analysis for a land-use and land-cover map. Photogramm. Eng. Remote Sens. 47 (1981), 343–351.
42. rd edition, 2003, Wiley, New Jersey.
43. Foley, J.A., DeFries, R., Asner, G.P., Barford, C., Bonan, G., Carpenter, S.R., Chapin, F.S., et al. Global consequences of land use. Science 309 (2005), 570–574.
44. Fonte, C.C., Bastin, L., See, L., Foody, G., Lupia, F., Usability of VGI for validation of land cover maps. Int. J. Geogr. Inf. Sci. 29 (2015), 1269–1291.
45. Foody, G.M., On the compensation for chance agreement in image classification accuracy assessment. Photogramm. Eng. Remote Sens. 58 (1992), 1459–1460.
46. Foody, G.M., Approaches for the production and evaluation of fuzzy land cover classifications from remotely-sensed data. Int. J. Remote Sens. 17 (1996), 1317–1340.
47. Foody, G.M., Status of land cover classification accuracy assessment. Remote Sens. Environ. 80 (2002), 185–201.
48. Foody, G.M., Thematic map comparison: evaluating the statistical significance of differences in classification accuracy. Photogramm. Eng. Remote Sens. 70 (2004), 627–633.
49. Foody, G.M., Local characterization of thematic classification accuracy through spatially constrained confusion matrices. Int. J. Remote Sens. 26 (2005), 1217–1228.
50. Foody, G.M., What is the difference between two maps? A remote senser's view. J. Geogr. Syst. 8 (2006), 119–130.
51. Foody, G.M., Harshness in image classification accuracy assessment. Int. J. Remote Sens. 29 (2008), 3137–3158.
52. Foody, G.M., The impact of imperfect ground reference data on the accuracy of land cover change estimation. Int. J. Remote Sens. 30 (2009), 3275–3281.
53. Foody, G.M., Classification accuracy comparison: hypothesis tests and the use of confidence intervals in evaluations of difference, equivalence and non-inferiority. Remote Sens. Environ. 113 (2009), 1658–1663.
54. Foody, G.M., Sample size determination for image classification accuracy assessment and comparison. Int. J. Remote Sens. 30 (2009), 5273–5291.
55. Foody, G.M., Assessing the accuracy of land cover change with imperfect ground reference data. Remote Sens. Environ. 114 (2010), 2271–2285.
56. Foody, G.M., Latent class modeling for site-and non-site-specific classification accuracy assessment without ground data. IEEE Trans. Geosci. Remote Sens. 50 (2012), 2827–2838.
57. Foody, G.M., Ground reference data error and the mis-estimation of the area of land cover change as a function of its abundance. Remote Sensing Letters 4 (2013), 783–792.
58. Foody, G.M., Valuing map validation: the need for rigorous land cover map accuracy assessment in economic valuations of ecosystem services. Ecol. Econ. 111 (2015), 23–28.
59. Foody, G.M., Boyd, D.S., Using volunteered data in land cover map validation: Mapping West African forests. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 6 (2013), 1305–1312.
60. Foody, G.M., Mathur, A., The use of small training sets containing mixed pixels for accurate hard image classification: training on mixed spectral responses for classification by a SVM. Remote Sens. Environ. 103 (2006), 179–189.
61. Foody, G.M., See, L., Fritz, S., Van der Velde, M., Perger, C., Schill, C., Boyd, D.S., Comber, A., Accurate attribute mapping from volunteered geographic information: issues of volunteer quantity and quality. Cartogr. J. 52 (2015), 336–344.
62. Foody, G.M., Pal, M., Rocchini, D., Garzon-Lopez, C.X., Bastin, L., The sensitivity of mapping methods to reference data quality: training supervised image classifications with imperfect reference data. ISPRS International Journal of Geo-Information, 5(11), 2016, 199.
63. Foody, G., See, L., Fritz, S., Moorthy, I., Perger, C., Schill, C., Boyd, D., Increasing the accuracy of crowdsourced information on land cover via a voting procedure weighted by information inferred from the contributed data. ISPRS International Journal of Geo-Information, 7(3), 2018, 80.
64. Friedl, M.A., Woodcock, C., Gopal, S., Muchoney, D., Strahler, A.H., Barker-Schaaf, C., A note on procedures used for accuracy assessment in land cover maps derived from AVHRR data. Int. J. Remote Sens. 21 (2000), 1073–1077.
65. Fritz, S., McCallum, I., Schill, C., Perger, C., Grillmayer, R., Achard, F., Kraxner, F., Obersteiner, M., Geo-Wiki.Org: the use of crowdsourcing to improve global land cover. Remote Sens. 1 (2009), 345–544.
66. Fritz, S., See, L., Perger, C., McCallum, I., Schill, C., et al. A global dataset of crowdsourced land cover and land use reference data. Scientific Data, 4, 2017, 170075.
67. Fuller, R.M., Groom, G.B., Jones, A.R., The land cover map of Great Britain: an automated classification of Landsat thematic mapper data. Photogramm. Eng. Remote Sens. 60 (1994), 553–562.
68. Gallego, F.J., Remote sensing and land cover area estimation. Int. J. Remote Sens. 25 (2004), 3019–3047.
69. Gallego, F.J., Validation of GIS layers in the EU: getting adapted to available reference data. International Journal of Digital Earth 4:Supplement 1 (2011), 42–57.
70. Gallego, F.J., The efficiency of sampling very high resolution images for area estimation in the European Union. Int. J. Remote Sens. 33 (2012), 1868–1880.
71. Gallego, J., Bamps, C., Using CORINE land cover and the point survey LUCAS for area estimation. Int. J. Appl. Earth Obs. Geoinf. 10 (2008), 467–475.
72. Gallego, F.J., Stibig, H.J., Area estimation from a sample of satellite images: the impact of stratification on the clustering efficiency. Int. J. Appl. Earth Obs. Geoinf. 22 (2013), 139–146.
73. Gong, P., Wang, J., Yu, L., Zhao, Y., Zhao, Y., Liang, L., et al. Finer resolution observation and monitoring of global land cover: first mapping results with Landsat TM and ETM+ data. Int. J. Remote Sens. 34 (2013), 2607–2654.
74. Gopal, S., Woodcock, C., Theory and methods for accuracy assessment of thematic maps using fuzzy sets. Photogramm. Eng. Remote Sens. 60 (1994), 181–188.
75. Gordon, S.I., Utilizing LANDSAT imagery to monitor land-use change: a case study in Ohio. Remote Sens. Environ. 9 (1980), 189–196.
76. Gorelick, N., Hancher, M., Dixon, M., Ilyushchenko, S., Thau, D., Moore, R., Google Earth Engine: planetary-scale geospatial analysis for everyone. Remote Sens. Environ. 202 (2017), 18–27.
77. Green, E.J., Strawderman, W.E., Determining accuracy of thematic maps. The Statistician 43 (1994), 77–85.
78. Hagen, A., Fuzzy set approach to assessing similarity of categorical maps. Int. J. Geogr. Inf. Sci. 17 (2003), 235–249.
79. Hagen-Zanker, A., Map comparison methods that simultaneously address overlap and structure. J. Geogr. Syst. 8 (2006), 165–185.
80. Hammond, T.O., Verbyla, D.L., Optimistic bias in classification accuracy assessment. Int. J. Remote Sens. 17 (1996), 1261–1266.
81. Haralick, R.M., Caspall, F., Simonett, D.S., Using radar imagery for crop discrimination: A statistical and conditional probability study. Remote Sens. Environ. 1 (1969-1970), 131–142.
82. Hay, A.M., Sampling designs to test land-use map accuracy. Photogramm. Eng. Remote Sens. 45 (1979), 529–533.
83. Healey, S.P., Cohen, W.B., Yang, Z., Brewer, C.K., Brooks, E.B., Gorelick, et al. Mapping forest change using stacked generalization: an ensemble approach. Remote Sens. Environ. 204 (2018), 717–728.
84. Herold, M., Mayaux, P., Woodcock, C.E., Baccini, A., Schmullius, C., Some challenges in global land cover mapping: an assessment of agreement and accuracy in existing 1 km datasets. Remote Sens. Environ. 112 (2008), 2538–2556.
85. Hollmann, R., Merchant, C.J., Saunders, R., Downy, C., Buchwitz, M., et al. The ESA climate change initiative: Satellite data records for essential climate variables. Bull. Am. Meteorol. Soc. 94 (2013), 1541–1552.
86. Hord, R.M., Brooner, W., Land-use map accuracy criteria. Photogramm. Eng. Remote Sens. 42 (1976), 671–677.
87. Iwao, K., Nishida, K., Kinoshita, T., Yamagata, Y., Validating land cover maps with degree confluence project information. Geophys. Res. Lett., 33, 2006, L23404.
88. Janssen, L.L.F., van der Wel, F.J.M., Accuracy assessment of satellite derived land-cover data: a review. Photogramm. Eng. Remote Sens. 60 (1994), 419–426.
89. Jensen, J.R., Christensen, E.J., Sharitz, R., Nontidal wetland mapping in South Carolina using airborne multispectral scanner data. Remote Sens. Environ. 16 (1984), 1–12.
90. Kempeneers, P., McInerney, D., Sedano, F., Gallego, J., Strobl, P., et al. Accuracy assessment of a remote sensing-based, pan-European forest cover map using multi-country national forest inventory data. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 6 (2013), 54–65.
91. Khatami, R., Mountrakis, G., Stehman, S.V., A meta-analysis of remote sensing research on supervised pixel-based land-cover image classification processes: general guidelines for practitioners and future research. Remote Sens. Environ. 177 (2016), 89–100.
92. Khatami, R., Mountrakis, G., Stehman, S.V., Mapping per-pixel predicted accuracy of classified remote sensing images. Remote Sens. Environ. 191 (2017), 156–167.
93. Kyriakidis, P.C., Dungan, J.L., A geostatistical approach for mapping thematic classification accuracy and evaluating the impact of inaccurate spatial data on ecological model predictions. Environ. Ecol. Stat. 8 (2001), 311–330.
94. Laso Bayas, J.-C., See, L., Fritz, S., Sturn, T., Perger, C., et al. Crowdsourcing in-situ data on land cover and land use using gamification and mobile technology. Remote Sens., 8(11), 2017, 905.
95. Latifovic, R., Olthof, I., Accuracy assessment using sub-pixel fractional error matrices of global land cover products derived from satellite data. Remote Sens. Environ. 90 (2004), 153–165.
96. Lesiv, M., See, L., Laso Bayas, J.C., Sturn, T., Schepaschenko, D., Karner, M., Moorthy, I., McCallum, I., Fritz, S., Characterizing the spatial and temporal availability of very high resolution satellite imagery in Google Earth and Microsoft Bing Maps as a source of reference data. Land, 7, 2018, 118.
97. Lewis, H.G., Brown, M., A generalized confusion matrix for assessing area estimates from remotely sensed data. Int. J. Remote Sens. 22 (2001), 3223–3235.
98. st-century annual forest loss: independent accuracy assessment and application in a temperate forest region of Atlantic Canada. Remote Sens. Environ. 188 (2017), 164–176.
99. Liu, C., Frazier, P., Kumar, L., Comparative assessment of the measures of thematic classification accuracy. Remote Sens. Environ. 107 (2007), 606–616.
100. Loveland, T.R., Reed, B.C., Brown, J.F., Ohlen, D.O., Zhu, Z., Yang, L., Merchant, J.W., Development of a global land cover characteristics database and IGBP DISCover from 1 km AVHRR data. Int. J. Remote Sens. 21 (2000), 1303–1330.
101. Lu, D., Weng, Q., A survey of image classification methods and techniques for improving classification performance. Int. J. Remote Sens. 28 (2007), 823–870.
102. Lyons, M.B., Keith, D.A., Phinn, S.R., Mason, T.J., Elith, J., A comparison of resampling methods for remote sensing classification and accuracy assessment. Remote Sens. Environ. 208 (2018), 145–153.
103. Magnussen, S., A Bayesian approach to classification accuracy inference. Forestry 82 (2009), 211–226.
104. Magnussen, S., Arguments for a model-dependent inference?. Forestry 88 (2015), 317–325.
105. Marsh, S.E., Walsh, J.L., Sobrevila, C., Evaluation of airborne video data for land-cover classification accuracy assessment in an isolated Brazilian forest. Remote Sens. Environ. 48 (1994), 61–69.
106. Mas, J.F., Monitoring land-cover changes: a comparison of change detection techniques. Int. J. Remote Sens. 20 (1999), 139–152.
107. Mather, P., Tso, B., Classification Methods for Remotely Sensed Data. 2016, CRC press.
108. Mayaux, P., Eva, H., Gallego, J., Strahler, A.H., Herold, M., et al. Validation of the Global Land Cover 2000 map. IEEE Trans. Geosci. Remote Sens. 44 (2006), 1728–1739.
109. McGwire, K.C., Fisher, P., Spatially variable thematic accuracy: beyond the confusion matrix. Hunsaker, C.T., Goodchild, M.F., Friedl, M.A., Case, T.J., (eds.) Spatial Uncertainty in Ecology: Implications for Remote Sensing and GIS Applications, 2001, Springer, New York, 308–329.
110. McRoberts, R.E., A model-based approach to estimating forest area. Remote Sens. Environ. 103 (2006), 56–66.
111. McRoberts, R.E., Satellite image-based maps: scientific inference or pretty pictures?. Remote Sens. Environ. 115 (2011), 715–724.
112. McRoberts, R.E., Walters, B.F., Statistical inference for remote sensing-based estimates of net deforestation. Remote Sens. Environ. 124 (2012), 394–401.
113. McRoberts, R.E., Stehman, S.V., Liknes, G.C., Næsset, E., Sannier, C., Walters, B.F., The effects of imperfect reference data on remote sensing-assisted estimators of land cover class proportions. ISPRS J. Photogramm. Remote Sens. 142 (2018), 292–300.
114. Munafò, M.R., Nosek, B.A., Bishop, D.V.M., Button, K.S., Chambers, C.D., et al. A manifesto for reproducible science. Nat. Hum. Behav., 1, 2017, 0021.
115. Navajas, J., Niella, T., Garbulsky, G., Bahrami, B., Sigman, M., Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds. Nat. Hum. Behav., 2(2), 2018, 126.
116. Nusser, S.M., Klaas, E.E., Survey methods for assessing land cover map accuracy. Environ. Ecol. Stat. 10 (2003), 309–331.
117. Olofsson, P., Stehman, S.V., Woodcock, C.E., Sulla-Menashe, D., Sibley, A.M., et al. A global land-cover validation data set, part I: fundamental design principles. Int. J. Remote Sens. 33 (2012), 5768–5788.
118. Olofsson, P., Foody, G.M., Stehman, S.V., Woodcock, C.E., Making better use of accuracy data in land change studies: estimating accuracy and area and quantifying uncertainty using stratified estimation. Remote Sens. Environ. 129 (2013), 122–131.
119. Olofsson, P., Foody, G.M., Herold, M., Stehman, S.V., Woodcock, C.E., Wulder, M.A., Good practices for estimating area and assessing accuracy of land change. Remote Sens. Environ. 148 (2014), 42–57.
120. Overton, W.S., Stehman, S.V., The Horvitz-Thompson theorem as a unifying perspective for probability sampling: with examples from natural resource sampling. Am. Stat. 49 (1995), 261–268.
121. Padilla, M., Olofsson, P., Stehman, S.V., Tansey, K., Chuvieco, E., Stratification and sample allocation for reference burned area data. Remote Sens. Environ. 203 (2017), 240–255.
122. Pal, M., Foody, G.M., Evaluation of SVM, RVM and SMLR for accurate image classification with limited ground data. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 5 (2012), 1344–1355.
123. Park, N.-W., Kyriakidis, P.C., Hong, S.-Y., Spatial estimation of classification accuracy using indicator kriging with an image-derived ambiguity index. Remote Sens., 8, 2016, 320.
124. Pla, M., Duane, A., Brotons, L., Potential of UAV images as ground-truth data for burn severity classification of Landsat imagery: approaches to a useful product for post-fire management. Revista de Teledetección 49 (2017), 91–102.
125. Pontius, R.G. Jr., Component intensities to relate difference by category with difference overall. Int. J. Appl. Earth Obs. Geoinf. 77 (2019), 94–99.
126. Pontius, R.G. Jr., Cheuk, M.L., A generalized cross-tabulation matrix to compare soft-classified maps at multiple spatial resolutions. Int. J. Geogr. Inf. Sci. 20 (2006), 1–30.
127. Pontius, R.G. Jr., Connors, J., Range of categorical associations for comparison of maps with mixed pixels. Photogramm. Eng. Remote Sens. 75 (2009), 696–963.
128. Pontius, R.G. Jr., Millones, M., Death to kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment. Int. J. Remote Sens. 32 (2011), 4407–4429.
129. Pontius, R.G. Jr., Santacruz, A., Quantity, exchange, and shift components of difference in a square contingency table. Int. J. Remote Sens. 35 (2014), 7543–7554.
130. Pontius, R.G. Jr., Krithivasan, R., Sauls, L., Yan, Y., Zhang, Y., Methods to summarize change among land categories across time intervals. J. Land Use Sci. 12:4 (2017), 218–230.
131. Potapov, P.V., Dempewolf, J., Talero, Y., Hansen, M.C., Stehman, S.V., et al. National satellite-based humid tropical forest change assessment in Peru in support of REDD+ implementation. Environ. Res. Lett., 9(2014), 2014, 124012 (13pp).
132. Powell, R.L., Matzke, N., de Souza, C. Jr., Clark, M., Numata, I., Hess, L.L., Roberts, D.A., Sources of error in accuracy assessment of thematic land-cover maps in the Brazilian Amazon. Remote Sens. Environ. 90 (2004), 221–234.
133. Prelec, D., Seung, H.S., McCoy, J., A solution to the single-question crowd wisdom problem. Nature, 541(7638), 2017, 532.
134. Radoux, J., Bogaert, P., Good practices for object-based accuracy assessment. Remote Sens., 9, 2017, 646.
135. Radoux, J., Bogaert, P., Fasbender, D., Defourny, P., Thematic accuracy assessment of geographic object-based image classification. Int. J. Geogr. Inf. Sci. 25 (2011), 895–911.
136. Ramankutty, N., Evan, A.T., Monfreda, C., Foley, J.A., Farming the planet: 1. Geographic distribution of global agricultural lands in the year 2000. Glob. Biogeochem. Cycles, 22, 2008, GB1003.
137. Riemann, R., Wilson, B.T., Lister, A., Parks, S., An effective assessment protocol for continuous geospatial datasets of forest characteristics using USFS Forest inventory and analysis (FIA) data. Remote Sens. Environ. 114 (2010), 2337–2352.
138. Sarmento, P., Fonte, C.C., Caetano, M., Stehman, S.V., Incorporating the uncertainty of linguistic-scale reference data to assess accuracy of land-cover maps using fuzzy intervals. Int. J. Remote Sens. 34 (2013), 4008–4024.
139. Särndal, C.E., Swensson, B., Wretman, J., Model-Assisted Survey Sampling. 1992, Springer-Verlag, New York.
140. Scepan, J., Thematic validation of high-resolution global land-cover data sets. Photogramm. Eng. Remote. Sens. 65 (1999), 1051–1060.
141. See, L., Mooney, P., Foody, G., Bastin, L., Comber, A., et al. Crowdsourcing, citizen science or volunteered geographic information? The current state of crowdsourced geographic information. ISPRS International Journal of Geo-information, 5(5), 2016, 55.
142. See, L., Laso Bayas, J.C., Schepaschenko, D., Perger, C., Dresel, C., Maus, V., Salk, C., Weichselbaum, J., Lesiv, M., McCallum, I., Moorthy, I., LACO-wiki: a new online land cover validation tool demonstrated using GlobeLand30 for Kenya. Remote Sens., 9(7), 2017, 754.
143. Singh, A., Digital change detection techniques using remotely-sensed data. Int. J. Remote Sens. 10 (1989), 989–1003.
144. Smits, P.C., Dellepiane, S.G., Schowengerdt, R.A., Quality assessment of image classification algorithms for land-cover mapping: a review and a proposal for a cost-based approach. Int. J. Remote Sens. 20 (1999), 1461–1486.
145. Song, X.-P., Potapov, P.V., Krylov, A., King, L., Di Bella, C.M., et al. National-scale soybean mapping and area estimation in the United States using medium resolution satellite imagery and field survey. Remote Sens. Environ. 190 (2017), 383–395.
146. Staquet, M., Rozencweig, M., Lee, Y.J., Muggia, F.M., Methodology for the assessment of new dichotomous diagnostic tests. J. Chronic Dis. 34 (1981), 599–610.
147. Steele, B.M., Winne, J.C., Redmond, R.L., Estimation and mapping of misclassification probabilities for thematic land cover maps. Remote Sens. Environ. 66 (1998), 192–202.
148. Steele, B.M., Patterson, D.A., Redmond, R.L., Toward estimation of map accuracy without a probability test sample. Environ. Ecol. Stat. 10 (2003), 333–356.
149. Stehman, S.V., Thematic map accuracy assessment from the perspective of finite population sampling. Int. J. Remote Sens. 16 (1995), 589–593.
150. Stehman, S.V., Selecting and interpreting measures of thematic classification accuracy. Remote Sens. Environ. 62 (1997), 77–89.
151. Stehman, S.V., Estimating standard errors of accuracy assessment statistics under cluster sampling. Remote Sens. Environ. 60 (1997), 258–269.
152. Stehman, S.V., Basic probability sampling designs for thematic map accuracy assessment. Int. J. Remote Sens. 20 (1999), 2423–2441.
153. Stehman, S.V., Comparing thematic map accuracy based on map value. Int. J. Remote Sens. 20 (1999), 2347–2366.
154. Stehman, S.V., Practical implications of design-based sampling inference for thematic map accuracy assessment. Remote Sens. Environ. 72 (2000), 35–45.
155. Stehman, S.V., Statistical rigor and practical utility in thematic map accuracy assessment. Photogramm. Eng. Remote Sens. 67 (2001), 727–734.
156. Stehman, S.V., A critical evaluation of the normalized error matrix in map accuracy assessment. Photogramm. Eng. Remote Sens. 70 (2004), 743–751.
157. Stehman, S.V., Comparing estimators of gross change derived from complete coverage mapping versus statistical sampling of remotely sensed data. Remote Sens. Environ. 96 (2005), 466–474.
158. Stehman, S.V., Design, analysis, and inference for studies comparing thematic accuracy of classified remotely sensed data: a special case of map comparison. J. Geogr. Syst. 8 (2006), 209–226.
159. Stehman, S.V., Sampling designs for accuracy assessment of land cover. Int. J. Remote Sens. 30 (2009), 5243–5272.
160. Stehman, S.V., Model-assisted estimation as a unifying framework for estimating the area of land cover and land-cover change from remote sensing. Remote Sens. Environ. 113 (2009), 2455–2462.
161. Stehman, S.V., Estimating area from an accuracy assessment error matrix. Remote Sens. Environ. 132 (2013), 202–211.
162. Stehman, S.V., Estimating area and map accuracy for stratified random sampling when the strata are different from the map classes. Int. J. Remote Sens. 35 (2014), 4923–4939.
163. Stehman, S.V., Czaplewski, R.L., Design and analysis for thematic map accuracy assessment: fundamental principles. Remote Sens. Environ. 64 (1998), 331–344.
164. Stehman, S.V., Selkowitz, D.J., A spatially stratified, multi-stage cluster sampling design for assessing accuracy of the Alaska (USA) National Land-Cover Data (NLCD). Int. J. Remote Sens. 31 (2010), 1877–1896.
165. Stehman, S.V., Wickham, J.D., Pixels, blocks of pixels, and polygons: choosing a spatial unit for thematic accuracy assessment. Remote Sens. Environ. 115 (2011), 3044–3055.
166. Stehman, S.V., Czaplewski, R.L., Nusser, S.M., Yang, L., Zhu, Z., Combining accuracy assessment of land-cover maps with environmental monitoring programs. Environ. Monit. Assess. 64 (2000), 115–126.
167. Stehman, S.V., Wickham, J.D., Smith, J.H., Yang, L., Thematic accuracy of the 1992 National Land-Cover Data (NLCD) for the Eastern United States: statistical methodology and regional results. Remote Sens. Environ. 86 (2003), 500–516.
168. Stehman, S.V., Olofsson, P., Woodcock, C.E., Herold, M., Friedl, M.A., A global land cover validation dataset, II: augmenting a stratified sampling design to estimate accuracy by region and land-cover class. Int. J. Remote Sens. 33 (2012), 6975–6993.
169. Stehman, S.V., Fonte, C.C., Foody, G.M., See, L., Using volunteered geographic information (VGI) in design-based statistical inference for area estimation and accuracy assessment of land cover. Remote Sens. Environ. 212 (2018), 47–59.
170. Stevens, D.L., Olsen, A.R., Spatially balanced sampling of natural resources. J. Am. Stat. Assoc. 99 (2004), 262–278.
171. Strahler, A.H., Boschetti, L., Foody, G.M., Friedl, M.A., Hansen, M.C., et al. Global Land Cover Validation: Recommendations for Evaluation and Accuracy Assessment of Global Land Cover Maps, EUR 22156 EN – DG. 2006, Office for Official Publications of the European Communities, Luxembourg (48 pp).
172. Tchuenté, A.T.K., Roujean, J.L., De Jong, S.M., Comparison and relative quality assessment of the GLC2000, GLOBCOVER, MODIS and ECOCLIMAP land cover data sets at the African continental scale. Int. J. Appl. Earth Obs. Geoinf. 13 (2011), 207–219.
173. Toll, D.L., Effect of Landsat thematic mapper sensor parameters on land cover classification. Remote Sens. Environ. 17 (1985), 129–140.
174. Toutin, T., Fine spatial resolution optical sensors. Warner, T.A., Nellis, M.D., Foody, G.M., (eds.) The SAGE Handbook of Remote Sensing, 2009, Sage, London, 139–150.
175. Tsendbazar, N.-E., de Bruin, S., Mora, B., Schouten, L., Herold, M., Comparitive assessment of thematic accuracy of GLC maps for specific applications using existing reference data. International Journal of Applied Earth Observation and Geoinfomation 44 (2016), 124–135.
176. Tsendbazar, N.-E., Herold, M., de Bruin, S., Lesiv, M., Fritz, S., et al. Developing and applying a muti-purpose land cover validation dataset for Africa. Remote Sens. Environ. 219 (2018), 298–309.
177. Tsutsumida, N., Comber, A.J., Measures of spatio-temporal accuracy for time series land cover data. Int. J. Appl. Earth Obs. Geoinf. 41 (2015), 46–55.
178. Turner, B.L., Lambin, E.F., Reenberg, A., The emergence of land change science for global environmental change and sustainability. Proc. Natl. Acad. Sci. 104 (2007), 20666–20671.
179. Valliant, R., Dorfman, A.H., Royall, R.M., Finite Population Sampling and Inference: A Prediction Approach. 2000, John Wiley & Sons, Inc, New York.
180. Van Oort, P.A.J., Interpreting the change detection error matrix. Remote Sens. Environ. 108 (2007), 1–8.
181. Verbyla, D.L., Boles, S.H., Bias in land cover change estimates due to misregistration. Int. J. Remote Sens. 21 (2000), 3553–3560.
182. Verbyla, D.L., Hammond, T.O., Conservative bias in classification accuracy assessment due to pixel-by-pixel comparison of classified images with reference grids. Int. J. Remote Sens. 16 (1995), 581–587.
183. Vitousek, P.M., Mooney, H.A., Lubchenco, J., Melillo, J.M., Human domination of earth's ecosystems. Science 277 (1997), 494–499.
184. Wagner, J.E., Stehman, S.V., Optimizing sample size allocation to strata for estimating area and map accuracy. Remote Sens. Environ. 168 (2015), 126–133.
185. Waldner, F., Defourny, P., Where can pixel counting area estimates meet user-defined accuracy requirements?. Int. J. Appl. Earth Obs. Geoinf. 60 (2017), 1–10.
186. Waldner, F., Schucknecht, A., Lesiv, M., Gallego, J., See, L., et al. Conflation of expert and crowd reference data to validate global binary thematic maps. Remote Sens. Environ. 221 (2019), 235–246.
187. Walsh, S.J., Coniferous tree species mapping using LANDSAT data. Remote Sens. Environ. 9 (1980), 11–26.
188. Whiteside, T.G., Maier, S.W., Boggs, G.S., Area-based and location-based validation of classified image objects. Int. J. Appl. Earth Obs. Geoinf. 28 (2014), 117–130.
189. Wickham, J.D., Stehman, S.V., Smith, J.H., Yang, L., Thematic accuracy of the 1992 National Land-cover Data for the western United States. Remote Sens. Environ. 91 (2004), 452–468.
190. Wickham, J.D., Stehman, S.V., Fry, J.A., Smith, J.H., Homer, C.G., Thematic accuracy of the NLCD 2001 land cover for the conterminous United States. Remote Sens. Environ. 114 (2010), 1286–1296.
191. Wickham, J.D., Stehman, S.V., Gass, L., Dewitz, J., Fry, J.A., Wade, T.G., Accuracy assessment of NLCD 2006 land cover and impervious surface. Remote Sens. Environ. 130 (2013), 294–304.
192. Wickham, J., Stehman, S.V., Gass, L., Dewitz, J.A., Sorenson, D.G., et al. Thematic accuracy assessment of the 2011 National Land Cover Database (NLCD). Remote Sens. Environ. 191 (2017), 328–341.
193. Woodcock, C.E., Gopal, S., Fuzzy set theory and thematic maps: accuracy assessment and area estimation. Int. J. Geogr. Inf. Sci. 14 (2000), 153–172.
194. Wulder, M.A., Franklin, S.E., White, J.C., Linke, J., Magnussen, S., An accuracy assessment framework for large-area land cover classification products derived from medium-resolution satellite data. Int. J. Remote Sens. 27 (2006), 663–683.
195. Wulder, M.A., White, J.C., Luther, J.E., Strickland, G., Remmel, T.K., Mitchell, S.W., Use of vector polygons for the accuracy assessment of pixel-based land cover maps. Can. J. Remote. Sens. 32 (2006), 268–279.
196. Wulder, M.A., Coops, N.C., Roy, D.P., White, J.C., Hermosilla, T., Land cover 2.0. Int. J. Remote Sens. 39 (2018), 4254–4284.
197. Yang, L., Brus, D.J., Zhu, A.-X., Li, X., Shi, J., Accounting for access costs in validation of soil maps: a comparison of design-based sampling strategies. Geoderma 315 (2018), 160–169.
198. Ye, S., Pontius, R.G. Jr., Rakshit, R., A review of accuracy assessment for object-based image analysis: from per-pixel to per-polygon approaches. ISPRS J. Photogramm. Remote Sens. 141 (2018), 137–147.
199. Zhao, Y., Gong, P., Yu, L., Hu, L., Li, X., et al. Towards a common validation sample set for global land-cover mapping. Int. J. Remote Sens. 35 (2014), 4795–4814.
200. Zhen, Z., Quackenbush, L.J., Stehman, S.V., Zhang, L., Impact of training and validation sample selection on classification accuracy and accuracy assessment when using reference polygons in object-based classification. Int. J. Remote Sens. 34 (2013), 6914–6930.
201. Zimmerman, P.L., Housman, I.W., Perry, C.H., Chastain, R.A., Webb, J.B., Finco, M.V., An accuracy assessment of forest disturbance mapping in the western Great Lakes. Remote Sens. Environ. 128 (2013), 176–185.