Assessing quality of volunteer crowdsourcing contributions: lessons from the Cropland Capture game

International Journal of Digital Earth

Volumn 9, Issue 4, 2016, Pages 410-426

  Salk, Carl F ^a,b   Sturn, Tobias ^a   See, Linda ^a   Fritz, Steffen ^a   Perger, Christoph ^a  

^a INTERNATIONAL INSTITUTE FOR APPLIED SYSTEMS ANALYSIS   (Austria)

^b SWEDISH UNIVERSITY OF AGRICULTURAL SCIENCES   (Sweden)

Author keywords

cropland; crowdsourcing; data quality; Geo Wiki; image classification; volunteered geographic information

Indexed keywords

CLASSIFICATION (OF INFORMATION); CROWDSOURCING;

ACTIVE CONTRIBUTORS; CROPLAND; DATA QUALITY; EXPERT EVALUATION; GEO-WIKI; QUALITY METRICS; SPATIAL INFORMATIONS; VOLUNTEERED GEOGRAPHIC INFORMATION;

IMAGE CLASSIFICATION;

ASSESSMENT METHOD; CROWDSOURCING; DATA QUALITY; IMAGE CLASSIFICATION;

EID: 84930328342     PISSN: 17538947     EISSN: 17538955     Source Type: Journal    
DOI: 10.1080/17538947.2015.1039609     Document Type: Article

References (42)

1. M.Allahbakhsh,, B.Benatallah, A.Ignjatovic, H.R.Motahari-nezhad, E.Bertino, and S.Dustdar. 2013. “Quality Control in Crowdsourcing Systems.” IEEE Internet Computing 17 (2): 76–81. doi:10.1109/MIC.2013.20.
2. Y.Bachrach,, T.Minka, J.Guiver, and T.Graepel. 2012. “How to Grade a Test without Knowing the Answers – A Bayesian Graphical Model for Adaptive Crowdsourcing and Aptitude Testing.” ArXiv, 1206.6386. http://arxiv.org/abs/1206.6386.
3. T.J.Bird,, A.E.Bates, J.S.Lefcheck, N.A.Hill, R.J.Thomson, G.J.Edgar, R.D.Stuart-Smith. 2014. “Statistical Solutions for Error and Bias in Global Citizen Science Datasets.” Biological Conservation 173: 144–154. doi:10.1016/j.biocon.2013.07.037.
4. R.Bonney,, C.B.Cooper, J.Dickinson, S.Kelling, T.Phillips, K.V.Rosenberg, and J.Shirk. 2009. “Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy.” BioScience 59: 977–984. doi:10.1525/bio.2009.59.11.9.
5. B.S.Cade,, and B.R.Noon. 2003. “A Gentle Introduction to Quantile Regression for Ecologists.” Frontiers in Ecology and the Environment 1: 412–420. doi:10.1890/1540-9295(2003)001[0412:AGITQR]2.0.CO;2.
6. R.Canavosio-Zuzelski,, P.Agouris, and P.Doucette. 2013. “A Photogrammetric Approach for Assessing Positional Accuracy of OpenStreetMap© roads.” ISPRS International Journal of Geo-Information 2: 276–301. doi:10.3390/ijgi2020276.
7. D.Clery, 2011. “Galaxy Zoo Volunteers Share Pain and Glory of Research.” Science 333 (6039): 173–175. doi:10.1126/science.333.6039.173.
8. D.Coleman, 2013. “Potential Contributions and Challenges of VGI for Conventional Topographic Base-mapping Programs.” In Crowdsourcing Geographic Knowledge, edited by D.S.E.Sui, D. S. Elwood, and M.Goodchild, 245–263. Dordrecht: Springer.
9. A.Comber,, L.See, S.Fritz, M.Van der Velde, C.Perger, and G.Foody. 2013. “Using Control Data to Determine the Reliability of Volunteered Geographic Information about Land Cover.” International Journal of Applied Earth Observation and Geoinformation 23: 37–48. doi:10.1016/j.jag.2012.11.002.
10. A.P.Dawid,, and A.M.Skene. 1979. “Maximum Likelihood Estimation of Observer Error-rates using the EM Algorithm.” Applied Statistics 28 (1): 20–28. doi:10.2307/2346806.
11. P.Defourny,, C.Vancustem, P.Bicheron, C.Brockmann, F.Nino, L.Schouten, and M.Leroy. 2006. “GLOBCOVER: A 300m Global Land Cover Product for 2005 using ENVISAT MERIS Time Series.” Proceedings of the ISPRS Commission VII Mid-Term Symposium: Remote Sensing: From Pixels to Processes. Enscede NL.
12. S.Deterding,, M.Sicart, L.Nacke, K.O’Hara, and D.Dixon. 2011. Gamification: using Game-design Elements in Non-gaming Contexts. New York: ACM Press, 2425.
13. Digital Globe. 2014. DigitalGlobe Crowdsourcing. https://www.digitalglobe.com/sites/default/files/Crowdsourcing-DS-CROWD.pdf
14. dotMobi. 2014. Global Mobile Statistics 2013 Section E: Mobile Apps, App Stores, Pricing and Failure Rates. Accessed 7 March 2014. http://mobithinking.com/mobile-marketing-tools/latest-mobile-stats/e#popularappcatagories.
15. A.Flanagin,, and M.Metzger. 2008. “The Credibility of Volunteered Geographic Information.” GeoJournal 72 (3–4): 137–148. doi:10.1007/s10708-008-9188-y.
16. C.C.Fonte,, L.Bastin, L.See, G.M.Foody, F.Lupia, and R.Vatseva. Submitted. “Usage of VGI for Validation of Land Cover Maps.” International Journal of Geographical Information Science.
17. G.M.Foody,, L.See, S.Fritz, M.Van der Velde, C.Perger, C.Schill, and D.S.Boyd. 2013. “Assessing the Accuracy of Volunteered Geographic Information Arising from Multiple Contributors to an Internet based Collaborative Project.” Transactions in GIS 17: 847–860. doi:10.1111/tgis.12033.
18. M.A.Friedl,, D.Sulla-Menashe, B.Tan, A.Schneider, N.Ramankutty, A.Sibley, and X.Huang. 2010. “MODIS Collection 5 Global Land Cover: Algorithm Refinements and Characterization of New Datasets.” Remote Sensing of Environment 114 (1): 168–182. doi:10.1016/j.rse.2009.08.016.
19. S.Fritz,, E.Bartholomé, A.Belward, A.Hartley, H.-J.Stibig, H.Eva, and P.Mayaux. 2003. Harmonisation, Mosaicing and Production of the Global Land Cover 2000 Database (Beta Version). Luxembourg: Office for Official Publications of the European Communities, No. EUR 20849EN.
20. S.Fritz,, I.McCallum, C.Schill, C.Perger, R.Grillmayer, F.Achard, F.Kraxner, and M.Obersteiner. 2009. “Geo-Wiki.Org: The Use of Crowdsourcing to Improve Global Land Cover.” Remote Sensing 1 (3): 345–354. doi:10.3390/rs1030345.
21. S.Fritz,, I.McCallum, C.Schill, C.Perger, L.See, D.Schepaschenko, M.van der Velde, F.Kraxner, and M.Obersteiner. 2012. “Geo-Wiki: An Online Platform for Improving Global Land Cover.” Environmental Modelling & Software 31: 110–123. doi:10.1016/j.envsoft.2011.11.015.
22. S.Fritz,, L.See, M.van der Velde, R.A.Nalepa, C.Perger, C.Schill, I.McCallum. 2013. “Downgrading Recent Estimates of Land Available for Biofuel Production.” Environmental Science & Technology 47: 1688–1694.
23. J.-F.Girres,, and G.Touya. 2010. “Quality Assessment of the French OpenStreetMap Dataset.” Transactions in GIS 14: 435–459. doi:10.1111/j.1467-9671.2010.01203.x.
24. M.F.Goodchild, 2007. “Citizens as Sensors: The World of Volunteered Geography.” GeoJournal 69 (4): 211–221. doi:10.1007/s10708-007-9111-y.
25. M.Haklay, 2010. “How Good is Volunteered Geographical Information? A Comparative Study of OpenStreetMap and Ordnance Survey Datasets.” Environment and Planning B: Planning and Design 37: 682–703. doi:10.1068/b35097.
26. M.Haklay,, S.Basiouka, V.Antoniou, and A.Ather. 2010. “How Many Volunteers does it take to Map an Area Well? The Validity of Linus’ Law to Volunteered Geographic Information.” The Cartographic Journal 47: 315–322. doi:10.1179/000870410X12911304958827.
27. R.Hecht,, C.Kunze, and S.Hahmann. 2013. “Measuring Completeness of Building Footprints in OpenStreetMap over Space and Time.” ISPRS International Journal of Geo-Information 2: 1066–1091. doi:10.3390/ijgi2041066.
28. J.Howe, 2006. “The Rise of Crowdsouring.” Wired Magazine, 14 (6). http://archive.wired.com/wired/archive/14.06/crowds_pr.html.
29. J.Jokar Arsanjani,, M.Helbich, M.Bakillah, and L.Loos. 2015. “The Emergence and Evolution of OpenStreetMap: A Cellular Automata Approach.” International Journal of Digital Earth 8: 1–15.
30. F.Khatib,, F.DiMaio, F.C.Group, F.V.C.Group, S.Cooper, M.Kazmierczyk, M.Gilski. 2011. “Crystal Structure of a Monomeric Retroviral Protease Solved by Protein Folding Game Players.” Nature Structural & Molecular Biology 18: 1175–1177. doi:10.1038/nsmb.2119.
31. R.Koenker,, and J.A.F.Machado. 1999. “Goodness of Fit and Related Inference Processes for Quantile Regression.” Journal of the American Statistical Association 94: 1296–1310. doi:10.1080/01621459.1999.10473882.
32. P.Legendre, and L.Legendre. 1988. Numerical Ecology. Number 20 in Developments in Environmental Modelling. 2nd ed. Amsterdam: Elsevier.
33. D.R.Michael,, and S.L.Chen. 2005. Serious Games: Games that Educate, Train, and Inform. Cincinnati, OH: Muska & Lipman/Premier-Trade.
34. A.Miller-Rushing,, R.Primack, and R.Bonney. 2012. “The History of Public Participation in Ecological Research.” Frontiers in Ecology and the Environment 10: 285–290. doi:10.1890/110278.
35. P.Neis,, D.Zielstra, and A.Zipf. 2012. “The Street Network Evolution of Crowdsourced Maps: OpenStreetMap in Germany 2007–2011.” Future Internet 4 (4): 1–21. doi:10.3390/fi4010001.
36. C.Perger,, S.Fritz, L.See, C.Schill, M.van der Velde, I.McCallum, and M.Obersteiner. 2012. “A Campaign to Collect Volunteered Geographic Information on Land Cover and Human Impact.” In GI_Forum 2012: Geovisualisation, Society and Learning, edited by T.Jekel,, A.Car, J.Strobl, and G.Griesebner, 83–91. Berlin/Offenbach: Herbert Wichmann Verlag.
37. F.Ramm,, J.Topf, and S.Chilton. 2011. OpenstreetMap: Using and Enhancing the Free Map of the World. Cambridge: UIT Cambridge.
38. L.See,, A.Comber, C.Salk, S.Fritz, M.van der Velde, C.Perger, C.Schill, I.McCallum, F.Kraxner, and M.Obersteiner. 2013. “Comparing the Quality of Crowdsourced Data Contributed by Expert and Non-experts.” PLoS One 8 (7): e69958. doi:10.1371/journal.pone.0069958.t012.
39. L.See,, S.Fritz, and J.de Leeuw. 2013. “The Rise of Collaborative Mapping: Trends and Future Directions.” ISPRS International Journal of Geo-Information 2: 955–958. doi:10.3390/ijgi2040955.
40. B.L.Sullivan,, J.L.Aycrigg, J.H.Barry, R.E.Bonney, N.Bruns, C.B.Cooper, T.Damoulas. 2014. “The eBird Enterprise: An Integrated Approach to Development and Application of Citizen Science.” Biological Conservation 169: 31–40. doi:10.1016/j.biocon.2013.11.003.
41. L.Yu,, J.Wang, N.Clinton, Q.Xin, L.Zhong, Y.Chen, and P.Gong. 2013. “FROM-GC: 30 m Global Cropland Extent Derived through Multisource Data Integration.” International Journal of Digital Earth 6: 550–562. doi:10.1080/17538947.2011.628414.
42. Y.Zhao,, P.Gong, L.Yu, L.Hu, X.Li, C.Li, H.Zhang. 2014. “Towards a Common Validation Sample Set for Global Land-cover Mapping.” International Journal of Remote Sensing 35: 4795–4814. doi:10.1080/01431161.2014.930202.