Assessing data quality in citizen science

Frontiers in Ecology and the Environment

Volumn 14, Issue 10, 2016, Pages 551-560

  Kosmala, Margaret ^a   Wiggins, Andrea ^b   Swanson, Alexandra ^c   Simmons, Brooke ^c,d  

^a HARVARD UNIVERSITY   (United States)

^b UNIVERSITY OF MARYLAND   (United States)

^c UNIVERSITY OF OXFORD   (United Kingdom)

^d UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ASSESSMENT METHOD; DATA QUALITY; DATA SET; KNOWLEDGE; PROJECT DESIGN; RESOURCE MANAGEMENT; TRAINING;

EID: 84999633667     PISSN: 15409295     EISSN: 15409309     Source Type: Journal    
DOI: 10.1002/fee.1436     Document Type: Article

References (65)

1. Alabri A and Hunter J. 2010. Enhancing the quality and trust of citizen science data. In: IEEE Sixth International Conference on e-Science; 7–10 Dec 2010. Brisbane, Australia: IEEE.
2. Atkinson GM and Wald DJ. 2007. “Did You Feel It?” intensity data: a surprisingly good measure of earthquake ground motion. Seismol Res Lett 78: 362–68.
3. Bergstedt J, Westerberg L, and Milberg P. 2009. In the eye of the beholder: bias and stochastic variation in cover estimates. Plant Ecol 204: 271–83.
4. Bird TJ, Bates AE, Lefcheck JS, et al. 2014. Statistical solutions for error and bias in global citizen science datasets. Biol Conserv 173: 144–54.
5. Bloniarz DV and Ryan HDP III. 1996. The use of volunteer initiatives in conducting urban forest resource inventories. J Arboric 22: 75–82.
6. Bonney R, Shirk JL, Phillips TB, et al. 2014. Next steps for citizen science. Science 343: 1436–37.
7. Bonter DN and Cooper CB. 2012. Data validation in citizen science: a case study from Project FeederWatch. Front Ecol Environ 10: 305–07.
8. Borer ET, Seabloom EW, Jones MB, and Schildhauer M. 2009. Some simple guidelines for effective data management. Bull Ecol Soc Am 90: 205–14.
9. Bowser A, Hansen D, He Y, et al. 2013. Using gamification to inspire new citizen science volunteers. In: Proceedings of the First International Conference on Gameful Design, Research, and Applications; 2–4 Oct 2013. Stratford, Canada: ACM.
10. Bowser-Livermore A and Wiggins A. 2015. Privacy in participatory research: advancing policy to support human computation. Hum Comput 2: 19–44.
11. Cohn JP. 2008. Citizen science: can volunteers do real research? BioScience 58: 192–97.
12. Conrad CC and Hilchey KG. 2011. A review of citizen science and community-based environmental monitoring: issues and opportunities. Environ Monit Assess 176: 273–91.
13. Cook RD and Jacobson JO. 1979. A design for estimating visibility bias in aerial surveys. Biometrics 35: 735.
14. Cooper CB, Shirk J, and Zuckerberg B. 2014. The invisible prevalence of citizen science in global research: migratory birds and climate change. PLoS ONE 9: e106508.
15. Courter JR, Johnson RJ, Stuyck CM, et al. 2013. Weekend bias in citizen science data reporting: implications for phenology studies. Int J Biometeorol 57: 715–20.
16. Cox TE, Philippoff J, Baumgartner E, and Smith CM. 2012. Expert variability provides perspective on the strengths and weaknesses of citizen-driven intertidal monitoring program. Ecol Appl 22: 1201–12.
17. Crall AW, Newman GJ, Jarnevich CS, et al. 2010. Improving and integrating data on invasive species collected by citizen scientists. Biol Invasions 12: 3419–28.
18. Crall AW, Newman GJ, Stohlgren TJ, et al. 2011. Assessing citizen science data quality: an invasive species case study. Conserv Lett 4: 433–42.
19. Danielsen F, Jensen PM, Burgess ND, et al. 2014. A multicountry assessment of tropical resource monitoring by local communities. BioScience 64: 236–51.
20. Delaney DG, Sperling CD, Adams CS, and Leung B. 2008. Marine invasive species: validation of citizen science and implications for national monitoring networks. Biol Invasions 10: 117–28.
21. Dickinson JL, Shirk J, Bonter D, et al. 2012. The current state of citizen science as a tool for ecological research and public engagement. Front Ecol Environ 10: 291–97.
22. Dickinson JL, Zuckerberg B, and Bonter DN. 2010. Citizen science as an ecological research tool: challenges and benefits. Annu Rev Ecol Evol Syst 41: 149–72.
23. Elkinton J, Ellison A, Preisser E, and Fitzpatrick M. 2009. Observer bias and the detection of low-density populations. Ecol Appl 19: 1673–79.
24. Engel SR and Voshell JRJ. 2002. Volunteer biological monitoring: can it accurately assess the ecological condition of streams? Am Entomol 48: 164–77.
25. Eveleigh A, Jennett C, Blandford A, et al. 2014. Designing for dabblers and deterring drop-outs in citizen science. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; 26 Apr–1 May 2014. Toronto, Canada: ACM Press.
26. Fuccillo KK, Crimmins TM, de Rivera CE, and Elder TS. 2015. Assessing accuracy in citizen science-based plant phenology monitoring. Int J Biometeorol 59: 917–26.
27. Galloway AWE, Tudor MT, and Vander Haegen WM. 2006. The reliability of citizen science: a case study of Oregon white oak stand surveys. Wildl Soc Bull 34: 1425–29.
28. Gardiner MM, Allee LL, Brown PM, et al. 2012. Lessons from lady beetles: accuracy of monitoring data from US and UK citizen-science programs. Front Ecol Environ 10: 471–76.
29. Hines G, Swanson A, Kosmala M, and Lintott C. 2015. Aggregating user input in ecology citizen science projects. In: Proceedings of the Twenty-Seventh Conference on Innovative Applications of Artificial Intelligence; 25–29 Jan 2015. Austin, TX: AAAI.
30. Hochachka WM, Fink D, Hutchinson RA, et al. 2012. Data-intensive science applied to broad-scale citizen science. Trends Ecol Evol 27: 130–37.
31. Jiguet F. 2009. Method learning caused a first-time observer effect in a newly started breeding bird survey. Bird Study 56: 253–58.
32. Kageyama M, Monk RR, Bradley RD, et al. 2007. The changing significance and definition of the biological voucher. In: Williams S and Hawks C (Eds). Museum studies: perspectives and innovations. Washington, DC: Society for the Preservation of Natural History Collections.
33. Kamar E, Hacker S, and Horvitz E. 2012. Combining human and machine intelligence in large-scale crowdsourcing. In: Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems; 4–8 Jun 2012. Valencia, Spain: International Foundation for Autonomous Agents and Multiagent Systems.
34. Kelling S, Johnston A, Hochachka WM, et al. 2015. Can observation skills of citizen scientists be estimated using species accumulation curves? PLoS ONE 10: e0139600.
35. Kendall WL, Peterjohn BG, and Sauer JR. 1996. First-time observer effects in the North American Breeding Bird Survey. The Auk 113: 823–29.
36. LeResche RE and Rausch RA. 1974. Accuracy and precision of aerial moose censusing. J Wildl Manag 38: 175–82.
37. McKinley DC, Miller-Rushing AJ, Ballard HL, et al. 2015. Investing in citizen science can improve natural resource management and environmental protection. Issues in Ecology 19: 1–27.
38. McShea WJ, Forrester T, Costello R, et al. 2015. Volunteer-run cameras as distributed sensors for macrosystem mammal research. Landsc Ecol 31: 55–66.
39. Michener WK and Jones MB. 2012. Ecoinformatics: supporting ecology as a data-intensive science. Trends Ecol Evol 27: 85–93.
40. Moon JTI, Guinan PE, Snider DJ, and Lupo AR. 2009. CoCoRaHs in Missouri: four years later, the importance of observations. Trans Missouri Acad Sci 43: 1–12.
41. Newman G, Wiggins A, Crall A, et al. 2012. The future of citizen science: emerging technologies and shifting paradigms. Front Ecol Environ 10: 298–304.
42. Ninio R, Delean J, Osborne K, and Sweatman H. 2003. Estimating cover of benthic organisms from underwater video images: variability associated with multiple observers. Mar Ecol-Prog Ser 265: 107–16.
43. Pipino LL, Lee YW, and Wang RY. 2002. Data quality assessment. Commun ACM 45: 211–18.
44. Prysby MD and Oberhauser KS. 2004. Temporal and geographic variation in monarch densities: citizen scientists document monarch population patterns. In: Oberhauser KS and Solensky MJ (Eds). Monarch butterfly biology & conservation. Ithaca, NY: Cornell University Press.
45. Resnik DB, Elliott KC, and Miller AK. 2015. A framework for addressing ethical issues in citizen science. Environ Sci Policy 54: 475–81.
46. Riesch H and Potter C. 2014. Citizen science as seen by scientists: methodological, epistemological and ethical dimensions. Public Underst Sci 23: 107–20.
47. Rosemartin AH, Denny EG, Weltzin JF, et al. 2015. Lilac and honeysuckle phenology data 1956–2014. Sci Data 2: 150038.
48. Sauer JR, Link WA, Fallon JE, et al. 2013. The North American Breeding Bird Survey 1966–2011: summary analysis and species accounts. North Am Fauna 79: 1–32.
49. Schwamb ME, Lintott CJ, Fischer DA, et al. 2012. Planet hunters: assessing the Kepler inventory of short-period planets. Astrophys J 754: 129.
50. Sheppard SA and Terveen L. 2011. Quality is a verb: the operationalization of data quality in a citizen science community. In: Proceedings of the 7th International Symposium on Wikis and Open Collaboration; 3–5 Oct 2011. Mountain View, CA: ACM.
51. Show H. 2015. Rise of the citizen scientist. Nature 524: 265.
52. Silvertown J. 2009. A new dawn for citizen science. Trends Ecol Evol 24: 467–71.
53. Sullivan BL, Aycrigg JL, Barry JH, et al. 2014. The eBird enterprise: an integrated approach to development and application of citizen science. Biol Conserv 169: 31–40.
54. Swanson A, Kosmala M, Lintott C, and Packer C. 2016. A generalized approach for producing, quantifying, and validating citizen science data from wildlife images: citizen science data quality. Conserv Biol 30: 520–31.
55. Theobald EJ, Ettinger AK, Burgess HK, et al. 2015. Global change and local solutions: tapping the unrealized potential of citizen science for biodiversity research. Biol Conserv 181: 236–44.
56. van der Wal R, Sharma N, Mellish C, et al. 2016. The role of automated feedback in training and retaining biological recorders for citizen science: automated feedback for citizen science. Conserv Biol 30: 550–61.
57. Weir LA, Royle JA, Nanjappa P, and Jung RE. 2005. Modeling anuran detection and site occupancy on North American Amphibian Monitoring Program (NAAMP) routes in Maryland. J Herpetol 39: 627–39.
58. Westphal AJ, von Korff J, Anderson DP, et al. 2006. Stardust@home: virtual microscope validation and first results. In: 37th Annual Lunar and Planetary Science Conference; 13–17 Mar 2006; League City, TX. Abstract no 2225.
59. Wiggins A. 2012. Crowdsourcing scientific work: a comparative study of technologies, processes, and outcomes in citizen science (PhD dissertation). Syracuse, NY: Syracuse University.
60. Wiggins A. 2013. Free as in puppies: compensating for ICT constraints in citizen science. In: Proceedings of the 16th ACM Conference on Computer-Supported Cooperative Work and Social Computing; 23–27 Feb 2013. San Antonio, TX: ACM.
61. Wiggins A and Crowston K. 2015. Surveying the citizen science landscape. First Monday 20: 1.
62. Wiggins A and He Y. 2016. Community-based data validation practices in citizen science. In: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work and Social Computing; 27 Feb–2 Mar 2016. San Francisco, CA: ACM.
63. Wiggins A, Bonney R, Graham E, et al. 2013. Data management guide for public participation in scientific research. Albuquerque, NM: DataONE.
64. Woz´niak M, Graña M, and Corchado E. 2014. A survey of multiple classifier systems as hybrid systems. Inf Fusion 16: 3–17.
65. Yu J, Wong W-K, and Hutchinson RA. 2010. Modeling experts and novices in citizen science data for species distribution modeling. In: 2010 IEEE International Conference on Data Mining; 14–17 Dec 2010. Sydney, Australia: IEEE.