Automated data verification in a large-scale citizen science project: A case study

2012 IEEE 8th International Conference on E-Science, e-Science 2012

Volumn , Issue , 2012, Pages

  Yu, Jun ^a   Kelling, Steve ^b   Gerbracht, Jeff ^b   Wong, Weng Keen ^a  

^a Oregon State University   (United States)

^b Fuller Evolutionary Biology Program   (United States)

Author keywords

Applications; Citizen science; Crowdsourcing; Data filters; Data quality; Species distribution modeling

Indexed keywords

CITIZEN SCIENCE; CROWDSOURCING; DATA FILTER; DATA QUALITY; SPECIES DISTRIBUTION MODELING;

APPLICATIONS; AUTOMATION; DATA REDUCTION; STATISTICS;

DIGITAL STORAGE;

EID: 84873663480     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/eScience.2012.6404472     Document Type: Conference Paper

References (27)

1. R. Bonney, C. Cooper, J. Dickinson, S. Kelling, T. Philllips, K. Rosenberg, and J. Shirk, Citizen science: a developing tool for expanding science knowledge and scientific literacy. BioScience, 59(11): 977-984, 2009.
2. A. P. Dawid and A. M. Skene, Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society, Series C, 28(1): 20-28, 1979.
3. A. P. Dempster, N. M. Laird, and D. B. Rubin, Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B, 39(1): 1-38, 1977.
4. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and F.-F. Li, ImageNet: a large-scale hierarchical image database. In Proceedings of Computer Vision and Pattern Recognition, pp. 248-255, 2009.
5. J. L. Dickenson, B. Zuckerberg, and D. N. Bonter, Citizen science as an ecological research tool: challenge and benefits. Annual Review of Ecology Evolution and Systematics, 41(1): 149-172, 2010.
6. K. A. Ericsson and N. Charness, Expert performance: its structure and acquisition. American Psychologist, 49: 525-747, 1994.
7. W. Hochachka, D. Fink, R. Hutchinson, D. Sheldon, W.-K. Wong, and S. Kelling, Data-intensive science applied to broad-scale citizen science. Trends in Ecology and Evolution, 27(2): 130-137, 2012.
8. J. Howe, Crowdsourcing: why the power of the crowd is driving the future of business. Crown Business, New York, 2008.
9. S. Kelling, Using bioinformatics in citizen science. Citizen Science: Public Collaboration in Environmental Research, Cornell University Press. Ithaca, NY, 2011.
10. S. Kelling, J. Yu, J. Gerbracht, and W.-K. Wong, Emergent filters: automated data verification in a large-scale citizen science project. In Proceedings of the e-Science 2011 Computing for Citizen Science Workshop, pp. 20-27, 2011.
11. D. Koller and N. Friedman, Probabilistic Graphical Models: Principles and Techniques. The MIT Press, Cambridge, MA, 2009.
12. C. J. Lintott, K. Schawinski, S. Anze, K. Land, S. Bamford, D. Thomas, M. J. Raddick, R. C. Nichol, A. Szalay, D. Andreescu, P. Murray, and J. Vandenberg, Galaxy Zoo: morphologies derived from visual inspection of galaxies from the sloan digital sky survey. Monthly Notices of the Royal Astronomical Society, 389: 1179-1189, 2008.
13. D. I. MacKenzie, J. D. Nichols, J. E. Hines, M. G. Knutson, and A. B. Franklin, Estimating site occupancy, colonization, and local extinction when a species is detected imperfectly.Ecology, 84(8): 2200-2207, 2003.
14. D. I. MacKenzie, J. D. Nichols, G. B. Lachman, S. Droege, J. A. Royle, and C. A. Langtimm, Estimating site occupancy rates when detection probabilities are less than one. Ecology, 83(8): 2248-2255, 2002.
15. M. A. Munson, K. Webb, D. Sheldon, D. Fink, W. M. Hochachka, M. Iliff, M. Riedewald, D. Sorokina, B. Sullivan, C. Wood, and S. Kelling, The eBird reference dataset, version 3.0. Cornell Lab of Ornithology and National Audubon Society, Ithaca, NY, December 2011.
16. M. A. Munson, R. Caruana, D. Fink, W. M. Hochachka, M. Iliff, K. V. Rosenberg, D. Sheldon, B. L. Sullivan, C. Wood, and S. Kelling, A method for measuring the relative information content of data from different monitoring protocols. Methods In Ecology and Evolution, 1(3): 263-273, 2010.
17. N. A. B. C. I. U. S. The state of the birds 2011 report on public lands and waters. Washington D.C., 2011.
18. V. C. Raykar, S. Yu, L. H. Zhao, A. Jerebko, C. Florin, G. H. Valadez, L. Bogoni, and L. Moy, Supervised learning from multiple experts: whom to trust when everyone lies a bit. In Proceedings of the 26th International Conference on Machine Learning, pp. 889-896, 2009.
19. th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 614-622, 2008.
20. P. Smyth, U. Fayyad, M. Burl, P. Perona, and P. Baldi, Inferring ground truth from subjective labeling of venus images. In Advances in Neural Information Processing System, pp. 1085-1092, 1995.
21. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng, Cheap and fast - but is it good? : evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 254-263, 2008.
22. B. L. Sullivan, C. L. Wood, M. J. Iliff, R. Bonney, D. Fink, and S. Kelling, eBird: a citizen-based bird observation network in the biological sciences. Biological Conservation, 142(10): 2282-2292, 2009.
23. D. J. Trumbull, R. Bonney, D. Bascom, and A. Cabral, Think scientifically during participation in a citizen-science project. Science Education, 84(2): 265-275, 2000.
24. P. Welinder, S. Branson, S. Belongie, and P. Perona, The multidimensional wisdom of crowds. In Advances in Neural Information Processing Systems 23, pp. 2424-2432, 2010.
25. J. Whitehill, P. Ruvolo, T. F. Wu, J. Bergsma, and J. Movellan, Whose vote should count more: optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems 22, pp. 2035-2043, 2009.
26. C. Wood, B. Sullivan, M. Iliff, D. Fink, and S. Kelling, eBird engaging birders in science and conservation. PLoS Biol, 9(12): e1001220, 2011.
27. J. Yu, W.-K. Wong, and R. A Hutchinson, Modeling experts and novices in citizen science data for species distribution modeling. In Proceedings of the 2010 IEEE International Conference on Data Mining, pp. 1157-1162, 2010.