Crowdsourcing as a tool for research: Implications of uncertainty

Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW

Volumn , Issue , 2017, Pages 1544-1561

  Law, Edith ^a   Gajos, Krzysztof Z ^b   Wiggins, Andrea ^c   Gray, Mary L ^d   Williams, Alex ^a  

^a UNIVERSITY OF WATERLOO   (Canada)

^b HARVARD UNIVERSITY   (United States)

^c UNIVERSITY OF MARYLAND   (United States)

^d MICROSOFT RESEARCH   (United Kingdom)

Author keywords

Citizen science; Crowdsourcing for research; Interviews

Indexed keywords

COMPUTER SUPPORTED COOPERATIVE WORK; GROUPWARE; INTERACTIVE COMPUTER SYSTEMS;

CITIZEN SCIENCE; CROWDSOURCING PLATFORMS; DESIGN IMPLICATIONS; INTERVIEWS; RESEARCH DATA; RESEARCH PROCESS; SEMI STRUCTURED INTERVIEWS;

CROWDSOURCING;

EID: 85014787889     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2998181.2998197     Document Type: Conference Paper

References (96)

1. Adam J. Berinsky, Gregory A. Huber, and Gabriel S. Lenz. 2012. Evaluating online labor markets for experimental research: Amazon.com's mechanical turk. Political Analysis 20, 3(2012), 351-368. DOI: http://dx.doi.org/10.1093/pan/mpr057
2. Jeremy P. Birnholtz and Matthew J. Bietz. 2003. Data at Work: Supporting Sharing in Science and Engineering. In Proceedings of the International ACM SIGGROUP Conference on Supporting Group Work (GROUP'03). ACM, New York, NY, USA, 339-348. DOI: http://dx.doi.org/10.1145/958160.958215
3. David Blumenthal, Eric G Campbell, Manjusha Gokhale, Recai Yucel, Brian Clarridge, Stephen Hilgartner, and Neil a Holtzman. 2006. Data withholding in genetics and the other life sciences: prevalences and predictors. Academic Medicine 81, 2(2006), 137-145. DOI: http://dx.doi.org/10.1097/00001888-200602000-00008
4. Sara Bly. 1998. Special Section on Collaboratories. interactions 5, 3(May 1998), 31. DOI: http://dx.doi.org/10.1145/275269.275273
5. John Bohannon. 2016. Psychologists grow increasingly dependent on online research subjects. (2016). Retrieved June 7, 2016 from http://www.sciencemag.org/news/2016/06/psychologistsgrow-increasingly-dependent-online-research-subjects.
6. Rick Bonney, Caren B. Cooper, Janis Dickinson, Steve Kelling, Tina Phillips, Kenneth V. Rosenberg, and Jennifer Shirk. 2009. Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy. BioScience 59, 11(2009), 977-984. DOI: http://dx.doi.org/10.1525/bio.2009.59.11.9
7. Anne Bowser, Derek Hansen, Jennifer Preece, Yurong He, Carol Boston, and Jen Hammock. 2014. Gamifying Citizen Science: A Study of Two User Groups. In Proceedings of the Companion Publication of the 17th ACM Conference on Computer Supported Cooperative Work (CSCW Companion'14). ACM, New York, NY, USA, 137-140. DOI: http://dx.doi.org/10.1145/2556420.2556502
8. Anne Bowser-Livermore and Andrea Wiggins. 2015. Privacy in Participatory Research: Advancing Policy to support Human Computation. Human Computation 2, 1(2015), 19-44. DOI: http://dx.doi.org/10.15346/hc.v2i1.3
9. Alex Bowyer, Chris Lintott, Greg Hines, Campbell Allan, and Ed Paget. 2015. Panoptes, a Project Building Tool for Citizen Science. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP'15). AAAI, San Diego, CA, USA, 1-2.
10. H. K. Burgess, L. B. DeBey, H. E. Froehlich, N. Schmidt, E. J. Theobald, A. K. Ettinger, J. HilleRisLambers, J. Tewksbury, and J. K. Parrish. 2016. The science of citizen science: Exploring barriers to use as a primary research tool. Biological Conservation (2016), 1-8. DOI: http://dx.doi.org/10.1016/j.biocon. 2016.05.014
11. Carolin Cardamone, Kevin Schawinski, Marc Sarzi, Steven P. Bamford, Nicola Bennert, C. M. Urry, Chris Lintott, William C. Keel, John Parejko, Robert C. Nichol, Daniel Thomas, Dan Andreescu, Phil Murray, M. Jordan Raddick, Anze Slosar, Alex Szalay, and Jan Vandenberg. 2009. Galaxy Zoo Green Peas: Discovery of a class of compact extremely star-forming galaxies. Monthly Notices of the Royal Astronomical Society 399, 3(2009), 1191-1205. DOI: http://dx.doi.org/10.1111/j.1365-2966.2009.15383.x
12. Casey Trees. 2016. Citizen Science. (2016). Retrieved January 15, 2016 from http://caseytrees.org/get-involved/citizen-science/.
13. Karin Knorr Cetina. 1999. Epistemic Communities: How the Sciences Make Knowledge. Harvard Education Press, Cambridge, MA.
14. Sayan Chakraborti, Naveen Yadav, Carolin Cardamone, and Alak Ray. 2012. Radio Detection of Green Peas: Implications for Magnetic Fields in Young Galaxies. The Astrophysical Journal 746, 1(2012), L6. DOI: http://dx.doi.org/10.1088/2041-8205/746/1/L6
15. Jeffrey P Cohn. 2008. Citizen Science: Can Volunteers Do Real Research? BioScience 58, 3(2008), 192-197. DOI: http://dx.doi.org/10.1641/B580303
16. Harry Collins. 1992. Changing Order: Replication and Induction in Scientific Practice. University of Chicago Press, Chicago, IL.
17. Harry Collins and Robert Evans. 2007. Rethinking Expertise. University of Chicago Press.
18. Colorado State University. 2016. Citsci. Org. (2016). Retrieved January 15, 2016 from http://www.citsci.org.
19. Caren Cooper. 2016. Zen in the Art of Citizen Science: Apps for Collective Discovery and the 1 Percent Rule of the Web. (2016). Retrieved January 15, 2016 from http://blogs.scientificamerican.com/guest-blog/zen-inthe-art-of-citizen-science-apps-for-collectivediscovery-and-the-1-percent-rule-of-the-web/.
20. Juliet M. Corbin and Anselm Strauss. 2008. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory (3rd ed.). Sage, Thousand Oaks, CA.
21. M. L. Cornwell and L. M. Campbell. 2012. Co-producing conservation and knowledge: Citizen-based sea turtle monitoring in North Carolina, USA. Social Studies of Science 42, 1(2012), 101-120. DOI: http://dx.doi.org/10.1177/0306312711430440
22. Joe Cox, Eun Young Oh, Brooke Simmons, Chris Lintott, Karen Masters, Anita Greenhill, Gary Graham, and Kate Holmes. 2015. Defining and Measuring Success in Online Citizen Science: A Case Study of Zooniverse Projects. IEEE Computing in Science & Engineering 17, 4(2015), 28-41. DOI: http://dx.doi.org/10.1109/MCSE.2015.65
23. M. H. Cragin, C. L. Palmer, J. R. Carlson, and M. Witt. 2010. Data sharing, small science and institutional repositories. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 368, 1926(2010), 4023-4038. DOI: http://dx.doi.org/10.1098/rsta.2010.0165
24. Schlagwein Daniel and Daneshgar Farhad. 2014. User requirements of a crowdsourcing platform for researchers: findings from a series of focus groups. In Proceedings of the 2014 Pacific Asia Conference on Information Systems. 1-11.
25. Janet Dickinson and Rick Bonney. 2012. Citizen Science: Public Participation in Environmental Research. Cornstock Pub. Associates, Ithaca, NY.
26. Michael Dummett. 1991. The Logical Basis of Metaphysics. Duckworth, London, UK.
27. Kevin Dunbar. 2000. How Scientists Think in the Real World. Journal of Applied Developmental Psychology 21, 1(2000), 49-58. DOI: http://dx.doi.org/10.1016/S0193-3973 (99) 00050-7
28. Alexandra Eveleigh, Charlene Jennett, Ann Blandford, Philip Brohan, and Anna L. Cox. 2014. Designing for Dabblers and Deterring Drop-outs in Citizen Science. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI'14). ACM, New York, NY, USA, 2985-2994. DOI: http://dx.doi.org/10.1145/2556288.2557262
29. Elizabeth J Farnsworth, Miyoko Chu, W John Kress, Amanda K Neill, Jason H Best, John Pickering, Robert D Stevenson, Gregory W Courtney, John K V Dyk, and Aaron M Ellison. 2013. Next-Generation Field Guides. BioScience 63, 11(2013), 891-899. DOI: http://dx.doi.org/10.1525/bio.2013.63.11.8
30. Karen Fort, Gilles Adda, and K. Bretonnel Cohen. 2011. Amazon Mechanical Turk: Gold Mine or Coal Mine? Computational Linguistics 37, 2(June 2011), 413-420. DOI: http://dx.doi.org/10.1162/COLI-a-00057
31. Amy Freitag and Max Pfeffer. 2013. Process, Not Product: Investigating Recommendations for Improving Citizen Science Success. PLoS One 8, 5(2013), e64079. DOI: http://dx.doi.org/10.1371/journal.pone.0064079
32. Bruno S. Frey and Reto Jegen. 2001. Motivation Crowding Theory. Journal of Economic Surveys 15, 5(2001), 589-611. DOI: http://dx.doi.org/10.1111/1467-6419.00150
33. Jay R Galbraith. 1974. Organization Design: An Information Processing View. Interfaces 4, 3(1974), 28-36. DOI: http://dx.doi.org/10.1287/inte.4.3.28
34. Thomas F. Giergyn. 1999. Cultural Boundaries of Science: Credibility on the Line. University of Chicago Press, Chicago, IL.
35. Thomas F Gieryn. 1983. Boundary-Work and the Demarcation of Science From Non-Science: Strains and Interests in Professional Ideologies of Scientists. American Sociological Review 48, 6(1983), 781-795. DOI: http://dx.doi.org/Doi10.2307/2095325
36. Elizabeth Goodman and Janet Vertesi. 2012. Design for X?: Distribution Choices and Ethical Design. In Extended Abstracts of the 30th Annual ACM Conference on Human Factors in Computing Systems (CHI EA'12). ACM, New York, NY, USA, 81-90. DOI: http://dx.doi.org/10.1145/2212776.2212786
37. David Grier. 2007. When Computers Were Humans. Princeton University Press, Princeton, NJ.
38. Katie Hafner and Matthew Lyon. 1996. Where Wizards Stay up Late: the Origins of the Internet. Simon & Schuster, New York.
39. Muki Haklay. 2013. Crowdsourcing Geographic Knowledge: Volunteered Geographic Information (VGI) in Theory and Practice. Springer, Berlin, Chapter Citizen Science and Volunteer Geographic Information: Overview and Typology of Participation, 105-122. DOI: http://dx.doi.org/10.1007/978-94-007-4587-2-7
40. Harvard School of Engineering and Applied Sciences. 2016. Privacy Tools for Sharing Research Data. (2016). Retrieved January 15, 2016 from http://privacytools.seas.harvard.edu.
41. Imperial College London. 2016. EpiCollect.net Website. (2016). Retrieved January 15, 2016 from http://www.epicollect.net.
42. Yuri I. Izotov, Natalia G. Guseva, and Trinh X. Thuan. 2011. Green Pea Galaxies and Cohorts: Luminous Compact Emission-Line Galaxies in the Sloan Digital Sky Survey. The Astrophysical Journal 728, 2(2011), 161. DOI: http://dx.doi.org/10.1088/0004-637X/728/2/161
43. Corey Jackson, Gabriel Mugar, Kevin Crowston, and Carsten Osterlund. 2016. Encouraging Work in Citizen Science: Experiments in Goal Setting and Anchoring. In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion (CSCW'16 Companion). ACM, New York, NY, USA, 297-300. DOI: http://dx.doi.org/10.1145/2818052.2869129
44. A. E. Jaskot and M. S. Oey. 2013. The Origin and Optical Depth of Ionizing Radiation in the "Green Pea" Galaxies. The Astrophysical Journal 766(2013), 91. DOI: http://dx.doi.org/10.1088/0004-637X/766/2/91
45. Marina Jirotka, Charlotte P. Lee, and Gary M. Olson. 2013. Supporting scientific collaboration: Methods, tools and concepts. Computer Supported Cooperative Work: CSCW: An International Journal 22, 4-6(2013), 667-715. DOI: http://dx.doi.org/10.1007/s10606-012-9184-0
46. Sunyoung Kim, Jennifer Mankoff, and Eric Paulos. 2013. Sensr: Evaluating a Flexible Framework for Authoring Mobile Data-collection Tools for Citizen Science. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW'13). ACM, New York, NY, USA, 1453-1462. DOI: http://dx.doi.org/10.1145/2441776.2441940
47. Thomas Kuhn. 2012. The Structure of Scientific Revolutions (4th ed.). University of Chicago Press, Chicago, IL.
48. Deepak Kulkarni and Herbert A. Simon. 1988. The processes of scientific discovery: The strategy of experimentation. Cognitive Science 12, 2(1988), 139-175. DOI: http://dx.doi.org/10.1016/0364-0213 (88) 90020-1
49. Bruno Latour and Steve Woolgar. 1986. Laboratory life: the Social Construction of Scientific Facts (2nd ed.). Princeton University Press, Beverly Hills, CA.
50. Edith Law, Connor Dalton, Nick Merrill, Alex Young, and Krzysztof Z. Gajos. 2013. Curio: a platform for supporting mixed-expertise crowdsourcing. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP'13). AAAI, 1-2.
51. Chris J. Lintott, Kevin Schawinski, Anze Slosar, Kate Land, Steven Bamford, Daniel Thomas, M. Jordan Raddick, Robert C. Nichol, Alex Szalay, Dan Andreescu, Phil Murray, and Jan Vandenberg. 2008. Galaxy Zoo: Morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey. Monthly Notices of the Royal Astronomical Society 389, 3(2008), 1179-1189. DOI: http://dx.doi.org/10.1111/j.1365-2966.2008.13689.x
52. Christopher S. Lowry and Michael N. Fienen. 2013. CrowdHydrology: Crowdsourcing hydrologic data and engaging citizen scientists. GroundWater 51, 1(2013), 151-156. DOI: http://dx.doi.org/10.1111/j.1745-6584.2012.00956.x
53. Winter Mason and Siddharth Suri. 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods 44, 1(2012), 1-23. DOI: http://dx.doi.org/10.3758/s13428-011-0124-6
54. R. Merton, E. Barber, and J. Shulman. 2006. The Travels and Adventures of Serendipity: A Study in Sociological Semantics and the Sociology of Science. Princeton University Press.
55. New York University. 2016. Project Noah. (2016). Retrieved January 15, 2016 from http://www.projectnoah.org.
56. Jon Noronha, Eric Hysen, Haoqi Zhang, and Krzysztof Z. Gajos. 2011. Platemate: Crowdsourcing Nutritional Analysis from Food Photographs. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST'11). ACM, New York, NY, USA, 1-12. DOI: http://dx.doi.org/10.1145/2047196.2047198
57. Gary Olson, Ann Zimmerman, Nathan Bos, and William Wulf. 2008. Scientific Collaboration on the Internet. MIT Press, Cambridge, MA.
58. Gary M. Olson, Daniel E. Atkins, Robert Clauer, Thomas A. Finholt, Farnam Jahanian, Timothy L. Killeen, Atul Prakash, and Terry Weymouth. 1998. The Upper Atmospheric Research Collaboratory (UARC). interactions 5, 3(May 1998), 48-55. DOI: http://dx.doi.org/10.1145/275269.275276
59. Johan Oomen and Lora Aroyo. 2011. Crowdsourcing in the Cultural Heritage Domain: Opportunities and Challenges. C&T'11 Proceedings of the 5th International Conference on Communities and Technologies July (2011), 138-149. DOI: http://dx.doi.org/10.1145/2103354.2103373
60. Michael Polanyi. 1967. The Tacit Dimension. Routledge and Kegan Paul, London, UK.
61. J. Preece and B. Schneiderman. 2009. The Reader-to-Leader Framework: Motivating Technology-Mediated Social Participation. AIS Transactions on Human-Computer Interaction 1, 1(2009), 13-32. DOI: http://dx.doi.org/10.5121/ijfcst.2014.4403
62. Nancy Proctor. 2013. Crowdsourcing-An Introduction: From Public Goods to Public Good. Curator: The Museum Journal 56, 1(January 2013), 105-106. DOI: http://dx.doi.org/DOI:10.1111/cura.12010
63. Danial Qaurooni, Ali Ghazinejad, Inna Kouper, and Hamid Ekbia. 2016. Citizens for Science and Science for Citizens: The View from Participatory Design. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI'16). ACM, New York, NY, USA, 1822-1826. DOI: http://dx.doi.org/10.1145/2858036.2858575
64. M. Jordan Raddick, Georgia Bracey, Pamela L. Gay, Chris J. Lintott, Phil Murray, Kevin Schawinski, Alexander S. Szalay, and Jan Vandenberg. 2010. Galaxy Zoo: Exploring the Motivations of Citizen Science Volunteers. Astronomy Education Review 9, 1(2010), 010103. DOI: http://dx.doi.org/10.3847/AER2009036
65. Katharina Reinecke and Krzysztof Z. Gajos. 2015. LabintheWild: Conducting Large-Scale Online Experiments With Uncompensated Samples. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work (CSCW'15). ACM, New York, NY, USA, 1364-1378. DOI: http://dx.doi.org/10.1145/2675133.2675246
66. Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Walter S. Lasecki, Jay Patel, Negar Rahmati, Tulsee Doshi, Melissa Valentine, and Michael S. Bernstein. 2014. Expert Crowdsourcing with Flash Teams. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST'14). ACM, New York, NY, USA, 75-85. DOI: http://dx.doi.org/10.1145/2642918.2647409
67. Hauke Riesch and Clive Potter. 2014. Citizen science as seen by scientists: Methodological, epistemological and ethical dimensions. Public Understanding of Science 23, 1(2014), 107-120. DOI: http://dx.doi.org/10.1177/0963662513497324
68. Dana Rotman, Jenny Preece, Jen Hammock, Kezee Procita, Derek Hansen, Cynthia Parr, Darcy Lewis, and David Jacobs. 2012. Dynamic Changes in Motivation in Collaborative Citizen-science Projects. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW'12). ACM, New York, NY, USA, 217-226. DOI: http://dx.doi.org/10.1145/2145204.2145238
69. Sharman Apt Russell. 2014. Diary of a Citizen Scientist. Oregon State University Press, Corvallis, OR.
70. SciFabric. 2016a. CrowdCrafting.org Website. (2016). Retrieved January 15, 2016 from http://crowdcrafting.org.
71. SciFabric. 2016b. Pybossa. (2016). Retrieved January 15, 2016 from http://www.pybossa.com.
72. SciStarter. 2016. Project Finder. (2016). Retrieved January 15, 2016 from http://scistarter.com/finder.
73. S. Shapin. 1995. Here and Everywhere: Sociology of Scientific Knowledge. Annual Review of Sociology 21, 1(1995), 289-321. DOI: http://dx.doi.org/10.1146/annurev. soc.21.1.289
74. Jennifer Shirk. 2014. Push the Edge of Science Forward: Expanding Considerations of Expertise Through Scientists' Citizen Science Work in Conservation. Ph. D. Dissertation. Cornell University, Ithaca, NY.
75. Jennifer L. Shirk, Heidi L. Ballard, Candie C. Wilderman, Tina Phillips, Andrea Wiggins, Rebecca Jordan, Ellen McCallie, Matthew Minarchek, Bruce V. Lewenstein, Marianne E. Krasny, and Rick Bonney. 2012. Public participation in scientific research: A framework for deliberate design. Ecology and Society 17, 2 (2012). DOI: http://dx.doi.org/org/10.5751/ES-04705-170229
76. Jonathan Silvertown. 2009. A new dawn for citizen science. Trends in ecology & evolution 24, 9(2009), 467-71. DOI: http://dx.doi.org/10.1016/j.tree.2009.03.017
77. Robert Simpson, Kevin R. Page, and David De Roure. 2014. Zooniverse: Observing the World's Largest Citizen Science Platform. In Proceedings of the 23rd International Conference on World Wide Web (WWW'14 Companion). ACM, New York, NY, USA, 1049-1054. DOI: http://dx.doi.org/10.1145/2567948.2579215
78. Ryo Suzuki, Niloufar Salehi, Michelle S. Lam, Juan C. Marroquin, and Michael S. Bernstein. 2016. Atelier: Repurposing Expert Crowdsourcing Tasks As Micro-internships. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI'16). ACM, New York, NY, USA, 2645-2656. DOI: http://dx.doi.org/10.1145/2858036.2858121
79. Carol Tenopir, Suzie Allard, Kimberly Douglass, Arsev Umur Aydinoglu, Lei Wu, Eleanor Read, Maribeth Manoff, and Mike Frame. 2011. Data sharing by scientists: Practices and perceptions. PLoS ONE 6, 6 (2011). DOI: http://dx.doi.org/10.1371/journal.pone.0021101
80. Ramine Tinati, Max Van Kleek, Elena Simperl, Markus Luczak-Rösch, Robert Simpson, and Nigel Shadbolt. 2015. Designing for Citizen Data Analysis: A Cross-Sectional Case Study of a Multi-Domain Citizen Science Platform. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI'15). ACM, New York, NY, USA, 4069-4078. DOI: http://dx.doi.org/10.1145/2702123.2702420
81. Sharon Traweek. 1988. Beamtimes and Lifetimes: the World of High Energy Physicists. Harvard University Press, Cambridge, MA.
82. Michael L Tushman and David A Nadler. 1978. Information Processing as an Integrating Concept in Organizational Design. Source: The Academy of Management Review 3, 3(1978), 613-624. DOI: http://dx.doi.org/10.5465/AMR.1978.4305791
83. Amos Tversky and Daniel Kahneman. 1974. Judgment under uncertainty: heuristics and biases. Science 185, 4157(1974), 1124-1131. DOI: http://dx.doi.org/10.1126/science.185.4157.1124
84. University College of London. 2016. Transcribe Bentham: A Participatory Initiative. (2016). Retrieved January 15, 2016 from http://blogs.ucl.ac.uk/transcribe-bentham/.
85. Andrew J Westphal, Anna L Butterworth, Christopher J Snead, Nahide Craig, David Anderson, Steven M Jones, Donald E Brownlee, Richard Farnsworth, and Michael E Zolensky. 2005. Stardust@home: A Massively Distributed Public Search for Interstellar Dust in the Stardust Interstellar Dust Collector. In Lunar and Planetary Science XXXVI. http://ntrs.nasa.gov/search.jsp?R=20050180792
86. Andrea Wiggins. 2012. Crowdsourcing Scientific Work: A Comparative Study of Technologies, Processes, and Outcomes in Citizen Science. Ph. D. Dissertation. Syracuse University, Syracuse, NY.
87. Andrea Wiggins and Kevin Crowston. 2011. From Conservation to Crowdsourcing: A Typology of Citizen Science. In Proceedings of the 44th Hawaii International Conference on System Sciences (HICSS'11). IEEE Computer Society, Washington, DC, USA, 1-10. DOI: http://dx.doi.org/10.1109/HICSS.2011.207
88. Andrea Wiggins and Kevin Crowston. 2012. Goals and Tasks: Two Typologies of Citizen Science Projects. In Proceedings of the 44th Hawaii International Conference on System Sciences (HICSS'12). IEEE Computer Society, Washington, DC, USA, 3426-3435.
89. A. Wiggins, G. Newman, R. D. Stevenson, and K. Crowston. 2011. Mechanisms for Data Quality and Validation in Citizen Science. In IEEE seventh International Conference on e-Science Workshops (eScienceW'11). 14-19.
90. Alex C Williams, John F Wallin, Haoyu Yu, Marco Perale, Hyrum D Carroll, Anne-Francoise Lamblin, Lucy Fortson, Dirk Obbink, Chris J Lintott, and James H Brusuelas. 2014. A computational pipeline for crowdsourced transcriptions of Ancient Greek papyrus fragments. In Proceedings of 2014 IEEE International Conference on Big Data. 100-105. DOI: http://dx.doi.org/10.1109/BigData.2014.7004460
91. Chris Wood, Brian Sullivan, Marshall Iliff, Daniel Fink, and Steve Kelling. 2011. eBird: Engaging birders in science and conservation. PLoS Biology 9, 12 (2011). DOI: http://dx.doi.org/10.1371/journal.pbio.1001220
92. A Zimmerman. 2008. New knowledge from old data - The role of standards in the sharing and reuse of ecological data. Science Technology & Human Values 33, 5(2008), 631-652. DOI: http://dx.doi.org/10.1177/0162243907306704
93. Ann Zimmerman, Bos Nathan, Judy Olson, and Gary Olson. 2009. e-Research: Transformation in Scholarly Practice. Routledge, Chapter The Promise of Data in e-Research: Many Challenges, Multiple Solutions, Diverse Outcomes, 222-239.
94. Zooniverse. 2016a. AnnoTate. (2016). Retrieved January 15, 2016 from https://anno. tate.org.uk/.
95. Zooniverse. 2016b. Old Weather Project. (2016). Retrieved January 15, 2016 from http://www.oldweather.org/.
96. Zooniverse. 2016c. Operation War Diary. (2016). Retrieved January 15, 2016 from http://www.operationwardiary.org/.