Citizen Science: An Information Quality Research Frontier

Information Systems Frontiers

Volumn 22, Issue 4, 2020, Pages 961-983

  Lukyanenko, Roman ^a   Wiggins, Andrea ^b   Rosser, Holly K ^b  

^a HEC MONTRÉAL   (Canada)

^b UNIVERSITY OF NEBRASKA AT OMAHA   (United States)

Author keywords

Citizen science; Crowdsourcing; Data quality; Humanities; Information quality; Information systems; Science

Indexed keywords

CROWDSOURCING; DATA SHARING; INFORMATION SYSTEMS; INFORMATION USE; QUALITY MANAGEMENT;

CITIZEN SCIENCE; DATA QUALITY; HUMANITIES; INFORMATION QUALITY; SCIENCE;

INFORMATION ANALYSIS;

EID: 85073447691     PISSN: 13873326     EISSN: 15729419     Source Type: Journal    
DOI: 10.1007/s10796-019-09915-z     Document Type: Article

References (242)

1. Abbasi, A., Sarker, S., & Chiang, R. H. (2016). Big data research in information systems: Toward an inclusive research agenda. Journal of the Association for Information Systems, 17(2), 3.
2. Abbasi, A., Zhou, Y., Deng, S., & Zhang, P. (2018). Text analytics to support sense-making in social media: A language-action perspective. MIS Quarterly, 42(2), 1–38.
3. Allahbakhsh, M., Benatallah, B., Ignjatovic, A., Motahari-Nezhad, H. R., Bertino, E., & Dustdar, S. (2013). Quality control in crowdsourcing systems: Issues and directions. IEEE Internet Computing, 17(2), 76–81.
4. Alonso, O. (2015). Challenges with label quality for supervised learning. Journal of Data and Information Quality, 6(1), 2.
5. Amir B, Ralph P (2014) Proposing a theory of gamification effectiveness. Hyderabad: ACM.
6. Amsterdamer Y, Davidson SB, Kukliansky A, Milo T, Novgorodov S, Somech A (2015) Managing general and individual knowledge in crowd mining applications. Seventh Biennial Conference on Innovative Data Systems Research. Asilomar: CIDR.
7. Andriole, S. J. (2010). Business impact of web 2.0 technologies. Communications of the ACM, 53(12), 67–79.
8. Arazy, O., & Kopak, R. (2011). On the measurability of information quality. Journal of the American Society for Information Science and Technology, 62(1), 89–99.
9. Arazy, O., Nov, O., Patterson, R., & Yeo, L. (2011). Information quality in Wikipedia: The effects of group composition and task conflict. Journal of Management Information Systems, 27(4), 71–98.
10. Arazy, O., Daxenberger, J., Lifshitz-Assaf, H., Nov, O., & Gurevych, I. (2016). Turbulent stability of emergent roles: The dualistic nature of self-organizing knowledge coproduction. Information Systems Research, 27(4), 792–812.
11. Arazy, O., Kopak, R., & Hadar, I. (2017). Heuristic principles and differential judgments in the assessment of information quality. Journal of the Association for Information Systems, 18(5), 403.
12. Awal GK, Bharadwaj K (2017) Leveraging collective intelligence for behavioral prediction in signed social networks through evolutionary approach. Information Systems Frontiers 1–23. 10.1007/s10796-017-9760-4
13. Azzurro, E., Broglio, E., Maynou, F., & Bariche, M. (2013). Citizen science detects the undetected: The case of Abudefduf saxatilis from the Mediterranean Sea. Management of Biological Invasions, 4(2), 167–170.
14. Ballou, D. P., & Pazer, H. L. (1995). Designing information systems to optimize the accuracy-timeliness tradeoff. Information Systems Research, 6(1), 51.
15. Ballou, D. P., & Pazer, H. L. (2003). Modeling completeness versus consistency tradeoffs in information decision contexts. IEEE Transactions on Knowledge and Data Engineering, 15(1), 240–243.
16. Ballou D, Madnick S, Wang R (Winter 2003) Special section: Assuring information quality. Journal of Management Information Systems. 20(3):9–11.
17. Barwise, P., & Meehan, S. (2010). The one thing you must get right when building a brand. Harvard Business Review, 88(12), 80–84.
18. Batini C, Scannapieca M (2006) Data quality: concepts, methodologies and techniques. Berlin Heidelberg: Springer-Verlag
19. Batini, C., Rula, A., Scannapieco, M., & Viscusi, G. (2015). From data quality to big data quality. Journal of Database Management, 26(1), 60–82.
20. Bauer, M. W., Petkova, K., & Boyadjieva, P. (2000). Public knowledge of and attitudes to science: Alternative measures that may end the “science war”. Science, Technology & Human Values, 25(1), 30–51.
21. Beath, C., Berente, N., Gallivan, M. J., & Lyytinen, K. (2013). Expanding the frontiers of information systems research: Introduction to the special issue. Journal of the Association for Information Systems, 14(4), 5.
22. Becker-Klein R, Peterman K, Stylinski C (2016) Embedded assessment as an essential method for understanding public engagement in citizen science. Citizen Science: Theory and Practice, 1(1), 1–10.
23. Belbin L (2011) The atlas of living Australia’s spatial portal. 28–29.
24. Belden, O. S., Baker, S. C., & Baker, B. M. (2015). Citizens unite for computational immunology! Trends in Immunology, 36(7), 385–387.
25. Watson-Manheim, M. B., Chudoba, K. M., & Crowston, K. (2002). Discontinuities and continuities: A new way to understand virtual work. Information Technology and People, 15(3), 191–209.
26. Bifet, A., & Frank, E. (2010). Sentiment knowledge discovery in twitter streaming data discovery science. In B. Pfahringer, G. Holmes, & A. Hoffmann (Eds.), Discovery science (pp. 1–15). Heidelberg: Springer Berlin.
27. Bird, T. J., Bates, A. E., Lefcheck, J. S., Hill, N. A., Thomson, R. J., Edgar, G. J., Stuart-Smith, R. D., Wotherspoon, S., Krkosek, M., & Stuart-Smith, J. F. (2014). Statistical solutions for error and bias in global citizen science datasets. Biological Conservation, 173, 144–154.
28. Blake R, Shankaranarayanan G (2015) Data and information quality: Research themes and evolving patterns. AMCIS 2015, 1–10.
29. Bodker, S. (1996). Creating conditions for participation: Conflicts and resources in systems development. Human Computer Interaction, 11(3), 215–236.
30. Bonabeau, E. (2009). Decisions 2.0: The power of collective intelligence. MIT Sloan Management Review, 50(2), 45.
31. Bonney, R., Cooper, C. B., Dickinson, J., Kelling, S., Phillips, T., Rosenberg, K. V., & Shirk, J. (2009). Citizen science: A developing tool for expanding science knowledge and scientific literacy. BioScience, 59(11), 977–984.
32. Bonney, R., Shirk, J. L., Phillips, T. B., Wiggins, A., Ballard, H. L., Miller-Rushing, A. J., & Parrish, J. K. (2014a). Next steps for citizen science. Science, 343(6178), 1436–1437.
33. Bonney, R., Shirk Jennifer, L., Phillips, T. B., Wiggins, A., Ballard, H. L., Miller-Rushing, A. J., & Parrish, J. K. (2014b). Next steps for citizen science. Science, 343(6178), 1436–1437.
34. Bonney R, Cooper C, Ballard H (2016) The theory and practice of Citizen Science: Launching a new journal. Citizen Science: Theory and Practice, 1(1), 1–13.
35. Bowser A, Hansen D, Preece J, He Y, Boston C, Hammock J (2014) Gamifying citizen science: a study of two user groups. Proceedings of the companion publication of the 17th ACM conference on Computer supportedcooperative work & social computing (pp. 137–140). Baltimore: ACM.
36. Bowser A, Shilton K, Warrick E, Preece J (2017) Accounting for privacy in citizen science: Ethical research in a context of openness. Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (pp. 2124–2136). Portland: ACM.
37. Boyajian, T., LaCourse, D., Rappaport, S., Fabrycky, D., Fischer, D., Gandolfi, D., Kennedy, G., Korhonen, H., Liu, M., & Moor, A. (2016). Planet hunters IX. KIC 8462852–where’s the flux? Monthly Notices of the Royal Astronomical Society, 457(4), 3988–4004.
38. Brabham, D. C. (2013). Crowdsourcing. Cambridge: MIT Press.
39. Bratteteig, T., & Wagner, I. (2014). Disentangling participation: Power and decision-making in participatory design. New York: Springer International Publishing.
40. Brossard, D., Lewenstein, B., & Bonney, R. (2005). Scientific knowledge and attitude change: The impact of a citizen science project. International Journal of Science Education, 27(9), 1099–1121.
41. Brynjolfsson, E., & McAfee, A. (2014). The second machine age: Work, progress, and prosperity in a time of brilliant technologies. New York: WW Norton & Company.
42. Burgess H, DeBey L, Froehlich H, Schmidt N, Theobald E, Ettinger A, HilleRisLambers J, Tewksbury J, Parrish J (2017) The science of citizen science: Exploring barriers to use as a primary research tool. Biological Conservation, 208, 113–120
43. Burton-Jones, A., & Volkoff, O. (2017). How can we develop contextualized theories of effective use? A demonstration in the context of community-care electronic health records. Information Systems Research, 28(3), 468–489.
44. Calero, C., Caro, A., & Piattini, M. (2008). An applicable data quality model for web portal data consumers. World Wide Web, 11(4), 465–484.
45. Chau, M., & Xu, J. (2012). Business intelligence in blogs: Understanding consumer interactions and communities. MIS Quarterly, 36(4), 1189–1216.
46. Chittilappilly, A. I., Chen, L., & Amer-Yahia, S. (2016). A survey of general-purpose crowdsourcing techniques. IEEE Transactions on Knowledge and Data Engineering, 28(9), 2246–2266.
47. Choy, K., & Schlagwein, D. (2016). Crowdsourcing for a better world: On the relation between IT affordances and donor motivations in charitable crowdfunding. Information Technology and People, 29(1), 221–247.
48. Clark, F., & Illman, D. L. (2001). Dimensions of civic science introductory essay. Science Communication, 23(1), 5–27.
49. Clark, C., Wu, J., Pletsch, H., Guillemot, L., Allen, B., Aulbert, C., Beer, C., Bock, O., Cuéllar, A., & Eggenstein, H. (2017). The Einstein@ home gamma-ray pulsar survey. I. Search methods, sensitivity, and discovery of new Young gamma-ray pulsars. The Astrophysical Journal, 834(2), 106.
50. Cohn, J. P. (2008). Citizen science: Can volunteers do real research? AIBS Bulletin, 58(3), 192–197.
51. Crall, A. W., Newman, G. J., Stohlgren, T. J., Holfelder, K. A., Graham, J., & Waller, D. M. (2011). Assessing citizen science data quality: An invasive species case study. Conservation Letters, 4(6), 433–442.
52. Crook T, Frasca B, Kohavi R, Longbotham R (2009) Seven pitfalls to avoid when running controlled experiments on the web. Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and datamining (pp. 1105–1114). Paris: ACM
53. Daniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., & Allahbakhsh, M. (2018). Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions. ACM Computing Surveys CSUR, 51(1), 7.
54. Daugherty, T., Eastin, M., & Bright, L. (2008). Exploring consumer motivations for creating user-generated content. Journal of Interactive Advertising, 8(2), 16–25.
55. Davenport, T., & Harris, J. (2017). Competing on analytics: Updated, with a new introduction: The new science of winning. Cambridge: Harvard Business Press.
56. Davis, A. K., & Howard, E. (2005). Spring recolonization rate of monarch butterflies in eastern North America: New estimates from citizen-science data. Journal of the Lepidopterists' Society, 59(1), 1–5.
57. Delaney, D. G., Sperling, C. D., Adams, C. S., & Leung, B. (2008). Marine invasive species: Validation of citizen science and implications for national monitoring networks. Biological Invasions, 10(1), 117–128.
58. DeLone, W. H., & McLean, E. R. (1992). Information systems success: The quest for the dependent variable. Information Systems Research, 3(1), 60–95.
59. Delort, J. Y., Arunasalam, B., & Paris, C. (2011). Automatic moderation of online discussion sites. International Journal of Electronic Commerce, 15(3), 9–30.
60. Deng, X., Joshi, K., & Galliers, R. D. (2016). The duality of empowerment and marginalization in microtask crowdsourcing: Giving voice to the less powerful through value sensitive design. MIS Quarterly, 40(2), 279–302.
61. Doan, A., Ramakrishnan, R., & Halevy, A. Y. (2011). Crowdsourcing systems on the world-wide web. Communications of the ACM, 54(4), 86–96.
62. Dwivedi, Y. K., Wastell, D., Laumer, S., Henriksen, H. Z., Myers, M. D., Bunker, D., Elbanna, A., Ravishankar, M., & Srivastava, S. C. (2015). Research on information systems failures and successes: Status update and future directions. Information Systems Frontiers, 17(1), 143–157.
63. Eitzel, M., Cappadonna, J. L., Santos-Lang, C., Duerr, R. E., Virapongse, A., West, S. E., Kyba, C. C. M., Bowser, A., Cooper, C. B., & Sforzi, A. (2017). Citizen science terminology matters: Exploring key terms. Citizen Science: Theory and Practice, 2(1).
64. Elevant K (2013) Why Share Weather? Motivational Model for" Share Weather" Online Communities and Three Empirical Studies. 46th Hawaii International Conference on System Sciences. 10.1109/HICSS.2013.26.
65. Endicott J, Larsen KR, Lukyanenko R, Bong CH (2017) Integrating Scientific Research: Theory and Design of Discovering Similar Constructs. AIS SIGSAND Symposium (Cincinnati, Ohio), 1–7.
66. ESCA (2015) 10 Principles of Citizen Science. European Citizen Science Association. ECSA. Retrieved (April 17, 2018), https://ecsa.citizen-science.net/engage-us/10-principles-citizen-science.
67. Eveleigh A, Jennett C, Lynn S, Cox AL (2013) I want to be a captain! i want to be a captain!: gamification in the old weather citizen science project. Proceedings of the First International Conference on Gameful Design, Research, and Applications (pp. 79–82). Toronto: ACM
68. Eveleigh A, Jennett C, Blandford A, Brohan P, Cox AL (2014) Designing for dabblers and deterring drop-outs in citizen science. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2985–2994). Toronto: ACM.
69. Fielden, M. A., Chaulk, A. C., Bassett, K., Wiersma, Y. F., Erbland, M., Whitney, H., & Chapman, T. W. (2015). Aedes japonicus japonicus (Diptera: Culicidae) arrives at the most easterly point in North America. Canadian Entomologist, 147(06), 737–740.
70. Flanagin, A., & Metzger, M. (2008). The credibility of volunteered geographic information. GeoJournal, 72(3), 137–148.
71. Fortson L, Masters K, Nichol R, Borne K, Edmondson E, Lintott C, Raddick J, Schawinski K, Wallin J (2011) Galaxy Zoo: Morphological Classification and Citizen Science. Advance Machine Learning Data Mineral Astronomy 1–11.
72. Franklin MJ, Kossmann D, Kraska T, Ramesh S, Xin R (2011) CrowdDB: answering queries with crowdsourcing. Proceedings 2011 ACM SIGMOD International Conference Management Data. SIGMOD ‘11. (ACM, Athens, Greece), 61–72.
73. French, A. M., Guo, C., & Shim, J. P. (2014). Current status, issues, and future of bring your own device (BYOD). CAIS, 35, 10.
74. Galloway, A. W. E., Tudor, M. T., & Haegen, W. M. V. (2006). The reliability of citizen science: A Case study of Oregon white oak stand surveys. Wildlife Society Bulletin, 34(5), 1425–1429.
75. Garcia-Molina, H., Joglekar, M., Marcus, A., Parameswaran, A., & Verroios, V. (2016). Challenges in data crowdsourcing. IEEE Transactions on Knowledge and Data Engineering, 28(4), 901–911.
76. Garrity, E. J. (2001). Synthesizing user centered and designer centered IS development approaches using general systems theory. Information Systems Frontiers, 3(1), 107–121.
77. Gemino, A., & Wand, Y. (2004). A framework for empirical evaluation of conceptual modeling techniques. Requirements Engineering, 9(4), 248–260.
78. Ghezzi, A., Gabelloni, D., Martini, A., & Natalicchio, A. (2017). Crowdsourcing: A review and suggestions for future research. International Journal of Management Reviews, 00, 1–21.
79. Girres, J. F., & Touya, G. (2010). Quality assessment of the French OpenStreetMap dataset. Transactions in GIS, 14(4), 435–459.
80. Goes, P. B. (2014). Editor’s comments: Design science research in top information systems journals. MIS Quarterly, 38(1), iii–viii.
81. Goodchild, M. (2007). Citizens as sensors: The world of volunteered geography. GeoJournal, 69(4), 211–221.
82. Goodchild, M. F., & Glennon, J. A. (2010). Crowdsourcing geographic information for disaster response: A research frontier. International Journal of Digital Earth, 3(3), 231–241.
83. Gray ML, Suri S, Ali SS, Kulkarni D (2016) The crowd is a collaborative network. Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (pp. 134–147). San Francisco: ACM
84. Gregor, S., & Hevner, A. R. (2013). Positioning and presenting design science research for maximum impact. MIS Quarterly, 37(2), 337–355.
85. Gura, T. (2013). Citizen science: amateur experts. Nature, 496(7444), 259–261.
86. Hailey, D., Roine, R., & Ohinmaa, A. (2002). Systematic review of evidence for the benefits of telemedicine. Journal of Telemedicine and Telecare, 8(1), 1–7.
87. Haklay M (2013) Citizen science and volunteered geographic information: Overview and typology of participation. Crowdsourcing Geographic Knowledge. Dordrecht: Springer.
88. Haklay, M., & Weber, P. (2008). OpenStreetMap: User-generated street maps. IEEE Pervasive Computing, 7(4), 12–18.
89. Hand, E. (2010). People power. Nature, 466(7307), 685–687.
90. He, Y., & Wiggins, A. (2017). Implementing an environmental citizen science project: Strategies and concerns from educators’ perspectives. International Journal of Environment Science Education, 12(6), 1459–1481.
91. Heath, T., & Bizer, C. (2011). Linked data: Evolving the web into a global data space. San Rafael: Morgan & Claypool Publishers.
92. Heikkila, T., & Isett, K. R. (2007). Citizen involvement and performance management in special-purpose governments. Public Administration Review, 67(2), 238–248.
93. Hevner, A., March, S., Park, J., & Ram, S. (2004). Design science in information systems research. MIS Quarterly, 28(1), 75–105.
94. Higgins, C. I., Williams, J., Leibovici, D. G., Simonis, I., Davis, M. J., Muldoon, C., van Genuchten, P., O’Hare, G., & Wiemann, S. (2016). Citizen OBservatory WEB (COBWEB): A generic infrastructure platform to facilitate the collection of citizen science data for environmental monitoring. International Journal Spatial Data Infrastructures Research, 11, 20–48.
95. Hochachka, W. M., Fink, D., Hutchinson, R. A., Sheldon, D., Wong, W. K., & Kelling, S. (2012). Data-intensive science applied to broad-scale citizen science. Trends in Ecology & Evolution, 27(2), 130–137.
96. Hopkins, N., Tate, M., Sylvester, A., & Johnstone, D. (2017). Motivations for 21st century school children to bring their own device to school. Information Systems Frontiers, 19(5), 1191–1203.
97. Howe, J. (2008). Crowdsourcing: How the power of the crowd is driving the future of business. New York: Random House.
98. Ipeirotis PG, Gabrilovich E (2014) Quizz: Targeted crowdsourcing with a billion (potential) users. Proceedings of the 23rd international conference on World wide web (pp. 143–154). Seoul: ACM.
99. Ipeirotis PG, Provost F, Wang J (2010) Quality management on amazon mechanical Turk. Proceedings of the ACM SIGKDD Workshop Human Computer (ACM), 64–67.
100. Irwin A (1995) Citizen science: A study of people, expertise and sustainable development (Psychology Press).
101. Irwin A, Michael M (2003) Science, social theory & public knowledge (McGraw-Hill Education (UK), London, UK).
102. Jabbari Sabegh MA, Lukyanenko R, Recker J, Samuel BM, Castellanos A (2017) Conceptual modeling research in information systems: What we now know and what we still do not know. AIS SIGSAND. (Cincinnati, Ohio), 1–12.
103. Jabbari MA, Lukyanenko R, Recker J, Samuel BM, Castellanos A (2018) Conceptual Modeling Research: Revisiting and Updating Wand and Weber’s 2002 Research Agenda. AIS SIGSAND. (Syracuse, NY), 1–12.
104. Jackson, M., Weyl, O., Altermatt, F., Durance, I., Friberg, N., Dumbrell, A. J., Piggott, J., Tiegs, S., Tockner, K., & Krug, C. (2016). Chapter twelve-recommendations for the next generation of global freshwater biological monitoring tools. Advances in Ecological Research, 55, 615–636.
105. Janssens, A. C. J., & Kraft, P. (2012). Research conducted using data obtained through online communities: Ethical implications of methodological limitations. PLoS Medicine, 9(10), e1001328.
106. Jordan, J. M. (2017). Challenges to large-scale digital organization: The case of Uber. Journal of Organization Design, 6(1), 11.
107. Kane, G. C., & Ransbotham, S. (2016). Research note—Content and collaboration: An affiliation network approach to information quality in online peer production communities. Information Systems Research, 27(2), 424–439.
108. Kane, G. C., Alavi, M., Labianca, G. J., & Borgatti, S. (2014). What’s different about social media networks? A framework and research agenda. MIS Quarterly, 38(1), 274–304.
109. Kapoor, K. K., Tamilmani, K., Rana, N. P., Patil, P., Dwivedi, Y. K., & Nerur, S. (2018). Advances in social media research: Past, present and future. Information Systems Frontiers, 20(3), 531–558.
110. Khatib, F., DiMaio, F., Cooper, S., Kazmierczyk, M., Gilski, M., Krzywda, S., Zabranska, H., Pichova, I., Thompson, J., & Popović, Z. (2011). Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nature Structural & Molecular Biology, 18(10), 1175–1177.
111. Khoury, G. A., Liwo, A., Khatib, F., Zhou, H., Chopra, G., Bacardit, J., Bortot, L. O., Faccioli, R. A., Deng, X., & He, Y. (2014). WeFold: A coopetition for protein structure prediction. Proteins Struct. Funct Bioinforma, 82(9), 1850–1868.
112. Kim S, Mankoff J, Paulos E (2013) Sensors: evaluating a flexible framework for authoring mobile data-collection tools for citizen science. (ACM), 1453–1462.
113. Klein, H. K., & Hirschheim, R. (2001). Choosing between competing design ideals in information systems development. Information Systems Frontiers, 3(1), 75–90.
114. Kohavi R, Deng A, Frasca B, Walker T, Xu Y, Pohlmann N (2013) Online controlled experiments at large scale. (ACM, ACM SIGKDD international conference on Knowledge discovery and data mining), 1168–1176.
115. Korpela, E. J. (2012). SETI@ home, BOINC, and volunteer distributed computing. Annual Review of Earth and Planetary Sciences, 40, 69–87.
116. Kosmala, M., Wiggins, A., Swanson, A., & Simmons, B. (2016). Assessing data quality in citizen science. Frontiers in Ecology and the Environment, 14(10), 551–560.
117. Kullenberg, C., & Kasperowski, D. (2016). What is citizen science?–a scientometric meta-analysis. PLoS One, 11(1), e0147152.
118. Kyng, M. (1995). Making representations work. Communications of the ACM, 38(9), 46–55.
119. Larsen, K., & Bong, C. H. (2016). A tool for addressing construct identity in literature reviews and meta-analyses. MIS Quarterly, 40(3), 1–23.
120. Law E, Gajos KZ, Wiggins A, Gray ML, Williams AC (2017) Crowdsourcing as a Tool for Research: Implications of Uncertainty. Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (pp. 1544–1561). Portland: ACM.
121. Lawrence, J. M. (2015). Rediscovery of the threatened Stoffberg widow butterfly, Dingana fraterna: The value of citizen scientists for African conservation. Journal of Insect Conservation, 19(4), 801–803.
122. Lee, Y. W. (2003). Crafting rules: Context-reflective data quality problem solving. Journal of Management Information Systems, 20(3), 93–119.
123. Lee, Y. W., Strong, D. M., Kahn, B. K., & Wang, R. Y. (2002). AIMQ: A methodology for information quality assessment. Information Management, 40(2), 133–146.
124. Levina, N., & Arriaga, M. (2014). Distinction and status production on user-generated content platforms: Using Bourdieu’s theory of cultural production to understand social dynamics in online fields. Information Systems Research, 25(3), 468–488.
125. Levitin, A., & Redman, T. (1995). Quality dimensions of a conceptual view. Information Processing and Management, 31(1), 81–88.
126. Levy, M., & Germonprez, M. (2017). The potential for citizen science in information systems research. Communications of the Association for Information Systems, 40(1), 2.
127. Lewandowski, E., & Specht, H. (2015). Influence of volunteer and project characteristics on data quality of biological surveys. Conservation Biology, 29(3), 713–723.
128. Li G, Wang J, Zheng Y, Franklin M (2016) Crowdsourced data management: A survey. IEEE Transactions on Knowledge and Data Engineering, 28(9), 2296–2319.
129. Light A, Miskelly C (2014) Design for sharing. North Umbria University Sustainable Social Network.
130. Liu J, Ram S (2011) Who does what: Collaboration patterns in the wikipedia and their impact on data quality. 19th Workshop Information Technology System 175–180.
131. Liu, K., Eatough, R., Wex, N., & Kramer, M. (2014). Pulsar–black hole binaries: Prospects for new gravity tests with future radio telescopes. Monthly Notices of the Royal Astronomical Society, 445(3), 3115–3132.
132. Losey, J., Perlman, J., & Hoebeke, E. (2007). Citizen scientist rediscovers rare nine-spotted lady beetle, Coccinella novemnotata, in eastern North America. Journal of Insect Conservation, 11(4), 415–417.
133. Louv, R., Dickinson, J. L., & Bonney, R. (2012). Citizen science: Public participation in environmental research. Ithaca: Cornell University Press.
134. Love J, Hirschheim R (2017) Crowdsourcing of information systems research. European Journal of Information Systems 1–18.
135. Lukyanenko R (2014) An information modeling approach to improve quality of user-generated content. PhD thesis. (Memorial University of Newfoundland, St. John’s, NL Canada).
136. Lukyanenko R, Parsons J (2014) Using Field Experimentation to Understand Information Quality in User-generated Content. CodeCon@MIT.
137. Lukyanenko R, Parsons J (2015a) Beyond Task-Based Crowdsourcing Database Research. AAAI Conf. Hum. Comput. Crowdsourcing AAAI HCOMP. (San Diego, CA, USA), 1–2.
138. Lukyanenko, R., & Parsons, J. (2015b). Information quality research challenge: Adapting information quality principles to user-generated content. ACM Journal of Data and Information Quality, 6(1), 1–3.
139. Lukyanenko, R., & Parsons, J. (2018). Beyond micro-tasks: Research opportunities in observational crowdsourcing. J Database Manag JDM, 29(1), 1–22.
140. Lukyanenko R, Parsons J, Wiersma Y (2014a) The Impact of Conceptual Modeling on Dataset Completeness: A Field Experiment. Proceedings of the International Conference Information System. ICIS. 1–18.
141. Lukyanenko, R., Parsons, J., & Wiersma, Y. (2014b). The IQ of the crowd: Understanding and improving information quality in structured user-generated content. Information Systems Research, 25(4), 669–689.
142. Lukyanenko, R., Parsons, J., & Wiersma, Y. (2016a). Emerging problems of data quality in citizen science. Conservation Biology, 30(3), 447–449.
143. Lukyanenko, R., Parsons, J., Wiersma, Y., Sieber, R., & Maddah, M. (2016b). Participatory Design for User-generated Content: Understanding the challenges and moving forward. Scandinavian Journal of Information Systems, 28(1), 37–70.
144. Lukyanenko R, Parsons J, Wiersma Y, Maddah M (2019) Expecting the Unexpected: Effects of Data Collection Design Choices on the Quality of Crowdsourced User-generated Content. Press MIS Quarterly.
145. MacDonald, E. A., Donovan, E., Nishimura, Y., Case, N. A., Gillies, D. M., Gallardo-Lacourt, B., Archer, W. E., Spanswick, E. L., Bourassa, N., & Connors, M. (2018). New science in plain sight: Citizen scientists lead to the discovery of optical structure in the upper atmosphere. Science Advances, 4(3), eaaq0030.
146. Maddah M, Lukyanenko R, VanderMeer D (2018) Impact of Collection Structures and Type of Data on Quality of User-Generated Content. WITS 2018. (Santa Clara, CA), 1–10.
147. Madnick, S. E., Wang, R. Y., Lee, Y. W., & Zhu, H. (2009). Overview and framework for data and information quality research. Journal of Data and Information Quality, 1(1), 1–22.
148. Malone, T. W., Laubacher, R., & Dellarocas, C. (2010). Harnessing crowds: Mapping the genome of collective intelligence. Sloan Management Review, 51(3), 21–31.
149. McAfee, A., & Brynjolfsson, E. (2017). Machine, platform, crowd: Harnessing our digital future. New York: WW Norton & Company.
150. McKinley, D. C., Miller-Rushing, A. J., Ballard, H. L., Bonney, R., Brown, H., Cook-Patton, S. C., Evans, D. M., French, R. A., Parrish, J. K., & Phillips, T. B. (2016). Citizen science can improve conservation science, natural resource management, and environmental protection. Biological Conservation.
151. Melville, N. P. (2010). Information systems innovation for environmental sustainability. MIS Quarterly, 34(1), 1–21.
152. Miller, D. A., Weir, L. A., McClintock, B. T., Grant, E. H. C., Bailey, L. L., & Simons, T. R. (2012). Experimental investigation of false positive errors in auditory species occurrence surveys. Ecological Applications, 22(5), 1665–1674.
153. Miranda, S. M., Young, A., & Yetgin, E. (2016). Are social media emancipatory or hegemonic? Societal effects of mass media digitization in the Case of the SOPA discourse. MIS Quarterly, 40(3), 303–329.
154. Mylopoulos, J. (1998). Information modeling in the time of the revolution. Information Systems, 23(3–4), 127–155.
155. Nelson AG, Stylinski C, Becker-Klein R, Peterman K (2017) Exploring embedded assessment to document scientic inquiry skills within citizen science. Citizen Inquiry. (Routledge), 81–100.
156. Newman, C., Buesching, C. D., & Macdonald, D. W. (2003). Validating mammal monitoring methods and assessing the performance of volunteers in wildlife conservation—“Sed quis custodiet ipsos custodies?”. Biological Conservation, 113(2), 189–197.
157. Newman, G., Zimmerman, D., Crall, A., Laituri, M., Graham, J., & Stapel, L. (2010). User-friendly web mapping: Lessons from a citizen science website. International Journal of Geographical Information Science, 24(12), 1851–1869.
158. Newman, G., Wiggins, A., Crall, A., Graham, E., Newman, S., & Crowston, K. (2012). The future of citizen science: Emerging technologies and shifting paradigms. Frontiers in Ecology and the Environment, 10(6), 298–304.
159. Nielsen, M. (2011). Reinventing discovery: The new era of networked science. Hoboken: Princeton University Press.
160. Nov O, Arazy O, Anderson D (2011) Dusting for science: motivation and participation of digital citizen science volunteers. 2011 IConference:68–74.
161. Nov, O., Arazy, O., & Anderson, D. (2014). Scientists@ home: What drives the quantity and quality of online citizen science participation. PLoS One, 9(4), 1–11.
162. Ogunseye S, Parsons J (2016) Can Expertise Impair the Quality of Crowdsourced Data? SIGOPEN Development Workshop ICIS 2016.
163. Ogunseye S, Parsons J, Lukyanenko R (2017) Do Crowds Go Stale? Exploring the Effects of Crowd Reuse on Data Diversity. WITS 2017. (Seoul, South Korea).
164. Osborn DA, Pearse JS, Roe CA, Magoon O, Converse H, Baird B, Jines B, Miller-Henson M (2005) Monitoring rocky intertidal shorelines: a role for the public in resource management. Conference Proceedings, California and the World Ocean 2002: Revisiting and Revising California’s Ocean Agenda. Reston: American Society of Civil Engineers.
165. Palacios, M., Martinez-Corral, A., Nisar, A., & Grijalvo, M. (2016). Crowdsourcing and organizational forms: Emerging trends and research implications. Journal of Business Research, 69(5), 1834–1839.
166. Parfitt I (2013) Citizen science in conservation biology: best practices in the geoweb era. (Univ. of British Columbia).
167. Parsons, J., Lukyanenko, R., & Wiersma, Y. (2011). Easier citizen science is better. Nature, 471(7336), 37–37.
168. Pattengill-Semmens CV, Semmens BX (2003) Conservation and management applications of the reef volunteer fish monitoring program. Coastal Monitoring Partnership (Springer), 43–50.
169. Pavlou, P. A., & Dimoka, A. (2006). The nature and role of feedback text comments in online marketplaces: Implications for trust building, price premiums, and seller differentiation. Information Systems Research, 17(4), 392–414.
170. Petter, S., DeLone, W., & McLean, E. R. (2013). Information systems success: The quest for the independent variables. Journal of Management Information Systems, 29(4), 7–62.
171. Pipino, L. L., Lee, Y. W., & Wang, R. Y. (2002). Data quality assessment. Communications of the ACM, 45(4), 211–218.
172. Poblet M, García-Cuesta E, Casanovas P (2017) Crowdsourcing roles, methods and tools for data-intensive disaster management. Information System Frontier 1–17.
173. Preece J, Boston C, Maher ML, Grace K, Yeh T (2016) From Crowdsourcing Technology Design to Participatory Design and Back Again! 315.
174. Prestopnik N, Crowston K (2011) Gaming for (Citizen) Science: Exploring Motivation and Data Quality in the Context of Crowdsourced Science through the Design and Evaluation of a Social-Computational System. IEEE International Conference E-Sci. Workshop EScienceW. 1–28.
175. Prestopnik N, Crowston K (2012a) Citizen science system assemblages: understanding the technologies that support crowdsourced science. (ACM), 168–176.
176. Prestopnik N, Crowston K (2012b) Purposeful gaming & socio-computational systems: a citizen science design case. (ACM), 75–84.
177. Prestopnik, N. R., Crowston, K., & Wang, J. (2013). Data quality in purposeful games. Computing, 1, 2.
178. Rai, A. (2017). Diversity of design science contributions. MIS Quarterly, 41(1), iii–xviii.
179. Rana, N. P., Dwivedi, Y. K., Williams, M. D., & Weerakkody, V. (2015). Investigating success of an e-government initiative: Validation of an integrated IS success model. Information Systems Frontiers, 17(1), 127–142.
180. Redman, T. C. (1996). Data quality for the information age. Norwood: Artech House.
181. Renault H (2018) How the web is helping gamers find new spider species. ABC News. Retrieved (April 9, 2018), http://www.abc.net.au/news/2018-01-05/seven-new-spider-species-discovered-by-gamers/9303710.
182. Richards C, Thompson CW, Graham N (2014) Beyond designing for motivation: the importance of context in gamification. (ACM), 217–226.
183. Riesch H, Potter C (2013) Citizen science as seen by scientists: Methodological, epistemological and ethical dimensions. Public Understanding Science 0963662513497324.
184. Robinson MR, Imran A (2015) A Design Framework for Technology-Mediated Public Participatory System for the Environment.
185. Rotman D, Preece J, Hammock J, Procita K, Hansen D, Parr C, Lewis D, Jacobs D (2012) Dynamic changes in motivation in collaborative citizen-science projects. (ACM), 217–226.
186. Rowland K (2012) Citizen science goes “extreme.” Nature News.
187. Sabou M, Bontcheva K, Scharl A (2012) Crowdsourcing research opportunities: lessons from natural language processing. (ACM), 17.
188. Scannapieco, M., Missier, P., & Batini, C. (2005). Data quality at a glance. Datenbank-Spektrum, 14, 6–14.
189. Schilthuizen M, Seip LA, Otani S, Suhaimi J, Njunjić I (2017) Three new minute leaf litter beetles discovered by citizen scientists in Maliau Basin, Malaysian Borneo (Coleoptera: Leiodidae, Chrysomelidae). Biodiversity Data Journal (5).
190. Schmeller, D. S., HENRY, P., Julliard, R., Gruber, B., Clobert, J., Dziock, F., Lengyel, S., Nowicki, P., Déri, E., & Budrys, E. (2009). Advantages of volunteer-based biodiversity monitoring in Europe. Conservation Biology, 23(2), 307–316.
191. Seidel, S., Recker, J. C., & Vom Brocke, J. (2013). Sensemaking and sustainable practicing: Functional affordances of information systems in green transformations. Management Information Systems Quarterly, 37(4), 1275–1299.
192. Shankaranarayanan, G., & Blake, R. (2017). From content to context: The evolution and growth of data quality research. Journal of Data and Information Quality, 8(2), 9.
193. Sheppard SA, Terveen L (2011) Quality is a verb: The operationalization of data quality in a citizen science community. (ACM), 29–38.
194. Sheppard S, Wiggins A, Terveen L (2014) Capturing quality: retaining provenance for curated volunteer monitoring data. 1234–1245.
195. Show, H. (2015). Rise of the citizen scientist. Nature, 524(7565), 265–265.
196. Siau, K., & Rossi, M. (2011). Evaluation techniques for systems analysis and design modelling methods – A review and comparative analysis. Information Systems Journal, 21(3), 249–268.
197. Sicari, S., Cappiello, C., De Pellegrini, F., Miorandi, D., & Coen-Porisini, A. (2016). A security-and quality-aware system architecture for internet of things. Information Systems Frontiers, 18(4), 665–677.
198. Sidorova A, Evangelopoulos N, Valacich JS, Ramakrishnan T (2008) Uncovering the intellectual core of the information systems discipline. MIS Q.:467–482.
199. Sieber, R. (2006). Public participation geographic information systems: A literature review and framework. Annals of the Association of American Geographers, 96(3), 491–507.
200. Silvertown, J. (2010). Taxonomy: Include social networking. Nature, 467(7317), 788–788.
201. Simpson E, Roberts S, Psorakis I, Smith A (2013) Dynamic bayesian combination of multiple imperfect classifiers. Decision Making Imperfection. (Springer), 1–35.
202. Simpson R, Page KR, De Roure D (2014) Zooniverse: observing the world’s largest citizen science platform. Proc. Companion Publ. 23rd International Conference World Wide Web Companion. (International World Wide Web Conferences Steering Committee), 1049–1054.
203. Sorokin A, Forsyth D (2008) Utility data annotation with Amazon Mechanical Turk. Comput. Vis. Pattern Recognit. Workshop 2008 CVPRW 08 IEEE Computer Social Conference On. 1–8.
204. Sousa, L., de Mello, R., Cedrim, D., Garcia, A., Missier, P., Uchôa, A., Oliveira, A., & Romanovsky, A. (2018). VazaDengue: An information system for preventing and combating mosquito-borne diseases with social networks. Information Systems, 75, 26–42.
205. Stevens, M., Vitos, M., Altenbuchner, J., Conquest, G., Lewis, J., & Haklay, M. (2014). Taking participatory citizen science to extremes. Pervasive Comput IEEE, 13(2), 20–29.
206. Stewart, N., Ungemach, C., Harris, A. J., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015). The average laboratory samples a population of 7,300 Amazon mechanical Turk workers. Judgment and Decision making, 10(5), 479–491.
207. Strien, A. J., Swaay, C. A., & Termaat, T. (2013). Opportunistic citizen science data of animal species produce reliable estimates of distribution trends if analysed with occupancy models. Journal of Applied Ecology, 50(6), 1450–1458.
208. Strong, D. M., Lee, Y. W., & Wang, R. Y. (1997). Data quality in context. Communications of the ACM, 40(5), 103–110.
209. Susarla, A., Oh, J. H., & Tan, Y. (2012). Social networks and the diffusion of user-generated content: Evidence from YouTube. Information Systems Research, 23(1), 23–41.
210. Tan M, Tripathi N, Zuiker SJ, Soon SH (2010) Building an online collaborative platform to advance creativity. (IEEE).
211. Tang, J., Zhang, P., & Wu, P. F. (2015). Categorizing consumer behavioral responses and artifact design features: The case of online advertising. Information Systems Frontiers, 17(3), 513–532.
212. Theobald, E. J., Ettinger, A. K., Burgess, H. K., DeBey, L. B., Schmidt, N. R., Froehlich, H. E., Wagner, C., HilleRisLambers, J., Tewksbury, J., & Harsch, M. A. (2015). Global change and local solutions: Tapping the unrealized potential of citizen science for biodiversity research. Biological Conservation, 181, 236–244.
213. Tilly R, Posegga O, Fischbach K, Schoder D (2016) Towards a Conceptualization of Data and Information Quality in Social Information Systems. Business & Information Systems Engineering, 59(1) 3–21.
214. Tremblay, M. C., Dutta, K., & Vandermeer, D. (2010). Using data mining techniques to discover bias patterns in missing data. Journal of Data and Information Quality, 2(1), 2.
215. Venkatesh, V., Davis, F., & Morris, M. G. (2007). Dead or alive? The development, trajectory and future of technology adoption research. Journal of the Association for Information Systems, 8(4), 1.
216. Vitos M, Lewis J, Stevens M, Haklay M (2013) Making local knowledge matter: supporting non-literate people to monitor poaching in Congo. (ACM), 1–10.
217. Voosen P, 2018, Am 11:00 (2018) Update: NASA confirms amateur astronomer has discovered a lost satellite. Science AAAS. Retrieved (April 9, 2018), http://www.sciencemag.org/news/2018/01/amateur-astronomer-discovers-revived-nasa-satellite.
218. Wahyudi, A., Kuk, G., & Janssen, M. (2018). A process pattern model for tackling and improving big data quality. Information Systems Frontiers, 20(3), 457–469.
219. Wand, Y., & Weber, R. (1995). On the deep-structure of information-systems. Information Systems Journal, 5(3), 203–223.
220. Wand, Y., & Weber, R. (2002). Research commentary: Information systems and conceptual modeling - a research agenda. Information Systems Research, 13(4), 363–376.
221. Wang, R. Y. (1998). A product perspective on total data quality management. Communications of the ACM, 41(2), 58–65.
222. Wang, R. Y., & Strong, D. M. (1996). Beyond accuracy: What data quality means to data consumers. Journal of Management Information Systems, 12(4), 5–33.
223. Wang, R. Y., Reddy, M. P., & Kon, H. B. (1995a). Toward quality data: An attribute-based approach. Decision Support Systems, 13(3–4), 349–372.
224. Wang, R. Y., Storey, V. C., & Firth, C. P. (1995b). A framework for analysis of data quality research. IEEE Transactions on Knowledge and Data Engineering, 7(4), 623–640.
225. Wang J, Ghose A, Ipeirotis P (2012) Bonus, Disclosure, and Choice: What Motivates the Creation of High-Quality Paid Reviews? International Conference Information System.
226. Wattal, S., Schuff, D., Mandviwalla, M., & Williams, C. B. (2010). Web 2.0 and politics: The 2008 U.S. presidential election and an E-politics research agenda. MIS Quarterly, 34(4), 669–688.
227. Weeger A, Wang X, Gewald H, Raisinghani M, Sanchez O, Grant G, Pittayachawan S (2015) Determinants of intention to participate in corporate BYOD-Programs: The case of digital natives. Information System Frontier 1–17.
228. Wells, J. D., Valacich, J. S., & Hess, T. J. (2011). What signals are you sending? How website quality influences perceptions of product quality and purchase intentions. MIS Quarterly, 35(2), 373–396.
229. Wiersma, Y. F. (2010). Birding 2.0: Citizen science and effective monitoring in the web 2.0 world. Avian Conservation Ecology, 5(2), 13.
230. Wiggins A, Crowston K (2011) From Conservation to Crowdsourcing: A Typology of Citizen Science. 44th Hawaii International Conference System Science 1–10.
231. Wiggins A, Crowston K (2014) Surveying the citizen science landscape. First Monday 20(1).
232. Wiggins A, He Y (2016) Community-based data validation practices in citizen science. (ACM), 1548–1559.
233. Wiggins A, Newman G, Stevenson RD, Crowston K (2011) Mechanisms for Data Quality and Validation in Citizen Science. Computer Citizen Science Workshop. (Stockholm, SE), 14–19.
234. Winter, S., Berente, N., Howison, J., & Butler, B. (2014). Beyond the organizational ‘container’: Conceptualizing 21st century sociotechnical work. Information and Organization, 24(4), 250–269.
235. Woo C (2011) The role of conceptual modeling in managing and changing the business. International Conference Conceptual Model. (Springer), 1–12.
236. Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N., & Malone, T. W. (2010). Evidence for a collective intelligence factor in the performance of human groups. Science, 330(6004), 686–688.
237. Zhang, P., & Li, N. (2005). The intellectual development of human-computer interaction research: A critical assessment of the MIS literature (1990-2002). Journal of the Association for Information Systems, 6(11), 227–292.
238. Zhao, Y., & Zhu, Q. (2014). Evaluation on crowdsourcing research: Current status and future direction. Information Systems Frontiers, 16(3), 417–434.
239. Zhou, M. J., Lu, B., Fan, W. P., & Wang, G. A. (2018). Project description and crowdfunding success: An exploratory study. Information Systems Frontiers, 20(2), 259–274.
240. Zhu, H., & Wu, H. (2011). Quality of data standards: Framework and illustration using XBRL taxonomy and instances. Electronic Markets, 21(2), 129–139.
241. Zuboff, S. (1988). In the age of the smart machine: The future of work and power. New York: Basic Books.
242. Zwass, V. (2010). Co-creation: Toward a taxonomy and an integrated research perspective. International Journal of Electronic Commerce, 15(1), 11–48.