Evaluation on crowdsourcing research: Current status and future direction

Information Systems Frontiers

Volumn 16, Issue 3, 2014, Pages 417-434

  Zhao, Yuxiang ^a   Zhu, Qinghua ^a  

^a NANJING UNIVERSITY   (China)

Author keywords

Collective intelligence; Crowdsourcing; Mass collaboration; Problem solving; Research progress; Socio technical systems; Web 2.0

Indexed keywords

WORLD WIDE WEB;

COLLECTIVE INTELLIGENCES; CROWDSOURCING; MASS COLLABORATION; SOCIOTECHNICAL SYSTEMS; WEB 2.0;

RESEARCH;

EID: 84903524315     PISSN: 13873326     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10796-012-9350-4     Document Type: Article

References (110)

1. Agarwal, R.; Henry Lucas, C. J. (2005). The information systems identify crisis: Focusing on high-visibility and high-impact research. MIS Quarterly, 29(3), 381-398.
2. Alavi, M.; Carlson, P. (1992). A review of MIS research and disciplinary development. Journal of Management Information Systems, 8(4), 45-62.
3. Albors, J.; Ramos, J. C.; Hervas, J. L. (2008). New learning network paradigms: communities of objectives, crowdsourcing, wikis and open source. International Journal of Information Management, 28(3), 194-202.
4. Alonso, O.; Rose, D.; Stewart, B. (2008). Crowdsourcing for relevance evaluation. SIGIR Forum, 42(2), 9-15.
5. Ambati, V.; Vogel, S.; Carbonell, J. (2010). Active learning and crowd-sourcing for machine translation. Proceedings of the LREC, 2010, 2169-2174.
6. Archak, N. (2010). Money, glory and cheap talk: Analyzing strategic behavior of contestants in simultaneous crowdsourcing contests on TopCoder.com. in Proceedings of the 19th international conference on World Wide Web ACM, pp. 21-30.
7. Archak, N.; Sundararajan, A. (2009). "Optimal Design of Crowdsourcing Contest," in Proceedings Thirtieth International Conference on Information Systems (ICIS 2009), Phoenix, USA.
8. Ba, S.; Stallaert, J.; Whinston, A. B. (2001). Introducing a third dimension in decentralized information systems design: the case for incentive alignment. Information Systems Research, 12(3), 225-239.
9. Bagozzi, R. P.; Dholakia, U. M. (2006). Open source software user communities: a study of participation in Linux user groups. Management Science, 52(7), 1099-1115.
10. Bayus, B. L. (2010). Crowdsourcing and individual creativity over time: the detrimental effects of past success. SSRN Quantitative Marketing eJournal, Retrieved from http://ssrn.com/abstract=1667101.
11. Benbasat, I.; Zmud, R. W. (2003). The identify crisis within the IS discipline: defining and communicating the discipline's core properties. MIS Quarterly, 27(2), 183-194.
12. Bigliardi, B.; Dormio, A. I.; Galati, F. (2012). The adoption of open innovation within the telecommunication industry. European Journal of Innovation Management, 15(1), 27-54.
13. Bonabeau, E. (2009). Decisions 2.0: the power of collective intelligence. MIT Sloan Management Review, 50(2), 45-52.
14. Boudreau, M.; Gefen, D.; Straub, D. (2001). Validation in IS research: a state-of-the-art assessment. MIS Quarterly, 25(1), 1-24.
15. Boudreau, K. J.; Nicola, L.; Karim, R. L. (2011). Incentives and problem uncertainty in innovation contests: an empirical Analysis. Management Science, 57(5), 843-863.
16. Brabham, D. C. (2008a). Crowdsourcing as a model for problem solving: an introduction and cases. Convergence: The International Journal of Research into New Media Technologies, 14(1), 75-90.
17. Brabham, D. C. (2008b). "Moving the crowd at iStockphoto: the composition of the crowd and motivations for participation in a crowdsourcing application. First Monday, 13(6).
18. Brabham, D. C. (2009). Crowdsourcing the public participation process for planning projects. Planning Theory, 8(3), 242-262.
19. Brabham, D. C. (2010). Moving the crowd at threadless: motivations for participation in a crowdsourcing application. Information, Communication & Society, 13(8), 1122-1145.
20. Brabham, D. C. (2011) forthcoming. Crowdsourcing: A model for leveraging online communities. In A. Delwiche & J. Henderson (Eds.), The Routledge Handbook of Participatory Culture.
21. Brew, A.; Greene, D.; Cunningham, P. (2010). Using crowdsourcing and active learning to track sentiment in online media. in Proceedings of ECAI 2010 19th European Conference on Artificial Intelligence, pp. 86-96.
22. Brito, J. (2008). Hack, Mash, & Peer: Crowdsourcing government transparency. The Columbia Science and Technology Law Review, 9, 119-157.
23. Chanal, V.; Caron-Fasan, & M. L. (2008). How to invent a new business model based on crowdsourcing: The CROWDSPIRIT Case. Conférence de l'Association Internationale de Management Stratégique, Nice, May.
24. Cheon, M. J.; Grover, V.; Teng, T. C. (1995). Theoretical perspectives on the outsourcing of information systems. Journal of Information Technology, 10(4), 209-219.
25. Chesbrough, H. W. (2003). The era of open innovation. MIT Sloan Management Review, 44(3), 35-41.
26. Chesbrough, H. (2006a). Open innovation: A new paradigm for understanding industrial innovation. In Chesbrough et al.; 2006. "Open Innovation: Researching a New Paradigm (pp 1-12). Oxford: Oxford University Press.
27. Chesbrough, H. (2006b). Open business models: How to thrive in the new innovation landscape. Cambridge, MA: Harvard Business School Press.
28. Chiaroni, D.; Chiesa, V.; Frattini. (2009). Investigating the adoption of open innovation in the bio-pharmaceutical industry: a framework and an empirical analysis. European Journal of Innovation Management, 12(3), 285-305.
29. Choffnes, D. R.; Bustamante, F. E.; Ge, Z. H. (2010). Crowdsourcing service-level network event monitoring. in Proceedings of the ACM SIGCOMM 2010 conference on SIGCOMM ACM, pp. 387-398.
30. Chris, C. B. (2009). "Fast, cheap, and creative: Evaluating translation quality using Amazons mechanical Turk," in Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, Vol. 1. PA, USA: Association for Computational Linguistics Stroudsburg
31. Delaney, D. G. (2008). Marine invasive species: validation of citizen science and implications for national monitoring networks. Biological Invasions, 10, 117-128.
32. Diana, R. (2010). Crowdsourcing is outsourcing Web 2.0 Style. Retrieved January 12, from http://regulargeek.com/2010/05/14/crowdsourcing-is-outsourcing- web-2-0-style /
33. DiPalantino, D.; Vojnovic, M. (2009). Crowdsourcing and all-pay auctions. in Proceedings of the Tenth ACM conference on Electronic commerce, pp. 119-128.
34. Doan, A.; Ramakrishnan, R.; Halevy, A. Y. (2011). Crowdsourcing systems on the world-wide web. Communications of the ACM, 54(4), 86-96.
35. Eckert, K.; Niepert, M.; Niemann, C.; Buckner, C.; Allen, C.; Stuckenschmidt, H. (2010). Crowdsourcing the Assembly of Concept Hierarchies. in Proceedings of the 10th annual joint conference on Digital libraries ACM, pp. 139-148.
36. Estellés-Arolas, E.; González Ladrón-de-Guevara, F. (2012). Towards an integrated crowdsourcing definition. Journal of Information Science (in press)
37. Franklin, M, Kossman, D.; Kraska, T, Ramesh, S, & Xin, R. (2011). CrowdDB: Answering Queries with Crowdsourcing. In Proceedings of SIGMOD 2011.
38. Gallivan, M. J. (2001). Striking a balance between trust anti control in a virtual organization: a content analysis of open source software case studies. Information Systems Journal, 11(4), 277-304.
39. Galloway, A. W. E.; et al. (2006). The reliability of citizen science: a case study of Oregon white oak stand surveys. Wildlife Society Bulletin, 34, 1425-1429.
40. Greengard, S. (2011). Following the Crowd. Communications of the ACM, 54(2), 20-22.
41. Gregg, D. G. (2010). Designing for collective intelligence. Communications of the ACM, 53(4), 134-138.
42. Gregor, S. (2006). The Nature of Theory in Information Systems. MIS Quarterly, 30(3), 611-642.
43. Hars, A.; Ou, S. S. (2002). Working for free? Motivations for participating in open-source projects. International Journal of Electronic Commerce, 6(3), 25-39.
44. Hauser, L. (1988). A conceptual analysis of information science. Library and Information Science Research, 10, 3-34.
45. Heer, J.; Bostock, M. (2010). Crowdsourcing Graphical Perception. in Proceedings of the 28th international conference on Human factors in computing systems ACM, pp. 286-295.
46. Hertel, G.; Niedner, S.; Herrmann, S. (2003). Motivation of software developers in open source projects: an internet-based survey of contributors to the Linux Kernel. Research Policy, 32(7), 1159-1177.
47. Hevner, A.; March, S.; Park, J.; Ram, S. (2004). Design science in information systems research. MIS Quarterly, 28(1), 75-105.
48. Holley, R. (2009). Crowdsourcing and social engagement: potential, power and freedom for libraries and users. in Proceedings of Pacific Rim Digital Library Alliance (PRDLA) Annual meeting and Conference: Libraries at the End of the World: Digital Content and Knowledge Creation, pp. 1-28.
49. Horton, J. J.; Chilton, L. B. (2010). The Labor Economics of Paid Crowdsourcing. in Proceedings of the 11th ACM conference on Electronic commerce ACM, pp. 209-218.
50. Howe, J. (2006). The rise of crowdsourcing. Wired Magazine, 14(6), 1-4.
51. Howe, J. (2008). Crowdsourcing. New York: Crown Publishing Group.
52. Hsueh, P.-Y.; Melville, P.; Sindhwani, V. (2009). Data quality from crowdsourcing: A study of annotation selection criteria. in Proceedings of the NAACL HLT 2009Workshop on Active Learning for Natural Language Processing, pp. 27-35.
53. Huberman, B. A.; Romero, D.; Wu, F. (2009). Crowdsourcing, attention and productivity. Journal of Information Science, 35(6), 758-765.
54. Hudson-Smith, A.; Batty, M.; Crooks, A.; Milton, R. 2009. Mapping for the Masses: Accessing Web 2.0 through Crowdsourcing. Social Science Computer Review, pp. 1066-1084.
55. Ipeirotis, P. G.; Provost, F.; Wang, J. (2010). Quality Management on Amazon Mechanical Turk. In KDD-HCOMP'10, July 2010, Washington DC, USA.
56. Jain, R. (2010). Investigation of Governance Mechanisms for Crowdsourcing Initiatives. in Proceedings of Americas Conference on Information Systems, pp. 557-563.
57. Johnson, J. (2007). Social networks and the wisdom of crowds. Network World, 24, 28-29.
58. Kazman, R.; Chen, H. (2009). The metropolis model a new logic for development of crowdsourced systems. Communications of the ACM, 52(7), 76-84.
59. Ke, W.; Zhang, P. (2009). Motivations in open source software communities: the mediating role of effort intensity and goal commitment. International Journal of Electronic Commerce, 13(4), 39-66.
60. Ke, W.; Zhang, P. (2010). The effects of extrinsic motivations and satisfaction in open source software development. Journal of the Association for Information Systems, 11(12), 784-808.
61. Keen, A. (2007). The cult of the amateur: How today's internet is killing our culture and assaulting our economy. London: Nicholas Brealey.
62. Kittur, A.; Chi, E. H.; Suh, B. (2008). Crowdsourcing User Studies with Mechanical Turk. in Proceedings of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, pp. 453-456.
63. Kleeman, F.; Voss, G. G.; Rieder, K. (2008). Un(der)paid innovators: the commercial utilization of consumer work through crowdsourcing. Science Technology & Innovation Studies, 4(1), 5-26.
64. Lakhani, K. R.; Wolf R. G. (2005). Why hackers do what they do: understanding motivation and effort in free/open source software projects. In J. Feller, B. Fitzgerald, S. Hissam, & K. R. Lakhani (Eds.). Perspectives on free and open source software. Cambridge, MA: MIT Press.
65. Lakhani, K. R.; Jeppesen, L. B.; Lohse, P. A.; Panetta, J. A. (2007). The value of openness in scientific problem solving. Harvard Business School Working Paper, (7: 50), Retrieved October 15, from http://www.hbs.edu/research/ pdf/07-050.pdf
66. Lane, S. (2010). Collective intelligence for competitive advantage: Crowd sourcing and open innovation. Continuing Education, 1-80.
67. Lee, J.-N.; Kim, Y.-G. (1999). Effect of partnership quality on is outsourcing success: conceptual framework and empirical validation. Journal of Management Information Systems, 15(4), 29-61.
68. Leimeister, J. M. (2010). Collective Intelligence. Business & Information Systems Engineering, April, pp. 245-248.
69. Leimeister, J. M.; Huber, M.; Bretschneider, U.; Krcmar, H. (2009). Leveraging crowdsourcing -activation-supporting components for IT-based idea competitions. Journal of Management Information Systems, 26(1), 197-224.
70. Lévy, P. (1995). Collective intelligence: Mankind's emerging world in cyberspace (R. Bononno, Trans.). New York: Plenum.
71. Maiolini, R.; Naggi, R. (2010). Crowdsourcing and SMEs: Opportunities and challenges. in Proceedings of ITAIS, 2010.
72. Malone, T. W.; Laubacher, R.; Dellarocas, C. (2010). Harnessing crowds: Mapping the genome of collective intelligence. MIT Sloan School Working Paper, no.4732-09.
73. Marcus, A.; Wu, E.; Madden, S. (2011). Crowdsourcing Databases: Query Processing with People. In Proceedings of CRDR 2011.
74. Mason, W.; Watts, D. J. (2010). Financial Incentives and the performance of crowds. SIGKDD Explorations Newsletter, 11(2), 100. ACM.
75. Mazzola, D.; Distefano, A. (2010). Crowdsourcing and the participation process for problem solving: The Case of BP. in Proceedings of ItAIS 2010 VII Conference of the Italian Chapter of AIS, pp. 42-49.
76. Meier, P. (2010). How to run a successful crowdsourcing project. Retrieved December 22, from http://irevolution.net/2010/05/05/towards-a-model- for-successful-crowdsourcing/
77. Munro, R.; Steven B.; Victor K.; Robin M.; Christopher P.; Tyler S.; et al. (2010). Crowdsourcing and language studies: The new generation of linguistic data. in Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Mechanical Turk, pp. 122-130.
78. Nov, O.; Arazy, O.; Anderson, D. (2011). Technology-mediated citizen science participation: a motivational model. Proceedings of the AAAI International Conference on Weblogs and Social Media (ICWSM 2011), Barcelona, Spain.
79. Orlikowski, W. J.; Iacono, C. S. (2001). Desperately seeking the 'IT' in IS research - a call to theorizing the IT artifact. Information Systems Research, 12(2), 121-134.
80. Pettigrew, K. E.; McKechnie, L. M. (2001). The use of theory in information science research. Journal of the American Society for Information Science & Technology, 52(1), 62-73.
81. Poetz, M. K.; Schreier, M. (2009). The value of crowdsourcing: can users really compete with professionals in generating new product ideas? Proceedings of the DRUID Summer Conference, 2009, 110-128.
82. Raddick, M.; et al. (2010). Galaxy zoo: exploring the motivations of citizen science volunteers. Astronomy Education Review, 9(1).
83. Reeve, J. (2005). Understanding motivation and emotion. New York: John Wiley & Sons.
84. Riedl, C.; Blohm, I.; Leimeister, J. M.; Krcmar, H. (2010). Rating scales for collective intelligence in innovation communities: Why quick and easy decision making does not get it right. In Proceedings of 2010 International Conference on Information Systems, St. Louis, Mi, USA: AIS.
85. Roberts, J. A.; Hann, I. H.; Slaughter, S. A. (2006). Understanding the motivations, participation, and performance of open source software developers: a longitudinal study of the apache projects. Management Science, 52(7), 984-999.
86. Roman, D. (2009). Crowdsourcing and the question of expertise. Communications of the ACM, 52(12), 12.
87. Rouse, A. (2010). A preliminary taxonomy of crowdsourcing. ACIS 2010 Proceedings, Paper 76. http://aisel.aisnet.org/acis2010/76.
88. Schenk, E.; Guittard, C. (2009). Crowdsourcing: What can be outsourced to the crowd, and why? Paper provided by HAL in its series of working papers with numbers halshs-00439256-v1
89. Shah, N.; Dhanesha, A.; Seetharam, D. (2009). Crowdsourcing for e-governance: Case study. in Proceedings of the 3rd International Conference on Theory and Practice of Electronic Governance (New York, NY, USA, 2009), ACM, pp. 253-258.
90. Sharma, A. (2010). Crowdsourcing critical success factor model: Strategies to harness the collective intelligence of the crowd. Retrieved January 27, from http://irevolution.files.wordpress.com/2010/05/working-paper1. pdf
91. Sims, J.; Crossland, C. (2010). "Partners or Pariahs? Firm engagement with open innovation communities. in: Academy of Management Conference. Montreal, Ca.
92. Stewart, O.; Huerta, J. M.; Sader, M. (2009). Designing crowdsourcing community for the enterprise. in Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 50-53.
93. Stewart, O.; Lubensky, D.; Huerta, J. M. (2010). Crowdsourcing participation inequality: A SCOUT model for the enterprise domain. in Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 30-33.
94. Stolee, K. T.; Elbaum, S. (2010). Exploring the use of crowdsourcing to support empirical studies in software engineering. in Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement ACM, pp. 35-38.
95. Strauss, A. (1987). Qualitative research for social scientists. Cambridge: Cambridge University Press.
96. Surowiecki, J. (2004). The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations. New York.
97. Tapscott, D.; Williams, A. D. (2007). Wikinomics: How mass collaboration changes everything. London, UK: Portfolio, Penguin.
98. Trompette, P. (2008). Crowdsourcing as a way to access external knowledge for innovation: Control, incentive and coordination in hybrid forms of innovation. in Proceedings of 24th EGOS Colloquium, pp. 1-29.
99. von Hippel, E. (2005). Democratizing innovation. Cambridge, Mass: MIT Press.
100. Vukovic, M. (2009). Crowdsourcing for enterprises. in Proceedings of Congress on Services, pp. 686-692.
101. Vukovic, M.; Kumara, S.; Greenshpan, O. (2010). Ubiquitous crowdsourcing. in Proceedings of the 12th ACM international conference adjunct papers on Ubiquitous computing ACM, pp. 523-525.
102. Webster, J.; Watson, R. T. (2002). Analyzing the past to prepare for the future: Writing a literature review. MIS Quarterly, 26(2), 13-23.
103. Whitla, P. (2009). Crowdsourcing and its application in marketing activities. Contemporary Management Research, 5(1), 15-28.
104. Wiggins, A.; Crowston, K. (2011). From conservation to crowdsourcing: A typology of citizen science. in Proceedings of the Forty-fourth Hawai'i International Conference on System Science (HICSS-44).
105. Wilcox, R. T. (2000). Experts and amateurs: the role of experience in internet auctions. Marketing Letters, 11, 363-374.
106. Yang, J.; Adamic, L. A.; Ackerman, M. S. (2008). Crowdsourcing and knowledge Sharing: Strategic user behavior on taskcn. in Proceedings of ACM Electronic Commerce'08, pp. 246-255.
107. Zhang, P. (2008). Motivational affordances: Reasons for ICT design and USE. Communications of the ACM, 61(11), 145-147.
108. Zhang, P.; Benjamin, R. I. (2007). Understanding information related fields: a conceptual framework. Journal of the American Society for Information Science and Technology, 58(13), 1934-1947.
109. Zhang, P.; Li, N. (2005). The intellectual development of human-computer interaction research: a critical assessment of the MIS literature (1990-2002). Journal of Association for Information Systems, 6(11), 227-292.
110. Zhang, P.; Scialdone, M.; Ku, M. (2011). IT artifacts and the state of IS research. forthcoming in the Proceedings of ICIS 2011, Shanghai, China, December.