Crowdsourcing: A taxonomy and systematic mapping study

Computer Science Review

Volumn 17, Issue , 2015, Pages 43-69

  Hosseini, Mahmood ^a   Shahri, Alimohammad ^a   Phalp, Keith ^a   Taylor, Jacqui ^a   Ali, Raian ^a  

^a BOURNEMOUTH UNIVERSITY   (United Kingdom)

Author keywords

Crowdsourcing; Crowdsourcing features; Systematic mapping; Taxonomy

Indexed keywords

MAPPING; TAXONOMIES;

APPLICATION AREA; CROWDSOURCING PLATFORMS; HUMAN INTELLIGENCE; MULTIPLE CRITERIA; NEW BUSINESS MODELS; REFERENCE MODELING; SYSTEMATIC MAPPING; SYSTEMATIC MAPPING STUDIES;

CROWDSOURCING;

EID: 84940447164     PISSN: 15740137     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.cosrev.2015.05.001     Document Type: Review

References (94)

1. J. Howe, Crowdsourcing: A definition, June 2006. URL: [Online]. http://www.crowdsourcing.com/cs/2006/06/crowdsourcing_a.html.
2. V. Chanal, M.-L. Caron-Fasan, How to invent a new business model based on crowdsourcing: the crowdspirit® case, in: Conférence de l'Association Internationale de Management Stratégique, 2008, pp. 1-27.
3. Djelassi S., Decoopman I. Customers' participation in product development through crowdsourcing: Issues and implications. Ind. Mark. Manag. 2013, 42(5):683-692. Business Models-Exploring value drivers and the role of marketing. 10.1016/j.indmarman.2013.05.006.
4. Poetz M.K., Schreier M. The value of crowdsourcing: Can users really compete with professionals in generating new product ideas?. J. Prod. Innov. Manage. 2012, 29(2):245-256. 10.1111/j.1540-5885.2011.00893.x.
5. R. Ali, C. Solis, I. Omoronyia, M. Salehie, B. Nuseibeh, Social adaptation: when software gives users a voice, in: ENASE 2012: 7th International Conference Evaluation of Novel Approaches to Software Engineering, 2012.
6. Ali R., Solis C., Salehie M., Omoronyia I., Nuseibeh B., Maalej W. Social sensing: When users become monitors. Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering 2011, 476-479. ACM, New York, NY, USA. 10.1145/2025113.2025196.
7. Pagano D., Brügge B. User involvement in software evolution practice: A case study. Proceedings of the 2013 International Conference on Software Engineering (ICSE) 2013, 953-962. IEEE Press, Piscataway, NJ, USA.
8. D. Pagano, W. Maalej, User feedback in the appstore: An empirical study, in: Proceedings of the 2013 21st IEEE International Requirements Engineering Conference, RE, 2013, pp. 125-134. http://dx.doi.org/10.1109/RE.2013.6636712.
9. Mortensen J.M., Musen M.A., Noy N.F. Crowdsourcing the verification of relationships in biomedical ontologies. AMIA Annual Symposium Proceedings, Vol. 2013 2013, 1020. American Medical Informatics Association.
10. Saunders D.R., Bex P.J., Woods R.L. Crowdsourcing a normative natural language dataset: A comparison of amazon mechanical turk and in-lab data collection. J. Med. Internet Res. 2013, 15(5). 10.2196/jmir.2620.
11. Yu B., Willis M., Sun P., Wang J. Crowdsourcing participatory evaluation of medical pictograms using amazon mechanical turk. J. Med. Internet Res. 2013, 15(6). 10.2196/jmir.2513.
12. Fraternali P., Castelletti A., Soncini-Sessa R., Ruiz C.V., Rizzoli A. Putting humans in the loop: Social computing for water resources management. Environ. Modell. Softw. 2012, 37(0):68-77. 10.1016/j.envsoft.2012.03.002.
13. Gao H., Barbier G., Goolsby R. Harnessing the crowdsourcing power of social media for disaster relief. IEEE Intell. Syst. 2011, 26(3):10-14. 10.1109/MIS.2011.52.
14. MacLean D., Yoshida K., Edwards A., Crossman L., Clavijo B., Clark M., Swarbreck D., Bashton M., Chapman P., Gijzen M., Caccamo M., Downie A., Kamoun S., Saunders D. Crowdsourcing genomic analyses of ash and ash dieback-power to the people. GigaScience 2013, 2(1):2. 10.1186/2047-217X-2-2.
15. Surowiecki J. The Wisdom of Crowds 2005, Random House LLC.
16. Larson M.A., Said A., Shi Y., Tikk D., Karatzoglou A., Baltrunas L., Geurts J., Anguera X., Hopfgarnter F. Activating the crowd: Exploiting user-item reciprocity for recommendation. ACM Conference Series on Recommender Systems 2013, ACM RECSYS, Hong Kong, China.
17. Mao K., Yang Y., Li M., Harman M. Pricing crowdsourcing-based software development tasks. Proceedings of the 2013 International Conference on Software Engineering (ICSE) 2013, 1205-1208. IEEE Press, Piscataway, NJ, USA.
18. J.A. Redi, T. Hoßfeld, P. Korshunov, F. Mazza, I. Povoa, C. Keimel, Crowdsourcing-based multimedia subjective evaluations: a case study on image recognizability and aesthetic appeal, in: Proceedings of the 2nd ACM International Workshop on Crowdsourcing for Multimedia, CrowdMM, 2013.
19. Franklin M.J., Kossmann D., Kraska T., Ramesh S., Xin R. CrowdDB: Answering queries with crowdsourcing. Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data 2011, 61-72. ACM, New York, NY, USA. 10.1145/1989323.1989331.
20. Kazai G. In search of quality in crowdsourcing for search engine evaluation. Lecture Notes in Computer Science 2011, vol. 6611:165-176. Springer, Berlin, Heidelberg. 10.1007/978-3-642-20161-5_17. P. Clough, C. Foley, C. Gurrin, G. Jones, W. Kraaij, H. Lee, V. Mudoch (Eds.).
21. Foncubierta Rodríguez A., Müller H. Ground truth generation in medical imaging: A crowdsourcing-based iterative approach. Proceedings of the ACM Multimedia 2012 Workshop on Crowdsourcing for Multimedia (CrowdMM) 2012, 9-14. ACM, New York, NY, USA. 10.1145/2390803.2390808.
22. Seltzer E., Mahmoudi D. Citizen participation, open innovation, and crowdsourcing: Challenges and opportunities for planning. J. Plan. Lit. 2013, 28(1):3-18. arxiv:http://jpl.sagepub.com/content/28/1/3.full.pdf+html, 10.1177/0885412212469112.
23. Bader M. Crowdsourcing election monitoring in the 2011-2012 Russian elections. East Eur. Polit. 2013, 29(4):521-535. arxiv:http://www.tandfonline.com/doi/pdf/10.1080/21599165.2013.818979, 10.1080/21599165.2013.818979.
24. Bani M. Crowdsourcing democracy: The case of icelandic social constitutionalism. Politics and Policy in the Information Age 2012, Springer.
25. Borch C. The Politics of Crowds: An Alternative History of Sociology 2012, Cambridge University Press.
26. Wexler M.N. Reconfiguring the sociology of the crowd: exploring crowdsourcing. Int. J. Sociol. Soc. Policy 2011, 31(1-2):6-20.
27. T. Erickson, Some thoughts on a framework for crowdsourcing, in: Workshop on Crowdsourcing and Human Computation, 2011, pp. 1-4.
28. H. Li, B. Yu, D. Zhou, Error rate analysis of labeling by crowdsourcing, in: Proceedings of the International Conference on Machine Learning 2013 Workshop: Machine Learning Meets Crowdsourcing, 2013.
29. K. Petersen, R. Feldt, S. Mujtaba, M. Mattsson, Systematic mapping studies in software engineering, in: 12th International Conference on Evaluation and Assessment in Software Engineering, Vol. 17, 2008, p. 1.
30. Engström E., Runeson P. Software product line testing-a systematic mapping study. Inf. Softw. Technol. 2011, 53(1):2-13. 10.1016/j.infsof.2010.05.011.
31. Fernandez A., Insfran E., Abrahão S. Usability evaluation methods for the web: A systematic mapping study. Inf. Softw. Technol. 2011, 53(8):789-817. advances in functional size measurement and effort estimation-Extended best papers. 10.1016/j.infsof.2011.02.007.
32. Bailey J., Budgen D., Turner M., Kitchenham B., Brereton P., Linkman S.G. Evidence relating to object-oriented software design: A survey. Proceedings of the First International Symposium on Empirical Software Engineering and Measurement 2007, 482-484. IEEE Computer Society.
33. Kitchenham B., Charters S. Guidelines for performing systematic literature reviews in software engineering, Tech. Rep. EBSE 2007-001 2007, Keele University and Durham University Joint Report.
34. Hosseini M., Phalp K., Taylor J., Ali R. The four pillars of crowdsourcing: A reference model. Proceedings of the 2014 IEEE Eighth International Conference on Research Challenges in Information Science (RCIS) 2014, 1-12. IEEE.
35. Petticrew M., Roberts H. Systematic Reviews in The Social Sciences: A Practical Guide 2008, John Wiley & Sons.
36. Wieringa R., Maiden N., Mead N., Rolland C. Requirements engineering paper classification and evaluation criteria: a proposal and a discussion. Requir. Eng. J. 2006, 11(1):102-107. 10.1007/s00766-005-0021-6.
37. Bederson B.B., Quinn A.J. Web workers unite! Addressing challenges of online laborers. CHI'11 Extended Abstracts on Human Factors in Computing Systems 2011, 97-106. ACM, New York, NY, USA. 10.1145/1979742.1979606.
38. Oleson D., Sorokin A., Laughlin G.P., Hester V., Le J., Biewald L. Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. Proceedings of the Third Human Computation Workshop, HCOMP, Vol. WS-11-11 of AAAI Workshops 2011, AAAI.
39. Alonso O., Lease M. Crowdsourcing 101: Putting the wsdm of crowds to work for you. Proceedings of the Fourth ACM International Conference on Web Search and Data Mining (WSDM) 2011, 1-2. ACM, New York, NY, USA. 10.1145/1935826.1935831.
40. DiPalantino D., Vojnovic M. Crowdsourcing and all-pay auctions. Proceedings of the 10th ACM Conference on Electronic Commerce 2009, 119-128. ACM, New York, NY, USA. 10.1145/1566374.1566392.
41. M. Hirth, T. Hossfeld, P. Tran-Gia, Anatomy of a crowdsourcing platform-using the example of microworkers.com, in: Proceedings of the 2011 Fifth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing, IMIS, 2011, pp. 322-329. http://dx.doi.org/10.1109/IMIS.2011.89.
42. Hirth M., Hoßfeld T., Tran-Gia P. Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms. Math. Comput. Modelling 2013, 57(11-12):2918-2932. information System Security and Performance Modeling and Simulation for Future Mobile Networks. 10.1016/j.mcm.2012.01.006.
43. Singer Y., Mittal M. Pricing mechanisms for crowdsourcing markets. Proceedings of the 22nd International Conference on World Wide Web (WWW) 2013, 1157-1166. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, URL: http://dl.acm.org/citation.cfm?id=2488388.248848.
44. Estellés-Arolas E., González-Ladrón-de Guevara F. Towards an integrated crowdsourcing definition. J. Inf. Sci. 2012, 38(2):189-200.
45. C. Ho, S. Jabbari, J.W. Vaughan, Adaptive task assignment for crowdsourced classification, in: S. Dasgupta, D. Mcallester (Eds.), Proceedings of the 30th International Conference on Machine Learning, ICML-13, Vol. 28, JMLR Workshop and Conference Proceedings, 2013, pp. 534-542.
46. Ribiere V.M., Tuggle F.D.D. Fostering innovation with km 2.0. Vine 2010, 40(1):90-101.
47. Grier D.A. Not for all markets. Computer 2011, 44(5):6-8. 10.1109/MC.2011.155.
48. Heipke C. Crowdsourcing geospatial data. ISPRS J. Photogramm. Remote Sens. 2010, 65(6):550-557. {ISPRS} Centenary Celebration Issue. 10.1016/j.isprsjprs.2010.06.005.
49. Heymann P., Garcia-Molina H. Turkalytics: Analytics for human computation. Proceedings of the 20th International Conference on World Wide Web (WWW) 2011, 477-486. ACM, New York, NY, USA. 10.1145/1963405.1963473.
50. Kern R., Thies H., Zirpins C., Satzger G. Dynamic and goal-based quality management for human-based electronic services. Int. J. Coop. Inf. Syst. 2012, 21(01):3-29. arXiv:http://www.worldscientific.com/doi/pdf/10.1142/S0218843012400011, http://dx.doi.org,10.1142/S0218843012400011.
51. Hansen D.L., Schone P.J., Corey D., Reid M., Gehring J. Quality control mechanisms for crowdsourcing: Peer review, arbitration, & expertise at familysearch indexing. Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW) 2013, 649-660. ACM, New York, NY, USA. 10.1145/2441776.2441848.
52. Stolee K.T., Elbaum S. Exploring the use of crowdsourcing to support empirical studies in software engineering. Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) 2010, 35:1-35:4. ACM, New York, NY, USA. 10.1145/1852786.1852832.
53. Zhao Y., Zhu Q. Evaluation on crowdsourcing research: Current status and future direction. Inf. Syst. Front. 2012, 1-18. 10.1007/s10796-012-9350-4.
54. Ambati V., Vogel S., Carbonell J. Towards task recommendation in micro-task markets. Human Computation 2011, 1-4.
55. T. Hossfeld, M. Hirth, P. Tran-Gia, Modeling of crowdsourcing platforms and granularity of work organization in future Internet, in: Proceedings of the 23rd International Teletraffic Congress, ITC'11, International Teletraffic Congress, 2011, pp. 142-149.
56. B. Li, D.S. Appling, S. Lee-urban, M.O. Riedl, Learning sociocultural knowledge via crowdsourced examples, in: Proceedings of the Fourth AAAI Workshop on Human Computation, 2012.
57. O'Neill J., Roy S., Grasso A., Martin D. Form digitization in BPO: From outsourcing to crowdsourcing?. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI) 2013, 197-206. ACM, New York, NY, USA. 10.1145/2470654.2470683.
58. D.E. Difallah, G. Demartini, P. Cudré-Mauroux, Mechanical cheat: Spamming schemes and adversarial techniques on crowdsourcing platforms, in: Proceedings of the First International Workshop on Crowdsourcing Web search (CrowdSearch) at 21st International World Wide Web Confernce, (WWW), 2012, pp. 26-30.
59. Alonso O., Rose D.E., Stewart B. Crowdsourcing for relevance evaluation. SIGIR Forum 2008, 42(2):9-15. 10.1145/1480506.1480508.
60. Wu W., Tsai W.-T., Li W. Creative software crowdsourcing: from components and algorithm development to project concept formations. Int. J. Creat. Comput. 2013, 1(1):57-91.
61. Baba Y., Kashima H. Statistical quality estimation for general crowdsourcing tasks. Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 2013, 554-562. ACM, New York, NY, USA. 10.1145/2487575.2487600.
62. J. Lehman, R. Miikkulainen, Leveraging human computation markets for interactive evolution, in: Proceedings of Machine Learning Meets Crowdsourcing Workshop at ICML 2013, 2013.
63. A. Adepetu, A. Khaja, Y.A. Abd, A.A. Zaabi, D. Svetinovic, CrowdREquire: A requirements engineering crowdsourcing platform, in: Proceedings of 2012 AAAI Spring Symposium: Wisdom of the Crowd, 2012.
64. Das R., Vukovic M. Emerging theories and models of human computation systems: A brief survey. Proceedings of the 2nd International Workshop on Ubiquitous Crowdsouring 2011, 1-4. ACM, New York, NY, USA. 10.1145/2030100.2030102.
65. Vukovic M. Crowdsourcing for enterprises. 2009 World Conference on Services-I 2009, 686-692. IEEE. 10.1109/SERVICES-I.2009.56.
66. Bozzon A., Brambilla M., Ceri S., Mauri A. Reactive crowdsourcing. Proceedings of the 22nd International Conference on World Wide Web (WWW) 2013, 153-164. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland.
67. L. Schmidt, Crowdsourcing for human subjects research, in: Proceedings of The World's First Conference on the Future of Distributed Work, CrowdConf.
68. Brabham D. Moving the crowd at istockphoto: The composition of the crowd and motivations for participation in a crowdsourcing application. First Monday 2008, 13(6).
69. Goncalves J., Ferreira D., Hosio S., Liu Y., Rogstadius J., Kukka H., Kostakos V. Crowdsourcing on the spot: Altruistic use of public displays, feasibility, performance, and behaviours. Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) 2013, 753-762. ACM, New York, NY, USA. 10.1145/2493432.2493481.
70. J. Pujara, B. London, L. Getoor, Reducing label cost by combining feature labels and crowdsourcing, in: ICML Workshop on Combining Learning Strategies to Reduce Label Cost, 2011.
71. Ross J., Irani L., Silberman M.S., Zaldivar A., Tomlinson B. Who are the crowdworkers?: Shifting demographics in mechanical turk. Proceedings of CHI'10 Extended Abstracts on Human Factors in Computing Systems 2010, 2863-2872. ACM, New York, NY, USA. 10.1145/1753846.1753873.
72. Mason W., Watts D.J. Financial incentives and the "performance of crowds". ACM SIGKDD Explor. Newsl. 2010, 11(2):100-108. 10.1145/1809400.1809422.
73. Mea V.D., Maddalena E., Mizzaro S. Crowdsourcing to mobile users: A study of the role of platforms and tasks. CEUR Workshop Proceedings 2013, vol. 1025:14-19. CEUR-WS.org. R. Cheng, A.D. Sarma, S. Maniu, P. Senellart (Eds.).
74. Brito J. Hack, mash & peer: Crowdsourcing government transparency. Columia Sci. Technol. Law Rev. 2007, 9:119-122.
75. E. Schenk, C. Guittard, Crowdsourcing: What can be outsourced to the crowd, and why, in: Workshop on Open Source Innovation, Strasbourg, France, 2009.
76. G. Hsieh, Understanding and designing for cultural differences on crowdsourcing marketplaces, in: Proceedings of Chi 2011: Workshop on Crowdsourcing and Human-Computation, 2011.
77. Mellebeek B., Benavent F., Grivolla J., Codina J., Costa-jussà M.R., Banchs R. Opinion mining of spanish customer comments with non-expert annotations on mechanical turk. Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk (CSLDAMT) 2010, 114-121. Association for Computational Linguistics, Stroudsburg, PA, USA.
78. Treiber M., Schall D., Dustdar S., Scherling C. Tweetflows: Flexible workflows with Twitter. Proceedings of the Third International Workshop on Principles of Engineering Service-Oriented Systems (PESOS) 2011, 1-7. ACM, New York, NY, USA. 10.1145/1985394.1985395.
79. Lukyanenko R., Parsons J. Conceptual modeling principles for crowdsourcing. Proceedings of the First International Workshop on Multimodal Crowd Sensing (CrowdSens) 2012, 3-6. ACM, New York, NY, USA. 10.1145/2390034.2390038.
80. Kleemann F., Voß G., Rieder K. Un(der)paid innovators: The commercial utiliza-tion of consumer work through crowdsourcing. Sci. Technol. Innov. Stud. 2008, 4(1).
81. J. Howe, The rise of crowdsourcing, Online, June 2006. URL: http://archive.wired.com/wired/archive/14.06/crowds.html.
82. Lofi C., Selke J., Balke W.-T. Information extraction meets crowdsourcing: A promising couple. Datenbank-Spektrum 2012, 12(2):109-120. 10.1007/s13222-012-0092-8.
83. Porta M., House B., Buckley L., Blitz A. Value 2.0: Eight new rules for creating and capturing value from innovative technologies. Strategy Leadersh. 2008, 36(4):10-18.
84. Crescenzi V., Merialdo P., Qiu D. Wrapper generation supervised by a noisy crowd. CEUR Workshop Proceedings 2013, vol. 1025:8-13. CEUR-WS.org. R. Cheng, A.D. Sarma, S. Maniu, P. Senellart (Eds.).
85. Whitla P. Crowdsourcing and its application in marketing activities. Contemp. Manag. Res. 2009, 5(1).
86. Cavallo R., Jain S. Winner-take-all crowdsourcing contests with stochastic production. Proceedings of the First AAAI Conference on Human Computation and Crowdsourcing (HCOMP) 2013, AAAI. B. Hartman, E. Horvitz (Eds.).
87. O. Alonso, Perspectives on infrastructure for crowdsourcing, in: Workshop on Crowdsourcing for Search and Data Mining, CSDM 2011, 2011.
88. Morris R. Crowdsourcing workshop: the emergence of affective crowdsourcing. Proceedings of the 2011 Annual Conference Extended Abstracts on Human Factors in Computing Systems 2011, ACM.
89. Abraham I., Alonso O., Kandylas V., Patel R., Shelford S., Slivkins A., Wu H. Crowdsourcing gold-hit creation at scale: Challenges and adaptive exploration approaches, Tech. Rep 2013, Microsoft Corporation.
90. R.M. Araujo, 99designs: An analysis of creative competition in crowdsourced design, in: Proceedings of the First AAAI Conference on Human Computation and Crowdsourcing, 2013.
91. Brabham D.C. Crowdsourcing as a model for problem solving: An introduction and cases. Converg.: Int. J. Res. New Media Technol. 2008, 14(1):75-90. arxiv:http://con.sagepub.com/content/14/1/75.full.pdf+html, 10.1177/1354856507084420.
92. Adda G., Mariani J.J., Besacier L., Gelas H. Economic and Ethical Background of Crowdsourcing for Speech 2013, 303-334. John Wiley & Sons, Ltd., (Chapter 11) URL: http://dx.doi.org/10.1002/9781118541241.ch11. 10.1002/9781118541241.ch11.
93. L.R. Varshney, Privacy and reliability in crowdsourcing service delivery, in: Proceedings of the 2012 Annual Service Research and Innovation Institute Global Conference, SRII, 2012, pp. 55-60. http://dx.doi.org/10.1109/SRII.2012.17.
94. Kang K.C., Cohen S.G., Hess J.A., Novak W.E., Peterson A.S. Feature-oriented domain analysis (foda) feasibility study, Tech. Rep 1990, Carnegie-Mellon University Software Engineering Institute.