The four pillars of crowdsourcing: A reference model

Proceedings - International Conference on Research Challenges in Information Science

Volumn , Issue , 2014, Pages

  Hosseini, Mahmood ^a   Phalp, Keith ^a   Taylor, Jacqui ^a   Ali, Raian ^a  

^a BOURNEMOUTH UNIVERSITY   (United Kingdom)

Author keywords

crowdsourcing; crowdsourcing configuration; crowdsourcing taxonomy

Indexed keywords

INFORMATION SCIENCE; SEARCH ENGINES;

BUSINESS MODELING; CROWDSOURCING; CROWDSOURCING PLATFORMS; DEGREE OF DIVERSITY; DESIGN AND EVALUATIONS; INFORMATION SYSTEMS DEVELOPMENT; REFERENCE MODELING; SEARCH ENGINE EVALUATIONS;

TAXONOMIES;

EID: 84906751082     PISSN: 21511349     EISSN: 21511357     Source Type: Conference Proceeding    
DOI: 10.1109/RCIS.2014.6861072     Document Type: Conference Paper

References (101)

1. J. Howe, "Crowdsourcing: a definition," [Online], 2006, available from: http://www.crowdsourcing.com/cs/2006/06/crowdsourcing-a.html [Accessed 24 February 2014].
2. V. Ambati, S. Vogel, and J. G. Carbonell, "Towards task recommendation in micro-task markets," in Human computation, 2011.
3. Y. Baba and H. Kashima, "Statistical quality estimation for general crowdsourcing tasks," in Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 554-562, ACM, 2013
4. R. Das and M. Vukovic, "Emerging theories and models of human computation systems: a brief survey," in Proceedings of the 2nd international workshop on Ubiquitous crowdsouring, pp. 1-4, ACM, 2011.
5. V. Della Mea, E. Maddalena, and S. Mizzaro, "Crowdsourcing to mobile users: a study of the role of platforms and tasks," in DBCrowd 2013, 14, 2013.
6. K. Mao, Y. Yang, M. Li, and M. Harman, "Pricing crowdsourcing-based software development tasks," in Proceedings of the 2013 international conference on Software engineering, pp. 1205-1208, IEEE Press, 2013.
7. D. MacLean, K. Yoshida, A. Edwards, L. Crossman, B. Clavijo, M. Clark,... and D. G. Saunders, "Crowdsourcing genomic analyses of ash and ash dieback-power to the people," GigaScience, 2(1), 2, 2013.
8. J. A. Redi, T. Hoßfeld, P. Korshunov, F. Mazza, I. Povoa, and C. Keimel, "Crowdsourcing-based multimedia subjective evaluations: a case study on image recognizability and aesthetic appeal," in Proceedings of the 2nd ACM international workshop on Crowdsourcing for multimedia, pp. 29-34, ACM, 2013.
9. Y. Singer and M. Mittal, "Pricing mechanisms for crowdsourcing markets," in Proceedings of the 22nd international conference on World Wide Web, pp. 1157-1166. International World Wide Web Conferences Steering Committee, 2013.
10. J. M. Mortensen, M. A. Musen, and N. F. Noy, "Crowdsourcing the verification of relationships in biomedical ontologies," in AMIA Annual Symposium, 2013.
11. J. Ross, L. Irani, M. Silberman, A. Zaldivar, and B. Tomlinson, "Who are the crowdworkers?: shifting demographics in mechanical turk," in CHI'10 extended abstracts on Human factors in computing systems, pp. 2863-2872, ACM, 2010.
12. D. E. Difallah, G. Demartini, and P. Cudré-Mauroux, "Mechanical cheat: spamming schemes and adversarial techniques on crowdsourcing platforms," in CrowdSearch, pp. 26-30, 2012.
13. J. Lehman and R. Miikkulainen, "Leveraging human computation markets for interactive evolution," in International conference on Machine learning 2013 workshop: Machine learning meets crowdsourcing, 2013.
14. J. Howe, "The rise of crowdsourcing," [Online], 2006, available from: http://www.wired.com/wired/archive/14.06/crowds.html [Accessed 24 February, 2014].
15. S. Faradani, B. Hartmann, and P. G. Ipeirotis, "What's the right price? pricing tasks for finishing on time," in Human computation, 2011.
16. B. B. Bederson and A. J. Quinn, "Web workers unite addressing challenges of online laborers," in Proceedings of the 2011 annual conference extended abstracts on Human factors in computing systems, pp. 97-106, ACM, 2011.
17. M. Hirth, T. Hoßfeld, and P. Tran-Gia, "Anatomy of a crowdsourcing platform-using the example of microworkers.com," in Fifth international conference on Innovative mobile and internet services in ubiquitous computing, IMIS 2011, pp. 322-329, IEEE, 2011.
18. D. R. Saunders, P. J. Bex, and R. L. Woods, "Crowdsourcing a normative natural language dataset: a comparison of amazon mechanical turk and in-lab data collection," Journal of medical Internet research, 15(5), 2013.
19. M. Buhrmester, T. Kwang, and S. D. Gosling, "Amazon's mechanical turk a new source of inexpensive, yet high-quality, data?," Perspectives on Psychological Science, 6(1), pp. 3-5, 2011.
20. A. Kittur, E. H. Chi, and B. Suh, "Crowdsourcing user studies with mechanical turk," in Proceedings of the SIGCHI conference on human factors in computing systems, pp. 453-456, ACM, 2008.
21. D. C. Brabham, "Moving the crowd at Threadless: motivations for participation in a crowdsourcing application," Information, Communication & Society, 13(8), pp. 1122-1145, 2010.
22. J. Wu, C. Damminga, K. K. Johnson, and H. Y. Kim, "Content analysis of online co-design community interactions: a case study of crowd-sourced Threadless," Journal of Global Academy of Marketing, 20(4), pp. 334-342, 2010.
23. V. Chanal and M. L. Caron-Fasan, "How to invent a new business model based on crowdsourcing: the Crowdspirit case," in Actes de la Conférence Internationale de Management Stratégique, Nice, pp. 1-27, 2008.
24. P. Whitla, "Crowdsourcing and its application in marketing activities," Contemporary Management Research, 5(1), 2009.
25. A. Foncubierta Rodríguez and H. Müller, "Ground truth generation in medical imaging: a crowdsourcing-based iterative approach," in Proceedings of the ACM multimedia 2012 workshop on Crowdsourcing for multimedia, pp. 9-14, ACM, 2012.
26. B. Yu, M. Willis, P. Sun, and J. Wang, "Crowdsourcing participatory evaluation of medical pictograms using Amazon mechanical turk," Journal of medical Internet research, 15(6), e108, 2013.
27. J. Heinzelman, R. Brown, and P. Meier, "Mobile technology, crowdsourcing and peace mapping: new theory and applications for conflict management," in Mobile Technologies for Conflict Management, pp. 39-53, Springer Netherlands, 2011.
28. M. N. Wexler, "Reconfiguring the sociology of the crowd: exploring crowdsourcing," International Journal of Sociology and Social Policy, 31(1/2), pp. 6-20, 2011.
29. H. Gao, G. Barbier, and R. Goolsby, "Harnessing the crowdsourcing power of social media for disaster relief," Intelligent Systems, IEEE, 26(3), pp. 10-14, 2011.
30. P. Fraternali, A. Castelletti, R. Soncini-Sessa, C. Vaca Ruiz, and A. E. Rizzoli, "Putting humans in the loop: Social computing for Water Resources Management," Environmental Modelling & Software, 37, pp. 68-77, 2012.
31. R. Ali, C. Solis, I. Omoronyia, M. Salehie, and B. Nuseibeh, "Social adaptation: when software gives users a voice," in ENASE'12, 2012.
32. R. Ali, C. Solis, M. Salehie, I. Omoronyia, B. Nuseibeh, and W. Maalej, "Social sensing: when users become monitors," in ESEC/FSE'11, 2011.
33. D. Pagano and B. Bruegge, "User involvement in software evolution practice: a case study," in Proceedings of the 2013 international conference on Software engineering, pp. 953-962, IEEE Press, 2013.
34. D. Pagano and W. Maalej, "User feedback in the appstore: an empirical study," in 21st IEEE international requirements engineering conference, pp. 125-134, IEEE, 2013.
35. E. Estellés-Arolas and F. González-Ladrón-de- Guevara, "Towards an integrated crowdsourcing definition," Journal of Information Science, 38(2), pp. 189-200, 2012.
36. Y. Zhao and Q. Zhu, "Evaluation on crowdsourcing research: current status and future direction," Information Systems Frontiers, pp. 1-18, 2012.
37. B. Mellebeek, F. Benavent, J. Grivolla, J. Codina, M. R. Costa-Jussa, and R. Banchs, "Opinion mining of Spanish customer comments with non-expert annotations on mechanical turk," in Proceedings of the NAACL HLT 2010 workshop on Creating speech and language data with Amazon's mechanical turk, pp. 114-121, Association for Computational Linguistics, 2010.
38. C. J. Ho, S. Jabbari, and J. W. Vaughan, "Adaptive task assignment for crowdsourced classification," in Proceedings of the 30th international conference on Machine learning, ICML-13, pp. 534-542, 2013.
39. V. M. Ribiere and F. D. D. Tuggle, "Fostering innovation with KM 2.0," Vine, 40(1), pp. 90-101, 2010.
40. M. Larson, P. Cremonesi, A. Said, D. Tikk, Y. Shi, and A. Karatzoglou, "Activating the crowd: exploiting user-item reciprocity for recommendation," in the first workshop on Crowdsourcing and human computation for recommender systems, ACM Conference Series on Recommender Systems, ACM RECSYS, 2013.
41. E. Schenk and C. Guittard, "Crowdsourcing: what can be outsourced to the crowd, and why?," in Workshop on open source innovation, Strasbourg, France, 2009.
42. F. Bry, F. Kneissl, T. Krefeld, S. Luecke, and C. Wieser, "Crowdsourcing for a geographical and social mapping of Italian dialects," in SoHuman 2: 2nd international workshop on Social media for crowdsourcing and human computation, 2013.
43. M. Vukovic, "Crowdsourcing for enterprises," in 2009 World conference on Services-I, pp. 686-692, IEEE, 2009.
44. T. Erickson, "Some thoughts on a framework for crowdsourcing," in CHI 2011: workshop on Crowdsourcing and human computation, pp. 1-4, 2011.
45. H. Li, B. Yu, and D. Zhou, "Error rate analysis of labeling by crowdsourcing," in International conference on Machine learning 2013 workshop: Machine learning meets crowdsourcing, 2013.
46. E. J. Chikofsky and J. H. Cross, "Reverse engineering and design recovery: a taxonomy," Software, IEEE, 7(1), pp. 13-17, 1990.
47. S. Franklin and A. Graesser, "Is it an agent, or just a program?: a taxonomy for autonomous agents," in Intelligent agents III agent theories, architectures, and languages, pp. 21-35, Springer Berlin Heidelberg, 1997.
48. A. Broder, "A taxonomy of web search," in ACM Sigir forum, Vol. 36, No. 2, pp. 3-10, ACM, 2002.
49. D. A. Grier, "Not for all markets," in Computer, Vol. 44, No. 5, pp. 6-8, 2011.
50. C. Heipke, "Crowdsourcing geospatial data," ISPRS Journal of Photogrammetry and Remote Sensing, 65(6), pp. 550-557, 2010.
51. P. Heymann and H. Garcia-Molina, "Turkalytics: analytics for human computation," in Proceedings of the 20th international conference on World wide web, pp. 477-486, ACM, 2011.
52. R. Kern, H. Thies, C. Zirpins, and G. Satzger, "Dynamic and goal-based quality management for human-based electronic services," International Journal of Cooperative Information Systems, 21(01), pp. 3-29, 2012.
53. K. T. Stolee and S. Elbaum, "Exploring the use of crowdsourcing to support empirical studies in software engineering," in Proceedings of the 2010 ACM-IEEE international symposium on Empirical software engineering and measurement, p. 35, ACM, 2010.
54. D. L. Hansen, P. J. Schone, D. Corey, M. Reid, and J. Gehring, "Quality control mechanisms for crowdsourcing: peer review, arbitration, & expertise at Familysearch indexing," in Proceedings of the 2013 conference on Computer supported cooperative work, pp. 649-660, ACM, 2013.
55. T. Hossfeld, M. Hirth, and P. Tran-Gia, "Modeling of crowdsourcing platforms and granularity of work organization in future internet," in Proceedings of the 23rd International teletraffic congress, pp. 142-149, ITCP, 2011.
56. B. Li, D. S. Appling, S. Lee-Urban, and M. O. Riedl, "Learning sociocultural knowledge via crowdsourced examples," in Proceedings of the 4th AAAI workshop on Human computation, 2012.
57. J. O'Neill, S. Roy, A. Grasso, and D. Martin, "Form digitization in BPO: from outsourcing to crowdsourcing?," in Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 197-206, ACM, 2013.
58. O. Alonso, D. E. Rose, and B. Stewart, "Crowdsourcing for relevance evaluation," in ACM SigIR Forum, Vol. 42, No. 2, pp. 9-15. ACM, 2008.
59. W. Wu, W. T. Tsai, and W. Li, "Creative software crowdsourcing: from components and algorithm development to project concept formations," International Journal of Creative Computing, 1(1), pp. 57-91, 2013.
60. A. Adepetu, K. A. Ahmed, Y. Al Abd, A. Al Zaabi, and D. Svetinovic, "CrowdREquire: a requirements engineering crowdsourcing platform," in AAAI Spring Symposium Series, 2012.
61. A. Bozzon, M. Brambilla, S. Ceri, and A. Mauri, "Reactive crowdsourcing," in Proceedings of the 22nd international conference on World Wide Web, pp. 153-164, International World Wide Web Conferences Steering Committee, 2013.
62. M. K. Poetz and M. Schreier, "The value of crowdsourcing: can users really compete with professionals in generating new product ideas?," Journal of Product Innovation Management, 29(2), 245-256, 2012.
63. L. Schmidt, "Crowdsourcing for human subjects research," in Proceedings of CrowdConf, 2010.
64. D. C. Brabham, "Moving the crowd at iStockphoto: the composition of the crowd and motivations for participation in a crowdsourcing application," First Monday, 13(6), 2008.
65. J. Goncalves, D. Ferreira, S. Hosio, Y. Liu, J. Rogstadius, H. Kukka, and V. Kostakos, "Crowdsourcing on the spot: altruistic use of public displays, feasibility, performance, and behaviours," in Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, pp. 753-762, ACM, 2013.
66. A. Singla and A. Krause, "Truthful incentives in crowdsourcing tasks using regret minimization mechanisms," in Proceedings of the 22nd international conference on World Wide Web, pp. 1167-1178, International World Wide Web Conferences Steering Committee, 2013.
67. G. Kazai, "In search of quality in crowdsourcing for search engine evaluation," in Advances in information retrieval, pp. 165-176, Springer Berlin Heidelberg, 2011.
68. J. Pujara, B. London, and L. Getoor, "Reducing label cost by combining feature labels and crowdsourcing," in ICML workshop on Combining learning strategies to reduce label cost, 2011.
69. W. Mason and D. J. Watts, "Financial incentives and the performance of crowds,"in ACM SigKDD explorations newsletter, 11(2), 100-108, 2010.
70. J. Brito, "Hack, mash & peer: crowdsourcing government transparency," 2007.
71. G. Hsieh, "Understanding and designing for cultural differences on crowdsourcing marketplaces," in CHI 2011 workshop on Crowdsourcing and human computation, 2011.
72. M. Treiber, D. Schall, S. Dustdar, and C. Scherling, "Tweetflows: flexible workflows with twitter," in Proceedings of the 3rd international workshop on Principles of engineering service-oriented systems, pp. 1-7, ACM, 2011.
73. S. Djelassi and I. Decoopman, "Customers' participation in product development through crowdsourcing: issues and implications," Industrial Marketing Management, 2013.
74. R. Lukyanenko and J. Parsons, "Conceptual modeling principles for crowdsourcing," in Proceedings of the 1st international workshop on Multimodal crowd sensing, pp. 3-6, ACM, 2012.
75. F. Kleemann, G. G. Voß, and K. Rieder, "Un(der) paid innovators: the commercial utilization of consumer work through crowdsourcing," Science, Technology & Innovation Studies, 4(1), pp. 1-5, 2008.
76. C. Lofi, J. Selke, and W. T. Balke, "Information extraction meets crowdsourcing: a promising couple," Datenbank-Spektrum, 12(2), pp. 109-120, 2012.
77. M. Porta, B. House, L. Buckley, and A. Blitz, "Value 2.0: eight new rules for creating and capturing value from innovative technologies," Strategy & Leadership, 36(4), pp. 10-18, 2008.
78. V. Crescenzi, P. Merialdo, and D. Qiu, "Wrapper generation supervised by a noisy crowd," in DBCrowd 2013, 8, 2013.
79. R. Cavallo and J. Jain, "Winner-take-all crowdsourcing contests with stochastic production," in Workshop on Social computing and user generated content, EC-12, Valencia, Spain, 2012.
80. O. Alonso, "Perspectives on infrastructure for crowdsourcing," in Crowdsourcing for search and data mining, CSDM 2011, 7, 2011.
81. R. Morris, "Crowdsourcing workshop: the emergence of affective crowdsourcing," in Proceedings of the 2011 annual conference extended abstracts on Human factors in computing systems. ACM, 2011.
82. D. Oleson, A. Sorokin, G. P. Laughlin, V. Hester, J. Le, and L. Biewald, "Programmatic gold: targeted and scalable quality assurance in crowdsourcing," in Human computation, 11, 2011.
83. I. Abraham, O. Alonso, V. Kandylas, R. Patel, S. Shelford, A. Slivkins, and H. Wu, "Crowdsourcing gold-HIT creation at scale: challenges and adaptive exploration approaches," in HCOMP 2013 workshop, 2013.
84. R. M. Araujo, "99designs: an analysis of creative competition in crowdsourced design," in First AAAI conference on Human computation and crowdsourcing, 2013.
85. D. C. Brabham, "Crowdsourcing as a model for problem solving an introduction and cases," in Convergence: the international journal of Research into new media technologies, 14(1), pp. 75-90, 2008.
86. M. J. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin, "CrowdDB: answering queries with crowdsourcing," in Proceedings of the 2011 ACM SIGMOD international conference on Management of data, pp. 61-72, ACM, 2011.
87. K. C. Kang, S. G. Cohen, J. A. Hess, W. E. Novak, and A. S. Peterson, "Feature-oriented domain analysis (FODA) feasibility study," Carnegie- Mellon University Pittsburgh, PA, Software Engineering Institute, 1990.
88. J. Rogstadius, V. Kostakos, A. Kittur, B. Smus, J. Laredo, and M. Vukovic, "An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets," In ICWSM, 2011.
89. A. Kittur, B. Smus, S. Khamkar, and R. E. Kraut, "Crowdforge: crowdsourcing complex work," in Proceedings of the 24th annual ACM symposium on User interface software and technology, pp. 43-52, ACM, 2011.
90. B. H. Cheng, R. De Lemos, H. Giese, P. Inverardi, J. Magee, J. Andersson,... and J. Whittle, "Software engineering for self-adaptive systems: a research roadmap," in Software engineering for self-adaptive systems, pp. 1-26, Springer Berlin Heidelberg, 2009.
91. S. Hallsteinsen, M. Hinchey, S. Park, and K. Schmid, "Dynamic software product lines," Computer, 41(4), pp. 93-95, 2008.
92. G. Stringhini, C. Kruegel, and G. Vigna, "Detecting spammers on social networks," in Proceedings of the 26th annual computer security applications conference, pp. 1-9, ACM, 2010.
93. J. Surowiecki, "The wisdom of crowds," Random House LLC, 2005.
94. T. Gruber, "Collective knowledge systems: where the social web meets the semantic web," Web semantics: science, services and agents on the World Wide Web, 6(1), pp. 4-13, 2008.
95. O. Z. Maimon and L. Rokach, "Eds.," Data mining and knowledge discovery handbook, Vol. 1, New York, Springer, 2005.
96. R. Olfati-Saber, J. A. Fax, and R. M. Murray, "Consensus and cooperation in networked multi-agent systems," in Proceedings of the IEEE, 95(1), pp. 215-233, 2007.
97. S. L. Lim, D. Quercia, and A. Finkelstein, "StakeNet: using social networks to analyse the stakeholders of large-scale software projects," in Proceedings of the 32nd ACM/IEEE international conference on Software engineering, Vol. 1, pp. 295-304, ACM, 2010.
98. J. Bell, S. Sheth, and G. Kaiser, "Secret ninja testing with HALO software engineering," in Proceedings of the 4th international workshop on Social software engineering, pp. 43-47, ACM, 2011.
99. A. Knauss, "On the usage of context for requirements elicitation: enduser involvement in IT ecosystems," in Proceedings of the 2012 IEEE 20th international requirements engineering conference, RE '12, IEEE Computer Society, Washington, DC, USA, pp. 345-348, 2012.
100. M. Hosseini, K. Phalp, J. Taylor, and R. Ali, "Towards crowdsourcing for requirements engineering," in 20th International working conference on Requirements engineering: foundation for software quality - Empirical Track, 2013.
101. A. B. Sayeed, T. J. Meyer, H. C. Nguyen, O. Buzek, and A. Weinberg, "Crowdsourcing the evaluation of a domain-adapted named entity recognition system," in Human language technologies: the 2010 annual conference of the North American chapter of the association for computational linguistics, pp. 345-348, Association for Computational Linguistics, 2010.