Managing crowdsourced software testing: a case study based insight on the challenges of a crowdsourcing intermediary

Journal of Business Economics

Volumn 84, Issue 3, 2014, Pages 375-405

  Zogaj, Shkodran ^a   Bretschneider, Ulrich ^a   Leimeister, Jan Marco ^a  

^a UNIVERSITY OF KASSEL   (Germany)

Author keywords

Case study; Crowdsourcing; Crowdsourcing business model; Intermediary; Software testing

Indexed keywords

EID: 85015094128     PISSN: 00442372     EISSN: 18618928     Source Type: Journal    
DOI: 10.1007/s11573-014-0721-9     Document Type: Article

References (82)

1. Afuah A, Tucci CL (2012) Crowdsourcing as a solution to distant search. Acad of Manag Rev 37(3):355–375. doi:10.5465/amr.2010.0146
2. Allison TH, Townsend DM (2012) Wisdom of the Crowd? Reputational cascades and emotional contagion in microlender crowdfunding. Paper presented at the 72nd academy of management annual meeting, Boston
3. Bacon DF, Chen Y, Parkes D, Rao M (2009) A market-based approach to software evolution. Paper presented at the proceedings of the 24th ACM SIGPLAN conference
4. Bederson B, Quinn A (2010) Web workers, unite! addressing challenges of online laborers. In: Proceedings of CHI '11 extended abstracts on human factors in computing systems, pp 97–106
5. Bittner E, Leimeister JM (2011) Towards CSR 2.0—potentials and challenges of Web 2.0 for corporate social responsibility communication. In: Proceedings of the 11th European academy of management annual meeting, Tallinn
6. Bjelland OM, Wood RC (2008) An inside view of IBM’s innovation jam. MIT Sloan Manag Rev 50(1):31–40. doi:10.1225/SMR291
7. Blohm I, Köroglu O, Leimeister JM, Krcmar H (2012) Absorptive capacity for open innovation communities—learnings from theory and practice. Paper presented at the academy of management annual meeting
8. Brabham DC (2010) Moving the crowd at threadless: motivations for participation in a crowdsourcing application. Inform Commun Soc 13(8):1122–1145. doi:10.1080/13691181003624090
9. Brabham DC (2012) Crowdsourcing: a model for leveraging online communities. In: Delwiche A, Henderson J (eds) The Routledge handbook of participatory culture. Routledge, London, pp 1–25
10. Brandel M (2008) Crowdsourcing: are you ready to ask the world for answers? Computerworld 42:24–26
11. Bretschneider U (2012) Die Ideen-Community zur Integration von Kunden in den Innovationsprozess: Empirische Analysen und Implikationen. Dissertation. Gabler, Wiesbaden. doi:10.1007/978-3-8349-7173-9
12. Bretschneider U, Leimeister JM (2011) Schöne neue Crowdsourcing Welt: Billige Arbeitskräfte, Weisheit der Massen? In: Meißner K, Engelien M (eds) Virtual enterprises, communities and social networks, Proceedings zum Workshop Gemeinschaft in Neuen Medien (GeNeMe 11). Dresden, pp 1–13
13. Bullinger AC, Neyer AK, Rass M, Möslein KM (2010) Community-based innovation contests: where competition meets cooperation. Creativ Innov Manag 19(3):290–303. doi:10.1111/j.1467-8691.2010.00565.x
14. Burger-Helmchen T, Penin J (2010) The limits of crowdsourcing inventive activities: What do transaction cost theory and the evolutionary theories of the firm teach us? Paper presented at the workshop on open source innovation, Strasbourg
15. Burt RS (2005) Brokerage and closure: An introduction to social capital. Oxford University Press, New York. doi:10.1093/esr/jcm030
16. Caraway B (2010) Online labour markets: an inquiry into oDesk providers. Work Organ Labour Global 4(2):111–125
17. Chanal AV, Caron-Fasan M (2010) The difficulties involved in developing business models open to innovation communities: the case of a crowdsourcing platform. Manag 13(4):318–340. doi:10.3917/mana.134.0318
18. Chesbrough H, Crowther A (2006) Beyond high tech: early adopters of open innovation in other industries. R&D Manag 36(3):229–236. doi:10.1111/j.1467-9310.2006.00428.x
19. Corney JR, Torres-Sanchez C, Jagadeesan AP, Regli WC (2009) Outsourcing labour to the cloud. Int J Innov Sustain Develop 4(4):294–313. doi:10.1504/IJISD.2009.033083
20. Darke P, Shanks G, Broadbent M (1998) Successfully completing case study research: combining rigour, relevance and pragmatism. Inform Syst J 8(4):273–289. doi:10.1046/j.1365-2575.1998.00040.x
21. Doan A, Ramakrishnan R, Halevy AY (2011) Crowdsourcing systems on the World-Wide Web. Commun ACM 54:86–96. doi:10.1145/1924421.1924442
22. Eisenhardt KM (1989) Building theories from case study research. Acad Manag Rev 14(4):532–550. doi:10.5465/AMR.1989.4308385
23. Ernst H (2002) Success factors of new product development: a review of the empirical literature. Intern J Manag Rev 4(1):1–40. doi:10.1111/1468-2370.00075
24. Estellés-Arolas E, González-Ladrón-de-Guevara F (2012) Towards an integrated crowdsourcing definition. J Inform Sci 38(2):189–200. doi:10.1177/0165551512437638
25. Faste H (2011) Opening “open” innovation. In: DPPI ‘11 proceedings of the 2011 conference on design pleasurable products and interfaces
26. Franke N, Piller F (2004) Value creation by toolkits for user innovation and design: the case of the watch market. J Prod Innov Manag 21:401–415. doi:10.1111/j.0737-6782.2004.00094.x
27. Füller J, Matzler K (2007) Virtual product experience and customer participation—a chance for customer-centred, really new products. Technovation 27(6–7):378–387. doi:10.1016/j.technovation.2006.09.005
28. Füller J, Hutter K, Faullant R (2011) Why co-creation experience matters? Creative experience and its impact on the quantity and quality of creative contributions. R&D Manag 41(3):259–273. doi:10.1111/j.1467-9310.2011.00640.x
29. Gassmann O, Enkel E (2004) Towards a theory of open innovation: three core process archetypes. Paper presented at the R&D management conference (RADMA), Lissabon
30. Geiger D, Seedorf S, Schulze T, Nickerson R, Schader M (2011) Managing the crowd: towards a taxonomy of crowdsourcing processes. In: Proceedings of the 7th American conference on information system, Detroit
31. Hartley SE (2010) Kiva.org: crowd-sourced microfinance and cooperation in group lending working paper, 25 March 2010 Available at SSRN: http://ssrn.com/abstract=1572182
32. Hoßfeld T, Hirth M, Tran-Gia P (2012) Aktuelles Schlagwort: crowdsourcing. Inf Spek 35(3):204–208. doi:10.1007/s11576-012-0321-7
33. Howe J (2006a) Crowdsourcing: a definition. http://crowdsourcing.typepad.com/cs/2006/06/crowdsourcing_a.html. Accessed 18 May 2012
34. Howe J (2006b) The rise of crowdsourcing. Wired Mag 14(6):1–4
35. Howe J (2008) Crowdsourcing. Why the power of the crowd is driving the future of business. Crown Business Publishing, New York
36. Howells J (2006) Intermediation and the role of intermediaries in innovation. Res Policy 35:715–728. doi:10.1016/j.respol.2006.03.005
37. Jain R (2010) Investigation of governance mechanisms for crowdsourcing initiatives. Paper presented at the AMCIS 2010 proceedings
38. Jayakanthan R, Sundararajan D (2011) Enterprise crowdsourcing solutions for software development and ideation. Paper presented at the proceedings 2nd international workshop on ubiquitous crowdsouring
39. Jeppesen LB, Lakhani KR (2010) Marginality and problem-solving effectiveness in broadcast search. Organ Sci 21(5):1016–1033. doi:10.1287/orsc.1090.0491
40. Kaufmann N, Schulze T, Veit D (2011) More than fun and money. Worker motivation in crowdsourcing—a study on mechanical turk. In: Proceedings of the 7th AMCIS, Detroit
41. Kirkels Y, Duysters G (2010) Brokerage in SME networks. Res Policy 39:375–385. doi:10.1016/j.respol.2010.01.005
42. Kleeman F, Voss GG, Rieder K (2008) Un(der)paid innovators: the commercial utilization of consumer work through crowdsourcing. Sci Tech Innov Stud 4(1):5–26
43. Klerkx L, Leeuwis C (2009) Establishment and embedding of innovation brokers at different innovation system levels: insights from the Dutch agricultural sector. Tech Forecast Soc Change 76:849–860. doi:10.1016/j.techfore.2008.10.001
44. Lakhani KR, Wolf B (2005) Why hackers do what they do. Understanding motivation and effort in free/open source software projects. In: Feller J, Fitzgerald B, Hissam S, Lakhani KR (eds) Perspectives on free and open source software. The MIT Press, Cambridge. doi:10.2139/ssrn.443040
45. Lakhani KR, Jeppesen LB, Lohse PA, Panetta JA (2007) The value of openness in scientific problem solving. Harvard Business School Working Paper No 07-050 [Online] http://www.hbs.edu/research/pdf/07-050pdf
46. Leimeister JM (2010) Collective intelligence. Bus Inf Syst Eng 4(2):245–248. doi:10.1007/s12599-010-0114-8
47. Leimeister JM, Huber M, Bretschneider U, Krcmar H (2009) Leveraging crowdsourcing: activation-supporting components for IT-based ideas competition. J Manag Inf Syst 26(1):197–224. doi:10.2753/MIS0742-1222260108
48. Malone TW, Laubacher R, Dellarocas C (2010) The collective intelligence genome. MIT Sloan Manag Rev 51(3):20–31
49. Malone T, Laubacher R, Johns T (2011) The big idea: the age of hyperspecialization. Harvard Bus Rev, July–August, pp 1–11. http://www.topcoder.com/wp-content/uploads/2011/09/Hyperspecialization.pdf. Accessed 13 June 2012
50. Mao K, Yang Y, Li M, Harman M (2013) Pricing Crowdsourcing-Based Software Development Tasks. Paper presented at the 26th international conf on software engineering, San Francisco
51. Meredith J (1998) Building operations management theory through case and field research. J Operat Manag 16(4):441–454. doi:10.1016/S0272-6963(98)00023-0
52. Muhdi L, Boutellier R (2011a) The crowdsourcing process: an intermediary mediated idea generation approach in the early phase of innovation. Int J Entrepren Innov Manag 14(4):315–332. doi:10.1504/IJEIM.2011.043052
53. Muhdi L, Boutellier R (2011b) Motivational factors affecting participation and collaboration of members in two different Swiss Innovation communities. Int J Innov Manag 15(3):543–562. doi:10.1142/S1363919611003477
54. Myers GJ, Sandler C, Badgett T (2011) The art of software testing. New Jersey
55. Oliveira F, Ramos I, Santos L (2010) Definition of a crowdsourcing innovation service for the European SMEs. In: Daniel F, Facca FM (eds) Current trends in web engineering. Springer, Heidelberg
56. Paulini M, Murty P, Maher ML (2012) Understanding collective design communication in open innovation communities. Working Paper. University of Sydney, Australia
57. Pénin J (2008) More open than open innovation? Rethinking the concept of openness in innovation studies. Paper presented at the DT BETA, Strasbourg
58. Quinn AJ, Bederson BB (2011) Human computation: a survey and taxonomy of a growing field. Paper presented at the CHI 2011—proceed of the SIGCHI conference, Vancouver, May 7–12
59. Rayna T, Striukova L (2010) Large-scale open innovation: open source vs. patent pools. Int J Techn Manag 52(3, 4):477–496
60. Riungu-Kalliosaari L, Taipale O, Smolander K (2012) Testing in the cloud: exploring the practice. IEEE Softw 29(2):46–51. doi:10.1109/MS.2011.132
61. Rouse AC (2010) A Preliminary taxonomy of crowdsourcing. In: Australian conference on information system (ACIS), Brisbane, 1–3 Dec 2010
62. Schenk E, Guittard C (2009) Crowdsourcing: what can be outsourced to the crowd, and why? In: HAL: sciences de l’Homme et de la Société
63. Schenk E, Guittard C (2011) Towards a characterization of crowdsourcing practices. J Innov Econ 7(1):93–107. doi:10.3917/jie.007.0093
64. Steinfield C, Markus ML, Wigand RT (2011) Through a glass clearly: standards, architecture, and process transparency in global supply chains. J Manag Inform Sys 28(2):75–108. doi:10.2753/MIS0742-1222280204
65. Stewart J, Hyysalo S (2008) Intermediaries, users and social learning in technological innovation. Int J Innov Manag 12(3):295–325. doi:10.1142/S1363919608002035
66. Surowiecki J (2004) The wisdom of crowds: why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations, 1st edn. Doubleday Books, New York
67. Tapscott D, Williams AD (2007) Wikinomics: how mass collaboration changes everything. New York. doi:10.1111/j.1468-0270.2008.864_2.x
68. Tung Y-H, Tseng S-S (2013) A novel approach to collaborative testing in a crowdsourcing environment. J Syst Softw 86(8):2143–2153. doi:10.1016/j.jss.2013.03.079
69. Verner JM, Abdullah LM (2012) Exploratory case study research: outsourced project failure. Inform Softw Technol 54(8):866–886. doi:10.1016/j.infsof.2011.11.001
70. von Hippel E (1986) Lead users: a source of novel product concepts. Manag Sci 32(7):791–805. doi:10.1287/mnsc.32.7.791
71. Vukovic M (2009) Crowdsourcing for Enterprises. Paper presented at the SERVICES ‘09 proceedings of the 2009 congress on services—I, Los Angeles
72. Vukovic M, Bartolini C (2010) Towards a research agenda for enterprise crowdsourcing. In: Margaria T, Steffen B (eds) Leveraging Applications of formal methods, verification, and validation, Lecture notes in computer science, vol 6415. Springer, Berlin Heidelberg, pp 425-434. doi:10.1007/978-3-642-16558-0_36
73. Walmsley A (2009) The art of delegation. Marketing, 22 July 2009, pp 12–12 (1 page)
74. Ward C, Ramachandra V (2010) Crowdfunding the next hit: microfunding online experience goods. Workshop on computational social science and the wisdom of crowds (NIPS 2010). http://people.cs.umass.edu/~wallach/workshops/nips2010css/papers/ward.pdf. Accessed 20 July 2012
75. West J, Lakhani K (2008) Getting clear about communities in open innovation. Ind Innov 15(2):223–231
76. Whitla P (2009) Crowdsourcing and its application in marketing activities. Contem Manag Res 5(1):15–28
77. Whittaker JA (2000) What is software testing? And why is it so hard? IEEE Softw 17(1):70–79. doi:10.1109/52.819971
78. Winch GM, Courtney R (2007) The organization of innovation brokers: an international review. Technol Analy Strategy Manag 19:747–763. doi:10.1080/09537320701711223
79. Yin RK (2003) Case study research: design and methods, vol 5. Applied social research methods series, 3rd edn. Sage Publications, Thousand Oaks. doi:10.1111/j.1540-4781.2011.01212_17.x
80. Yuen M-C, King I, Leung K-S (2011) A survey of crowdsourcing systems. Paper presented at the 2011 IEEE international conference on privacy, security, risk, and trust, Boston
81. Zhao Y, Zhu Q (2012) Evaluation on crowdsourcing research: current status and future direction. Inf Syst Front. doi:10.1007/s10796-012-9350-4
82. Zwass V (2010) Co-creation: toward a taxonomy and an integrated research perspective. Int J Electr Com 15(1):11–48. doi:10.2753/JEC1086-4415150101