How (not) to Incent Crowd Workers: Payment Schemes and Feedback in Crowdsourcing

Business and Information Systems Engineering

Volumn 57, Issue 3, 2015, Pages 167-179

  Straub, Tim ^a   Gimpel, Henner ^b   Teschner, Florian ^a   Weinhardt, Christof ^a  

^a KARLSRUHE INSTITUTE OF TECHNOLOGY   (Germany)

^b UNIVERSITY OF AUGSBURG   (Germany)

Author keywords

Crowdsourcing; Experimental techniques; Exploratory study; Feedback; Incentives; Online labor; Piece rate; Rank order tournament; Real effort task

Indexed keywords

EID: 84958953675     PISSN: None     EISSN: 18670202     Source Type: Journal    
DOI: 10.1007/s12599-015-0384-2     Document Type: Article

References (51)

1. Baron RM, Kenny DA (1986) The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. J Pers Soc Psychol 51(6):1173-1182
2. Berinsky AJ, Huber GA, Lenz GS (2012) Evaluating online labor markets for experimental research: Amazon.com's mechanical turk. Polit Anal 20(3):351-368
3. Boudreau M-C, Gefen D, Straub DW (2001) Validation in information systems research: a state-of-the-art assessment. MIS Q 25(1):1-16
4. Bracha A, Fershtman C (2013) Competitive incentives: working harder or working smarter? Manag Sci 59(4):771-781
5. Briggs RO, Schwabe G (2011) On expanding the scope of design science in is research. In: Jain H, Sinha AP, Vitharana P (eds) DESRIST 2011, LNCS 6629. Springer, Heidelberg, pp 92-106
6. Buhrmester M, Kwang T, Gosling SD (2011) Amazon's mechanical turk: a new source of inexpensive, yet high-quality, data? Perspect Psychol Sci 6(1):3-5
7. Bull C, Schotter A, Weigelt K (1987) Tournaments and piece rates: an experimental study. J of Polit Econ 95(1):1-33
8. Chilton LB, Horton JJ, Miller RC, Azenkot S (2010) Task search in a human computation market. In: ACM SIGKDD workshop on hum comput (HCOMP 2010), New York, pp 1-9
9. Cohen J (1988) Statistical power analysis for the behavioral sciences. Lawrence Erlbaum Associates, Hillsdale
10. Donabedian A (1980) Explorations in quality assessment and monitoring: the definition of quality and approaches to its assessment, vol 1. Health Administration Press, Ann Arbor
11. Donabedian A (2003) An introduction to quality assurance in health care. Oxford University Press, New York
12. Eccles JS, Wigfield A (2002) Motivational beliefs, values, and goals. Annu Rev of Psychol 53(1):109-132
13. Ehrenberg RG, Bognanno ML (1990) Do tournaments have incentive effects? J of Polit Econ 98(6):1307-1324
14. Eriksson T, Teyssier S, Villeval MC (2009a) Self-selection and the efficiency of tournaments. Econ Inq 47(3):530-548
15. Eriksson T, Poulsen A, Villeval MC (2009b) Feedback and incentives: experimental evidence. Labour Econ 16:679-688
16. Fershtman C, Gneezy U (2011) The tradeoff between performance and quitting in high power tournaments. J Europ Econ Assoc 9(2):318-336
17. Fischbacher U (2007) Z-Tree: zurich toolbox for ready-made economic experiments. Exp Econ 10(2):171-178
18. Gill D, Prowse V (2012) A structural analysis of disappointment aversion in a real effort competition. Am Econ Rev 102(1):469-503
19. Hammon L, Hippner H (2012) Crowdsourcing. Bus Inf Syst Eng 4(3):163-166
20. Harrison GW, List JA (2004) Field experiments. J Econ Lit 42(4):1009-1055
21. Hayes AF (2009) Beyond baron and kenny: statistical mediation analysis in the new millennium. Commun Monogr 76(4):408-420
22. Horton JJ, Rand DG, Zeckhauser RJ (2011) The online laboratory: conducting experiments in a real labor market. Exp Econ 14(3):399-425
23. Ipeirotis PG (2010) Analyzing the Amazon mechanical turk marketplace. XRDS 17(2):16-21
24. Ipeirotis PG, Provost F, Wang J (2010) Quality management on amazon mechanical turk. In: ACM SIGKDD workshop on hum comput (HCOMP 2010), Washington DC, pp 64-67
25. Kaufmann N, Schulze T, Veit D (2011) More than fun and money. Worker motivation in crowdsourcing - a study on mechanical turk. In: 17th Am conf on inf syst (AMCIS 2011), Detroit, paper 340
26. Kittur A, Khamkar S, André P, Kraut RE (2012) CrowdWeaver: visually managing complex crowd work. In: ACM 2012 conf on comput support coop work (CSCW 2012), Seattle, pp 1033-1036
27. Kittur A, Nickerson JV, Bernstein MS, Gerber EM, Shaw A, Zimmerman J, Lease M, Horton JJ (2013) The future of crowd work. In: 2013 conf on comput support coop work (CSCW 2013), San Antonio, pp 1301-1318
28. Kokkodis M, Ipeirotis PG (2013) Have you done anything like that? Predicting performance using inter-category reputation. In: 6th ACM int conf on web search and data min (WSDM 2013), Rome, pp 435-444
29. Kuhnen CM, Tymula A (2012) Feedback, self-esteem, and performance in organizations. Manag Sci 58(1):94-113
30. Lazear EP, Rosen S (1981) Rank-order tournaments as optimum labor contracts. J of Polit Econ 89(5):841-864
31. Leimeister JM (2010) Collective intelligence. Bus Inf Syst Eng 2(4):245-248
32. Malone TW, Laubacher R, Dellarocas C (2010) The collective intelligence genome. MIT Sloan Manag Rev 51(3):21-31
33. Mao A, Chen Y, Gajos KZ, Parkes D, Procaccia AD, Zhang H (2012) TurkServer: enabling synchronous and longitudinal online experiments. In: HCOMP (2012)
34. Mason W, Suri S (2012) Conducting behavioral research on Amazon's mechanical turk. Behav Res Methods 44(1):1-23
35. Mason W, Watts DJ (2009) Financial incentives and the performance of crowds. ACM SigKDD Explor Newsl 11(2):100-108
36. Paolacci G, Chandler J, Ipeirotis PG (2010) Running experiments on amazon mechanical turk. Judgm Decis Mak 5(5):411-419
37. Pederson EC, Denson TF, Goss RJ, Vasquez EA, Kelley NJ, Miller N (2011) The impact of rumination on aggressive thoughts, feelings, arousal, and behavior. Br J Soc Psychol 50:281-301
38. Pilz D, Gewald H (2013) Does money matter? Motivational factors for participation in paid- and non-profit-crowdsourcing communities. In: 11th Int Conf on Wirtschaftsinformatik (WI2013), Leipzig
39. Preacher KJ, Hayes AF (2004) SPSS and SAS procedures for estimating indirect effects in simple mediation models. Behav Res Methods, Instrum Comp 36(4):717-731
40. Pull K, Bäker H, Bäker A (2013) The ambivalent role of idiosyncratic risk in asymmetric tournaments. Theor Econ Lett 3(3A):16-22
41. Roth AE (1986) Laboratory experimentation in economics. Econ Philos 2:245-273
42. Roth AE (1987) Introduction and overview. In: Roth AE (ed) Laboratory experimentation in economics: six points of view. Cambridge University Press, Cambridge, pp 1-13
43. Ryan RM, Deci EL (2000) Intrinsic and extrinsic motivations: classic definitions and new directions. Contemp Educ Psychol 25(1):54-67
44. Shaw AD, Horton JJ, Chen DL (2011) Designing incentives for inexpert human raters. In: ACM 2011 conf on comput support coop work (CSCW 2011), Hangzhou, pp 275-284
45. Stebbins R (2001) Exploratory research in the social sciences. Sage Pubs, Thousand Oaks
46. Straub T, Gimpel H, Teschner F, Weinhardt C (2014a) Feedback and performance in crowd work: a real effort experiment. In: ECIS 2014 Proc, Tel Aviv
47. Straub T, Gimpel H, Teschner F, (2014b) The negative effect of feedback on performance in crowd labor tournaments. In: Nickerson J, Malone T (eds) Proc of collective intell 2014
48. Teschner F, Gimpel H (2013a) Crowd labor markets as platform for IS research: first evidence from electronic markets. In: 2013 Int conf on inf syst (ICIS 2013), Milan
49. Teschner F, Gimpel H (2013b) Validity of MTurk experiments in IS research: results from electronic markets. Working paper
50. Van Dijk F, Sonnemans J, van Winden F (2001) Incentive systems in a real effort experiment. Europ Econ Rev 45(2):187-214
51. Wang J, Ipeirotis PG, Provost F (2013) Quality-based pricing for crowdsourced workers. Working Paper. http://ssrn.com/abstract=2283000. Accessed 13 Mar 2015