Incentivizing high quality crowdwork

WWW 2015 - Proceedings of the 24th International Conference on World Wide Web

Volumn , Issue , 2015, Pages 419-429

  Ho, Chien Ju ^a   Slivkins, Aleksandrs ^b   Suri, Siddharth ^b   Vaughan, Jennifer Wortman ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b MICROSOFT RESEARCH   (United States)

Author keywords

Crowdsourcing; Incentives; Performance Based Payments

Indexed keywords

CROWDSOURCING;

AMAZON MECHANICAL TURKS; BEHAVIORAL EXPERIMENT; CROWDSOURCING PLATFORMS; FINANCIAL INCENTIVES; INCENTIVES; PAYMENT STRUCTURE; PERFORMANCE BASED; PRINCIPAL AGENT MODELS;

WORLD WIDE WEB;

EID: 84968835134     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2736277.2741102     Document Type: Conference Paper

References (37)

1. O. Alonso. Implementing crowdsourcing-based relevance experimentation: An industrial perspective. Information Retrieval, 16(2):101-120, 2013.
2. R. M. Araujo. 99designs: An analysis of creative competition in crowdsourced design. In HCOMP, 2013.
3. S. E. Bonner, R. Hastie, S. G. B., and S. M. Young. A review of the effects of financial incentives on performance in laboratory tasks: Implications for management accounting. Journal of Management Accounting Research, 12(1):19-64, 2000.
4. M. Buhrmester, T. Kwang, and S. D. Gosling. Amazons Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 2011.
5. C. F. Camerer and R. Hogarth. The effects of financial incentives in economics experiments: A review and capital-labor-production framework. Journal of Risk and Uncertainty, 19(1):7-42, 1999.
6. J. Cameron and W. D. Pierce. Reinforcement, reward, and intrinsic motivation: A meta-Analysis. Review of Educational Research, 64(3):363-423, 1994.
7. J. Chandler, P. Mueller, and G. Paolacci. Nonnäivete among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46(1):112-130, 2014.
8. R. Eisenberger and J. Cameron. Detrimental effects of reward: Reality or myth? American psychologist, 51 (11):1153, 1996.
9. B. S. Frey and F. Oberholzer-Gee. The cost of price incentives: An empirical analysis of motivation crowding-out. The American Economic Review, pages 746-755, 1997.
10. D. Gilchrist, M. Luca, and D. Malhotra. When 3+1 4: Gift structure and reciprocity in the field. Technical report, 2014. Working Paper.
11. U. Gneezy and A. Rustichini. Pay enough or dont pay at all. Quarterly Journal of Economics, August, pages 791-810, 2000.
12. C. G. Harris. Youre hired! An examination of crowdsourcing incentive models in human resource tasks. In WSDM 2011 Workshop on Crwdsourcing for Search and Data Mining, 2011.
13. R. Hertwig and A. Ortmann. Experimental practices in economics: A methodological challenge for psychologists? Behavioral and Brain Sciences, 24(3): 383-403, 2001.
14. C.-J. Ho and J. W. Vaughan. Online task assignment in crowdsourcing markets. In AAAI, 2012.
15. C.-J. Ho, S. Jabbari, and J. W. Vaughan. Adaptive task assignment for crowdsourced classification. In ICML, 2013.
16. C.-J. Ho, A. Slivkins, and J. W. Vaughan. Adaptive contract design for crowdsourcing markets: Bandit algorithms for repeated principal-Agent problems. In ACM EC, 2014.
17. J. J. Horton and L. B. Chilton. The labor economics of paid crowdsourcing. In ACM EC, 2010.
18. J. J. Horton, D. Rand, and R. Zeckhauser. The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14(3):399-425, 2011.
19. P. G. Ipeirotis, F. Provost, and J. Wang. Quality management on Amazon Mechanical Turk. In HCOMP, 2010.
20. G. D. Jenkins Jr., N. Gupta, A. Gupta, and J. D. Shaw. Are financial incentives related to performance? A meta-Analytic review of empirical research. Journal of Applied Psychology, 83(5):777-787, 1998.
21. D. Karger, S. Oh, and D. Shah. Iterative learning for reliable crowdsourcing systems. In NIPS, 2011.
22. A. Kittur, E. Chi, and B. Suh. Crowdsourcing user studies with Mechanical Turk. In CHI, 2008.
23. J.-J. Laffont and D. Martimort. The Theory of Incentives: The Principal-Agent Model. Princeton University Press, 2002.
24. E. P. Lazear. Performance pay and productivity. American Economic Review, pages 1346-1361, 2000.
25. L. Litman, J. Robinson, and C. Rosenzweig. The relationship between motivation, monetary compensation, and data quality among US-And India-based workers on Mechanical Turk. Behavioral Research Methods, 2014. To appear.
26. Q. Liu, J. Peng, and A. Ihler. Variational inference for crowdsourcing. In NIPS, 2012.
27. U.-V. Marti and H. Bunke. The iam-database: An english sentence database for offine handwriting recognition. International Journal on Document Analysis and Recognition, 5(1):39-46, 2002.
28. W. Mason and S. Suri. Conducting behavioral research on Amazons Mechanical Turk. Behavior Research Methods, 44(1):1-23, 2012.
29. W. Mason and D. J. Watts. Financial incentives and the "performance of crowds". In HCOMP, 2009.
30. J. Rogstadius, V. Kostakos, A. Kittur, B. Smus, J. Laredo, and M. Vukovic. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In ICWSM, 2011.
31. D. Schall. Service-Oriented Crowdsourcing-Architecture, Protocols and Algorithms. Springer Briefs in Computer Science. Springer, 2012.
32. H. J. Seltman. Experimental design and analysis. http://www.stat.cmu.edu/~hseltman/309/Book, 2014.
33. A. D. Shaw, J. J. Horton, and D. L. Chen. Designing incentives for inexpert human raters. In CSCW, 2011.
34. V. Sheng, F. Provost, and P. Ipeirotis. Get another label? Improving data quality using multiple, noisy labelers. In KDD, 2008.
35. C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.
36. M. Yin, Y. Chen, and Y.-A. Sun. The effects of performance-contingent financial incentives in online labor markets. In AAAI, 2013.
37. M. Yin, Y. Chen, and Y.-A. Sun. Monetary interventions in crowdsourcing task switching. In HCOMP, 2014.