Designing incentives for inexpert human raters

Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW

Volumn , Issue , 2011, Pages 275-284

  Shaw, Aaron D ^a   Horton, John J ^b   Chen, Daniel L ^c  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b HARVARD UNIVERSITY   (United States)

^c DUKE UNIVERSITY   (United States)

Author keywords

Amazon mechanical turk; Content analysis; Crowdsourcing; Experimentation; Human computation; Search; Sociology

Indexed keywords

CROWDSOURCING; EMPLOYMENT; SOCIOLOGY;

AMAZON MECHANICAL TURKS; CONTENT ANALYSIS; EXPERIMENTATION; HUMAN COMPUTATION; SEARCH;

COMMERCE;

EID: 79955134548     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1958824.1958865     Document Type: Conference Paper

References (31)

1. World values survey 1981-2008 official aggregate v.20090901.World Values Survey Association (www.worldvaluessurvey.org), Aggregate File Producer: ASEP/JDS, Madrid, 2009.
2. J. Antin and A. Shaw. Social Desirability Bias in Reports of Motivation on Mturk. In CSCW Horizons Workshop, 2011.
3. D. J. Benjamin and J. M. Shapiro. Thin-slice forecasts of gubernatorial elections. Review of Economics and Statistics (Forthcoming), 2009.
4. Y. Benkler and A. Shaw. A tale of two blogospheres: Discursive practices on the left and right. Berkman Center for Internet and Society Working Paper Series, 2010.
5. D. Chandler and A. Kapelner. Breaking monotony with meaning: Motivation in crowdsourcing markets. University of Chicago mimeo, 2010.
6. D. L. Chen. Markets and Morality: How Does Competition Affect Moral Judgment? Working Paper, Duke University School of Law, 2010.
7. D. L. Chen and J. J. Horton. The wages of paycuts: Evidence from a field experiment. Working Paper, 2009.
8. J. Cohen. A coefficient of agreement for nominal scales. Educational & Psychological Measurement, 20(1):37-46, 1960.
9. J. Downs, M. Holbrook, S. Sheng, and L. Cranor. Are your participants gaming the system?: screening mechanical turk workers. In Proceedings of the 28th international conference on Human factors in computing systems, pages 2399-2402. ACM, 2010.
10. J. Duarte, S. Siegel, and L. A. Young. Trust and credit. SSRN eLibrary, 2009.
11. D. A. Freedman. On regression adjustments in experiments with several treatments. The Annals of Applied Statistics, 2(1):176-196, 2008.
12. D. A. Freedman. Randomization does not justify logistic regression. Statistical Science, 23(2):237-249, 2008.
13. E. Hargittai. An update on survey measures of Web-Oriented digital literacy. Social Science Computer Review, 27(1):130-137, 2009.
14. D. Hopkins and G. King. A method of automated nonparametric content analysis for social science. American Journal of Political Science, 54(1):229-247, 2010.
15. J. Horton and L. Chilton. The labor economics of paid crowdsourcing. Proceedings of the ACM Conference on Electronic Commerce 2010, 2010.
16. J. J. Horton, D. Rand, and R. J. Zeckhauser. The Online Laboratory: Conducting Experiments in a Real Labor Market. SSRN eLibrary, 2010.
17. J. C. Hsu. Multiple comparisons: Theory and Methods. CRC Press, 1996.
18. B. A. Huberman, D. Romero, and F. Wu. Crowdsourcing, attention and productivity. Journal of Information Science (in press), 2009.
19. P. Ipeirotis. Demographics of mechanical turk. New York University Working Paper, 2010.
20. A. Kapelner and D. Chandler. Preventing Satisficing in Online Surveys. In Proceedings of 2010 CrowdConf, 2010.
21. A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies with mechanical turk. Proceedings of Computer Human Interaction (CHI-2008), 2008.
22. K. H. Krippendorff. Content Analysis: An Introduction to Its Methodology. Sage Publications, Inc, 2nd edition, 2003.
23. G. Little, L. B. Chilton, R. Miller, and M. Goldman. Turkit: Tools for iterative tasks on mechanical turk. Working Paper, MIT, 2009.
24. W. Mason and D. J. Watts. Financial incentives and the'performance of crowds'. In Proc. ACM SIGKDD Workshop on Human Computation (HCOMP), 2009.
25. D. Prelec. A bayesian truth serum for subjective data. Science, 306(5695):462-466, Oct. 2004. (Pubitemid 39372440)
26. R. Rosenthal and D. B. Rubin. Multiple contrasts and ordered bonferroni procedures. Journal of Educational Psychology, 76(6):1028-1034, 1984.
27. J. P. Shaffer. Multiple hypothesis testing. Annual Review of Psychology, 46:561-584, 1995.
28. V. S. Sheng, F. Provost, and P. G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. Knowledge Discovery and Data Minding 2008 (KDD-2008), 2008.
29. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng. Cheap and fast-but is it good? evaluating non-expert annotations for natural language tasks. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2008), 2008.
30. L. Von Ahn and L. Dabbish. Labeling images with a computer game. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 319-326. ACM, 2004.
31. H. Wickham. ggplot2: An implementation of the grammar of graphics. R package version 0.7, URL: http://CRAN.R-project.org/package=ggplot2, 2008.