Risks and rewards of crowdsourcing marketplaces

Handbook of Human Computation

Volumn , Issue , 2013, Pages 377-392

  Chandler, Jesse ^a   Paolacci, Gabriele ^b   Mueller, Pam ^c  

^a UNIVERSITY OF MICHIGAN   (United States)

^b ERASMUS UNIVERSITY ROTTERDAM   (Netherlands)

^c PRINCETON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85014799833     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1007/978-1-4614-8806-4_30     Document Type: Chapter

References (87)

1. Akerlof GA (1970) The market for" lemons": quality uncertainty and the market mechanism. Q J Econ 84:488-500
2. Allio RJ (2004) CEO interview: the InnoCentive model of open innovation. Strategy Leadersh 32(4):4-9
3. Anderson LR, Holt CA (1997) Information cascades in the laboratory. Am Econ Rev 87:847-862
4. Bao J, Sakamoto Y, Nickerson JV (2011) Evaluating design solutions using crowds. In: Proceedings of the 17th Americas conference on information systems, Detroit, MI, USA
5. Becker GS, Murphy KM (1992) The division of labor, coordination costs, and knowledge. Q J Econ 107(4):1137-1160
6. Berinsky AJ, Huber GA, Lenz GS (2012) Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Polit Anal 20:351-368. doi: 10.1093/pan/mpr057
7. Bernstein MS, Little G, Miller RC, Hartmann B, Ackerman MS, Karger DR, Crowell D, Panovich K (2010) Soylent: a word processor with a crowd inside. In: Proceeding UIST 2010, ACM Press, pp 313-322
8. Bigham JP, Jayant C, Ji H, Little G, Miller A, Miller RC,... Yeh T (2010) VizWiz: nearly real-time answers to visual questions. In: Proceedings of the 23nd annual ACM symposium on user interface software and technology, ACM, New York, pp 333-342
9. Case SM, Swanson DB (2001) Constructing written test questions for the basic and clinical sciences, 3rd edn. National Board of Medical Examiners, Philadelphia
10. Chandler D, Horton J (2011) Labor allocation in paid crowdsourcing: experimental evidence on positioning, nudges and prices. In: Workshops at the twenty-fifth AAAI conference on artificial intelligence. AAAI Press, Menlo Park, California
11. Chandler D, Kapelner A (2013) Breaking monotony with meaning: motivation in crowdsourcing markets. J Econ Behav Organ 90:123-133
12. Chandler J, Mueller P, Paolacci G (in press) Methodological concerns and advanced uses of Amazon mechanical Turk in psychological research. Manuscript submitted for publication
13. Chilton LB, Horton JJ, Miller RC, Azenkot S (2010) Task search in a human computation market. In:Proceedings of the ACM SIGKDD workshop on human computation, ACM, New York, pp 1-9
14. Chua CC, Milosavljevic M, Curran JR (2009) A sentiment detection engine for internet stock message boards. In Pizzato LA, Schwitter R (eds) Proceedings of the Australasian language technology association workshop 2009, Sydney, pp 89-93
15. Collins A, Joseph D, Bielaczyc K (2004) Design research: theoretical and methodological issues. J Learn Sci 13(1):15-42
16. Cooper S, Khatib F, Treuille A, Barbero J, Lee J, Beenen M, Leaver-Fay A, Baker D, Popović Z (2010) Predicting protein structures with a multiplayer online game. Nature 466(7307):756-760
17. Couper M (2008) Designing effective web surveys. Cambridge University Press, New York
18. Davis LE (1965) Pacing effects on manned assembly lines. Int J Prod Res 4(3):171-184
19. Dominowski RL, Dallob PI (1995) Insight and problem solving. In: Sternberg RJ, Davidson JE (eds) The nature of insight. MIT Press, Cambridge, pp 33-62
20. Elson DK, McKeown KR (2010) Automatic attribution of quoted speech in literary narrative. In:Proceedings of the twenty-fourth AAAI conference on artificial intelligence. The AAAI Press, Menlo Park, pp 1013-1019
21. Estellés-Arolas E, González-Ladrón-de-Guevara F (2012) Towards an integrated crowdsourcing definition. J Info Sci 38(2):189-200
22. Galton F 1907 Vox populi. Nature 75:450-451
23. Gneezy U, Meier S, Rey-Biel P (2011) When and why incentives (don't) work to modify behavior. J Econ Perspect 25:191-209
24. Goldin G, Darlow A (2013) TurkGate (Version 0.4.0) [Software]. Available from http://gideongoldin. github.com/TurkGate/ )
25. Goodman JK, Cryder CE, Cheema A (2012) Data collection in a flat world: the strengths and weaknesses of mechanical Turk samples. J Behav Decis Making 26:213-224
26. Grady C, Lease M (2010) Crowdsourcing document relevance assessment with mechanical Turk. In: Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon's mechanical Turk. Association for Computational Linguistics, pp 172-179
27. Grice HP (1989) Studies in the way of words. Harvard University Press, Cambridge
28. Gruenstein A, McGraw I, Sutherland A (2009) A self-transcribing speech corpus: collecting continuous speech with an online educational game. In: Proceedings of the speech and language technology in education (SLaTE) workshop. Warwickshire
29. Hayes AF, Krippendorff K (2007) Answering the call for a standard reliability measure for coding data. Commun Methods Meas 1:77-89. doi: 10.1080/19312450709336664
30. Horton JJ (2010) Online labor markets. Springer Berlin Heidelberg, pp 515-522
31. Horton JJ (2011) The condition of the Turking class: are online employers fair and honest? Econ Lett 111(1):10-12
32. Horton JJ, Chilton LB (2010) The labor economics of paid crowdsourcing. In: Proceedings of the 11th ACM conference on electronic commerce, ACM, pp 209-218
33. Hosseini M, Cox I, Milić-Frayling N, Kazai G, Vinay V (2012) On aggregating labels from multiple crowd workers to infer relevance of documents. Adv Inf Retr 182-194
34. Hsieh G, Kraut RE, Hudson SE (2010) Why pay?: exploring how financial incentives are used for question & answer. In: Proceedings of the 28th international conference on human factors in computing systems, pp 305-314. doi: 10.1145/1753326.1753373
35. Hullman J, Adar E, Shah P (2011) The impact of social information on visual judgments. In:Proceedings of the 2011 annual conference on human factors in computing systems, ACM, New York, pp 1461-1470
36. Ipeirotis P (2010) Demographics of mechanical Turk. CeDER-10-01 working paper, New York University
37. Ipeirotis PG, Horton JJ (2011) The need for standardization in crowdsourcing. CHI
38. Jung HJ, Lease M (2011) Improving consensus accuracy via Z-score and weighted voting. In: Proceedings of the 3rd Human Computation Workshop (HCOMP) at AAAI Press, Menlo Park, California
39. Kahneman D (2011) Thinking, fast and slow. Farrar, Straus and Giroux, New York
40. Kapelner A, Chandler D (2010) Preventing satisficing in online surveys: a 'kapcha' to ensure higher quality data. In: The world's first conference on the future of distributed work, San Francisco, CA (CrowdConf2010)
41. Kaufmann N, Schulze T, Veit D (2011) More than fun and money. worker motivation in crowdsourcing- a study on mechanical turk. In: Proceedings of the seventeenth Americas conference on information systems, Detroit
42. Kazai G, Milic-Frayling N (2009) On the evaluation of the quality of relevance assessments collected through crowdsourcing. In: SIGIR 2009 workshop on the future of IR evaluation, Boston, MA, p 21
43. Kazai G, Kamps J, Milic-Frayling N (2012) The face of quality in crowdsourcing relevance labels:Demographics, personality and labeling accuracy. In: Proceedings of the 21st ACM international conference on Information and knowledge, ACM, New York, pp 2583-2586
44. Khanna S, Ratan A, Davis J, Thies W (2010) Evaluating and improving the usability of mechanical turk for low-income workers in India. In: Proceedings of the first ACM symposium on computing for development, ACM, New York, p 12
45. Kittur A, Chi EH, Suh, B (2008) Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, pp 453-456
46. Kittur A, Nickerson J, Bernstein M, Gerber E, Shaw A, Zimmerman J,... Horton J (2013) The future of crowd work. In: Sixteenth ACM conference on Computer Supported Coooperative Work (CSCW 2013), Forthcoming
47. Krippendorff K (2004) Reliability in content analysis. Hum Commun Res, 30(3):411-433
48. Krosnick JA (2006) Response strategies for coping with the cognitive demands of attitude measures in surveys. Appl Cogn Psychol 5(3):213-236
49. Krug S (2009) Don't make me think: a common sense approach to web usability. New Riders, Berkeley, CA
50. Lakhani KR (2008) InnoCentive. com (A). Harvard Business School Case, 608-170
51. Lane I, Weibel A, Eck M, Rottmann K (2010) Tools for collecting speech corpora via Mechanical-Turk. In: Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon's mechanical turk, Association for Computational Linguistics, Stroudsbug, PA, pp 184-187
52. Lau T, Drews C, Nichols J (2009) Interpreting written how-to instructions. In: Kitano H (ed) Proceedings of the 21st international joint conference on artificial intelligence, Morgan Kaufmann, San Francisco, pp 1433-1438
53. Li B, Liu Y, Agichtein E (2008) CoCQA: co-training over questions and answers with an application to predicting question subjectivity orientation. In: Proceedings of the 2008 conference on empirical methods in natural language processing. Association for Computational Linguistics, Stroudsburg. doi: 10.3115/1613715.1613836, pp 937-946
54. Lintott CJ, Schawinski K, Slosar A, Land K, Bamford S, Thomas D,... Vandenberg J (2008) Galaxy zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey. Mon Not R Astron Soc 389(3):1179-1189
55. Marge M, Banerjee S, Rudnicky AI (2010) Using the Amazon mechanical turk for transcription of spoken language. In: Acoustics Speech and Signal Processing (ICASSP), 2010 IEEE international conference on (5270-5273), Institute of Electronics and Electrical Engineers, Washington, DC. doi: 10.1109/ICASSP.2010.5494979
56. Mason W, Suri S (2012) Conducting behavioral research on Amazon's Mechanical Turk. Behav Res Methods 44(1):1-23
57. Mason W, Watts DJ (2009) Financial incentives and the performance of crowds. In: Proceedings of the ACM SIGKDD workshop on human computation, ACM, New York, pp 77-85
58. Molla D, Santiago-Martinez ME (2011) Development of a corpus for evidence based medicine summarisation. In: Proceedings of Australasian language technology association workshop, Australasian Language Technology Association, Melbourne, pp 86-94
59. Nelson L, Held C, Pirolli P, Hong L, Schiano D, Chi EH (2009) With a little help from my friends:examining the impact of social annotations in sensemaking tasks. In: Proceedings of the 27th international conference on human factors in computing systems, ACM, New York, pp 1795-1798. doi: 10.1145/1518701.1518977
60. Nickerson JV, Sakamoto Y, Yu L (2011) Structures for creativity: the crowdsourcing of design. In:CHI workshop on crowdsourcing and human computation, pp 1-4
61. Open Science Collaboration (2013) The reproducibility project: a model of large-scale collaboration for empirical research on reproducibility. In: Stodden V, Leisch F, Peng R (eds) Implementing reproducible computational research (A Volume in The R Series). Taylor and Francis, New York
62. Paolacci G, Chandler J, Ipeirotis P (2010) Running experiments on Amazon mechanical turk. Judgm Decis Making 5:411-419
63. Pe'er E, Paolacci G, Chandler J, Mueller P (2012) Screening participants from previous studies on Amazon mechanical turk and qualtrics. Available at SSRN 2100631
64. Prelec D (2004) A bayesian truth serum for subjective data. Science 306(5695):462-466
65. Pontin J (2007) Artificial intelligence, with help from the humans. New York Times, 25. Retrieved from http://www.nytimes.com/2007/03/25/business/yourmoney/25Stream.html
66. Rand DG (2012) The promise of mechanical turk: how online labor markets can help theorists run behavioral experiments. J Theor Biol 299:172-179
67. Resnick P, Kuwabara K, Zeckhauser R, Friedman E (2000) Reputation systems. Commun ACM 43(12):45-48
68. Rochet JC, Tirole J (2003) Platform competition in two-sided markets. J Eur Econ Assoc 1(4):990-1029
69. Rogstadius J, Kostakos V, Kittur A, Smus B, Laredo J, Vukovic M (2011) An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In ICWSM
70. Sayeed AB, Rusk B, Petrov M, Nguyen HC, Meyer TJ, Weinberg A (2011) Crowdsourcing syntactic relatedness judgements for opinion mining in the study of information technology adoption. In: Proceedings of the 5th ACL-HLT workshop on language technology for cultural heritage, social sciences, and humanities, Association for Computational Linguistics, Stroudsburg, pp 69-77
71. Schwarz N (1999) Self-reports: how the questions shape the answers. Am psychol 54(2):93
72. Shapiro DN, Chandler J, Mueller PA (2013) Using mechanical turk to study clinical populations. Clin Psychol Sci 1:213-220
73. Shaw AD, Horton JJ, Chen DL (2011) Designing incentives for inexpert human raters. In:Proceedings of the ACM 2011 conference on computer supported cooperative work, ACM, New York, pp 275-284
74. Sheerman-Chase T, Ong EJ, Bowden R (2011) Cultural factors in the regression of non-verbal communication perception. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain, pp 1242-1249
75. Silberman M, Irani L, Ross J (2010) Ethics and tactics of professional crowdwork. XRDS Crossroads ACM Mag Stud 17(2):39-43
76. Simon HA (1972) Theories of bounded rationality. Decis Organ 1:161-176
77. Suri S, Goldstein DG, Mason WA (2011) Honesty in an online labor market. In: von Ahn L, Ipeirotis PG (eds) Papers from the 2011 AAAI workshop. AAAI Press, Menlo
78. Park Tang W, Lease M (2011) Semi-supervised consensus labeling for crowdsourcing. In: Proceedings of the ACM SIGIR workshop on crowdsourcing for information retrieval, ACM, New York
79. Tetlock P (2005) Expert political judgment: how good is it? How can we know? Princeton University Press, Princeton
80. Tetreault JR, Filatova E, Chodorow M (2010) Rethinking grammatical error annotation and evaluation with the Amazon mechanical turk. In: Proceedings of the NAACL HLT 2010 fifth workshop on innovative use of NLP for building educational applications, Association for Computational Linguistics, pp 45-48
81. Tversky A, Kahneman D (1974) Judgment under uncertainty: heuristics and biases. Science 211(January):453-458
82. von Ahn L (2006) Games with a purpose. Computer 39(6):92-94
83. von Ahn L, Maurer B, McMillen C, Abraham D, Blum M (2008) reCAPTCHA: human-based character recognition via web security measures. Science 321(5895):1465-1468
84. Wenger E (1998) Communities of practice: learning, meaning, and identity. Cambridge University Press, Cambridge
85. Winchester S (2004) The meaning of everything: The story of the Oxford English Dictionary.
86. Oxford University Press Yu L, Nickerson JV (2011) Cooks or cobblers?: crowd creativity through combination. In:Proceedings of the 2011 annual conference on human factors in computing systems, ACM, New York, pp 1393-1402
87. Zhou DX, Resnick P, Mei Q (2011) Classifying the political leaning of news articles and users from user votes. In: Proceedings of the fifth international AAAI conference on weblogs and social media. The AAAI Press, Menlo Park, pp 417-424