To Play or Not to Play: Interactions between Response Quality and Task Complexity in Games and Paid Crowdsourcing

Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2015

Volumn , Issue , 2015, Pages 102-109

  Krause, Markus ^a   Kizilcec, René ^a  

^a STANFORD UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE;

COMPLEX TASK; CONTROLLED EXPERIMENT; DIGITAL GAMES; HIGH RESPONSE; SIMPLE++; TASK COMPLEXITY;

CROWDSOURCING;

EID: 84995473547     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1609/hcomp.v3i1.13226     Document Type: Conference Paper

References (34)

1. Anastasiou, D., and R. Gupta. 2011. “Comparison of Crowdsourcing Translation with Machine Translation.” Journal of Information Science 37 (6): 637–59.
2. Aras, Hidir, Markus Krause, Andreas Haller, and Rainer Malaka. 2010. “Webpardy: Harvesting QA by HC.” In HComp’10 Proceedings of the ACM SIGKDD Workshop on Human Computation, 49–53. New York, New York, USA: ACM Press.
3. Barrington, Luke, Douglas Turnbull, and Gert Lanckriet. 2012. “Game-Powered Machine Learning.” Proceedings of the National Academy of Sciences of the United States of America 109 (17): 6411–16.
4. Batram, Nils, Markus Krause, and Paul-olivier Dehaye. 2014. “Comparing Human and Algorithm Performance on Estimating Word-Based Semantic Similarity.” In SoHuman’14 Proceedings of the 3rd International Workshop on Social Media for Crowdsourcing and Human Computation, 131–39. Barcelona, Spain: Springer Berlin Heidelberg.
5. Bigham, JP, Chandrika Jayant, Hanjie Ji, and Greg Little. 2010. “Vizwiz: Nearly Real-Time Answers to Visual Questions.” In UIST’10 Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, 333–42. ACM Press.
6. Cooper, Seth, Firas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, Andrew Leaver-Fay, David Baker, and Zoran Popović. 2010. “Predicting Protein Structures with a Multiplayer Online Game.” Nature 466 (7307): 756–60.
7. Dow, Steven, Anand Kulkarni, Scott Klemmer, and Björn Hartmann. 2012. “Shepherding the Crowd Yields Better Work.” Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work - CSCW’12. New York, New York, USA: ACM Press, 1013.
8. Efron, Bradley, and Robert Tibshirani. 1986. “Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy.” Statistical Science 2 (1): 54–75.
9. Findley, MG, MC Gleave, RN Morello, and DL Nielson. 2012. “Extrinsic, Intrinsic, and Social Incentives for Crowdsourcing Development Information in Uganda: A Field Experiment.” In WebSci’13 Proceedings of the 5th Annual ACM Web Science Conference, 326–35. Provo, UT, USA: ACM Press.
10. Harwell, M. R., E. N. Rubinstein, W. S. Hayes, and C. C. Olds. 1992. “Summarizing Monte Carlo Results in Methodological Research: The One- and Two-Factor Fixed Effects ANOVA Cases.” Journal of Educational and Behavioral Statistics.
11. Holm, Sture. 1979. “A Simple Sequentially Rejective Multiple Test Procedure.” Scandinavian Journal of Statistics 6 (2): 65–70.
12. Hsieh, Chu-cheng, Christopher Moghbel, Jianhong Fang, and Junghoo Cho. 2013. “Experts vs the Crowd: Examining Popular News Prediction Perfomance on Twitter.” In Proceedings of ACM KDD Conference. Chicago, USA: ACM Press.
13. Jeong, JW, MR Morris, Jaime Teevan, and Dan Liebling. 2013. “A Crowd-Powered Socially Embedded Search Engine.” In Proceedings of ICWSM 2013, 263–72.
14. Kandasamy, DM, K Curtis, Armando Fox, and David Patterson. 2012. “Diversity within the Crowd.” In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work - CSCW’12, 115–18. Seatle, Washington, USA: ACM Press.
15. Kilian, Niklas, Markus Krause, Nina Runge, and Jan Smeddinck. 2012. “Predicting Crowd-Based Translation Quality with Language-Independent Feature Vectors.” In HComp’12 Proceedings of the AAAI Workshop on Human Computation, 114–15. Toronto, ON, Canada: AAAI Press.
16. Kincaid, J Peter, Robert P Fishburne, Richard L Rogers, and Brad S Chissom. 1975. Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel. Naval Air Station Memphis-Millington, TN, USA: National Technical Information Service, Springfield, Virginia.
17. Krause, Markus. 2014. Homo Ludens in the Loop: Playful Human Computation Systems. Hamburg, Germany: tredition GmbH, Hamburg.
18. Krause, Markus, Aneta Takhtamysheva, Marion Wittstock, and Rainer Malaka. 2010. “Frontiers of a Paradigm.” In HComp’10 Proceedings of the ACM SIGKDD Workshop on Human Computation, 22–25. New York, New York, USA: ACM Press.
19. Krippendorff, Klaus. 1970. “Estimating the Reliability, Systematic Error and Random Error of Interval Data.” Educational and Psychological Measurement 30 (61): 61–70.
20. Mao, Andrew, Yiling Chen, Eric Horvitz, Megan E Schwamb, Chris J Lintott, and Arfon M Smith. 2013. “Volunteering Versus Work for Pay: Incentives and Tradeoffs in Crowdsourcing.” In HComp’13 Proceedings of the AAAI Conference on Human Computation, 94–102. Palm Springs, CA, USA: AAAI Press.
21. Musat, CC, and Boi Faltings. 2013. “A Novel Human Computation Game for Critique Aggregation.” In Workshops at the Twenty-Seventh AAAI Conference on Artificial Intelligence, 86–88. AAAI Press.
22. Noll, Paul, and Bernice Noll. 2006. “3000 Most Commonly Used Words in the English Language (USA).” “Clear English” a Book on Teaching and Learning English. Pleasent Hill, OR, USA. http://www.paulnoll.com/Books/Clear-English/English-3000-common-words.html.
23. Oosten, Philip Van, and V Hoste. 2011. “Readability Annotation: Replacing the Expert by the Crowd.” In Proceedings of the 6th Workshop on Innovative Use of NLP for Building Educational Applications, 120–29. Association for Computational Linguistics.
24. Pe-Than, Ei Pa Pa, Dion Hoe-Lian Goh, and Chei Sian Lee. 2014. “A Typology of Human Computation Games: An Analysis and a Review of Current Games.” Behaviour & Information Technology 00 (00). Taylor & Francis: 1–16..
25. Prestopnik, Nathan, Kevin Crowston, and Jun Wang. 2012. “Exploring Data Quality in Games With a Purpose Theory: Gamification and Games With a Purpose.” In Proceedings of the 2012 iConference, 1–15. ACM Press.
26. Rashtchian, C, and P Young. 2010. “Collecting Image Annotations Using Amazon’s Mechanical Turk.” In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, 139–47. Stroudsburg, PA, USA.
27. Schmider, Emanuel, Matthias Ziegler, Erik Danay, Luzi Beyer, and Markus Bühner. 2010. “Is It Really Robust?: Reinvestigating the Robustness of ANOVA against Violations of the Normal Distribution Assumption.” Methodology 6 (4): 147–51.
28. Simko, J, M Tvarozek, and M Bielikova. 2011. “Little Search Game: Term Network Acquisition via a Human Computation Game.” In Proceedings of the 22nd International Conference on Hypertext HT’11, 57–61. Eindhoven, The Netherlands: ACM Press.
29. Sjöberg, L. 2009. “Are All Crowds Equally Wise? A Comparison of Political Election Forecasts by Experts and the Public.” Journal of Forecasting.
30. Snow, Rion, Brendan O Connor, Daniel Jurafsky, Andrew Y Ng, Dolores Labs, and Capp St. 2008. “Cheap and Fast — But Is It Good ? Evaluating Non-Expert Annotations for Natural Language Tasks.” In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 254–63. Association for Computational Linguistics.
31. Thisone, Musat, Claudiu-Cristian, Alireza Ghasemi, and Boi Faltings. 2012. “Sentiment Analysis Using a Novel Human Computation Game.” In Proceedings of the 3rd Workshop on the People’s Web Meets NLP: Collaboratively Constructed Semantic Resources and Their Applications to NLP, 1–9. Association for Computational Linguistics.
32. Von Ahn, Luis, and Laura Dabbish. 2004. “Labeling Images with a Computer Game.” In CHI’04 Proceedings of the 22nd International Conference on Human Factors in Computing Systems, 319–26. Vienna, Austria: ACM Press.
33. Von Ahn, Luis, and Laura Dabbish. 2008. “Designing Games with a Purpose.” Communications of the ACM 51 (8). ACM: 58–67.
34. Von Ahn, Luis, Mihir Kedia, and Manuel Blum. 2006. “Verbosity: A Game for Collecting Common-Sense Facts.” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 22:78. Montreal, Quebec, Canada: ACM Press.