Quality through flow and immersion: Gamifying crowdsourced relevance assessments

SIGIR'12 - Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2012, Pages 871-880

  Eickhoff, Carsten ^a   Harris, Christopher G ^b   De Vries, Arjen P ^b   Srinivasan, Padmini ^c  

^a DELFT UNIVERSITY OF TECHNOLOGY   (Netherlands)

^b UNIVERSITY OF IOWA   (United States)

^c CWI   (Netherlands)

Author keywords

clustering; crowdsourcing; gamification; relevance assessments; serious games

Indexed keywords

CLUSTERING; CROWDSOURCING; GAMIFICATION; RELEVANCE ASSESSMENTS; SERIOUS GAMES;

COMMERCE; PROFITABILITY;

INFORMATION RETRIEVAL;

EID: 84866626350     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2348283.2348400     Document Type: Conference Paper

References (44)

1. Gambit - Payment Solutions for Virtual Currency. http://www.getgambit. com/, 2011.
2. O. Alonso and R. Baeza-Yates. Design and implementation of relevance assessments using crowdsourcing. In ECIR, 2011.
3. O. Alonso and M. Lease. Crowdsourcing 101: putting the WSDM of crowds to work for you. In WSDM, 2011.
4. O. Alonso, D.E. Rose, and B. Stewart. Crowdsourcing for relevance evaluation. In ACM SIGIR Forum, 2008.
5. B. Baldwin and B. Carpenter. LingPipe. Available from World Wide Web: http://alias-i. com/lingpipe, 2003.
6. C. Buckley and E.M. Voorhees. Retrieval evaluation with incomplete information. In SIGIR 2004. ACM.
7. J. Callan, M. Hoy, C. Yoo, and L. Zhao. Clueweb09 data set. http://boston.lti.cs.cmu.edu, 2009.
8. B. Carterette, J. Allan, and R. Sitaraman. Minimal test collections for retrieval evaluation. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 268-275. ACM, 2006.
9. B. Carterette, V. Pavlu, H. Fang, and E. Kanoulas. Million Query track 2009 overview. In Proceedings of TREC, volume 9, 2009.
10. V.R. Carvalho, M. Lease, and E. Yilmaz. Crowdsourcing for search evaluation. In ACM SIGIR Forum, 2011.
11. CrowdFlower. Crowdsourcing - Labor on demand. http://crowdflower.com/, 2012.
12. M. Csikszentmihalyi. Flow: The psychology of optimal experience. Harper Perennial, 1991.
13. C. Eickhoff and A. de Vries. How crowdsourcable is your task. In WSDM Workshop on Crowdsourcing for Search and Data Mining (CSDM), pages 11-14, 2011.
14. C. Eickhoff and A.P. de Vries. Increasing Cheat Robustness of Crowdsourcing Tasks. Information Retrieval To appear, 2012.
15. C. Grady and M. Lease. Crowdsourcing document relevance assessment with Mechanical Turk. In NAACL HLT Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, 2010.
16. C. Harris. You're Hired! An Examination of Crowdsourcing Incentive Models in Human Resource Tasks. In WSDM Workshop on Crowdsourcing for Search and Data Mining (CSDM), pages 15-18, 2011.
17. M. Hirth, T. Hoßfeld, and P. Tran-Gia. Cheat-Detection Mechanisms for Crowdsourcing. University of Würzburg, Tech. Rep, 2010.
18. P. Ipeirotis. Crowdsourcing using Mechanical Turk: quality management and scalability. In Proceedings of the 8th International Workshop on Information Integration on the Web: in conjunction with WWW 2011. ACM, 2011.
19. T. Joachims. Optimizing search engines using clickthrough data. In SIGKDD, 2002.
20. G. Kazai. In search of quality in crowdsourcing for search engine evaluation. ECIR, 2011.
21. G. Kazai, J. Kamps, and N. Milic-Frayling. Worker Types and Personality Traits in Crowdsourcing Relevance Labels. 2011.
22. G. Kazai and N. Milic-Frayling. On the evaluation of the quality of relevance assessments collected through crowdsourcing. In SIGIR 2009 Workshop on the Future of IR Evaluation, 2009.
23. C.H. Ko, J.Y. Yen, C.C. Chen, S.H. Chen, and C.F. Yen. Gender differences and related factors affecting online gaming addiction among taiwanese adolescents. The Journal of nervous and mental disease, 2005.
24. J. Lampel and A. Bhalla. The role of status seeking in online communities: Giving the gift of experience. Journal of Computer-Mediated Communication, 2007.
25. M. Lease and G. Kazai. Overview of the TREC 2011 Crowdsourcing Track (Conference Notebook). 2011.
26. M. Lease and E. Yilmaz. Crowdsourcing for information retrieval. In ACM SIGIR Forum, number 2. ACM, 2012.
27. H. Ma, R. Chandrasekar, C. Quirk, and A. Gupta. Improving search engines using human computation games. In CIKM 2009.
28. C.C. Marshall and F.M. Shipman. The ownership and reuse of visual media. JCDL, 2011.
29. J. McGonigal. Reality is broken: Why games make us better and how they can change the world. Penguin Pr, 2011.
30. David Pennock. The Wisdom of the Probability Sports Crowd. http://blog.oddhead.com/2007/01/04/the-wisdom-of-the-probabilitysports-crowd/, 2007.
31. C. Richards. Teach the world to twitch: An interview with Marc Prensky, CEO and founder Games2train. com. Futurelab, 2003.
32. F. Scholer, A. Turpin, and M. Sanderson. Quantifying test collection quality based on the consistency of relevance judgements. In SIGIR 2011.
33. R. Snow, B. O'Connor, D. Jurafsky, and A.Y. Ng. Cheap and fast - but is it good?: evaluating non-expert annotations for natural language tasks. In EMNLP, 2008.
34. J. Surowiecki, M.P. Silverman, et al. The wisdom of crowds. American Journal of Physics, 2007.
35. Amazon Mechanical Turk. Artificial Artificial Intelligence. http://mturk.com, 2012.
36. J. Urbano, M. Marrero, D. Martín, J. Morato, K. Robles, and J. Lloréns. The University Carlos III of Madrid at TREC 2011 Crowdsourcing Track: Notebook Paper. 2011.
37. L. von Ahn and L. Dabbish. Labeling images with a computer game. In SIGCHI, 2004.
38. L. von Ahn and L. Dabbish. Designing games with a purpose. Communications of the ACM, 2008.
39. L. von Ahn, S. Ginosar, M. Kedia, and M. Blum. Improving image search with phetch. In ICASSP, 2007.
40. L. von Ahn, M. Kedia, and M. Blum. Verbosity: a game for collecting common-sense facts. In SIGCHI 2006.
41. L. von Ahn, R. Liu, and M. Blum. Peekaboom: a game for locating objects in images. In SIGCHI 2006.
42. E. Voorhees. The philosophy of information retrieval evaluation. In Evaluation of cross-language information retrieval systems, 2002.
43. E. Voorhees, D.K. Harman, National Institute of Standards, and Technology (US). TREC: Experiment and evaluation in information retrieval. MIT press USA, 2005.
44. P. Welinder, S. Branson, S. Belongie, and P. Perona. The multidimensional wisdom of crowds. In NIPS, 2010.