Crowdsourcing for book search evaluation: Impact of HIT design on comparative system ranking

SIGIR'11 - Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2011, Pages 205-214

  Kazai, Gabriella ^a   Kamps, Jaap ^b   Koolen, Marijn ^b   Milic Frayling, Natasa ^a  

^a MICROSOFT RESEARCH   (United Kingdom)

^b UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

Book search; Crowdsourcing quality; Prove it

Indexed keywords

CROWDSOURCING; INFORMATION RETRIEVAL; SEARCH ENGINES;

BOOK SEARCH; DESIGN AND IMPLEMENTATIONS; HUMAN INTELLIGENCE; ORDERING STRATEGY; PERFORMANCE METRICS; PROVE IT; RELEVANCE JUDGMENT; SYSTEM RANKINGS;

QUALITY CONTROL;

EID: 80052132873     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2009916.2009947     Document Type: Conference Paper

References (30)

1. O. Alonso and R. A. Baeza-Yates. Design and implementation of relevance assessments using crowdsourcing. In Advances in Information Retrieval - 33rd European Conference on IR Research (ECIR 2011), volume 6611 of LNCS, pages 153-164. Springer, 2011.
2. O. Alonso and S. Mizzaro. Can we get rid of TREC assessors? using Mechanical Turk for relevance assessment. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 15-16, 2009.
3. O. Alonso, D. E. Rose, and B. Stewart. Crowdsourcing for relevance evaluation. SIGIR Forum, 42:9-15, November 2008.
4. C. W. Cleverdon. The Cranfield tests on index language devices. Aslib, 19:173-192, 1967.
5. G. V. Cormack, C. R. Palmer, and C. L. A. Clarke. Efficient construction of large test collections. In Proceedings of the 21st annual international ACM SIGIR conference, SIGIR '98, pages 282-289, 1998.
6. J. S. Downs, M. B. Holbrook, S. Sheng, and L. F. Cranor. Are your participants gaming the system?: screening mechanical turk workers. In CHI, pages 2399-2402. ACM, 2010.
7. C. Grady and M. Lease. Crowdsourcing document relevance assessment with mechanical turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, CSLDAMT '10, pages 172-179, 2010.
8. J. Howe. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business. Crown Publishing Group, 2008.
9. J. Kamps, S. Geva, C. Peters, T. Sakai, A. Trotman, and E. Voorhees. Report on the SIGIR 2009 workshop on the future of IR evaluation. SIGIR Forum, 43(2):17-36, December 2009.
10. A. Kapelner and D. Chandler. Preventing satisficing in online surveys: A 'kapcha' to ensure higher quality data. In The World's First Conference on the Future of Distributed Work (CrowdConf2010), 2010.
11. G. Kazai. In search of quality in crowdsourcing for search engine evaluation. In Advances in Information Retrieval - 33rd European Conference on IR Research (ECIR 2011), volume 6611 of LNCS, pages 165-176. Springer, 2011.
12. G. Kazai, N. Milic-Frayling, and J. Costello. Towards methods for the collective gathering and quality control of relevance assessments. In Proceedings of the 32nd international ACM SIGIR conference, SIGIR '09, pages 452-459, 2009. ACM.
13. G. Kazai, M. Koolen, J. Kamps, A. Doucet, and M. Landoni. Overview of the INEX 2010 book track: Scaling up the evaluation using crowdsourcing. In Comparative Evaluation of Focused Retrieval: 9th International Workshop of the Initiative for the Evaluation of XML Retrieval (INEX 2010), LNCS. Springer, 2011.
14. A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies with mechanical turk. In Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, CHI '08, pages 453-456, 2008. ACM.
15. J. Le, A. Edmonds, V. Hester, and L. Biewald. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation, pages 21-26, 2010.
16. W. Mason and D. J. Watts. Financial incentives and the "performance of crowds". In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP '09, pages 77-85, 2009. ACM.
17. S. Nowak and S. Rüger. How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation. In Proceedings of the international conference on Multimedia information retrieval, MIR '10, pages 557-566, 2010. ACM.
18. A. N. Oppenheim. Questionnaire Design, Interviewing and Attitude Measurement. Pinter Publishers, London, 1992.
19. B. Piwowarski and M. Lalmas. Providing consistent and exhaustive relevance assessments for XML retrieval evaluation. In Proceedings of the thirteenth ACM international conference on Information and knowledge management, CIKM'04, pages 361-370, 2004. ACM. (Pubitemid 40673475)
20. A. J. Quinn and B. B. Bederson. Human computation: A survey and taxonomy of a growing field. In Proceedings of CHI 2011, 2011.
21. F. Radlinski, M. Kurup, and T. Joachims. How does clickthrough data reflect retrieval quality? In CIKM, pages 43-52. ACM, 2008.
22. V. S. Sheng, F. Provost, and P. G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceeding of the 14th ACM SIGKDD international conference, KDD '08, pages 614-622, 2008. ACM.
23. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng. Cheap and fast - but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP '08, pages 254-263, 2008.
24. I. Soboroff, C. Nicholas, and P. Cahan. Ranking retrieval systems without relevance judgments. In Proceedings of the 24th annual international ACM SIGIR conference, SIGIR '01, pages 66-73, 2001. (Pubitemid 32951057)
25. L. von Ahn and L. Dabbish. Labeling images with a computer game. In Proceedings of the SIGCHI conference on Human factors in computing systems, CHI '04, pages 319-326, 2004. ACM.
26. E. M. Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing & Management, 36(5):697 - 716, 2000.
27. E. M. Voorhees and D. K. Harman, editors. TREC: Experimentation and Evaluation in Information Retrieval. MIT Press, 2005.
28. P. Welinder, S. Branson, S. Belongie, and P. Perona. The multidimensional wisdom of crowds. In Advances in Neural Information Processing Systems 23, pages 2424-2432, 2010.
29. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Proceedings of the 2009 Neural Information Processing Systems (NIPS) Conference, 2009.
30. D. Zhu and B. Carterette. An analysis of assessor behavior in crowdsourced preference judgments. In SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation, pages 17-20, 2010.