Design and implementation of relevance assessments using crowdsourcing

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 6611 LNCS, Issue , 2011, Pages 153-164

  Alonso, Omar ^a   Baeza Yates, Ricardo ^b  

^a MICROSOFT   (United States)

^b YAHOO RESEARCH   (Spain)

Author keywords

[No Author keywords available]

Indexed keywords

BUDGET CONTROL; FIXED PLATFORMS; INFORMATION RETRIEVAL; USER INTERFACES;

AMAZON MECHANICAL TURKS; CROWDSOURCING PLATFORMS; DESIGN AND IMPLEMENTATIONS; DETAILED FEEDBACKS; DOCUMENT DESIGN; RELEVANCE ASSESSMENTS; RELEVANCE JUDGMENT; USER INTERFACE GUIDELINES;

CROWDSOURCING;

EID: 84930177119     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-642-20161-5_16     Document Type: Conference Paper

References (15)

1. von Ahn, L.: Games with a purpose. IEEE Computer 39(6), 92–94 (2006)
2. Alonso, O., Mizzaro, S.: Can we get rid of TREC Assessors? Using Mechanical Turk for Relevance Assessment. In: SIGIR Workshop Future of IR Evaluation (2009)
3. Alonso, O., Schenkel, R., Theobald, M.: Crowdsourcing Assessments for XML Ranked Retrieval. In: 32 ECIR, Milton Keynes, UK (2010)
4. Alonso, O., Baeza-Yates, R.: An Analysis of Crowdsourcing Relevance Assessments in Spanish. In: CERI 2010, Madrid, Spain (2010)
5. Bradburn, N., Sudman, S., Wansink, B.: Asking Questions: The Definitive Guide to Questionnaire Design. Josey-Bass (2004)
6. Callison-Burch, C.: Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk. In: Proceedings of EMNLP (2009)
7. Grady, C., Lease, M.: Crowdsourcing Document Relevance Assessment with Mechanical Turk. In: NAACL HLT Workshop on Creating Speech and Language Data with Amazons Mechanical Turk (2010)
8. Kazai, G., Milic-Frayling, N., Costello, J.: Towards Methods for the Collective Gathering and Quality Control of Relevance Assessments. In: 32 SIGIR (2009)
9. Kinney, K., Huffman, S., Zhai, J.: How Evaluator Domain Expertise Affects Search Result Relevance Judgments. In: 17 CIKM (2008)
10. Malone, T.W., Laubacher, R., Dellarocas, C.: Harnessing Crowds: Mapping the Genome of Collective Intelligence. MIT Press, Cambridge (2009)
11. Mason, W., Watts, D.: Financial Incentives and the ‘Performance of Crowds’. In: HCOMP Workshop at KDD, Paris, France (2009)
12. Nov, O., Naaman, M., Ye, C.: What Drives Content Tagging: The Case of Photos on Flickr. In: CHI, Florence, Italy (2008)
13. Snow, R., O’ Connor, B., Jurafsky, D., Ng, A.Y.: Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. In: EMNLP (2008)
14. Tang, J., Sanderson, M.: Evaluation and User Preference Study on Spatial Diversity. In: 32 ECIR, Milton Keynes, UK (2010)
15. Voorhees, E.: Personal communication (2009)