The effects of choice in routing relevance judgments

SIGIR'11 - Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2011, Pages 1127-1128

  Law, Edith ^a   Bennett, Paul N ^b   Horvitz, Eric ^b  

^a CARNEGIE MELLON UNIVERSITY   (United States)

^b MICROSOFT RESEARCH   (United States)

Author keywords

Human computation; Relevance judgments; Task routing

Indexed keywords

GAMES WITH A PURPOSE; HUMAN COMPUTATION; MECHANICAL TURKS; RELEVANCE JUDGMENT; TASK ROUTING;

EID: 80052105609     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2009916.2010082     Document Type: Conference Paper

References (8)

1. Bailey, P., Craswell, N., Soboroff, I., Thomas, P., de Vries, A. and Yilmaz, E. Relevance assessment: are judges exchangeable and does it matter? In SIGIR 2008.
2. Carterette, B., Allan, J., & Sitaraman, R. Minimal Test Collections for Retrieval Evaluation. In SIGIR 2006.
3. Carterette, B., Bennett, P. N., Chickering, D. M., & Dumais, S. T. Here or there: preference judgments for relevance. In ECIR 2008.
4. Cleverdon, C. The cranfield tests on index language devices. Readings in IR, Morgan Kaufmann, 1997.
5. Russell, D. and C. Grimes. Assigned tasks are not the same as self-chosen web search tasks. In HICSS 2007.
6. Teevan, J., Dumais, S. and Horvitz, E. Characterizing the Value of Personalizing Search. In SIGIR 2007
7. Teevan, J., Morris, M. and Bush, S. Discovering and Using Groups to Improve Personalized Search. In WSDM 2009.
8. Yilmaz, E. and Aslam, J. Estimating average precision with incomplete and imperfect judgments. In CIKM 2006.