Towards methods for the collective gathering and quality control of relevance assessments

Proceedings - 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009

Volumn , Issue , 2009, Pages 452-459

  Kazai, Gabriella ^a   Milic Frayling, Natasa ^a   Costello, Jamie ^a  

^a MICROSOFT RESEARCH   (United Kingdom)

Author keywords

Relevance assessments; Social game; Test collection construction

Indexed keywords

AFFORDANCES; DIGITAL BOOKS; DIGITAL ITEM; GAME DESIGN; GAME MODELS; INCENTIVE STRUCTURE; ONLINE COLLECTION; PILOT STUDIES; RELEVANCE ASSESSMENTS; RELEVANCE JUDGMENT; RETRIEVAL SYSTEMS; REVIEW PROCESS; SOCIAL GAME; TEST COLLECTION; TUNE SYSTEMS; VIDEO CONTENTS;

COMPUTER GRAPHICS; INFORMATION RETRIEVAL; INFORMATION SERVICES; ONLINE SYSTEMS; QUALITY ASSURANCE; QUALITY FUNCTION DEPLOYMENT; TOTAL QUALITY MANAGEMENT; VISUAL COMMUNICATION;

QUALITY CONTROL;

EID: 72449196273     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1571941.1572019     Document Type: Conference Paper

References (17)

1. Bailey, P., Craswell, N., Soboroff, I., Thomas, P., de Vries, A. P., and Yilmaz, E. 2008. Relevance assessment: are judges exchangeable and does it matter. In Proc. of 31st ACM SIGIR (Singapore). ACM, New York, NY, 667-674.
2. Clark, P. B. and J. Q. Wilson. 1961. "Incentive Systems: A Theory of Organizations." Administrative Science Quarterly 6:129-26.
3. Cormack, G. V. and Lynam, T. R. 2007. Power and bias of subset pooling strategies. In Proc. of 30th ACM SIGIR (Amsterdam). ACM, New York, NY, 837-838.
4. Fuhr, N., Kamps, J., Lalmas, M., Malik, S., and Trotman, A. 2007. Overview of the INEX 2007 ad hoc track. In Proc. of INEX'07. 1-22.
5. Kazai, G., Doucet, A., Landoni, M. 2009. Overview of the INEX 2008 Book Track. In Proc. of INEX'08. LNCS Vol. 5613, Springer.
6. Piwowarski, B., Trotman, A., and Lalmas, M. 2008. Sound and complete relevance assessment for XML retrieval. ACM Trans. Inf. Syst. 27(1), 1-37.
7. Sanderson, M. and Joho, H. 2004. Forming test collections with no system pooling. In Proc. of 27th ACM SIGIR (Sheffield).ACM, New York, NY, 33-40.
8. Soboroff, I., Nicholas, C., and Cahan, P. 2001. Ranking retrieval systems without relevance judgments. In Proc. of 24th ACM SIGIR (New Orleans). ACM, New York, 66-73.
9. Spink, A. and Greisdorf, H. 2001. Regions and levels: measuring and mapping users' relevance judgments. J. Am. Soc. Inf. Sci. Technol. 52(2), 161-173.
10. Trotman, A., Pharo, N. & Lehtonen (2006). XML-IR users and use cases. In Pre-Proc. of INEX'06, 274-286
11. Trotman, A. and Jenkinson, D. 2007. IR evaluation using multiple assessors per topic. In Proc. of ADCS.
12. von Ahn, L. and Dabbish, L. 2008. Designing games with a purpose. Commun. ACM 51(8), 58-67.
13. von Ahn, L. and Dabbish, L. 2004. Labeling images with a computer game. In Proc. of SIGCHI Conference on Human Factors in Comp. Systems (Vienna). ACM, NY, 319-326.
14. Voorhees, E. M. and Tice, D. M. 2000. Building a question answering test collection. In Proc. of 23rd ACM SIGIR (Athens). ACM, New York, NY, 200-207.
15. Voorhees, E. M. and Harman, D. K. 2005 TREC: Experiment and Evaluation in Information Retrieval. The MIT Press.
16. Yilmaz, E., Kanoulas, E., and Aslam, J. A. 2008. A simple and efficient sampling method for estimating AP and NDCG. In Proc. of 31st ACM SIGIR. ACM, New York, 603-610.
17. Zobel, J. 1998. How reliable are the results of large-scale information retrieval experiments?. In Proc. of 21st ACM SIGIR (Melbourne). ACM, New York, NY, 307-314.