Exploring Cost-Effective Approaches to Human Evaluation of Search Engine Relevance

Lecture Notes in Computer Science

Volumn 3408, Issue , 2005, Pages 360-374

  Ali, Kamal ^a   Chang, Chi Chao ^a   Juan, Yunfang ^a  

^a YAHOO RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CORRELATION METHODS; COST EFFECTIVENESS; DECISION MAKING; INFORMATION RETRIEVAL SYSTEMS; SECURITY SYSTEMS; TESTING;

CORRELATION ANALYSIS; FRAMEWORKS; TEST METHODS;

SEARCH ENGINES;

EID: 24644505753     PISSN: 03029743     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1007/978-3-540-31865-1_26     Document Type: Conference Paper

References (15)

1. Cleverdon, C. The significance of the cranfield tests on index languages. Proceedings of the SIGIR Conference on Research and Development in Information Retrieval, pages 3-12, 1991.
2. Amitay, E., Carmel, D., Lempel, R., Soffer, A. Scaling IR-System Evaluatin using Term Relevance Sets. In Proceedings of SIGIR 2004, pages 10-17, Sheffield, UK.
3. Buckley, C., Voorhees, E., Retrieval Evaluation with Incomplete Information. In Proceedings of SIGIR 2004, pages 25-32, Sheffield, UK.
4. Voorhees, E., The philosophy of information retrieval evaluation. In Proceedings of the Second Workshop of the Cross-Language Evaluation Forum (CLEF 2001), pages 355-370, 2001.
5. Buckley, C., Voorhees, E., Evaluating evaluation measure stability. In Proceedings of SIGIR 2000, pages 33-40.
6. Zobel, J., How reliable are the results of large-scale information retrieval experiments? In Proceedings of SIGIR 1998, pages 307-314, Melbourne, Australia.
7. Gabrieli, S., and Mizzaro, S., Negotiating a Multidimensional Framework for Relevance Space. In Proceedings of MIRA Conference, 1999.
8. Jarvelin, K. and Kekalainen, J. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems(ACM TOIS) 20(4), 422-446.
9. Joachims, T. Evaluating Retrieval Performance Using Clickthrough Data, Proceedings of the SIGIR Workshop on Mathematical/Formal Models in Information Retrieval, 2002.
10. Mizarro, S. How Many Relevances in Information Retrieval? Interacting With Computers, 10(3):305-322, 1998.
11. Chang, C. and Ali, K. How much correlation is there from one judge to another? Yahoo! Technical Report, 2004-12.
12. Silverstein, C., Henzinger, M., Marais, H. and Moricz, M. Analysis of a Very Large AltaVista Query Log, SRC Technical Note #1998-14.
13. Amento, B. Terveen L. and Hill W. D. "Does 'Authority' Mean Quality? Predicting Expert Quality Ratings of Web Sites". Proceedings of SIGIR 2000(Athens, Greece).
14. Harter, S. Variations in Relevance Assessments and the Measurement of Retrieval Effectiveness." JASIS, 47(1):37-49, 1996.
15. Brin, S. and Page L. 1998. The anatomy of a large-scale hypertextual Web search engine. Computer Networks and ISDN Systems 30(1-7): 107-117.