Select-the-Best-Ones: A new way to judge relative relevance

Information Processing and Management

Volumn 47, Issue 1, 2011, Pages 37-52

  Song, Ruihua ^a,b   Guo, Qingwei ^b   Zhang, Ruochi ^b   Xin, Guomao ^b   Wen, Ji Rong ^b   Yu, Yong ^a   Hon, Hsiao Wuen ^b  

^a SHANGHAI JIAO TONG UNIVERSITY   (China)

^b MICROSOFT RESEARCH ASIA   (China)

Author keywords

Human judgment; Preference judgment; Relative relevance judgment

Indexed keywords

INFORMATION RETRIEVAL; INFORMATION RETRIEVAL SYSTEMS; SEARCH ENGINES;

EVALUATION METHODOLOGIES; HUMAN JUDGMENTS; JUDGMENT METHOD; LEARNING TO RANK; PREFERENCE JUDGMENT; RELEVANCE JUDGMENT; USER STUDY;

LEARNING SYSTEMS;

EID: 77958501794     PISSN: 03064573     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ipm.2010.02.005     Document Type: Article

References (38)

1. Bollmann, P., & Wong, S. K. M. (1987). Adaptive linear information retrieval models. In SIGIR '87: Proceedings of the 10th annual international ACM SIGIR conference on research and development in information retrieval (pp. 157-163).
2. Borlund, P. (2000). Evaluation of interactive information retrieval systems. Doctoral dissertation. bo Akademi University. bo: bo Akademi University Press.
3. P. Borlund The concept of relevance in IR Journal of the American Society for Information Science and Technology 54 10 2003 913 925
4. H.W. Bruce A cognitive view of the situational dynamism of user-centered relevance estimation Journal of the American Society for Information Science 45 1994 142 148
5. Buckley, C., & Voorhees, E. M. (2004). Retrieval evaluation with incomplete information. In SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on research and development in information retrieval (pp. 25-32).
6. Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., et al. (2005). Learning to rank using gradient descent. In ICML'05: Proceedings of the 22nd international conference on machine learning.
7. Cao, Z., Qin, T., Liu, T.-Y., Tsai, M.-F., & Li, H. (2007). Learning to rank: From pairwise approach to listwise approach. In ICML '07: Proceedings of the 24th international conference on machine learning (pp. 129-136).
8. Carterette, B., Bennett, P. N., Chickering, D. M., & Dumais, S. T. (2008). Here or there preference judgments for relevance. In ECIR'08: Proceedings of European conference on information retrieval (pp. 16-27).
9. E.P. Cox III The optimal number of response alternatives for a scale: A review Journal of Marketing Research XVII 1980 407 422
10. Cross Language Evaluation Forum. .
11. Cuadra, C. A., & Katter, R. V. (1967). Experimental studies of relevance judgments: Final report. NSF Report No. TM-3520/001/00. Santa Monica, CA: System Development Corp.
12. M.B. Eisenberg Measuring relevance judgments Information Processing & Management 24 1988 373 389
13. M.B. Eisenberg, and C. Barry Order effects: A study of the possible influence of presentation order on user judgments of document relevance Journal of the American Society for Information Science 39 1988 293 300
14. Y. Freund, R. Iyer, R.E. Schapire, and Y. Singer An efficient boosting algorithm for combining preferences Journal of Machine Learning Research 4 2003 933 969
15. S.P. Harter Psychological relevance and information science Journal of the American Society for Information Science 43 1992 602 615
16. S.P. Harter Variations in relevance assessments and the measurement of retrieval effectiveness Journal of the American Society for Information Science 47 1 1996 37 49
17. Hoffman, J. M. (1965). Experimental design for measuring the intra- and inter-group consistence of human judgment for relevance. Master's thesis. Georgia Institute of Technology, Atlanta, Georgia.
18. J. Jacoby, and M.S. Matell The neutral point on a Likert scale Journal of Psychology 95 1971 199 204
19. Järvelin, K., & Kekäläinen, J. (2000). IR evaluation methods for retrieving highly relevant documents. In SIGIR'00: Proceedings of the 23rd ACM SIGIR conference on research and development of information retrieval (pp. 41-48).
20. Joachims, T. (2002). Evaluating retrieval performance using clickthrough data. In Proceedings of the SIGIR workshop on mathematical/formal methods in information retrieval.
21. Latin Square. square.
22. M.E. Lesk, and G. Salton Relevance assessment and retrieval system evaluation Information Storage and Retrieval 4 3 1968 343 359
23. Mizzaro, S. (1999). Measuring the agreement among relevance judges. In MIRA conference proceedings.
24. NII test collections for IR systems. .
25. J. O'Connor Relevance disagreements and unclear request forms American Documentation 18 3 1967 165 177
26. J. O'Connor Some independent agreements and resolved disagreements about answer-providing documents American Documentation 20 4 1969 311 319
27. Rees, A. M., & Schultz, D. G. (1967). A field experiment approach to the study of relevance assessments in relation to document searching (Vol. 2). Cleveland, OH: Center for Documentation and Communication Research, School of Library Science, Case Western Reserve University.
28. M.E. Rorvig The simple scalability of documents Journal of the American Society for Information Science 41 8 1990 590 598
29. Sanderson, M., & Zobel, J. (2005). Information retrieval system evaluation: Effort, sensitivity, and reliability. In SIGIR'05 conference proceedings.
30. Saracevic, T. (1969). Comparative effects of titles, abstracts and full texts on relevance judgments. In ASIS proceedings (pp. 293-299).
31. T. Saracevic Relevance: A review of and a framework for the thinking on the notion in information science Journal of the American Society for Information Science 26 1975 321 343
32. Saracevic, T. (1996). Relevance reconsidered '96. In P. Ingwersen, & N. O. Pors (Eds.). Proceedings of CoLIS 2, second international conference on conceptions of library and information science: Integration in perspective, Copenhagen (pp. 201-218).
33. S.S. Stevens A metric for the social consensus Science 151 1966 530 541
34. D.R. Swanson Subjective versus objective relevance in bibliographic retrieval systems Library Quaterly 56 1986 389 398
35. R. Tang, W.M. Shaw Jr., and J.L. Vevea Towards the identification of the optimal number of relevance categories Journal of the American Society for Information Science 50 1999 254 264
36. Text Retrieval Conference. .
37. Voorhees, E. M. (1998). Variations in relevance judgments and the measurement of retrieval effectiveness. In SIGIR'98 conference proceedings.
38. Zheng, Z., Chen, K., Sun, G., & Zha, H. (2007). A regression framework for learning ranking functions using relative relevance judgments. In SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on research and development in information retrieval.