Dissimilarity based query selection for efficient preference based IR evaluation

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 8416 LNCS, Issue , 2014, Pages 172-183

  Kazai, Gabriella ^a,b   Sung, Homer ^a,b  

^a MICROSOFT   (United States)

^b Bellevue ^*  (United States)

Author keywords

judging cost reduction; preference judgments; query prioritization; Side by side evaluation; weighted ranked difference measure

Indexed keywords

DATA PROCESSING; COST REDUCTION; INFORMATION RETRIEVAL;

EVALUATION EXPERIMENTS; PREFERENCE JUDGMENTS; PREFERENCE-BASED; PRIORITIZATION; RELEVANCE CRITERIA; SIDE BY SIDES; WEIGHTED RANKED DIFFERENCE MEASURE; WEIGHTED RANKINGS;

INFORMATION RETRIEVAL; SEARCH ENGINES;

EID: 84899995068     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-319-06028-6_15     Document Type: Conference Paper

References (18)

1. Aslam, J.A., Pavlu, V., Yilmaz, E.: A statistical method for system evaluation using incomplete judgments. In: Proc. of the 29th ACM SIGIR Conference, SIGIR 2006, pp. 541-548. ACM, New York (2006)
2. Bailey, P., Craswell, N., White, R.W., Chen, L., Satyanarayana, A., Tahaghoghi, S.M.: Evaluating search systems using result page context. In: Proc. of the Third Symposium on Information Interaction in Context, IIiX 2010, pp. 105-114. ACM, New York (2010)
3. Bar-Ilan, J., Mat-Hassan, M., Levene, M.: Methods for comparing rankings of search engine results. Comput. Netw. 50(10), 1448-1463 (2006)
4. Carterette, B., Allan, J., Sitaraman, R.: Minimal test collections for retrieval evaluation. In: Proc. of the 29th ACM SIGIR Conference, SIGIR 2006, pp. 268-275. ACM, New York (2006)
5. Chandar, P., Carterette, B.: Using preference judgments for novel document retrieval. In: Proc. of the 35th ACM SIGIR Conference, SIGIR 2012, pp. 861-870. ACM, New York (2012)
6. Fagin, R., Kumar, R., Sivakumar, D.: Comparing top k lists. SIAM J. Discret. Math. 17(1), 134-160 (2004)
7. Guiver, J., Mizzaro, S., Robertson, S.: A few good topics: Experiments in topic set reduction for retrieval evaluation. ACM Trans. Inf. Syst. 27(4), 21:1-21:26 (2009)
8. Hosseini, M., Cox, I.J., Milic-Frayling, N., Vinay, V., Sweeting, T.: Selecting a subset of queries for acquisition of further relevance judgements. In: Amati, G., Crestani, F. (eds.) ICTIR 2011. LNCS, vol. 6931, pp. 113-124. Springer, Heidelberg (2011)
9. Kim, J., Kazai, G., Zitouni, I.: Relevance dimensions in preference-based ir evaluation. In: Proc. of the 36th ACM SIGIR Conference, SIGIR 2013, pp. 913-916. ACM, New York (2013)
10. Radlinski, F., Bennett, P.N., Carterette, B., Joachims, T.: Redundancy, diversity and interdependent document relevance. SIGIR Forum 43(2), 46-52 (2009)
11. Radlinski, F., Craswell, N.: Comparing the sensitivity of information retrieval metrics. In: Crestani, F., Marchand-Maillet, S., Chen, H.-H., Efthimiadis, E.N., Savoy, J. (eds.) SIGIR 2010, pp. 667-674. ACM (2010)
12. Sanderson, M., Paramita, M.L., Clough, P., Kanoulas, E.: Do user preferences and evaluation measures line up? In: Proc. of the 33rd ACM SIGIR Conference, SIGIR 2010, pp. 555-562. ACM, New York (2010)
13. Shieh, G.: A weighted kendall's tau statistic. Statistics and Probability Letters 39, 17-24 (1998)
14. Thomas, P., Hawking, D.: Evaluation by comparing result sets in context. In: Proc. of the 15th ACM International Conference on Information and Knowledge Management, CIKM 2006, pp. 94-101. ACM, New York (2006)
15. Voorhees, E.M., Harman, D.K. (eds.): TREC: Experimentation and Evaluation in Information Retrieval. MIT Press, Cambridge (2005)
16. Webber, W., Moffat, A., Zobel, J.: A similarity measure for indefinite rankings. ACM Trans. Inf. Syst. 20, 1-20 (2010)
17. Yilmaz, E., Aslam, J.A., Robertson, S.: A new rank correlation coefficient for information retrieval. In: Proc. of the 31st ACM SIGIR Conference, SIGIR 2008, pp. 587-594. ACM, New York (2008)
18. Zhu, J., Wang, J., Vinay, V., Cox, I.J.: Topic (query) selection for IR evaluation. In: Proc. of the 32nd ACM SIGIR Conference, SIGIR 2009, pp. 802-803. ACM, New York (2009)