Diversified search evaluation: Lessons from the NTCIR-9 INTENT task

Information Retrieval

Volumn 16, Issue 4, 2013, Pages 504-529

  Sakai, Tetsuya ^a   Song, Ruihua ^a  

^a MICROSOFT RESEARCH ASIA   (China)

Author keywords

Diversity; Evaluation; Intents; NTCIR; Search result diversification; Test collections; TREC; Web search

Indexed keywords

EID: 84880838418     PISSN: 13864564     EISSN: 15737659     Source Type: Journal    
DOI: 10.1007/s10791-012-9208-x     Document Type: Article

References (28)

1. Brandt, C., Joachims, T., Yue, Y., & Bank, J. (2011). Dynamic ranked retrieval. In Proceedings of ACM WSDM 2011 (pp. 247-256).
2. Carterette, B. (2012). Multiple testing in statistical analysis of systems-based information retrieval experiments. ACM TOIS 30(1). doi: 10. 1145/2094072. 2094076.
3. Carterette, B., & Chandar, P. (2009). Probabilistic models of novel document rankings for faceted topic retrieval. In Proceedings of ACM CIKM 2009 (pp. 1287-1296).
4. Carterette, B., & Smucker, M. (2007). Hypothesis testing with incomplete relevance judgments. In Proceedings of ACM CIKM 2007 (pp. 643-652).
5. Chandar, P., & Carterette, B. (2011). Analysis of various evaluation measures for diversity. In Proceedings of the DDR workshop 2011 (pp. 21-28).
6. Chapelle, O., Ji, S., Liao, C., Velipasaoglu, E., Lai, L., & Wu, S. L. (2011). Intent-based diversification of web search results: Metrics and algorithms. Information Retrieval, 14(6), 572-592.
7. Clarke, C. L., Craswell, N., & Soboroff. I. (2010). Overview of the TREC 2009 web track. In Proceedings of TREC 2009.
8. Clarke, C. L., Craswell, N., Soboroff, I., & Ashkan, A. (2011a). A comparative analysis of cascade measures for novelty and diversity. In Proceedings of ACM WSDM 2011 (pp. 75-84).
9. Clarke, C. L., Craswell, N., Soboroff, I., & Cormack, G. V. (2011b). Overview of the TREC 2010 web track. In Proceedings of TREC 2010.
10. Clarke, C. L., Craswell, N., Soboroff, I., & Voorhees, E. (2012). Overview of the TREC 2011 web track. In Proceedings of TREC 2011.
11. Clarke, C. L., Kolla, M., & Vechtomova, O. (2009). An effectiveness measure for ambiguous and underspecified queries. In Advances in information retrieval theory (ICTIR 2009) (pp. 188-199). LNCS 5766.
12. Kanoulas, E., Carterette, B., Hall, M., Clough, P., & Sanderson, M. (2012). Overview of the trec 2011 session track. In Proceedings of TREC 2011.
13. Macdonald, C., Ounis, I., & Soboroff, I. (2010). Overview of the TREC-2009 blog track. In Proceedings of TREC 2009.
14. Pavlu, V., Rajput, S., Golbus, P. B., & Aslam, J. A. (2012). IR system evaluation using nugget-based test collections. In Proceedings of ACM WSDM 2012.
15. Robertson, S. E., Kanoulas, E., & Yilmaz, E. (2010). Extending average precision to graded relevance judgments. In Proceedings of ACM SIGIR 2010 (pp. 603-610).
16. Sakai, T. (2006). Evaluating evaluation metrics based on the bootstrap. In Proceedings of ACM SIGIR 2006 (pp. 525-532).
17. Sakai, T. (2012). Evaluation with informational and navigational intents. In Proceedings of WWW 2012 (pp. 499-508).
18. Sakai, T., & Mitamura, T. (2010). Boiling down information retrieval test collections. In: Proceedings of RIAO 2010.
19. Sakai, T., & Song, R. (2011). Evaluating diversified search results using per-intent graded relevance. In Proceedings of ACM SIGIR 2011 (pp. 1043-1042).
20. Sakai, T., Kato, M. P., Song, Y. I. (2011). Click the search button and be happy: Evaluating direct and immediate information access. In Proceedings of ACM CIKM 2011 (pp. 621-630).
21. Sanderson, M., & Zobel, J. (2005). Information retrieval evaluation: Effort, sensitivity, and reliability. In Proceedings of ACM SIGIR 2005 (pp. 162-169).
22. Sanderson, M., Paramita, M. L., Clough, P., & Kanoulas, E. (2010). Do user preferences and evaluation measures line up? In Proceedings of ACM SIGIR 2010 (pp. 555-562).
23. Soboroff, I. (2010). Test collection diagnosis and treatment. In: Proceedings of EVIA 2010 (pp. 34-41).
24. Song, R., Zhang, M., Sakai, T., Kato, M. P., Liu, Y., Sugimoto, M., Wang, Q., & Orii, N. (2011). Overview of the NTCIR-9 INTENT task. In Proceedings of NTCIR-9 (pp. 82-105).
25. Webber, W., Moffat, A., & Zobel, J. (2008). Statistical power in retrieval experimentation. In Proceedings of ACM CIKM 2008 (pp. 571-580).
26. Webber, W., Moffat, A., & Zobel, J. (2010). The effect of pooling and evaluation depth on metric stability. In Proceedings of EVIA 2010 (pp. 7-15).
27. Yilmaz, E., Aslam, J., & Robertson, S. (2008). A new rank correlation coefficient for information retrieval. In Proceedings of ACM SIGIR 2008 (pp. 587-594).
28. Zhai, C., Cohen, W. W., & Lafferty, J. (2003). Beyond independent relevance: Methods and evaluation metrics for subtopic retrieval. In Proceedings of ACM SIGIR 2003 (pp. 10-17).