On the reliability and intuitiveness of aggregated search metrics

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2013, Pages 689-698

  Zhou, Ke ^a   Lalmas, Mounia ^b   Sakai, Tetsuya ^c   Cummins, Ronan ^d   Jose, Joemon M ^a  

^a UNIVERSITY OF GLASGOW   (United Kingdom)

^b YAHOO RESEARCH   (United States)

^c WASEDA UNIVERSITY   (Japan)

^d UNIVERSITY OF GREENWICH   (United Kingdom)

Author keywords

Aggregated search; Discriminative power; Diversity; Evaluation; Intuitiveness; Metric; Reliability

Indexed keywords

AGGREGATED SEARCH; DISCRIMINATIVE POWER; DIVERSITY; EVALUATION; INTUITIVENESS; METRIC;

KNOWLEDGE MANAGEMENT; RELIABILITY;

AGGREGATES;

EID: 84889588155     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2505515.2505691     Document Type: Conference Paper

References (28)

1. R. Agrawal, S. Gollapudi, A. Halverson, and S. Ieong. Diversifying search results. WSDM, 2009.
2. J. Arguello, F. Diaz, and J. Callan. Learning to aggregate vertical results into web search results. In CIKM, pages 201-210, 2011.
3. J. Arguello, F. Diaz, J. Callan, and B. Carterette. A methodology for evaluating aggregated search results. In ECIR'11, pages 141-152, Berlin, Heidelberg, 2011. Springer-Verlag.
4. J. Arguello, F. Diaz, J. Callan, and J.-F. Crespo. Sources of evidence for vertical selection. In SIGIR '09, pages 315-322, New York, NY, USA, 2009. ACM.
5. P. Bailey, N. Craswell, R. W. White, L. Chen, A. Satyanarayana, and S. M. M. Tahaghoghi. Evaluating whole-page relevance. In SIGIR, pages 767-768, 2010.
6. J. P. Callan, Z. Lu, and W. B. Croft. Searching distributed collections with inference networks. In SIGIR '95, pages 21-28, New York, NY, USA, 1995. ACM.
7. B. Carterette. Multiple Testing in Statistical Analysis of Systems-Based Information Retrieval Experiments. TOIS, 30(1), 2012.
8. O. Chapelle, D. Metlzer, Y. Zhang, and P. Grinspan. Expected reciprocal rank for graded relevance. In CIKM '09, pages 621-630, New York, NY, USA, 2009. ACM.
9. C. L. Clarke, M. Kolla, G. V. Cormack, O. Vechtomova, A. Ashkan, S. Büttcher, and I. MacKinnon. Novelty and diversity in information retrieval evaluation. In SIGIR '08, pages 659-666, New York, NY, USA, 2008. ACM.
10. C. L. Clarke, N. Craswell, I. Soboroff, and A. Ashkan. A comparative analysis of cascade measures for novelty and diversity. In Proceedings of ACM WSDM 2011, 2011.
11. K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst., 20:422-446, October 2002.
12. X. Li, Y.-Y. Wang, and A. Acero. Learning query intent from regularized click graphs. In SIGIR, 2008.
13. A. Moffat and J. Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Inf. Syst., 27:2:1-2:27, December 2008.
14. D. Nguyen, T. Demeester, D. Trieschnigg, and D. Hiemstra. Federated Search in the Wild, The Combined Power of over a Hundred Search Engines In CIKM 2012, 2012.
15. A. K. Ponnuswami, K. Pattabiraman, Q. Wu, R. Gilad-Bachrach, and T. Kanungo. On composition of a federated web search result page: using online users to provide pairwise preference for heterogeneous verticals. In WSDM '11, pages 715-724, New York, NY, USA, 2011. ACM.
16. T. Sakai and R. Song. Evaluating diversified search results using per-intent graded relevance. In SIGIR, pages 1043-1052, 2011.
17. T. Sakai. Evaluating evaluation metrics based on the bootstrap. In Proceedings of ACM SIGIR 2006, pages 525-532, 2006.
18. T. Sakai and R. Song. Diversified Search Evaluation: Lessons from the NTCIR-9 INTENT Task In Information Retrieval, 2013.
19. M. Sanderson, M. L. Paramita, P. Clough, and E. Kanoulas. Do user preferences and evaluation measures line up? In SIGIR, pages 555-562, 2010.
20. R. L. T. Santos, C. Macdonald, and I. Ounis. Aggregated search result diversification. In Proceedings of the 3rd International Conference on the Theory of Information Retrieval, pages 250-261, Bertinoro, Italy, 2011. Springer.
21. T. Sakai. Evaluation with informational and navigational intents. In WWW, 499-508, 2012.
22. M. Shokouhi. Central-Rank-Based Collection Selection in Uncooperative Distributed Information Retrieval. ECIR, pages 160-172, 2007.
23. L. Si and J. Callan. Relevant document distribution estimation method for resource selection. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in information retrieval, pages 298-305, 2003.
24. C. Zhai, W. Cohen and J. Lafferty. Beyond independent relevance: Methods and evaluation metrics for subtopic retrieval. In Proceedings of ACM SIGIR 2003, pages 10-17, 2003.
25. K. Zhou, R. Cummins, M. Lalmas, and J. Jose. Evaluating large-scale distributed vertical search. In LSDS-IR workshop in CIKM11, 2011.
26. K. Zhou, R. Cummins, M. Halvey, M. Lalmas and J. Jose. Assessing and Predicting Vertical Intent for Web Queries. In ECIR 2012., Barcelona, Spain, 1-5 April 2012.
27. K. Zhou, R. Cummins, M. Lalmas and J.M. Jose. Evaluating aggregated search pages. In SIGIR, 115-124, 2012.
28. K. Zhou, R. Cummins, M. Lalmas, and J. M. Jose. Evaluating reward and risk for vertical selection. CIKM 2012: pages 2631-2634.