On statistical analysis and optimization of information retrieval effectiveness metrics

SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2010, Pages 226-233

  Wang, Jun ^a   Zhu, Jianhan ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

Algorithms; Experimentation; Measurement; Performance

Indexed keywords

EXPECTED VALUES; EXPERIMENTATION; IR MODELS; JOINT PROBABILITY; OPTIMIZATION FRAMEWORK; OPTIMIZATION OBJECTIVE FUNCTION; PERFORMANCE; PROBABILITY MEASURES; RANKING PROBLEMS; RELEVANCE PREDICTION; RETRIEVAL EFFECTIVENESS; RETRIEVAL MODELS; STATISTICAL ANALYSIS;

EXPERIMENTS; NATURAL LANGUAGE PROCESSING SYSTEMS; OPTIMIZATION; PROBABILITY DENSITY FUNCTION; UNCERTAINTY ANALYSIS;

INFORMATION RETRIEVAL;

EID: 77956031442     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1835449.1835489     Document Type: Conference Paper

References (35)

1. G. Amati and C. J. V. Rijsbergen. Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Trans. Inf. Syst., 20(4):357-389, 2002.
2. K. Arrow. Aspects of the Theory of Risk-Bearing. Helsinki: Yrjö Hahnsson Foundation, 1965.
3. J. A. Aslam, V. Pavlu, and E. Yilmaz. A statistical method for system evaluation using incomplete judgments. In SIGIR, 2006.
4. J. A. Aslam, E. Yilmaz, and V. Pavlu. The maximum entropy method for analyzing retrieval measures. In SIGIR, 2005.
5. C. M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
6. P. F. Brown, V. J. D. Pietra, S. A. D. Pietra, and R. L. Mercer. The mathematics of statistical machine translation: parameter estimation. Comput. Linguist., 1993.
7. C. Buckley and E. M. Voorhees. Evaluating evaluation measure stability. In SIGIR, 2000.
8. C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In ICML '05, 2005.
9. J. Carbonell and J. Goldstein. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In SIGIR, 1998.
10. H. Chen and D. R. Karger. Less is more: probabilistic models for retrieving fewer relevant documents. In SIGIR, 2006.
11. G. V. Cormack and T. R. Lynam. Statistical precision of information retrieval evaluation. In SIGIR, 2006.
12. W. B. Croft and D. J. Harper. Using probabilistic models of document retrieval without relevance information. Document Retrieval Systems, 1988.
13. D. Harman. Overview of the second text retrieval conference (trec-2). In HLT '94, 1994.
14. K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst., 2002.
15. J. Lafferty and C. Zhai. Document language models, query models, and risk minimization for information retrieval. In SIGIR, 2001.
16. C. D. Manning, P. Raghavan, and H. Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008.
17. M. E. Maron and J. L. Kuhns. On relevance, probabilistic indexing and information retrieval. J. ACM, 1960.
18. S. Mizzaro. Relevance: The whole history. Journal of the American Society of Information Science, 1997.
19. S. E. Robertson. The probability ranking principle in IR. Journal of Documentation, pages 294-304, 1977.
20. S. E. Robertson and K. Spärck Jones. Relevance weighting of search terms. Journal of the American Society for Information Science, 27(3):129-46, 1976.
21. S. E. Robertson and S. Walker. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In SIGIR, 1994.
22. A. Singhal, C. Buckley, and M. Mitra. Pivoted document length normalization. In SIGIR, pages 21-29, 1996.
23. M. Taylor, J. Guiver, S. Robertson, and T. Minka. Softrank: optimizing non-smooth rank metrics. In WSDM, 2008.
24. S. Tomlinson. Early precision measures: implications from the downside of blind feedback. In SIGIR, 2006.
25. C. J. van Rijsbergen. Information Retrieval. Butterworths, London, London, UK, 1979.
26. M. N. Volkovs and R. S. Zemel. Boltzrank: learning to maximize expected ranking gain. In ICML '09, 2009.
27. E. Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. In Information Processing and Management, pages 315-323. ACM Press, 1998.
28. E. M. Voorhees. The TREC-8 question answering track report. In TREC-8, pages 77-82, 1999.
29. J. Wang and J. Zhu. Portfolio theory of information retrieval. In SIGIR, 2009.
30. Y. Wang and A. Waibel. Decoding algorithm in statistical machine translation. In EACL, 1997.
31. E. Yilmaz, E. Kanoulas, and J. A. Aslam. A simple and efficient sampling method for estimating ap and ndcg. In SIGIR, 2008.
32. E. Yilmaz and S. Robertson. On the choice of effectiveness measures for learning to rank. Information Retrieval, 2009.
33. Y. Yue, T. Finley, F. Radlinski, and T. Joachims. A support vector method for optimizing average precision. In SIGIR, 2007.
34. C. Zhai. Statistical language models for information retrieval a critical review. Found. Trends Inf. Retr., 2(3):137-213, 2008.
35. C. Zhai and J. D. Lafferty. A risk minimization framework for information retrieval. Inf. Process. Manage., 42(1):31-55, 2006.