A few good topics: Experiments in topic set reduction for retrieval evaluation

ACM Transactions on Information Systems

Volumn 27, Issue 4, 2009, Pages

  Guiver, John ^a   Mizzaro, Stefano ^b   Robertson, Stephen ^a  

^a MICROSOFT RESEARCH   (United Kingdom)

^b UNIVERSITY OF UDINE   (Italy)

Author keywords

Evaluation experiments; Search effectiveness; Test corpora; Topic selection

Indexed keywords

ACCURATE PREDICTION; EFFECTIVENESS METRICS; EVALUATION EXPERIMENTS; RETRIEVAL EVALUATION; SAMPLE SIZES; SYSTEM EFFECTIVENESS; TEST CORPUS;

INFORMATION RETRIEVAL; INFORMATION RETRIEVAL SYSTEMS; INFORMATION SERVICES;

EXPERIMENTS;

EID: 75149143907     PISSN: 10468188     EISSN: 15582868     Source Type: Journal    
DOI: 10.1145/1629096.1629099     Document Type: Article

References (17)

1. BANKS, D., OVER, P., AND ZHANG, N.-F. 1999. Blind men and elephants: Six approaches to TREC data. Inform. Retriev. 1, 1-2, 7-34.
2. BOLDI, P., SANTINI, M., AND VIGNA, S. 2005. Paradoxical effects in PageRank incremental computations. Internet Math. 2, 3, 387-404.
3. BUCKLEY, C. AND VOORHEES, E. 2000. Evaluating evaluation measure stability. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. N. J. Belkin, P. Ingwersen, and M.-K. Leong, Eds. ACM Press, New York, 33-40.
4. CARTERETTE, B., ALLAN, J., AND SITARAMAN, R. 2006. Minimal test collections for retrieval evaluation. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM Press, New York, 268-275
5. EFTHIMIADIS, E. N., DUMAIS, S. T., HAWKING, D., AND JARVELIN, K., EDS. 2006. Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM Press, New York.
6. MIZZARO, S. AND ROBERTSON, S. 2007. HITS hits TREC -Exploring IR evaluation results with network analysis. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. C.L.A. Clarke, N. Fuhr, N. Kando, W. Kraaij, and A. P. de Vries, Eds. ACM Press, New York, 479-486.
7. ROBERTSON, S. 2006. On GMAP-And other transformations. In Proceedings of the 13th ACM Conference on Information and Knowledge Management (CIKM'06). 78-83.
8. SAKAI, T. 2006. Evaluating evaluation metrics based on the bootstrap. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM Press, New York, 525-532.
9. SAKAI, T. 2007. On the reliability of information retrieval metrics based on graded relevance. Inform. Process. Manag. 43, 531-548.
10. SANDERSON, M. AND ZOBEL, J. 2005. Information retrieval system evaluation: Effort, sensitivity, and reliability. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval G. Marchionini. A. Moffat, J. Tait, R. Baeza-Yates, and N. Ziviani, Eds. ACM Press, New York, 162-169.
11. SPARCK JONES, K. AND VAN RIJSBERGEN, C. J. 1976. Information retrieval test collections. J. Doc. 32, 59-75.
12. VOORHEES, E. AND BUCKLEY, C. 2002. The effect of topic set size on retrieval experiment error. In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. M. Beaulieu, R. Baeza-Yates, S. H. Myaeng, and K. Jarvelin, Eds. ACM Press, New York, 316-323.
13. WEBBER, W., MOFFAT, A., AND ZOBEL, J. 2008a. Statistical power in retrieval experimentation. In Proceedings of the 17th ACM Conference on Information and Knowledge Management. ACM, New York, 571-580.
14. WEBBER, W., MOFFAT, A., ZOBEL, J., AND SAKAI, T. 2008b. Precision-at-Ten considered redundant. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. S.-H. Myaeng, D. W. Oard, F. Sebastiani, T. Chua, and M.-K. Leong, Eds. ACM Press, New York, 695-696.
15. YILMAZ, E. AND ASLAM, J. A. 2006. Estimating average precision with incomplete and imperfect judgements. In Proceedings of the 13th ACM Conference on Information and Knowledge Management (CIKM'06). 102-111
16. YU, P. S., TSOTRAS, V. J., FOX, E. A., AND LIU, B., EDS. 2006. Proceedings of the 13th ACM Conference on Information and Knowledge Management. ACM Press, New York.
17. ZOBEL, J. 1998. How reliable are the results of large-scale information retrieval experiments? In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. W. B. Croft, A. Moffat, C. J. van Rijsbergen, R. Wilkinson, and J. Zobel, Eds. ACM Press, New York, 307-314.