Differences in effectiveness across sub-collections

ACM International Conference Proceeding Series

Volumn , Issue , 2012, Pages 1965-1969

  Sanderson, Mark ^a   Turpin, Andrew ^b   Zhang, Ying ^a   Scholer, Falk ^a  

^a RMIT UNIVERSITY   (Australia)

^b UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

information retrieval evaluation; search engines; sub collections

Indexed keywords

RELATIVE PERFORMANCE; RETRIEVAL EFFECTIVENESS; RETRIEVAL SYSTEMS; SUBCOLLECTIONS; TEST COLLECTION;

SEARCH ENGINES;

KNOWLEDGE MANAGEMENT;

EID: 84871098973     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2396761.2398553     Document Type: Conference Paper

References (13)

1. C. W. Cleverdon, "The Evaluation of Systems Used in Information Retrieval (1958: Washington)," in Proceedings of the International Conference on Scientific Information - Two Volumes, 1959, pp. 687-698.
2. E. M. Voorhees, "Variations in relevance judgments and the measurement of retrieval effectiveness," in Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, 1998, pp. 315-323.
3. B. Carterette, V. Pavlu, E. Kanoulas, J. A. Aslam, and J. Allan, "Evaluation over thousands of queries," in Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, 2008, pp. 651-658.
4. J. Tague-Sutcliffe, "The pragmatics of information retrieval experimentation, revisited," Information Processing & Management, vol. 28, no. 4, pp. 467-490, Jul. 1992.
5. I. Soboroff, "Dynamic test collections: measuring search effectiveness on the live web," in Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, 2006, pp. 276-283.
6. E. M. Voorhees and D. K. Harman, TREC: Experiment and Evaluation in Information Retrieval, illustrated ed. The MIT Press, 2005.
7. S. E. Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford, "Okapi at TREC-3," NIST SPECIAL PUBLICATION SP, pp. 109-126, 1995.
8. A. Singhal, C. Buckley, and M. Mitra, "Pivoted document length normalization," in Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, 1996, pp. 21-29.
9. F. Scholer, A. Turpin, and M. Sanderson, "Quantifying test collection quality based on the consistency of relevance judgements," in Proceedings of the 34th international ACM SIGIR conference on Research and development in Information, 2011, pp. 1063-1072.
10. C. Clarke, N. Craswell, and I. Soboroff, "Overview of the TREC 2004 Terabyte Track," in Proceedings of TREC, 2004, vol. 2004.
11. E. M. Voorhees and C. Buckley, "The effect of topic set size on retrieval experiment error," in Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, 2002, pp. 316-323.
12. M. D. Smucker, J. Allan, and B. Carterette, "A comparison of statistical significance tests for information retrieval evaluation," in Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, 2007, pp. 623-632.
13. J. Lin, "Divergence measures based on the Shannon entropy," Information Theory, IEEE Transactions on, vol. 37, no. 1, pp. 145-151, 1991.