Evaluating the performance of information retrieval systems using test collections

Information Research

Volumn 18, Issue 2, 2013, Pages

  Clough, Paul ^a   Sanderson, Mark ^b  

^a UNIVERSITY OF SHEFFIELD   (United Kingdom)

^b RMIT UNIVERSITY   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84879146779     PISSN: None     EISSN: 13681613     Source Type: Journal    
DOI: None     Document Type: Article

References (54)

1. Al-Maskari, A., Sanderson, M., Clough, P. & Airio, E. (2008). The good and the bad system: does the test collection predict users' effectiveness? In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. (pp. 59-66). New York, NY: ACM Press. Retrieved 5 May, 2013 from http://ir.shef.ac.uk/cloughie/papers/fp440-almaskari.pdf Archived by WebCite® at http://www.webcitation.org/6H0NCnbHd)
2. Alonso, O. & Mizzaro, S. (2009). Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment. In Shlomo Geva, Jaap Kamps, Carol Peters, Tetsuya Sakai, Andrew Trotman & Ellen Voorhees, (Eds.). Proceedings of the SIGIR 2009 Workshop on the Future of information retrieval Evaluation. (pp. 15-16). Amsterdam: IR Publications. Retrieved 5 May, 2013 from http://staff.science.uva.nl/%7Ekamps/ireval/papers/paper_22.pdf (Archived by WebCite® at http://www.webcitation.org/6H1mV6taR)
3. Aslam, J.A., Pavlu, V. & Yilmaz, E. (2006). A statistical method for system evaluation using incomplete judgments. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, (pp. 541-548). New York, NY: ACM Press
4. Azzopardi, L., Järvelin, K., Kamps, J., & Smucker, M.D. (2010). SIGIR Forum, 44(2), 35-47. Retrieved 5 May, 2013 from http://www.sigir.org/forum/2010D/sigirwksp/2010d_sigirforum_azzopardi.pdf (Archived by WebCite® at http://www.webcitation.org/6H1miejTW)
5. Bailey, P., Craswell, N., Soboroff, I., Thomas, P., de Vries, A.P. & Yilmaz, E. (2008). Relevance assessment: are judges exchangeable and does it matter? In Sung-Hyon Myaeng, Douglas W. Oard, Fabrizio Sebastiani, Tat-Seng Chua, Mun-Kew Leong (Eds.). Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Singapore, July 20-24, 2008. (pp. 667-674). New York, NY: ACM Press. Retrieved 5 May, 2013 from http://es.csiro.au/pubs/bailey_sigir08.pdf (Archived by WebCite® at http://www.webcitation.org/6GOBCHiqK)
6. Bates, M.J. (1989). The design of browsing and berrypicking techniques for the online search interface. Online Review, 13(5), 407-424. Retrieved 5 May, 2013 from http://pages.gseis.ucla.edu/faculty/bates/berrypicking.html Archived by WebCite® at http://www.webcitation.org/6H1mrhwxF)
7. Borlund: (2009). User-centred evaluation of information retrieval systems. In A. Göker and J. Davies (eds), Information retrieval: searching in the 21st Century, Chicester, UK: John Wiley & Sons
8. Carterette, B., Pavlu, V., Kanoulas, E., Aslam, J.A. & Allan, J. (2008). Evaluation over thousands of queries. In Sung-Hyon Myaeng, Douglas W. Oard, Fabrizio Sebastiani, Tat-Seng Chua, Mun-Kew Leong (Eds.). Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Singapore, July 20-24, 2008. (pp. 651-658). New York, NY: ACM Press. Retrieved 5 May, 2013 from http://ir.cis.udel.edu/~carteret/papers/sigir08.pdf Archived by WebCite® at http://www.webcitation.org/6H1n04iqc)
9. Carvalho, V. R., Lease, M. & Yilmaz, E. (2011). Crowdsourcing for search evaluation. SIGIR Forum, 44(2), 17-22. Retrieved 5 May, 2013 from http://www.sigir.org/forum/2010D/sigirwksp/2010d_sigirforum_carvalho.pdf Archived by WebCite® at http://www.webcitation.org/6H1n8yVN1)
10. Cleverdon, C.W. (1967). The Cranfield tests on index language devices. Aslib Proceedings, 19(6), 173-192
11. Cleverdon, C. W. (1970). The effect of variations in relevance assessments in comparative experimental tests of index languages. Cranfield, UK: Cranfield Institute of Technology. (Cranfield Library Report, No. 3)
12. Cleverdon, C.W. (1991). The significance of the Cranfield tests on index languages. In Abraham Bookstein, Yves Chiaramella, Gerard Salton, Vijay V. Raghavan (Eds.). Proceedings of 14th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Chicago, IL, USA October 13-16, 1991. (pp. 3-12). New York, NY: ACM Press
13. Clough, P., Sanderson, M., Tang, J., Gollins, T. & Warner, A. (2012). Examining the limits of crowdsourcing for relevance assessment. IEEE Internet Computing, In Print. Retrieved 5 May, 2013 from http://www.seg.rmit.edu.au/mark/publications/my_papers/IEEE-IC-2012.pdf Archived by WebCite® at http://www.webcitation.org/6H1nE0LNX)
14. Cormack, G.V., Palmer, C.R. & Clarke, C.L.A. (1998). Efficient construction of large test collections. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August 24-28 1998, Melbourne, Australia. (pp. 282-289). New York, NY: ACM Press.
15. Croft, B., Metzler, D. & Strohman, T. (2009). Search engines: information retrieval in practice. Boston, MA, Pearson Education
16. Harman, D. (2011). Information retrieval evaluation. San Raphael, CA: Morgan & Claypool Publishers. (Synthesis Lectures on Information Concepts, Retrieval, and Services, Volume 3, No. 2).
17. Hersh, W., Turpin, A., Price, S., Chan, B., Kramer, D., Sacherek, L. & Olson, D. (2000). Do batch and user evaluations give the same results? In Nicholas J. Belkin, Peter Ingwersen, Mun-Kew Leong (Eds.). Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, July 24-28, 2000, Athens, Greece. (pp. 17-24). New York, NY: ACM Press. Retrieved 5 May, 2013 from http://skynet.ohsu.edu/~hersh/sigir-00-batcheval.pdf Archived by WebCite® at http://www.webcitation.org/6H1nNaYid)
18. Ingwersen: & Järvelin, K. (2005). The turn: integration of information seeking and retrieval in context, New York, NY: Springer-Verlag
19. Jansen, B.J., Booth, D.L., & Spink, A. (2009). Patterns of query reformulation during web searching. Journal of the American Society for Information Science and Technology, 60(7), 1358-1371. Retrieved 5 May, 2013 from http://faculty.ist.psu.edu/jjansen/academic/pubs/jansen_patterns_query_reformulation.pdf Archived by WebCite® at http://www.webcitation.org/6H1nU1bRD)
20. Järvelin, K. & Kekäläinen, J. (2002). Cumulated gain-based evaluation of information retrieval techniques. ACM Transactions on Information Systems, 20(4), 422-446
21. Järvelin, K., Price, S.L., Delcambre, L.M.L. & Nielsen, M.L. (2008). Discounted cumulated gain based evaluation of multiple-query IR sessions. In Proceedings of the 30th European Conference on Advances in Information Retrieval. (pp. 4-15). Berlin: Springer. (Lecture Notes in Computer Science, 4956).
22. Järvelin, K. (2009). Explaining user performance in information retrieval: challenges to IR evaluation. In Proceedings of the 2nd International Conference on Theory of Information Retrieval: Advances in Information Retrieval Theory. (pp. 289-296). Berlin: Springer.
23. Järvelin, K. (2011). Evaluation. In Ruthven, I. and Kelly, D. (eds.), Interactive information seeking, behaviour and retrieval. (pp. 113-138). London, UK: Facet Publishing.
24. Kamps, J., Geva, S., Peters, C., Sakai, T., Trotman, A. & Voorhees, E. (2009). Report on the SIGIR 2009 workshop on the future of information retrieval evaluation. SIGIR Forum,43(2), 13-23. Retrieved 5 May, 2013 from http://www.sigir.org/forum/2009D/sigirwksp/2009d_sigirforum_kamps.pdf Archived by WebCite® at http://www.webcitation.org/6H1nmVq7g)
25. Kanoulas, E., Carterette, B., Clough, P. & Sanderson, M. (2011). Evaluating multi-query sessions. In Wei-Ying Ma, Jian-Yun Nie, Ricardo A. Baeza-Yates, Tat-Seng Chua, W. Bruce Croft (Eds.). Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Beijing, China July 24-28, 2011. (pp. 1053-1062). New York, NY: ACM Press. Retrieved 5 May, 2013 from http://ir.cis.udel.edu/~carteret/papers/sigir11b.pdf Archived by WebCite® at http://www.webcitation.org/6H1nvBDqm)
26. Kanoulas, E., Carterette, B., Hall, M., Clough: & Sanderson, M. (2012). Session Track 2012 Overview. In Proceedings of the 21st Text REtrieval Conference. Washington, DC: National Institute of Standards and Technology
27. Kazai, G. (2011). In search of quality in crowdsourcing for search engine evaluation. InProceedings of the 33rd European Conference on Advances in Information Retrieval. (pp. 165-176). Berlin: Springer
28. Kelly D. (2009). Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval, 3(1-2), 1-224. Retrieved 5 May, 2013 from http://www.ils.unc.edu/~dianek/FnTIR-Press-Kelly.pdf Archived by WebCite® at http://www.webcitation.org/6H1o2TKSU)
29. Keskustalo, H., Järvelin, K., Pirkola, A., Sharma, T., and Lykke, M. (2009). Test Collection-Based information retrieval Evaluation Needs Extension toward Sessions-A Case of Extremely Short Queries. In Proceedings of the 5th Asia Information Retrieval Symposium on Information Retrieval Technology, (pp. 63-74). Berlin:Springer. Retrieved 5 May, 2013 from http://www.sis.uta.fi/infim/julkaisut/fire/2009/HK-SessEval-AIRS09-prep.pdf Archived by WebCite® at http://www.webcitation.org/6H1o7fRB8)
30. Kiewitt, E.L. (1979). Evaluating information retrieval systems: the PROBE program. Westport, CT: Greenwood Press
31. Kinney, K. A., Huffman, S.B. & Zhai, J. (2008). How evaluator domain expertise affects search result relevance judgments. In Proceedings of the 17th ACM Conference on Information and Knowledge Management. (pp. 591-598). New York, NY: ACM Press.
32. Magdy, W., & Jones, G. (2010). PRES: a score metric for evaluating recall-oriented information retrieval applications. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Ddevelopment in Information Retrieval, Geneva, Switzerland July 19-23, 2010. (pp. 611-618). New York, NY: ACM Press.
33. Manning, C.D., Raghavan: & Schütze, H. (2008). Introduction to information retrieval. Cambridge: Cambridge University Press. Retrieved 5 May, 2013 from http://nlp.stanford.edu/IR-book/ Archived by WebCite® at http://www.webcitation.org/6H1okAxEy)
34. Mizzaro, S. & Robertson, S. (2007). HITS hits TREC: exploring IR evaluation results with network analysis. In Wessel Kraaij, Arjen P. de Vries, Charles L. A. Clarke, Norbert Fuhr, Noriko Kando (Eds.). Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Amsterdam, The Netherlands, July 23-27, 2007. (pp. 479-486). New York, NY: ACM Press. Retrieved 5 May, 2013 from http://www.soi.city.ac.uk/~ser/papers/MizzaroRobertsonSIGIR07.pdf Archived by WebCite® at http://www.webcitation.org/6H1oEKke1)
35. Robertson, S. (2008). On the history of evaluation in IR. Journal of Information Science, 34(4), 439-456
36. Robertson, S.E. & Hancock-Beaulieu, M. (1992). On the evaluation of information retrieval systems. Information Processing and Management, 28(4), 457-466
37. Sanderson, M. (2010). Test collection based evaluation of information retrieval systems. Foundations and Trends in Information Retrieval, 4(4), 247-375. Retrieved 5 May, 2013 from http://www.seg.rmit.edu.au/mark/publications/my_papers/FnTIR.pdf Archived by WebCite® at http://www.webcitation.org/6H1oISFSP)
38. Sanderson, M. & Braschler, M. (2009). Best practices for test collection creation and information retrieval system evaluation. Pisa, Italy: TrebleCLEF. (TrebleCLEF technical report). Retrieved 5 May, 2013 from http://www.seg.rmit.edu.au/mark/publications/my_papers/T-CLEF-test-collection-report.pdf Archived by WebCite® at http://www.webcitation.org/6H1oLiTdt)
39. Sanderson, M., Paramita, M., Clough, P. & Kanoulas, E. (2010). Do user preferences and evaluation measures line up? In Fabio Crestani, Stéphane Marchand-Maillet, Hsin-Hsi Chen, Efthimis N. Efthimiadis, Jacques Savoy (Eds.). Proceedings of the 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Geneva, Switzerland, July 19-23, 2010. (pp. 555-562). New York, NY: ACM Press. Retrieved 5 May, 2013 from http://ir.shef.ac.uk/cloughie/papers/p555-sanderson.pdf Archived by WebCite® at http://www.webcitation.org/6H1oSVAAf)
40. Saracevic, T. (1995). Evaluation of evaluation in information retrieval. In Edward A. Fox, Peter Ingwersen, Raya Fidel (Eds.). Proceedings of 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, Washington, USA, July 9-13, 1995. (pp. 138-146). New York, NY: ACM Press. Retrieved 5 May, 2013 from http://comminfo.rutgers.edu/~muresan/IR/Docs/Articles/sigirSaracevic1995.pdf Archived by WebCite® at http://www.webcitation.org/6H1oVU1EI)
41. Schamber, L. (1994). Relevance and information behavior. Annual Review of Information Science and Technology, 29, 3-48
42. Smucker, M.D. & Clarke, C.L.A. (2012). Time-based calibration of effectiveness measures. In William R. Hersh, Jamie Callan, Yoelle Maarek, Mark Sanderson (Eds.). Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, OR, USA. (pp. 95-104). New York, NY: ACM Press. Retrieved 5 May, 2013 from http://www.mansci.uwaterloo.ca/~msmucker/publications/smucker-clarke-sigir2012.pdf Archived by WebCite® at http://www.webcitation.org/6H1oa1knA)
43. Tague-Sutcliffe, J.M. (1996). Some perspectives on the evaluation of information retrieval systems. Journal of the American Society for Information Science, 47(1), 1-3. Retrieved 5 May, 2013 from http://comminfo.rutgers.edu/~muresan/IR/Docs/Articles/jasisTagueSutcliffe1996.pdf Archived by WebCite® at http://www.webcitation.org/6H1oe6O3B)
44. Taube, M. (1956). Cost as the measure of efficiency of storage and retrieval systems. In Mortimer Taube, Studies in Coordinate Indexing Volume 3. (pp. 18-33). Washington, DC: Documentation Inc.
45. Turpin, A. & Hersh, W. (2001). Why batch and user evaluations do not give the same results. In W. Bruce Croft, David J. Harper, Donald H. Kraft, Justin Zobel (Eds.). Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, September 9-13, 2001, New Orleans, Louisiana, USA. (pp. 225-231). New York, NY: ACM Press.
46. van Rijsbergen, C.J. (1979). Information retrieval. (2nd ed.) London: Butterworths. Retrieved 5 May, 2013 from http://www.dcs.gla.ac.uk/Keith/Preface.html Archived by WebCite® at http://www.webcitation.org/6H1ool2uv)
47. Voorhees, E. M. (1998). Variations in relevance judgments and the measurement of retrieval effectiveness. In Hans-Peter Frei, Donna Harman, Peter Schäuble, Ross Wilkinson (Eds.). Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August 18-22, 1996, Zurich, Switzerland. (pp. 315-323). New York, NY: ACM Press. Retrieved 5 May, 2013 from http://nlp.cs.swarthmore.edu/~richardw/papers/vorhees1999-variations.pdf Archived by WebCite® at http://www.webcitation.org/6H1otD9Jl)
48. Voorhees, E.M. (2000). Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing & Management, 36(5), 697-716
49. Voorhees, E.M. (2002). The philosophy of information retrieval evaluation. In Evaluation of cross-language information retrieval systems: 2nd workshop of the Cross-Language Evaluation Forum (CLEF 2001). (pp. 355-370). Berlin: Springer. (Lecture Notes in Computer Science 2406).
50. Voorhees, E.M. & Harman, D.K. (2005). TREC: experiments and evaluation in information retrieval. Cambridge, MA: MIT Press.
51. Yilmaz, E., Shokouhi, M., Craswell, N. & Robertson, S. (2010). Expected browsing utility for web search evaluation. In Proceedings of the 19th ACM International Conference on Information and Knowledge Management. (pp. 1561-1564). New York, NY: ACM Press.
52. Zhai, C., Cohen, W.W. & Lafferty, J. (2003). Beyond independent relevance: methods and evaluation metrics for subtopic retrieval. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, July 28-August 1, 2003, Toronto, Canada. (pp. 10-17). New York, NY: ACM Press.
53. Zhang, J. & Kamps, J. (2010). A search log-based approach to evaluation. In Proceedings of the 14th European Conference on Research and Advanced Technology for Digital Libraries. (pp. 248-260). Berlin: Springer. (Lecture Notes in Computer Science, 6273)
54. Zobel, J. (1998). How reliable are the results of large-scale information retrieval experiments?In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August 24-28 1998, Melbourne, Australia. (pp. 307-314). New York, NY: ACM Press. Retrieved 5 May, 2013 from http://goanna.cs.rmit.edu.au/~jz/fulltext/sigir98.pdf Archived by WebCite® at http://www.webcitation.org/6H1p6aqJh)