Relatively relevant: Assessor shift in document judgements

ADCS 2010 - Proceedings of the Fifteenth Australasian Document Computing Symposium

Volumn , Issue , 2010, Pages 60-67

  Sanderson, Mark ^a   Scholer, Falk ^a   Turpin, Andrew ^b  

^a RMIT UNIVERSITY   (Australia)

^b UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

Cranfield approach; Information retrieval evaluation; Relevance judgements; Trec

Indexed keywords

CRANFIELD; HUMAN ASSESSMENT; RELEVANCE JUDGEMENT; RELEVANT DOCUMENTS; TEST COLLECTION; TREC;

INFORMATION RETRIEVAL SYSTEMS;

EID: 84872873938     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Article

References (24)

1. Y. Bernstein and J. Zobel. Redundant documents and search effectiveness. In Proceedings of the 14th ACM international conference on Information and knowledge management, pages 736-743. ACM, 2005.
2. Chris Buckley and Ellen M. Voorhees. Retrieval system evaluation. In Ellen M. Voorhees and Donna K. Harman (editors), TREC: experiment and evaluation in information retrieval. MIT Press, 2005.
3. S. Bttcher, C. L. A. Clarke, P. C. K. Yeung and I. Soboroff. Reliable information retrieval evaluation with incomplete and biased judgements. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 63-70. ACM Press New York, NY, USA, 2007.
4. H. Chen and D. R. Karger. Less is more: probabilistic models for retrieving fewer relevant documents. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 429-436. ACM New York, NY, USA, 2006.
5. Cyril Cleverdon. The cranfield tests on index language devices. Aslib Proceedings, Volume 19, pages 173- 192, 1967. (Reprinted in K. Sparck Jones and P. Willett, editors. Readings in Information Retrieval. Morgan Kaufmann Publishers Inc., 1997).
6. Carlos Cuadra and Robert Katter. The relevance of relevance assessment. In Proceedings of the American Documentation Institute, Volume 4, pages 95-99, 1967.
7. Benedetto de Martino, Dharshan Kumaran, Ben Seymour and Raymond J. Dolan. Frames, biases and rational decision-making in the human brain. Science, Volume 313, pages 684-687, 2006.
8. Donna K. Harman. The TREC test collection. In Ellen M. Voorhees and Donna K. Harman (editors), TREC: experiment and evaluation in information retrieval. MIT Press, 2005.
9. M. Lesk and G. Salton. Relevance assessments and retrieval system evaluation. Information storage and retrieval, Volume 4, Number 4, pages 343-359, 1968.
10. M. Sanderson. Accurate user directed summarization from existing tools. In Proceedings of the seventh international conference on Information and knowledge management, pages 45-51. ACM New York, NY, USA, 1998.
11. T. Saracevic. Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance. Journal of the American Society for Information Science and Technology, Volume 58, Number 13, pages 1915- 1933, 2007.
12. M. N. K. Saunders and S. M. Davis. The use of assessment criteria to ensure consistency of marking: some implications for good practice. Quality Assurance in Education, Volume 6, Number 3, pages 162-171, 1998.
13. James Shanteau. Psychological characteristics and strategies of expert decision makers. Acta Psychologica, Volume 68, Number 1-3, pages 203 - 215, 1988.
14. David Sheskin. Handbook of parametric and nonparametric statistical procedures. CRC Press, 1997.
15. M. D. Smucker, J. Allan and B. Carterette. A comparison of statistical significance tests for information retrieval evaluation. In Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, pages 623-632. ACM New York, NY, USA, 2007.
16. Eero Sormunen. Liberal relevance criteria of TREC - counting on negligible documents? In Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval, pages 324- 330, Tampere, Finland, 2002.
17. I. Suto, R. Ńadas and J. Bell. Who should mark what? a study of factors affecting marking accuracy in a biology examination. Research Papers in Education, pages 1- 31, 2009.
18. Amos Tversky and Daniel Kahneman. The framing of decisions and the psychology of choice. Science, Volume 211, pages 453-458, 1981.
19. E. M. Voorhees and D. Harman. Overview of the fifth text retrieval conference (TREC-5). In The Fifth Text Retrieval Conference (TREC-5), Gaithersburg, MD, USA, NIST Special Publication, Gaithersburg, MD, USA, 1996. Department of Commerce, National Institute of Standards and Technology.
20. Ellen M. Voorhees. Variations in relevance judgements and the measurement of retrieval effectiveness. Information Processing and Management, Volume 36, Number 5, pages 697-716, 2000.
21. Ellen M. Voorhees and Chris Buckley. The effect of topic set size on retrieval experiment error. In Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval, pages 316-323, Tampere, Finland, 2002.
22. Ellen M. Voorhees and Donna K. Harman. TREC: experiment and evaluation in information retrieval. MIT Press, 2005.
23. William Webber, Alistair Moffat and Justin Zobel. Statistical power in retrieval experimentation. In Proceedings of the ACM CIKM International Conference on Information and Knowledge Management, pages 571- 580, Napa Valley, California, USA, 2008.
24. Justin Zobel. How reliable are the results of large-scale information retrieval experiments? In Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval, pages 307- 314, Melbourne, Australia, 1998.