User performance versus precision measures for simple search tasks

Proceedings of the Twenty-Ninth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn 2006, Issue , 2006, Pages 11-18

  Turpin, Andrew ^a   Scholer, Falk ^a  

^a RMIT UNIVERSITY   (Australia)

Author keywords

Information retrieval evaluation; Search engines; User study

Indexed keywords

DATA ACQUISITION; INFORMATION RETRIEVAL SYSTEMS; PRECISION ENGINEERING; QUERY LANGUAGES; SEARCH ENGINES; USER INTERFACES; QUERY PROCESSING;

DOCUMENTS; INFORMATION RETRIEVAL EVALUATION; MEAN AVERAGE PRECISION (MAP);

ONLINE SEARCHING; SEARCH ENGINES;

ONE-SHOT QUERY EVALUATION; PRECISION MEASURES; QUESTION ANSWERING TASK; WEB-TRACK DATA;

EID: 33750351305     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1148170.1148176     Document Type: Conference Paper

References (21)

1. J. Allan, B. Carterette, and J. Lewis. When will information retrieval be "good enough"? In Proc. ACM SIGIR, pages 433-440, Salvador, Brazil, 2005.
2. P. Bailey, N. Craswell, and D. Hawking. Engineering a multi-purpose test collection for web retrieval experiments. Information Processing and Management, 39(6):853-871, 2003.
3. A. Broder. A taxonomy of web search. SIGIR Forum, 36(2):3-10, 2002.
4. C. Buckley and E. Voorhees. Evaluating evaluation measure stability. In Proc. ACM SIGIR, pages 33-40, Athens, Greece, 2000.
5. C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In Proc. ACM SIGIR, pages 25-32, Sheffield, UK, 2004.
6. C. Buckley and E. M. Voorhees. Retrieval system evaluation. In E. M. Voorhees and D. K. Harman, editors, TREC: experiment and evaluation in information retrieval. MIT Press, 2005.
7. C. Cleverdon. The cranfield tests on index language devices. Aslib Proceedings, 19:173-192, 1967.
8. (Reprinted in K. Sparck Jones and P. Willett, editors. Readings in Information Retrieval. Morgan Kaufmann Publishers Inc., 1997).
9. N. Craswell, D. Hawking, R. Wilkinson, and M. Wu. Overview of the TREC 2003 web track. In The Twelfth Text REtrieval Conference (TREC 2003), pages 78-92, Gaithersburg, MD, 2003. NIST Special Publication 500-255.
10. M. Elsenberg and C. Barry. Order effects: A study of the possible influence of presentation order on user judgments of document relevance. Journal of the American Society for Information Science and Technology, 39:293-301, 1988.
11. D. K. Harman. The TREC test collection. In E. M. Voorhees and D. K. Harman, editors, TREC: experiment and evaluation in information retrieval. MIT Press, 2005.
12. S. P. Harter. Variations in relevance assessments and the measurement of retrieval effectiveness. Journal of the American Society for Information Science, 47(1):37-49, 1996.
13. W. Hersh and P. Over. TREC-9 interactive track report. In The Ninth Text REtrieval Conference (TREC-9), pages 41-50, Gaithersburg, MD, 2000. NIST Special Publication 500-249.
14. W. Hersh, A. Turpin, S. Price, B. Chan, D. Kraemer, L. Sacherek, and D. Olson. Do batch and user evaluations give the same results? In Proc. ACM SIGIR, pages 17-24, Athens, Greece, 2000.
15. W. R. Hersh. Tree 2002 interactive track report. In The Eleventh Text REtrieval Conference (TREC 2002), Gaithersburg, MD, 2002. NIST Special Publication 500-251.
16. T. Joachims, L. A. Granka, B. Pan, H. Hembrooke, and G. Gay. Accurately interpreting clickthrough data as implicit feedback. In Proc. ACM SIGIR, pages 154-161, Salvador, Brazil, 2005.
17. A. Tombros and M. Sanderson. Advantages of query biased summaries in information retrieval. In Proc. ACM SIGIR, pages 2-10, Melbourne, Australia, 1998.
18. A. Turpin and W. Hersh. Why batch and user evaluations do not give the same results. In Proc. ACM SIGIR, pages 225-231, New Orleans, LA, 2001.
19. E. M. Voorhees. Variations in relevance judgements and the measurement of retrieval effectiveness. Information Processing and Management, 36(5):697-716, 2000.
20. E. M. Voorhees. The philosophy of information retrieval evaluation. In CLEF, volume 2406 of Lecture Notes in Computer Science, pages 355-370. Springer, 2001.
21. J. Zobel. How reliable are the results of large-scale information retrieval experiments? In Proc. ACM SIGIR, pages 307-314, Melbourne, Australia, 1998.