When will information retrieval be "good enough"? User effectiveness as a function of retrieval accuracy

SIGIR 2005 - Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2005, Pages 433-440

  Allan, James ^a   Carterette, Ben ^a   Lewis, Joshua ^a  

^a Biologically Inspired Neural and Dynamical Systems Laboratory   (United States)

Author keywords

information retrieval; passage retrieval; performance evaluation; user study

Indexed keywords

PASSAGE RETRIEVAL; PERFORMANCE EVALUATION; RETRIEVAL ACCURACY; RETRIEVAL SYSTEMS; SYSTEM ACCURACY; TIME ON TASKS; USER STUDY; USER UTILITY;

INFORMATION RETRIEVAL;

BLENDING;

EID: 84885591216     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1076034.1076109     Document Type: Conference Paper

References (22)

1. N. Abdul-Jaleel, J. Allan, W. Croft, F. Diaz, L. Larkey, X. Li, D. Metzler, M. Smucker, T. Strohman, H. Turtle, and C. Wade. UMass at TREC 2004: Notebook. In E. Voorhees, editor, The Thirteenth Text Retrieval Conference (TREC 2004) Notebook, pages 657-670, 2004. Available online at http://trec.nist.gov.
2. J. Allan. HARD track overview in TREC 2004 (notebook), high accuracy retrieval from documents. In E. Voorhees, editor, The Thirteenth Text Retrieval Conference (TREC 2004) Notebook, pages 226-235, 2004. Available online at http://trec.nist.gov.
3. Apache Software Foundation. Jakarta Lucene. Can be found at http://jakarta.apache.org/lucene/.
4. D. C. Blair and M. E. Maron. An evaluation of retrieval effectiveness for a full-text document-retrieval system. Communications of the ACM, 28(3):289-299, 1985.
5. S. Boyce and A. Gorin. User interface issues for natural spoken dialog systems. In Proceedings of the International Symposium on Spoken Dialogue 96, pages 65-68, 1996.
6. C. Buckley. Why current IR engines fail. Proceedings of the 27th annual ACM SIGIR conference, pages 584-585, 2004.
7. C. Buckley and E. Voorhees. Retrieval evaluation with incomplete information. Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 25-32, 2004.
8. J. P. Callan. Passage-level evidence in document retrieval. Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, pages 302-310, 1994.
9. Z. Chen and Y. Xu. User-oriented relevance judgment: A conceptual model. In Proceedings of the 38th Hawaii International Conference on System Sciences, page 101, 2005. Abstract is in proceedings; full paper in accompanying CD-ROM.
10. N. Dahlback, A. Jonsson, and L. Ahrenberg. Wizard of Oz studies: why and how. In Proceedings of the 1st international conference on Intelligent user interfaces, pages 193-200, 1993.
11. O. Drori and N. Alon. Using documents classification for displaying search results list. Technical Report 2002-46, Leibniz Center for Research in Computer Science, Hebrew University, Jerusalem, Israel, 2002.
12. M. Kaszkiel and J. Zobel. Passage retrieval revisited. Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, pages 178-185, 1997.
13. A. Kehler, J. Martin, A. Cheyer, L. Julia, J. Hobbs, and J. Bear. On representing salience and reference in multimodal human-computer interaction. In AAAI '98 (Representations for Multi-Model Human Computer Interaction), pages 33-39, 1998.
14. J. Koenemann and N. J. Belkin. A case for interaction: A study of interactive information retrieval behavior and effectiveness. In CHI, pages 205-212, 1996.
15. P. Over. TREC-6 interactive track report. In E. Voorhees and D. Harman, editors, The Sixth Text Retrieval Conference (TREC 6), pages 73-82, 1998. NIST Special Publication 500-240.
16. G. Salton. Automatic text processing: the transformation and analysis and and retrieval of information by computer. Addison-Wesley, 1989.
17. I. Soboroff. Overview of the TREC 2004 novelty track. In E. Voorhees, editor, The Thirteenth Text Retrieval Conference (TREC 2004) Notebook, pages 57-70, 2004.
18. R. Swan and J. Allan. Improving interactive information retrieval effectiveness with 3-d graphics. Technical Report IR-100, University of Massachusetts, Amherst, Center for Intelligent Information Retrieval, 1996.
19. E. Tanin, A. Lotem, I. Haddadin, B. Shneiderman, C. Plaisant, and L. Slaughter. Facilitating network data exploration with query previews: A study of user performance and preference. Technical Report CS-TR-3879, University of Maryland, College Park, 1998.
20. E. Voorhees. Overview of the TREC 2003 question answering track. In E. Voorhees, editor, The Twelfth Text Retrieval Conference (TREC 2003), pages 73-82, 2003. NIST Special Publication 500-255.
21. W. J. Weiland and B. Shneiderman. A graphical query interface based on aggregation/generalization hierarchies. Information Systems, 18(4):215-232, 1993.
22. M. Wu, M. Fuller, and R. Wilkinson. Searcher performance in question answering. Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages 375-381, 2001.