Scaling IR-system evaluation using term relevance sets

Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2004, Pages 10-17

  Amitay, Einat ^a   Carmel, David ^a   Lempel, Ronny ^a   Soffer, Aya ^a  

^a IBM HAIFA RESEARCH LAB   (Israel)

Author keywords

Ir system evaluation; Trels; Web search evaluation

Indexed keywords

DATA REDUCTION; INFORMATION RETRIEVAL; QUERY LANGUAGES; RELIABILITY; SEARCH ENGINES; WORLD WIDE WEB;

DATA SOURCES; INFORMATION SEARCH; POOLING TECHNIQUES; WEB SEARCH EVALUATION;

SYSTEMS ANALYSIS;

EID: 8644259297     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1008992.1008997     Document Type: Conference Paper

References (22)

1. Steven M. Beitzel, Eric C. Jensen, Abdur Chowdhury, and David Grossman. Using titles and category names from editor-driven taxonomies for automatic evaluation. In Proceedings of the twelfth international conference on Information and knowledge management, pages 17-23. ACM Press, 2003.
2. Andrei Z. Broder, Steve C. Glassman, and Mark S. Manasse. Syntactic clustering of the web. In Proceedings of the sixth International World Wide Web Conference, 1997.
3. Chris Buckley and Ellen M. Voorhees. Evaluating evaluation measure stability. In Proceedings of the 23rd annual international A CM SIGIR conference on Research and development in information retrieval, pages 33-40. ACM Press, 2000.
4. David Carmel, Einat Amitay, Miki Herscovici, Yoelle S. Maarek, Yael Petruschka, and Aya Soffer. Juru at TREC 10 - Experiments with Index Pruning. In Proceeding of Tenth Text REtrieval Conference (TREC-10). National Institute of Standards and Technology. NIST, 2001.
5. Junghoo Cho, Hector García-Molina, and Lawrence Page. Efficient crawling through URL ordering. Computer Networks and ISDN Systems, 30(1-7):161-172, 1998.
6. Cyril W. Clevedron. The significance of the cranfield tests on index languages. In Proceedings of the Fourteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3-12, 1991.
7. Ron Fagin, Ravi Kumar, and D. Sivakumar. Comparing top k lists. In ACM-SIAM Symposium on Discrete Algorithms (SODA '03), pages 28-36, 2003.
8. Dennis Fetterly, Mark Manasse, Marc Najork, and Janet Wiener. A large-scale study of the evolution of web pages. In Proceedings of the sixth International World Wide Web Conference (WWW2003), Budapest, Hungary, pages 669-678, 2003.
9. D. Hawking and N. Craswell. Overview of the TREC-2001 Web Track. In E. M. Voorhees and D. K. Harman, editors, Proceedings of the Tenth Text Retrieval Conference (TREC-10). National Institute of Standards and Technology. NIST, 2001.
10. David Hawking, Nick Craswell, Paul Thistlewaite, and Donna Harman. Results and challenges in Web search evaluation. Computer Networks (Amsterdam, Netherlands: 1999), 31(11-16):1321-1330, 1999.
11. Steve Lawrence and C. Lee Giles. Searching the world wide web. Science, 280, April 1998.
12. Yoelle Maarek and Prank Smadja. Full text indexing based on lexical relations: An application: Software libraries. In Proceedings of the Twelfth International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 198-206, Cambridge, MA, June 1989.
13. Marc Najork and Janet L. Wiener. Breadth-First Crawling Yields High-Quality Pages. In Proceedings of the tenth International World Wide Web Conference, pages 114-118, Hong Kong, May 2001.
14. Rabia Nuray and Fazli Can. Automatic ranking of retrieval systems in imperfect environments. In Proceedings of the 26th annual international ACM SIGIR conference on Research and Development in Information Retrieval, pages 379-380. ACM Press, 2003.
15. John O'Connor. Automatic subject recognition in scientific papers: An empirical study. Journal of the ACM, 12(4):490-515, October 1965.
16. Ian Soboroff, Charles Nicholas, and Patrick Cahan. Ranking retrieval systems without relevance judgments. In Proceedings of the 24th annual international ACM SIGIR conference on Research and Development in Information Retrieval, pages 66-73. ACM Press, 2001.
17. Ian Soboroff, Ellen Voorhees, and Nick Craswell. Summary of the SIGIR 2003 workshop on defining evaluation methodologies for terabyte-scale test collections. In SIGIR Forum, volume 37 (2), 2003.
18. M. M. Soubbotin. Patterns of potential answer expressions as clues to the right answers. In Proceeding of Tenth Text REtrieval Conference (TREC-10). National Institute of Standards and Technology. NIST, 2001.
19. Ellen Voorhees. The philosophy of information retrieval evaluation. In Proceedings of the Second Workshop of the Cross-Language Evaluation Forum, (CLEF 2001), pages 355-370, 2001.
20. Ellen M. Voorhees and Donna Harman. Overview of the eighth text retrieval conference (trec-8). Information Processing and Management, 36(1):3-35, 2000.
21. Shengli Wu and Fabio Crestani. Methods for ranking information retrieval systems without relevance judgments. In Proceedings of the 2003 ACM Symposium on Applied Computing (SAC), pages 811-816. ACM Press, 2003.
22. Justin Zobel. How reliable are the results of large-scale information retrieval experiments? In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August, Melbourne, Austrailia, pages 307-314, August 1998.