Tasks, topics and relevance judging for the TREC Genomics Track: five years of experience evaluating biomedical text information retrieval systems

Information Retrieval

Volumn 12, Issue 1, 2009, Pages 81-97

  Roberts, Phoebe M ^a   Cohen, Aaron M ^b   Hersh, William R ^b  

^a PFIZER INC   (United States)

^b OREGON HEALTH AND SCIENCE UNIVERSITY   (United States)

Author keywords

Evaluation; Information retrieval; Inter annotator agreement; Reference standards; Text mining

Indexed keywords

EID: 58149218476     PISSN: 13864564     EISSN: 15737659     Source Type: Journal    
DOI: 10.1007/s10791-008-9072-x     Document Type: Article

References (22)

1. P Borlund 2003 The concept of relevance in IR Journal of the American Society for Information Science and Technology 54 10 913 925
2. K Burkhardt B Schneider 2006 A biocurator perspective: Annotation at the Research Collaboratory for Structural Bioinformatics Protein Data Bank PLoS Computational Biology 2 10 e99
3. AM Cohen WR Hersh 2006 The TREC 2004 genomics track categorization task: Classifying full text biomedical documents Journal of Biomedical Discovery and Collaboration 1 4
4. Cohen, K. B., Fox, L., et al. (2005). Empirical data on corpus design and usage in biomedical natural language processing. AMIA Annual Symposium Proceedings, 156-160.
5. ME Colosimo AA Morgan 2005 Data preparation and interannotator agreement: BioCreAtIvE task 1B BMC Bioinformatics 6 Suppl 1 S12 Epub 2005 May 24
6. S Dingare J Finkel 2005 A system for identifying named entities in biomedical text: How results from two evaluations reflect on both the system and the evaluations Comparative and Functional Genomics 6 77 85
7. M Gerstein M Seringhaus 2007 Structured digital abstract makes text mining easy Nature 447 7141 142
8. U Hahn J Wermter 2007 Text mining: Powering the database revolution Nature 448 7150 130
9. Hersh, W., & Bhupatiraju, R. T. (2003). TREC genomics track overview. The Twelfth Text Retrieval Conference (TREC 2004). Gaithersburg, MD: National Institute for Standards & Technology.
10. Hersh, W. R., Cohen, A., et al. (2006). TREC 2006 genomics track overview. The Fifteenth Text Retrieval Conference (TREC 2006). Gaithersburg, MD: National Institute for Standards & Technology.
11. Hersh, W. R., Cohen, A., et al. (2007). TREC 2007 genomics track overview. The Sixteenth Text Retrieval Conference (TREC 2007). Gaithersburg, MD: National Institute for Standards & Technology.
12. L Hirschman AA Morgan 2002 Rutabaga by any other name: Extracting biological names Journal of Biomedical Informatics 35 4 247 259
13. L Hirschman A Yeh 2005 Overview of BioCreAtIvE: Critical assessment of information extraction for biology BMC Bioinformatics 6 Suppl 1 S1
14. G Hripcsak A Wilcox 2002 Reference standards, judges, and comparison subjects: Roles for experts in evaluating system performance Journal of the American Medical Informatics Association 9 1 1 15
15. NC Ide RF Loane 2007 Essie: A concept-based search engine for structured biomedical text Journal of the American Medical Informatics Association 14 3 253 263
16. CE Lipscomb 2000 Medical subject headings (MeSH) Bulletin of the Medical Library Association 88 3 265 266
17. B Medlock 2008 Exploring hedge identification in biomedical literature Journal of Biomedical Informatics 41 4 636 654
18. S Pyysalo A Airola 2008 Comparative analysis of five protein-protein interaction corpora BMC Bioinformatics 9 Suppl 3 S6
19. N Salimi R Vita 2006 The biocurator: Connecting and enhancing scientific data PLoS Computational Biology 2 10 e125
20. EM Voorhees 2000 Variations in relevance judgments and the measurement of retrieval effectiveness Information Processing and Management 36 697 716
21. YC Xu Z Chen 2006 Relevance judgment: What do information users consider beyond topicality? Journal of the American Society for Information Science and Technology 57 7 961 973
22. AS Yeh L Hirschman 2003 Evaluation of text data mining for database curation: Lessons learned from the KDD Challenge Cup Bioinformatics 19 Suppl 1 i331 i339