Building a semantically annotated corpus of clinical texts

Journal of Biomedical Informatics

Volumn 42, Issue 5, 2009, Pages 950-966

  Roberts, Angus ^a   Gaizauskas, Robert ^a   Hepple, Mark ^a   Demetriou, George ^a   Guo, Yikun ^a   Roberts, Ian ^a   Setzer, Andrea ^a  

^a UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

Annotation guidelines; Clinical text; Corpora; Evaluation; Gold standards; Information extraction; Natural language processing; Semantic annotation; Temporal annotation; Text mining

Indexed keywords

ANNOTATION GUIDELINES; CLINICAL TEXT; CORPORA; EVALUATION; GOLD STANDARDS; INFORMATION EXTRACTION; NATURAL LANGUAGE PROCESSING; SEMANTIC ANNOTATION; TEMPORAL ANNOTATION; TEXT MINING;

COMPUTATIONAL LINGUISTICS; INFORMATION ANALYSIS; LINGUISTICS; NATURAL LANGUAGE PROCESSING SYSTEMS; SEMANTICS; TEXT PROCESSING; WORD PROCESSING;

SOFTWARE AGENTS;

ARTICLE; BIOMEDICINE; CONTROLLED STUDY; DATA ANALYSIS; DATA EXTRACTION; DATA SYNTHESIS; INFORMATION RETRIEVAL; INFORMATION SYSTEM; MEDICAL RECORD; METHODOLOGY; PRACTICE GUIDELINE; PRIORITY JOURNAL; RANDOM SAMPLE; SEMANTICS;

ABSTRACTING AND INDEXING AS TOPIC; BIOMEDICAL RESEARCH; GUIDELINES AS TOPIC; HUMANS; INFORMATION STORAGE AND RETRIEVAL; INTERNET; MEDICAL RECORDS; MODELS, STATISTICAL; NATURAL LANGUAGE PROCESSING; NEOPLASMS; SEMANTICS; TERMINOLOGY AS TOPIC; USER-COMPUTER INTERFACE;

EID: 70349466623     PISSN: 15320464     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jbi.2008.12.013     Document Type: Article

References (50)

1. Rector A, Rogers J, Taweel A, Ingram D, Kalra D, Milan J, et al. CLEF-joining up healthcare with clinical and post-genomic research. In: Proceedings of UK e-science all hands meeting 2003, Nottingham, UK; 2003. p. 264-7.
2. Grishman R. Information extraction. In: Mitkov R, editor. The Oxford handbook of computational linguistics; 2003 [Chapter 30].
3. Harkema H, Roberts I, Gaizauskas R, Hepple M. Information extraction from clinical records. In: Cox SJ, Walker DW, editors. Proceedings of the UK e-science all hands meeting 2005, Nottingham, UK; 2005. p. 254-8.
4. Riloff E. Automatically generating extraction patterns from untagged text. In: AAAI/IAAI, vol. 2; 1996. p. 1044-9.
5. Roberts A, Gaizauskas R, Hepple M, Davis N, Demetriou G, Guo Y, et al. The CLEF corpus: semantic annotation of clinical text. In: Proc AMIA Symp. Chicago, IL, USA; 2007. p. 625-9.
6. Roberts A, Gaizauskas R, Hepple M, Demetriou G, Guo Y, Setzer A, et al. Semantic annotation of clinical text: the CLEF corpus. In: Proceedings of building and evaluating resources for biomedical text mining: workshop at sixth international conference on language resources and evaluation (LREC) 2008. Marrakech, Morocco: ELRA; 2008.
7. Kim J.D., Ohta T., Tateisi Y., and Tsujii J. GENIA corpus - a semantically annotated corpus for bio-textmining. Bioinformatics 19 1 (2003) i180-i182
8. Kim JD, Ohta T, Tsujii J. Corpus annotation for mining biomedical events from literature. BMC Bioinform 2008;9(10).
9. Kulick S, Bies A, Liberman M, Mandel M, McDonald R, Palmer M, et al. Integrated annotation for biomedical information extraction. In: Hirschman L, Pustejovsky J, editors. HLT-NAACL 2004 workshop: BioLINK 2004, linking biological literature, ontologies and databases. Boston, Massachusetts, USA: Association for Computational Linguistics; 2004. p. 61-8.
10. Franzén K., Gunnar, Eriksson, Olsson F., Asker L., Lidén P., et al. Protein names and how to find them. Int J Med Inform 67 1-3 (2002) 49-61
11. Rosario B, Hearst MA. Classifying semantic relations in bioscience texts. In: Proceedings of the 42nd annual meeting on association for computational linguistics. Morristown, NJ, USA: Association for Computational Linguistics; 2004. p. 430.
12. Rosario B, Hearst MA. Multi-way relation classification: application to protein-protein interactions. In: Proceedings of the conference on human language technology and empirical methods in natural language processing. Morristown, NJ, USA: Association for Computational Linguistics; 2005. p. 732-9.
13. Alex B, Grover C, Haddow B, Kabadjov M, Klein E, Matthews M, et al. The ITI TXM corpora: tissue expressions and protein-protein interactions. In: Proceedings of building and evaluating resources for biomedical text mining: workshop at sixth international conference on language resources and evaluation, LREC 2008, Marrakech, Morocco; 2008. p. 11-8.
14. Tanabe L., Xie N., Thom L.H., Matten W., and Wilbur W.J. GENETAG: a tagged corpus for gene/protein named entity recognition. BMC Bioinform 6 Suppl. 1 (2005) S3
15. Nédellec C. Learning language in logic-genic interaction extraction challenge. In: Proceedings of the ICML05 workshop on learning language in logic, Bonn, Germany; 2005. p. 31-7.
16. TREC genomics track. Available from: http://ir.ohsu.edu/genomics [cited 6 June 2008].
17. Pestian J.P., Brew C., Matykiewicz P., Hovermale D., Johnson N., Cohen K.B., et al. A shared task involving multi-label classification of clinical free text. Biological, translational, and clinical language processing (2007), Association for Computational Linguistics, Prague, Czech Republic 97-104
18. Hersh W.R., Muller H., Jensen J.R., Yang J., Gorman P.N., and Ruch P. Advancing biomedical image retrieval: development and analysis of a test collection. J Am Med Inform Assoc 13 5 (2006) 488-496
19. Mller H, Deselaers T, Lehmann TM, Clough PD, Hersh W. Overview of the ImageCLEFmed 2006 medical retrieval and annotation tasks. In: Cross language evaluation forum (CLEF) workshop 2006, vol. 4730. Alicante, Spain: Springer; 2007. p. 595-608.
20. i2b2 NLP shared task. Available from: http://ir.ohsu.edu/genomics/ [cited 6 June 2008].
21. Ogren PV, Savova G, Buntrock JD, Chute CG. Building and evaluating annotated corpora for medical NLP systems. In: Proc AMIA Symp; 2006. p. 1050.
22. Meystre S., and Haug P.J. Natural language processing to extract medical problems from electronic clinical documents: performance evaluation. J Biomed Inform 39 6 (2006) 589-599
23. Denny J.C., Smithers J.D., Miller R.A., and Spickard A. "Understanding" medical school curriculum content using KnowledgeMap. J Am Med Inform Assoc 10 4 (2003) 351-362
24. Elkin PL, Brown SH, Bauer BA, Husser CS, Carruth W, Bergstrom LR, et al. A controlled trial of automated classification of negation from clinical notes. BMC Med Inform Decision Making 2005;5(13).
25. Friedman C., and Hripcsak G. Evaluating natural language processors in the clinical domain. Methods Inform Med 37 4-5 (1998) 334-344
26. International classification of diseases (ICD). Available from: http://www.who.int/classifications/icd [cited 6 June 2008].
27. Rogers J, Puleston C, Rector A. The CLEF chronicle: patient histories derived from electronic health records. In: Data engineering workshops, 2006 proceedings 22nd international conference on. 2006; p. x109-x109.
28. Hallett C, Power R, Scott D. Summarisation and visualisation of e-health data repositories. In: Proceedings of the UK e-science all hands meeting, Nottingham, UK; 2006. p. 69-77.
29. Gennari J.H., Musen M.A., Fergerson R.W., Grosso W.E., Crubézy M., Eriksson H., et al. The evolution of Protégé: an environment for knowledge-based systems development. Int J Hum-Comput Studies 58 1 (2003) 89-123
30. Ogren PV. Knowtator: a Protégé plug-in for annotated corpus construction. In: Proceedings of the 2006 conference of the North American chapter of the association for computational linguistics on human language technology. Morristown, NJ, USA: Association for Computational Linguistics; 2006. p. 273-5.
31. Defense Advanced Research Projects Agency. Proceedings of the seventh message understanding conference (MUC-7); 1998. Available at: http://www.itl.nist.gov/iaui/894.02/related_projects/muc/.
32. Boisen S, Crystal MR, Schwartz R, Stone R, Weischedel R. Annotating resources for information extraction. In: Proceedings of the second language resources and evaluation, LREC 2000; 2000. p. 1211-4.
33. Demetriou G, Gaizauskas R, Sun H, Roberts A. ANNALIST - ANNotation ALIgnment and scoring tool. In: Proceedings of the sixth international conference on language resources and evaluation, LREC 2008. Marrakech, Morocco: ELRA; 2008.
34. Hripcsak G., and Rothschild A. Agreement, F-measure and reliability in information retrieval. J Am Med Inform Assoc 12 3 (2005) 296-298
35. Cunningham H, Maynard D, Bontcheva K, Tablan V. GATE: a framework and graphical development environment for robust NLP tools and applications. In: Proceedings of the 40th anniversary meeting of the association for computational linguistics. Philadelphia, PA, USA; 2002. p. 168-175.
36. GATE-general architecture for text engineering. Available from: http://gate.ac.uk [cited 6 June 2008].
37. UMLS knowledge sources, 2007AB; 2007.
38. Pustejovsky J, Castaño JC, Ingria R, Saurí R, Gaizauskas R, Setzer A, et al. TimeML: Robust specification of event and temporal expressions in text. In: Proceedings of the fifth international workshop on computational semantics (IWCS-5). Tilburg; 2003.
39. Verhagen M, Gaizauskas R, Schilder F, Hepple M, Katz G, Pustejovsky J. SemEval-2007 task 15: TempEval temporal relation identification. In: Proceedings of the fourth international workshop on semantic evaluations, Prague; 2007. p. 75-80.
40. Mani I, Wilson G. Robust temporal processing of news. In: Proceedings of the 38th annual meeting of the association for computational linguistics (ACL 2000), New Brunswick, New Jersey; 2000. p. 69-76.
41. Harkema H, Gaizauskas R, Hepple M, Davis N, Guo Y, Roberts A, et al. A large-scale resource for storing and recognizing technical terminology. In: Proceedings of fourth international conference on language resources and evaluation, Lisbon, Portugal; 2004. p. 83-6.
42. Lindberg D., Humphreys B., and McCray A. The unified medical language system. Methods Inform Med 32 4 (1993) 281-291
43. Li Y, Bontcheva K, Cunningham H. SVM based learning system for information extraction. In: Deterministic and statistical methods in machine learning: first international workshop. No. 3635 in Lecture Notes in Computer Science. Springer; 2005. p. 319-39.
44. Roberts A, Gaizauskas R, Hepple M, Guo Y. Combining terminology resources and statistical methods for entity recognition: an evaluation. In: Proceedings of the sixth international conference on language resources and evaluation, LREC 2008, Marrakech, Morocco; 2008.
45. Roberts A., Gaizauskas R., Hepple M., and Guo Y. Mining clinical relationships from patient narratives. BMC Bioinform 9 Suppl. 11 (2008) S3
46. Thompson CA, Califf ME, Mooney RJ. Active learning for natural language parsing and information extraction. In: Proceedings 16th international conference on machine learning. San Francisco, CA: Morgan Kaufmann; 1999. p. 406-14.
47. Ghani R, Jones R, Mitchell T, Riloff E. Active learning for information extraction with multiple view feature sets. In: Proceedings of the 20th international conference on machine learning (ICML 2003) workshop on adaptive text extraction and mining; 2003.
48. SAFE, the semantic annotation factory environment. Available from: http://gate.ac.uk/safe/ [cited 2 October 2008].
49. BioNotate. Available from: http://sourceforge.net/projects/bionotate/ [cited 2 October 2008].
50. Clinical e-science framework: Sheffield NLP. Available from: http://nlp.shef.ac.uk/clef/ [cited 2 October 2008].