Evaluating the impact of pre-annotation on annotation speed and potential bias: Natural language processing gold standard development for clinical named entity recognition in clinical trial announcements

Journal of the American Medical Informatics Association

Volumn 21, Issue 3, 2014, Pages 406-413

  Lingren, Todd ^a   Deleger, Louise ^a   Molnar, Katalin ^a   Zhai, Haijun ^a   Meinzen Derr, Jareen ^a   Kaiser, Megan ^a   Stoutenborough, Laura ^a   Li, Qi ^a   Solti, Imre ^a  

^a CINCINNATI CHILDREN'S HOSPITAL MEDICAL CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANNOTATION; ARTICLE; BOOK; CLINICAL TRIAL (TOPIC); GOLD STANDARD; HUMAN; INFORMATION PROCESSING; LEARNING ALGORITHM; MACHINE LEARNING; NATURAL LANGUAGE PROCESSING; PREANNOTATION; STATISTICAL SIGNIFICANCE; ANALYSIS OF VARIANCE; CLINICAL TRIAL ANNOUNCEMENTS; EVALUATION STUDY; INFORMATION EXTRACTION; INFORMATION RETRIEVAL; METHODOLOGY; NAMED ENTITY RECOGNITION; PRE-ANNOTATION; TASK PERFORMANCE; UNIFIED MEDICAL LANGUAGE SYSTEM;

CLINICAL TRIAL ANNOUNCEMENTS; INFORMATION EXTRACTION; NAMED ENTITY RECOGNITION; NATURAL LANGUAGE PROCESSING; PRE-ANNOTATION; UMLS;

ANALYSIS OF VARIANCE; CLINICAL TRIALS AS TOPIC; HUMANS; INFORMATION STORAGE AND RETRIEVAL; NATURAL LANGUAGE PROCESSING; TIME AND MOTION STUDIES;

EID: 84901847800     PISSN: 10675027     EISSN: 1527974X     Source Type: Journal    
DOI: 10.1136/amiajnl-2013-001837     Document Type: Article

References (29)

1. Tomanek K, Wermter J, Hahn U. Efficient annotation with the jena annotation environment ( JANE). Linguistic Annotation Workshop-A Merger of NLPXML 2007 and FLAC 2007; Prague, Czech Republic: Association for Computational Linguistics (ACL), 2007:9-16.
2. Ganchev K, Pereira F, Mandel M, et al. Semi-automated named entity annotation. Proceedings of the Linguistic Annotation Workshop; Prague, Czech Republic: Association for Computational Linguistics, 2007:53-6.
3. Aramaki E, Miura Y, Tonoike M, et al. Extraction of adverse drug effects from clinical records. Stud Health Technol Inform 2010;160(Pt 1):739-43.
4. Ogren P, Savova G, Chute C. Constructing evaluation corpora for automated clinical named entity recognition. Proceedings of the Language Resources and Evaluation Conference (LREC); 2008:28-30.
5. Hachey B, Alex B, Becker M. Investigating the effects of selective sampling on the annotation task. Proceedings of the Ninth Conference on Computational Natural Language Learning; Ann Arbor, Michigan: Association for Computational Linguistics, 2005:144-51.
6. Ringger E, Carmen M, Haertel R, et al. Assessing the costs of machine-assisted corpus annotation through a user study. Proceedings of the Language Resources and Evaluation Conference (LREC); 2008.
7. Marcus MP, Marcinkiewicz MA, Santorini B. Building a large annotated corpus of English: the Penn Treebank. Comput Linguist 1993;19:313-30.
8. Fort K, Sagot B. Influence of pre-annotation on POS-tagged corpus development. Proceedings of the Fourth Linguistic Annotation Workshop; Uppsala, Sweden: Association for Computational Linguistics, 2010:56-63.
9. Chiou FD, Chiang D, Palmer M. Facilitating treebank annotation using a statistical parser. Proceedings of the first international conference on human language technology research; San Diego: Association for Computational Linguistics, 2001:1-4.
10. Névéol A, Islamaj Dogan R, Lu Z. Semi-automatic semantic annotation of PubMed queries: a study on quality, efficiency, satisfaction. J Biomed Inform 2011;44:310-18.
11. Jiang J, Zhai CX. A two-stage approach to domain adaptation for statistical classifiers. Proceedings of the Sixteenth Conference on Information and Knowledge Management; Lisbon, Portugal: ACM, 2007:401-10.
12. Aronson AR. Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program. Proceedings of the AMIA Symposium: American Medical Informatics Association; 2001:17.
13. Salgado D, Krallinger M, Depaule M, et al. MyMiner: a web application for computer-assisted biocuration and text annotation. Bioinformatics 2012;28:2285-7.
14. Felt P, Merkling O, Carmen M, et al. CCASH: a web application framework for efficient distributed language resource development. Proceedings of Language Resources and Evaluation Conference (LREC); 2010.
15. South BR, Shen S, Leng J, et al. A prototype tool set to support machine-assisted annotation. Proceedings of the Workshop on Biomedical Natural Language Processing (BioNLP 2012); 8 June 2012, 2012:130-9.
16. Fragkou P, Petasis G, Theodorakos A, et al. BOEMIE ontology-based text annotation tool. Proceedings of the Language Resources and Evaluation Conference (LREC); 2008:28-30.
17. Settles B. ABNER: an open source tool for automatically tagging genes, proteins and other entity names in text. Bioinformatics 2005;21:3191-2.
18. Stenetorp P, Pyysalo S, Topic G, et al. BRAT: a web-based tool for NLP-assisted text annotation. EACL 2012;2012:102.
19. Krippendorff K. Content analysis: an introduction to its methodology. Sage Publications, 2004.
20. Carmen M, Felt P, Haertel R, et al. Tag dictionaries accelerate manual annotation. Proceedings of Language Resources and Evaluation Conference (LREC); 2010.
21. South BR, Shen S, Friedlin FJ, et al. Enhancing annotation of clinical text using pre-annotation of common PHI. Proceedings of the AMIA Annual Symposium; 2010:1267.
22. Albright D, Lanfranchi A, Fredriksen A, et al. Towards comprehensive syntactic and semantic annotations of the clinical narrative. J Am Med Inform Assoc 2013; 20:922-30.
23. Chapman WW, Dowling JN, Hripcsak G. Evaluation of training with an annotation schema for manual annotation of clinical conditions from emergency department reports. Int J Med Inform 2008;77:107-13.
24. UMLS Terminology Services. https://uts.nlm.nih.gov/home.html (accessed 23 Mar 2013).
25. Ogren PV. Knowtator: a protégé plug-in for annotated corpus construction. In: Proceedings of the 2006 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology; 2006:273-5. http://knowtator.sourceforge.net/.
26. http://www.clinicaltrials.gov/ct2/home/.
27. Li Q, Zhai H, Deleger L, et al. A sequence labeling approach to link medications and their attributes in clinical notes and clinical trial announcements for information extraction. J Am Med Inform Assoc 2013;20:915-21.
28. Hripcsak G, Rothschild AS. Agreement, the F-measure, and reliability in information retrieval. J Am Med Inform Assoc 2005;12:296-8.
29. Bland JM, Altman DG. Multiple significance tests: the Bonferroni method. BMJ 1995;310:170.