Preparing an annotated gold standard corpus to share with extramural investigators for de-identification research

Journal of Biomedical Informatics

Volumn 50, Issue , 2014, Pages 173-183

  Deleger, Louise ^a   Lingren, Todd ^a   Ni, Yizhao ^a   Kaiser, Megan ^a   Stoutenborough, Laura ^a   Marsolo, Keith ^a   Kouril, Michal ^a   Molnar, Katalin ^a   Solti, Imre ^a  

^a CINCINNATI CHILDREN'S HOSPITAL MEDICAL CENTER   (United States)

Author keywords

Automated de identification; De identification gold standard; Health insurance portability and accountability act; Natural language processing; Privacy of patient data; Protected health information

Indexed keywords

ARTIFICIAL INTELLIGENCE; GOLD; HEALTH INSURANCE; HOSPITAL DATA PROCESSING; LEARNING ALGORITHMS; LEARNING SYSTEMS; NATURAL LANGUAGE PROCESSING SYSTEMS; STATISTICAL TESTS;

DE-IDENTIFICATION; GOLD STANDARDS; HEALTH INSURANCE PORTABILITY AND ACCOUNTABILITY ACTS; NATURAL LANGUAGE PROCESSING; PATIENT DATA; PROTECTED HEALTH INFORMATIONS;

LINGUISTICS;

ARTICLE; CROSS TRAINING; DE IDENTIFICATION; GOLD STANDARD; MACHINE LEARNING; MEDICAL INFORMATICS; MEDICAL INFORMATION; MEDICAL RESEARCH; PRIORITY JOURNAL; COMPUTER SECURITY; ELECTRONIC MEDICAL RECORD; HEALTH INSURANCE; UNITED STATES;

COMPUTER SECURITY; ELECTRONIC HEALTH RECORDS; HEALTH INSURANCE PORTABILITY AND ACCOUNTABILITY ACT; MEDICAL INFORMATICS; UNITED STATES;

EID: 84905195863     PISSN: 15320464     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jbi.2014.01.014     Document Type: Article

References (24)

1. Meystre S.M., Friedlin F.J., South B.R., Shen S., Samore M.H. Automatic de-identification of textual documents in the electronic health record: a review of recent research. BMC Med Res Methodol 2010, 10:70. [Epub 2010/08/04].
2. Neamatullah I., Douglass M.M., Lehman L.W., Reisner A., Villarroel M., Long W.J., et al. Automated de-identification of free-text medical records. BMC Med Inform Decis Mak 2008, 8:32. [Epub 2008/07/26].
3. Uzuner O., Luo Y., Szolovits P. Evaluating the state-of-the-art in automatic de-identification. J Am Med Inform Assoc: JAMIA 2007, 14(5):550-563. [Epub 2007/06/30].
4. Beckwith B.A., Mahaadevan R., Balis U.J., Kuo F. Development and evaluation of an open source software tool for deidentification of pathology reports. BMC Med Inform Decis Mak 2006, 6:12. [Epub 2006/03/07].
5. Gupta D., Saul M., Gilbertson J. Evaluation of a deidentification (De-Id) software engine to share pathology reports and clinical documents for research. Am J Clin Pathol 2004, 121(2):176-186. [Epub 2004/02/27].
6. Thomas SM, Mamlin B, Schadow G, McDonald C. A successful technique for removing names in pathology reports using an augmented search and replace method. In: Proceedings of the AMIA annual symposium; 2002. p. 777-81 [Epub 2002/12/05].
7. Morrison F.P., Li L., Lai A.M., Hripcsak G. Repurposing the clinical record: can an existing natural language processing system de-identify clinical notes?. J Am Med Inform Assoc: JAMIA 2009, 16(1):37-39. [Epub 2008/10/28].
8. Benton A., Hill S., Ungar L., Chung A., Leonard C., Freeman C., et al. A system for de-identifying medical message board text. BMC Bioinformatics 2011, 12(Suppl. 3):S2. [Epub 2011/06/17].
9. Gardner J., Xiong L. HIDE: an integrated system for health information DE-identification. Comp Med Syst 2008, 254-259.
10. Goldberger A.L., Amaral L.A.N, Glass L., Hausdorff J.M., Ivanov P.Ch., Mark R.G., et al. PhysioBank, PhysioToolkit, and Physionet: Components of a New Research Resource for Complex Physiologic Signals. Circulation 2000, 101(23):e215-e220.
11. Yeniterzi R., Aberdeen J., Bayer S., Wellner B., Clark C., Hirschman L., et al. Effects of personal identifier resynthesis on clinical text De-identification. J Am Med Inform Assoc 2010, 17(2):159-168.
12. Alfalahi A, Brissman S, Dalianis H. Pseudonymisation of personal names and other PHIs in an annotated clinical Swedish corpus. In: Third workshop on building and evaluating resources for biomedical text mining workshop programme; 2012.
13. Aberdeen J., Bayer S., Yeniterzi R., Wellner B., Clark C., Hanauer D., et al. The MITRE Identification Scrubber Toolkit: design, training, and assessment. Int J Med Inform 2010, 79(12):849-859. [Epub 2010/10/19].
14. Deleger L., Molnar K., Savova G., Xia F., Lingren T., Li Q., et al. Large-scale evaluation of automated clinical note de-identification and its impact on information extraction. J Am Med Inform Assoc 2012, doi:10.1136/amiajnl-2012-001012 [August 2, Epub ahead of print].
15. Deleger L, Li Q, Lingren T, Kaiser M, Molnar K, Stoutenborough L, et al. Building gold standard corpora for medical natural language processing tasks. In: American medical informatics association annual symposium proceedings; 2012. p 144-53.
16. US Census Bureau. http://www.census.gov/genealogy/names/.
17. Sibanda T, Uzuner O. Role of local context in automatic deidentification of ungrammatical, fragmented text. In: Proceedings of the main conference on human language technology conference of the North American chapter of the association of computational linguistics; 2006. p. 65-73.
18. McCallum AC. MALLET: a machine learning for language toolkit; 2002.
19. Friedman C., Hripcsak G. Evaluating natural language processors in the clinical domain. Methods Inf Med 1998, 37(4-5):334-344. [Epub 1998/12/29].
20. Hripcsak G., Rothschild A.S. Agreement, the f-measure, and reliability in information retrieval. J Am Med Inform Assoc: JAMIA 2005, 12(3):296-298. doi: 10.1197/jamia.M1733 [Epub 2005/02/03, PubMed PMID: 15684123; PubMed Central PMCID: PMC1090460].
21. Noreen E.W. Computer-intensive methods for testing hypotheses: an introduction 1989, Wiley, New-York.
22. Cinchor N. The statistical significance of MUC4 results. In: MUC4 '92 Proceedings of the 4th conference on message understanding; 1992. p. 30-50.
23. Bland J.M., Altman D.G. Multiple significance tests: the Bonferroni method. BMJ 1995, 310(6973):170. [Epub 1995/01/21. PubMed PMID: 7833759; PubMed Central PMCID: PMC2548561].
24. Lingren T., Deléger L., Molnar K., Zhai H., Meinzen-Derr J., Kaiser M., et al. Evaluating the impact of pre-annotation on annotation speed and potential bias: NLP gold standard development for clinical named entity recognition in clinical trial announcements. J Am Med Inform Assoc 2013, [September 3, Epub ahead of print]. 10.1136/amiajnl-2013-001837.