Evaluating the effects of machine pre-annotation and an interactive annotation interface on manual de-identification of clinical text

Journal of Biomedical Informatics

Volumn 50, Issue , 2014, Pages 162-172

  South, Brett R ^a,b   Mowery, Danielle ^d,e,f   Suo, Ying ^a,b   Leng, Jianwei ^a,b   Ferrández, Óscar ^c   Meystre, Stephane M ^a,b   Chapman, Wendy W ^a,b,d,e,f  

^a UNIVERSITY OF UTAH SCHOOL OF MEDICINE   (United States)

^b VETERANS AFFAIRS MEDICAL CENTER   (United States)

^c NUANCE COMMUNICATIONS   (United States)

^d UNIVERSITY OF PITTSBURGH SCHOOL OF MEDICINE   (United States)

^e VA SAN DIEGO HEALTHCARE SYSTEM   (United States)

^f UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Anonymization; Clinical corpora; Confidentiality; De identification; Electronic health records; Medical informatics; Natural language processing; Patient data privacy

Indexed keywords

DATA PRIVACY; EXPERIMENTS; HEALTH INSURANCE; HOSPITAL DATA PROCESSING; NATURAL LANGUAGE PROCESSING SYSTEMS; RISK ASSESSMENT;

ANONYMIZATION; CLINICAL CORPORA; CONFIDENTIALITY; DE-IDENTIFICATION; ELECTRONIC HEALTH RECORD; MEDICAL INFORMATICS; NATURAL LANGUAGE PROCESSING; PATIENT DATA;

LINGUISTICS;

ARTICLE; AUTOMATION; CONFIDENTIALITY; DATA PROCESSING; ELECTRONIC MEDICAL RECORD; EVALUATION STUDY; HUMAN; INFORMATION PROCESSING; INTERACTIVE ANNOTATION INTERFACE; MACHINE; MACHINE ASSISTED ANNOTATION; MEASUREMENT ACCURACY; MEDICAL DOCUMENTATION; MEDICAL INFORMATICS; MEDICAL INFORMATION; NOMENCLATURE; ORGANIZATION AND MANAGEMENT; PATIENT IDENTIFICATION; PRACTICE GUIDELINE; PRIORITY JOURNAL; PRIVACY; RELIABILITY; TRAINING; COMPUTER INTERFACE; ELECTRONIC HEALTH RECORD; HEALTH INSURANCE; UNITED STATES;

ELECTRONIC HEALTH RECORDS; HEALTH INSURANCE PORTABILITY AND ACCOUNTABILITY ACT; UNITED STATES; USER-COMPUTER INTERFACE;

EID: 84905184800     PISSN: 15320464     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jbi.2014.05.002     Document Type: Article

References (35)

1. GPO USA. CFR Title 45 Subtitle A Part 46: Protection of Human Subjects [Internet]. GPO; October 1, 2008. http://www.access.gpo.gov/nara/cfr/waisidx_08/45cfr46_08.html.
2. Meystre S.M., Friedlin F.J., South B.R., Shen S., Samore M.H. Automatic de-identification of textual documents in the electronic health record: a review of recent research. BMC Med Res Methodol 2010, 10:70.
3. Aberdeen J., Bayer S., Yeniterzi R., Wellner B., Clark C., Hanauer D., et al. The MITRE identification scrubber toolkit: design, training, and assessment. Int J Med Inform 2010, 79(12):849-859.
4. Friedlin F.J., McDonald C.J. A software tool for removing patient identifying information from clinical documents. J Am Med Inform Assoc 2008, 15(5):601-610.
5. Beckwith B.A., Mahaadevan R., Balis U.J., Kuo F. Development and evaluation of an open source software tool for deidentification of pathology reports. BMC Med Inform Decis Mak 2006, 6:12.
6. Gardner J., Xiong L. HIDE: an integrated system for health information DE-identification. IEEE Symp Comput Med Syst Proc 2008, 254-259.
7. Neamatullah I., Douglass M.M., Lehman L.W., Reisner A., Villarroel M., Long W.J., et al. Automated de-identification of free-text medical records. BMC Med Inform Decision Making 2008, 8(1):32.
8. Ferrández O., South B.R., Shen S., Friedlin F.J., Samore M.H., Meystre S.M. BoB, a best-of breed automated text de-identification system for VHA clinical documents. J Am Med Inform Assoc 2013, 20(1):77-83.
9. Sweeney L. Replacing personally-identifying information in medical records, the Scrub system. Proc AMIA Annu Fall Symp 1996, 333-337.
10. Taira R.K., Bui A.A., Kangarloo H. Identification of patient name references within medical documents using semantic selectional restrictions. AMIA Annu Symp Proc 2002, 757-761.
11. Ferrández O., South B.R., Shen S., Friedlin F.J., Samore M.H., Meystre S.M. Evaluating current automatic de-identification methods with Veteran's health administration clinical documents. BMC Med Res Methodol 2012, 12(1):109.
12. Berman J.J. Concept-match medical data scrubbing. How pathology text can be used in research. Arch Pathol Lab Med 2003, 127(6). 680-586.
13. Ruch P., Baud R.H., Rassinoux A.M., Bouillon P., Robert G. Medical document anonymization with a semantic lexicon. AMIA Ann Symp 2000, 729-733.
14. Deleger L., Molnar K., Savova G., Xia F., Lingren T., Li Q., et al. Large-scale evaluation of automated clinical note de-identification and its impact on information extraction. J Am Med Inform Assoc 2012.
15. GPO USA. CFR 45 Subtitle A Part 164: Security and Privacy [Internet]. GPO; October 1, 2008. http://www.access.gpo.gov/nara/cfr/waisidx_08/45cfr164_08.html.
16. Uzuner Ö., Luo Y., Szolovits P. Evaluating the state-of-the-art in automatic de-identification. J Am Med Inform Assoc 2007, 14(5):550-563.
17. Sibanda T., Uzuner Ö. Role of local context in de-identification of ungrammatical. Fragmented Text NAACL-HLT 2006, 65-73.
18. Carrel D., Malin B., Aberdeen J., Bayer S., Clark C., Wellner B., et al. Hiding in plain sight: use of realistic surrogates to reduce exposure of protected health information in clinical text. J Am Med Inform Assoc 2013, 20(2):342-348.
19. Fort K, Sagot B. Influence of pre-annotation on pos-tagged corpus development. In: Proceedings of the fourth linguistic annotation workshop, LAW IV. 2010. p. 56-63.
20. Hanauer D., Aberdeen J., Bayer S., Wellner B., Clark C., Zheng K., et al. Bootstrapping a de-identification system for narrative patient records: cost-performance tradeoffs. Int J Med Inform 2013, 82:821-831.
21. Aronson AR. Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program. In: Proceedings of the AMIA symposium: American medical informatics association; 2001. p. 17.
22. Ganchev K, Pereira F, Mandel M, Carrol S, White P. Semi-automated named entity annotation. In: Proceedings of the linguistic annotation workshop; Prague, Czech Republic: Association for, Computational Linguistics, 2007. p. 53-6.
23. Ogren PV, Savova GK, Chute CG. Constructing evaluation corpora for automated clinical named entity recognition. In: Proceedings of the sixth international conference on language resources and evaluation LREC 2008. p. 3143-50.
24. Lingren T, Deleger L, Molnar K, Zhai H, Meinzen-Derr J, Kaiser M, Stoutenborough L, Li Q, Solte I. Evaluating the impact of pre-annotation on annotation speed and potential bias: natural language processing gold standard development for clinical named entity recognition in clinical trial announcements. J Am Med Inform Assoc doi:10.1136/amiajnl-2013-001837.
25. South BR, Shen S, Friedlin FJ, Samore M, Meystre SM. Enhancing annotation of clinical text using pre-annotation of common PHI. In: Proceedings of the AMIA annual symposium; 2010. p. 1267.
26. Stenetorp P, Pyysalo S, Topic G, Ananiadou S, Tsujii J. BRAT: a web-based tool for NLP-assisted text annotation. EACL 2012; 2012. p. 102.
27. South B, Shen S, Leng J, Forbush T, DuVall S, Chapman WW. A prototype tool set to support machine-assisted annotation. In: Proceedings of the 2012 workshop on biomedical natural language processing. BioNLP '12, Stroudsburg, PA, USA, Association for Computational Linguistics. 2012. p. 130-39.
28. Settles B. ABNER: an open source tool for automatically tagging genes, proteins and other entity names in text. Bioinformatics 2005, 21:3191-3192.
29. Mayer J.M., Shen S., South B.R., Meystre S., Friedlin F.J., Ray W.R., et al. Inductive creation of an annotation schema and a reference standard for de-identification of VA electronic clinical notes. AMIA Annu Symp Proc 2009, 14(2009):416-420.
30. Hripcsak G., Rothschild A. Agreement, the F-measure, and reliability in information retrieval. J Am Med Inform Assoc 2005, 12(3):296-298. May-June.
31. Wilcoxon Rank Sum Test. [Accessed 21.08.13]. http://analytics.ncsu.edu/sesug/2004/TU04-Pappas.pdf.
32. Dorr D.A., Phillips W.F., Phansalkar S., Sims S.A., Hurdle J.F. Assessing the difficulty and time costs of de-identification in clinical narratives. Method Inform Med 2006, 45(3):246-252.
33. South B., Shen S., Maw M., Ferrández O., Friedlin F.J., Meystre S. Prevalence estimates of clinical eponyms in de-identified clinical documents. AMIA Summits Transl Sci Proc CRI 2012, 136.
34. Brooke J. SUS: A "quick and dirty" usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland, editors. Usability evaluation in industry. London UK: Taylor & Francis; 1996, p. 189-94.
35. Felt P, Ringger E, Seppi K, Heal K, Haertel R, Londsdale D. First results in a study evaluating pre-annotation and correction propagation for machine-assisted syriac morphological analysis. In: Proceedings of the tenth international conference on language resources and evaluation. LREC 2012. p. 878-85.