Benchmarking Clinical Speech Recognition and Information Extraction: New Data, Methods, and Evaluations

JMIR Medical Informatics

Volumn 3, Issue 2, 2015, Pages

  Suominen, Hanna ^a,b,c,d   Zhou, Liyuan ^a,b   Hanlen, Leif ^a,b,c   Ferraro, Gabriela ^a,b  

^a CSIRO   (Australia)

^b AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

^c UNIVERSITY OF CANBERRA   (Australia)

^d UNIVERSITY OF TURKU   (Finland)

Author keywords

Computer systems evaluation; Data collection; Information extraction; Nursing records; Patient handoff; Records as topic; Speech recognition software

Indexed keywords

EID: 85019543571     PISSN: None     EISSN: 22919694     Source Type: Journal    
DOI: 10.2196/medinform.4321     Document Type: Article

References (75)

1. Glaser SR, Zamanou S, Hacker K. Measuring and interpreting organizational culture. Management Communication Quarterly 1987 Nov 01;1(2):173-198. [doi: 10.1177/0893318987001002003].
2. Patterson ES, Roth EM, Woods DD, Chow R, Gomes JO. Handoff strategies in settings with high consequences for failure: Lessons for health care operations. Int J Qual Health Care 2004 Apr;16(2):125-132 [FREE Full text] [doi: 10.1093/intqhc/mzh026] [Medline: 15051706].
3. Tran DT, Johnson M. Classifying nursing errors in clinical management within an Australian hospital. Int Nurs Rev 2010 Dec;57(4):454-462. [doi: 10.1111/j.1466-7657.2010.00846.x] [Medline: 21050197].
4. Matic J, Davidson PM, Salamonson Y. Review: Bringing patient safety to the forefront through structured computerisation during clinical handover. J Clin Nurs 2011 Jan;20(1-2):184-189. [doi: 10.1111/j.1365-2702.2010.03242.x] [Medline: 20815861].
5. Finlayson SG, LePendu P, Shah NH. Building the graph of medicine from millions of clinical narratives. Sci. Data 2014 Sep 16;1:140032. [doi: 10.1038/sdata.2014.32].
6. Australian NSW Department of Health. 2009. Implementation toolkit: Standard key principles for clinical handoverURL:http://www.archi.net.au/documents/resources/qs/clinical/clinical-handover/implementation-toolkit.pdf [accessed 2015-02-04] [WebCite Cache ID 6W4wf1AzA].
7. Western Australian Department of Health. 2013. WA health clinical handover policyURL:http://www. safetyandquality.health.wa.gov.au/docs/initiative/CLINICAL-HANDOVER-Policy.pdf [accessed 2015-02-04] [WebCite Cache ID 6W4wNBaKD].
8. National Health Medical Research Council, Australian Research Council, Australian Vice-Chancellors' Committee. National statement on ethical conduct in human research. Canberra, ACT, Australia: Australian National Health and Medical Research Council; 2007. URL:https://www.nhmrc.gov.au/guidelines-publications/e72 [accessed 2015-04-01] [WebCite Cache ID 6XTHQ5iOr].
9. Friedlin FJ, McDonald CJ. A software tool for removing patient identifying information from clinical documents. J Am Med Inform Assoc 2008;15(5):601-610 [FREE Full text] [doi: 10.1197/jamia.M2702] [Medline: 18579831].
10. Meystre SM, Savova GK, Kipper-Schuler KC, Hurdle JF. Extracting information from textual documents in the electronic health record: A review of recent research. Yearb Med Inform 2008:128-144. [Medline: 18660887].
11. Neamatullah I, Douglass MM, Lehman LWH, Reisner A, Villarroel M, Long WJ, et al. Automated de-identification of free-text medical records. BMC Med Inform Decis Mak 2008;8:32 [FREE Full text] [doi: 10.1186/1472-6947-8-32] [Medline: 18652655].
12. El EK, Arbuckle L, Koru G, Eze B, Gaudette L, Neri E, et al. De-identification methods for open health data: The case of the Heritage Health Prize claims dataset. J Med Internet Res 2012;14(1):e33 [FREE Full text] [doi: 10.2196/jmir.2001] [Medline: 22370452].
13. Pathak MA. Privacy-preserving machine learning for speech processing. Berlin Heidelberg, Germany: Springer Theses; 2013:978-971.
14. Hrynaszkiewicz I, Norton ML, Vickers AJ, Altman DG. Preparing raw clinical data for publication: Guidance for journal editors, authors, and peer reviewers. BMJ 2010;340:c181 [FREE Full text] [Medline: 20110312].
15. BioGrid Australia: About us. URL:https://www.biogrid.org.au/page/3/about-us [accessed 2015-02-04] [WebCite Cache ID 6W5BweRCc].
16. I2b2: Informatics for integrating biology & the bedside, data sets. URL:https://www.i2b2.org/NLP/DataSets/Main.php [accessed 2015-03-25] [WebCite Cache ID 6XIGjCDeA].
17. Linguistic Data Consortium. Linguistic data consortium, data access. URL:https://www.ldc.upenn.edu/language-resources/data/obtaining [accessed 2015-02-04] [WebCite Cache ID 6W4yL9sG8].
18. Dunn AG, Day RO, Mandl KD, Coiera E. Learning from hackers: Open-source clinical trials. Sci Transl Med 2012 May 2;4(132):132cm5 [FREE Full text] [doi: 10.1126/scitranslmed.3003682] [Medline: 22553248].
19. Morita M, Kono Y, Ohkuma T. Overview of the NTCIR-10 MedNLP task. In: Proceedings of the 10th NTCIR Conference. Tokyo, Japan: NII Testbeds and Community for Information access Research (NTCIR); 2013 Presented at: 10th NTCIR; 2013 June 18-21; Tokyo, Japan p. 696-701 URL:http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings10/pdf/NTCIR/MedNLP/04-NTCIR10-MEDNLP-ImachiH.pdf.
20. Poissant L, Pereira J, Tamblyn R, Kawasumi Y. The impact of electronic health records on time efficiency of physicians and nurses: A systematic review. J Am Med Inform Assoc 2005;12(5):505-516 [FREE Full text] [doi: 10.1197/jamia.M1700] [Medline: 15905487].
21. Hakes B, Whittington J. Assessing the impact of an electronic medical record on nurse documentation time. Comput Inform Nurs 2008;26(4):234-241. [doi: 10.1097/01.NCN.0000304801.00628.ab] [Medline: 18600132].
22. Banner L, Olney CM. Automated clinical documentation: Does it allow nurses more time for patient care? Comput Inform Nurs 2009;27(2):75-81. [doi: 10.1097/NCN.0b013e318197287d] [Medline: 21685832].
23. Zweigenbaum P, Demner-Fushman D, Yu H, Cohen KB. Frontiers of biomedical text mining: Current progress. Brief Bioinform 2007 Sep;8(5):358-375 [FREE Full text] [doi: 10.1093/bib/bbm045] [Medline: 17977867].
24. Demner-Fushman D, Chapman WW, McDonald CJ. What can natural language processing do for clinical decision support? J Biomed Inform 2009 Oct;42(5):760-772 [FREE Full text] [doi: 10.1016/j.jbi.2009.08.007] [Medline: 19683066].
25. Nadkarni PM, Ohno-Machado L, Chapman WW. Natural language processing: An introduction. J Am Med Inform Assoc 2011;18(5):544-551 [FREE Full text] [doi: 10.1136/amiajnl-2011-000464] [Medline: 21846786].
26. Friedman C, Rindflesch TC, Corn M. Natural language processing: State of the art and prospects for significant progress, a workshop sponsored by the National Library of Medicine. J Biomed Inform 2013 Oct;46(5):765-773 [FREE Full text] [doi: 10.1016/j.jbi.2013.06.004] [Medline: 23810857].
27. Friedman C, Elhadad N. Natural language processing in health care and biomedicine. In: Shortliffe EH, Cimino JJ, editors. Biomedical Informatics: Computer Applications in Health Care and Biomedicine. London, UK: Springer-Verlag; 2014:255-284.
28. Suominen H. Text mining and information analysis of health documents. Artif Intell Med 2014 Jul;61(3):127-130. [doi: 10.1016/j.artmed.2014.06.001] [Medline: 24998391].
29. Chapman WW, Nadkarni PM, Hirschman L, D'Avolio LW, Savova GK, Uzuner O. Overcoming barriers to NLP for clinical text: The role of shared tasks and the need for additional creative solutions. J Am Med Inform Assoc 2011;18(5):540-543 [FREE Full text] [doi: 10.1136/amiajnl-2011-000465] [Medline: 21846785].
30. Voorhees EM, Hersh W. NIST Special Publication 500-298: The Twenty-First Text REtrieval Conference Proceedings (TREC 2012). Gaithersburg, MD, USA: Department of Commerce, National Institute of Standards and Technology; 2012. Overview of the TREC 2012 medical records trackURL:http://trec.nist.gov/pubs/trec21/papers/MED12OVERVIEW.pdf [accessed 2015-03-28] [WebCite Cache ID 6XMdUZS3E].
31. Suominen H, Salanterä S, Velupillai S, Chapman MM, Savova G, Elhadad N. In: Forner P, Müller H, Paredes R, Rosso P, Stein B, editors. Information access evaluation. Multilinguality, multimodality, and visualization, lecture notes in computer science 8138. Berlin, Germany: Springer-Verlag; 2013. Overview of the ShARe/CLEF eHealth Evaluation Lab 2013URL:http://www.springer.com/us/book/9783642408014 [accessed 2015-04-01] [WebCite Cache ID 6XTIz8aom].
32. The patient record. Health design challengeURL:http://healthdesignchallenge.com/[accessed 2015-01-23] [WebCite Cache ID 6VmhTcVol].
33. Kuniavsky M. Observing the user experience: A practitioner's guide to user research. San Francisco, CA, USA: Morgan Kaufmann Publishers; 2003.
34. Australian Government, Department of Health. Chronic disease: Chronic diseases are leading causes of death and disability in Australia. 2015. URL:http://www.health.gov.au/internet/main/publishing.nsf/Content/chronic [accessed 2015-01-23] [WebCite Cache ID 6Vmix5u6i].
35. Chaboyer W. Clinical handover. Slides.: NHMRC Centre of Resear Excellence in Nursing Care for Hospitalised Patients; 2011. URL:http://www.health.qld.gov.au/psq/handover/docs/ch-presentation2.pdf [accessed 2015-02-04] [WebCite Cache ID 6W5IQSZY2].
36. Suominen H, Johnson M, Zhou L, Sanchez P, Sirel R, Basilakis J, et al. Capturing patient information at nursing shift changes: Methodological evaluation of speech recognition and information extraction. J Am Med Inform Assoc 2014 Oct 21. [doi: 10.1136/amiajnl-2014-002868] [Medline: 25336589].
37. American Nurses Association. 2015. The nursing process URL:http://www.nursingworld.org/EspeciallyForYou/What-is-Nursing/Tools-You-Need/[accessed 2015-01-23] [WebCite Cache ID 6VmkclWzr].
38. Ogren PV. Knowtator: A Protégé plug-in for annotated corpus construction. Stroudsburg, PA, USA: Association for Computational Linguistics; 2006. Proceedings of the 2006 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology; New York, NY, USA; 2006 JunURL:http://www.aclweb.org/anthology/N06-4006 [accessed 2015-03-27] [WebCite Cache ID 6XNQAerTu].
39. Audacity® is free, open source, cross-platform software for recording and editing sounds. URL:http://audacity.sourceforge.net/[accessed 2015-02-04] [WebCite Cache ID 6W5If78xf].
40. US National Institute of Standards and Technology (NIST), Information Technology Laboratory, Information Access Division (IAD). Tools. URL:http://www.itl.nist.gov/iad/mig/tools/[accessed 2015-02-04] [WebCite Cache ID 6W5ImdeUH].
41. Mermelstein P. Pattern Recognition and Artificial Intelligence. 1976. Distance measures for speech recognition-psychological and instrumentalURL:http://web.haskins.yale.edu/sr/SR047/SR047-07.pdf [accessed 2015-03-28] [WebCite Cache ID 6XNPoVqG1].
42. Kaki S, Sumita E, Iidar H. Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics (ACL 1998) and 17th International Conference on Computational Linguistics. Stroudsburg, PA, USA: Association for Computational Linguistics; 1998. A method for correcting errors in speech recognition using the statistical features of character co-occurrenceURL:http://www.aclweb.org/anthology/P98-1107 [accessed 2015-03-28] [WebCite Cache ID 6XNPgyBjd].
43. Jeong M, Kim B, Lee GG. Proceedings of the HLTNAACL special workshop on Higher-Level Linguistic Information for Speech Processing. Stroudsburg, PA, USA: Association for Computational Linguistics; 2004. Using higher-level linguistic knowledge for speech recognition error correction in a spoken Q/A dialogURL:http://www.aclweb.org/anthology/W04-3009. pdf [accessed 2015-03-28] [WebCite Cache ID 6XNPQAgm3].
44. Pucher M, Tu rk A, Ajmera J. Proceedings of the 3rd Congress of the Alps Adria Acoustics Association. Graz, Austria: Alps Adria Acoustics Association; 2007. Phonetic distance measures for speech recognition vocabulary and grammar optimizationURL:http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.396.4647&rep=rep1&type=pdf[WebCite Cache ID 6XTJp0TPl].
45. Philips L. Computer Language. 1990. Hanging on the metaphoneURL:ftp://ftp.math.utah.edu/pub/tex/bib/toc/complang. html#7(12):December:1990 [accessed 2015-04-13] [WebCite Cache ID 6Xkxwa4PQ].
46. Philips L. C/C++ Users Journal. 2000. The double metaphone search algorithmURL:http://dl.acm.org/citation.cfm?id=349132 [accessed 2015-04-01] [WebCite Cache ID 6XTK6JzYc].
47. Class metaphone. URL:http://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/language/Metaphone.html [accessed 2015-02-04] [WebCite Cache ID 6W6Kt5T8G].
48. SourceForge. SimMetrics. URL:http://sourceforge.net/projects/simmetrics/[accessed 2015-02-04] [WebCite Cache ID 6W6KwJ8XY].
49. Lafferty JD, McCallum A, Pereira FCN. Proceedings of the 18th International Conference on Machine Learning, ICML 2001; Williamstown, MA, USA. Burlington, MA, USA: Morgan Kaufmann; 2001. Conditional random fields: Probabilistic models for segmenting and labelling sequence dataURL:http://repository.upenn.edu/cgi/viewcontent. cgi?article=1162&context=cis-papers[WebCite Cache ID 6XNOQY9i4].
50. CRF++, Yet another CRF Toolkit. URL:http://taku910.github.io/crfpp/[accessed 2015-04-01] [WebCite Cache ID 6XTKSIu3X].
51. Manning CD, Surdeanu M, Bauer J, Finkel J, Bethard SJ, McClosky D. Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations; Baltimore, MA, USA. Stroudsburg, PA, USA: Association for Computational Linguistics; 2014 Jun. The Stanford CoreNLP natural language processing toolkitURL:http://nlp.stanford.edu/pubs/StanfordCoreNlp2014.pdf [accessed 2015-03-27] [WebCite Cache ID 6XNOAYHPB].
52. MetaMap-A tool for recognizing UMLS concepts in text. URL:http://metamap.nlm.nih.gov/[accessed 2015-02-04] [WebCite Cache ID 6W6MDPZVh].
53. Ontoserver. URL:http://ontoserver.csiro.au:8080/[accessed 2015-03-25] [WebCite Cache ID 6XIIHDUMk].
54. Suominen H, Pyysalo S, Hiissa M. Performance evaluation measures for text mining. In: Song M, Wu YFB, editors. Handbook of Research on TextWeb Mining Technologies. Hershey, PA, USA: IGI Global; 2008:724-747.
55. Cardie C, Daelemans W, Nédellec C, Tjong Kim San E, editors. Introduction to the CoNLL-2000 shared task: Chunking. In: Proceedings of CoNLL-2000LLL-2000, Lisbon, Portugal. Stroudsburg, PA, USA: Association for Computational Linguistics; 2000:127-132.
56. The NICTA synthetic nursing handover dataset. Open NICTA: Datasets for download. URL:http://www.opennicta.com/datasets [accessed 2015-01-23] [WebCite Cache ID 6Vmoi8MNy].
57. Attribution-noncommercial-noderivatives 4.0 international (CC BY-NC-ND 4.0). URL:http://creativecommons.org/licenses/by-nc-nd/4.0/[accessed 2015-01-23] [WebCite Cache ID 6VmpgnGnB].
58. Attribution 4.0 international (CC BY 4.0). URL:http://creativecommons.org/licenses/by/4.0/[accessed 2015-01-23] [WebCite Cache ID 6VmpZB1a0].
59. CLEFeHealth 2015: Lab overview. 2015. URL:https://sites.google.com/site/clefehealth2015/[accessed 2015-01-23] [WebCite Cache ID 6Vmpr4eWy].
60. Johnson M, Sanchez P, Suominen H, Basilakis J, Dawson L, Kelly B, et al. Comparing nursing handover and documentation: Forming one set of patient information. Int Nurs Rev 2014 Mar;61(1):73-81. [doi: 10.1111/inr.12072] [Medline: 24308444].
61. Dawson L, Johnson M, Suominen H, Basilakis J, Sanchez P, Estival D, et al. A usability framework for speech recognition technologies in clinical handover: A pre-implementation study. J Med Syst 2014 Jun;38(6):56. [doi: 10.1007/s10916-014-0056-7] [Medline: 24827759].
62. Johnson M, Lapkin S, Long V, Sanchez P, Suominen H, Basilakis J, et al. A systematic review of speech recognition technology in health care. BMC Med Inform Decis Mak 2014;14:94 [FREE Full text] [doi: 10.1186/1472-6947-14-94] [Medline: 25351845].
63. Suominen H, Ferraro G. In: Karimi S, Verspoor K, editors. Proceedings of the Australasian Language Technology Association Workshop 2013 (ALTA 2013); Brisbane, QLD, Australia. Stroudsburg, PA, USA: Association for Computational Linguistics; 2013 Dec. Noise in speech-to-text voice: Analysis of errors and feasibility of phonetic similarity for their correctionURL:http://aclweb.org/anthology/U/U13/U13-1006.pdf [accessed 2015-03-28] [WebCite Cache ID 6XNNIzEvh].
64. NICTA speech to clinical text demonstration. URL:http://nicta-stct.s3-website-ap-southeast-2.amazonaws.com/[accessed 2015-01-23] [WebCite Cache ID 6VmuZsah1].
65. Spärck JK, Galliers JR. Evaluating natural language processing systems: An analysis and review. In: Lecture Notes in Computer Science 1083. New York, NY, USA: Springer-Verlag; 1996.
66. Hirschman L, Thompson HS. Overview of evaluation in speech and natural language processing. In: Cole R, editor. Survey of the State of the Art in Human Language Technology. New York, NY, USA: Cambridge University Press; 1997:409-414.
67. Zick RG, Olsen J. Voice recognition software versus a traditional transcription service for physician charting in the ED. Am J Emerg Med 2001 Jul;19(4):295-298. [doi: 10.1053/ajem.2001.24487] [Medline: 11447517].
68. Callaway EC, Sweet CF, Siegel E, Reiser JM, Beall DP. Speech recognition interface to a hospital information system using a self-designed visual basic program: Initial experience. J Digit Imaging 2002 Mar;15(1):43-53 [FREE Full text] [doi: 10.1007/s10278-001-0007-y] [Medline: 12134214].
69. Langer SG. Impact of speech recognition on radiologist productivity. J Digit Imaging 2002 Dec;15(4):203-209 [FREE Full text] [doi: 10.1007/s10278-002-0014-7] [Medline: 12415463].
70. Kauppinen T, Koivikko MP, Ahovuo J. Improvement of report workflow and productivity using speech recognition-a follow-up study. J Digit Imaging 2008 Dec;21(4):378-382 [FREE Full text] [doi: 10.1007/s10278-008-9121-4] [Medline: 18437491].
71. Singh M, Pal TR. Voice recognition technology implementation in surgical pathology: Advantages and limitations. Arch Pathol Lab Med 2011 Nov;135(11):1476-1481. [doi: 10.5858/arpa.2010-0714-OA] [Medline: 22032576].
72. Devine EG, Gaehde SA, Curtis AC. Comparative evaluation of three continuous speech recognition software packages in the generation of medical reports. J Am Med Inform Assoc 2000;7(5):462-468 [FREE Full text] [Medline: 10984465].
73. Al-Aynati MM, Chorneyko KA. Comparison of voice-automated transcription and human transcription in generating pathology reports. Arch Pathol Lab Med 2003 Jun;127(6):721-725. [doi: 10.1043/1543-2165(2003)127<721:COVTAH>2.0.CO;2] [Medline: 12741898].
74. Pothier D, Monteiro P, Mooktiar M, Shaw A. Pilot study to show the loss of important data in nursing handover. Br J Nurs 2005;14(20):1090-1093. [doi: 10.12968/bjon.2005.14.20.20053] [Medline: 16301940].
75. Matic J, Davidson PM, Salamonson Y. Review: Bringing patient safety to the forefront through structured computerisation during clinical handover. J Clin Nurs 2011 Jan;20(1-2):184-189. [doi: 10.1111/j.1365-2702.2010.03242.x] [Medline: 20815861].