Scalable and accurate deep learning with electronic health records

npj Digital Medicine

Volumn 1, Issue 1, 2018, Pages

  Rajkomar, Alvin ^a,b   Oren, Eyal ^a   Chen, Kai ^a   Dai, Andrew M ^a   Hajaj, Nissan ^a   Hardt, Michaela ^a   Liu, Peter J ^a   Liu, Xiaobing ^a   Marcus, Jake ^a   Sun, Mimi ^a   Sundberg, Patrik ^a   Yee, Hector ^a   Zhang, Kun ^a   Zhang, Yi ^a   Flores, Gerardo ^a   Duggan, Gavin E ^a   Irvine, Jamie ^a   Le, Quoc ^a   Litsch, Kurt ^a   Mossin, Alexander ^a   Tansuwan, Justin ^a   Wang, De ^a   Wexler, James ^a   Wilson, Jimbo ^a   Ludwig, Dana ^b   Volchenboum, Samuel L ^c   Chou, Katherine ^a   Pearson, Michael ^a   Madabushi, Srinivasan ^a   Shah, Nigam H ^d   Butte, Atul J ^b   Howell, Michael D ^a   Cui, Claire ^a   Corrado, Greg S ^a   Dean, Jeffrey ^a   more..

^a GOOGLE INC   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c UNIVERSITY OF CHICAGO MEDICAL CENTER   (United States)

^d STANFORD UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DEEP LEARNING; DIAGNOSIS; DIGITAL STORAGE; E-LEARNING; EHEALTH; HOSPITALS; RECORDS MANAGEMENT;

ELECTRONIC HEALTH; HEALTH RECORDS; HEALTHCARE INTEROPERABILITY; HEALTHCARE QUALITY; LABOR INTENSIVE PROCESS; PERSONALIZED MEDICINES; PREDICTIVE MODELS; PREDICTOR VARIABLES; RECEIVER OPERATOR CURVES; STATISTIC MODELING;

FORECASTING;

ADULT; ARTICLE; BODY WEIGHT; CONTROLLED STUDY; DIAGNOSIS; ELECTRONIC HEALTH RECORD; FEMALE; HOSPITAL MORTALITY; HOSPITAL READMISSION; HUMAN; LEARNING; LENGTH OF STAY; MAJOR CLINICAL STUDY; MALE; MULTICENTER STUDY; PREDICTION; UNIVERSITY HOSPITAL;

EID: 85127431078     PISSN: None     EISSN: 23986352     Source Type: Journal    
DOI: 10.1038/s41746-018-0029-1     Document Type: Article

References (89)

1. The Digital Universe: Driving Data Growth in Healthcare. Available at: https://www.emc.com/analyst-report/digital-universe-healthcare-vertical-report-ar.pdf (Accessed 23 Feb 2017)
2. Parikh, R. B., Schwartz, J. S. & Navathe, A. S. Beyond genes and molecules - a precision delivery initiative for precision medicine. N. Engl. J. Med. 376, 1609–1612 (2017)
3. Parikh, R. B., Kakad, M. & Bates, D. W. Integrating predictive analytics into high-value care: the dawn of precision delivery. JAMA 315, 651–652 (2016)
4. Bates, D. W., Saria, S., Ohno-Machado, L., Shah, A. & Escobar, G. Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Aff. 33, 1123–1131 (2014)
5. Krumholz, H. M. Big data and new knowledge in medicine: the thinking, training, and tools needed for a learning health system. Health Aff. 33, 1163–1170 (2014)
6. Jameson, J. L. & Longo, D. L. Precision medicine--personalized, problematic, and promising. N. Engl. J. Med. 372, 2229–2234 (2015)
7. Goldstein, B. A., Navar, A. M., Pencina, M. J. & Ioannidis, J. P. A. Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review. J. Am. Med. Inform. Assoc. 24, 198–208 (2017)
8. Press, G. Cleaning big data: most time-consuming, least enjoyable data science task, survey says. Forbes (2016). Available at: https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/ (Accessed 22 Oct 2017)
9. Lohr, S. For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights. (NY Times, 2014)
10. Drew, B. J. et al. Insights into the problem of alarm fatigue with physiologic monitor devices: a comprehensive observational study of consecutive intensive care unit patients. PLoS ONE 9, e110274 (2014)
11. Chopra, V. & McMahon, L. F. Jr. Redesigning hospital alarms for patient safety: alarmed and potentially dangerous. JAMA 311, 1199–1200 (2014)
12. Kaukonen, K.-M., Bailey, M., Pilcher, D., Cooper, D. J. & Bellomo, R. Systemic inflammatory response syndrome criteria in defining severe sepsis. N. Engl. J. Med. 372, 1629–1638 (2015)
13. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015)
14. Frome, A. et al. DeViSE: a deep visual-semantic embedding model. In Advances in Neural Information Processing Systems 26 (eds Burges, C. J. C., Bottou, L., Welling, M., Ghahramani, Z. & Weinberger, K. Q.), pp 2121–2129 (Curran Associates, Inc. Red Hook, NY, 2013)
15. Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316, 2402–2410 (2016)
16. Wu, Y. et al. Google’s neural machine translation system: bridging the gap between human and machine translation. arXi v [cs.CL] (2016)
17. Dai, A. M. & Le, Q. V. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems 28 (eds Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M. & Garnett, R.), pp 3079–3087 (Curran Associates, Inc. Red Hook, NY, 2015)
18. Bengio, Y., Courville, A. & Vincent, P. Representation learning: a review and new perspectives. IEEE. Trans. Pattern Anal. Mach. Intell. 35, 1798–1828 (2013)
19. Weed, L. L. Medical records that guide and teach. N. Engl. J. Med. 278, 652–657 (1968). concl
20. Adler-Milstein, J. et al. Electronic health record adoption In US hospitals: progress continues, but challenges persist. Health Aff. 34, 2174–2180 (2015)
21. Mandell, L. A. et al. Infectious Diseases Society of America/American Thoracic Society consensus guidelines on the management of community-acquired pneumonia in adults. Clin. Infect. Dis. 44, S27–S72 (2007). Suppl 2
22. Lim, W. S., Smith, D. L., Wise, M. P. & Welham, S. A. British Thoracic Society community acquired pneumonia guideline and the NICE pneumonia guideline: how they fit together. BMJ Open Respir. Res. 2, e000091 (2015)
23. Churpek, M. M. et al. Multicenter comparison of machine learning methods and conventional regression for predicting clinical deterioration on the wards. Crit. Care. Med. 44, 368–374 (2016)
24. Howell, M. D. et al. Sustained effectiveness of a primary-team-based rapid response system. Crit. Care. Med. 40, 2562–2568 (2012)
25. Sun, H. et al. Semantic processing of EHR data for clinical research. J. Biomed. Inform. 58, 247–259 (2015)
26. Newton, K. M. et al. Validation of electronic medical record-based phenotyping algorithms: results and lessons learned from the eMERGE network. J. Am. Med. Inform. Assoc. 20, e147–e154 (2013)
27. OHDSI. OMOP common data model. Observational health data sciences and informatics. Available at: https://www.ohdsi.org/data-standardization/the-common-data-model/ (Accessed 23 Jan 2018)
28. Mandel, J. C., Kreda, D. A., Mandl, K. D., Kohane, I. S. & Ramoni, R. B. SMART on FHIR: a standards-based, interoperable apps platform for electronic health records. J. Am. Med. Inform. Assoc. 23, 899–908 (2016)
29. Miotto, R., Li, L., Kidd, B. A. & Dudley, J. T. Deep patient: an unsupervised representation to predict the future of patients from the electronic health records. Sci. Rep. 6, 26094 (2016)
30. Lipton, Z. C., Kale, D. C., Elkan, C. & Wetzel, R. Learning to diagnose with LSTM recurrent neural networks. arXiv [cs.LG] (2015)
31. Aczon, M. et al. Dynamic mortality risk predictions in pediatric critical care using recurrent neural networks. arXiv [stat.ML] (2017)
32. Choi, E., Bahadori, M. T., Schuetz, A., Stewart, W. F. & Sun, J. Doctor AI: predicting clinical events via recurrent neural networks. In Proceedings of the 1st Machine Learning for Healthcare Conference, vol 56 (eds F. Doshi-Velez, J. Fackler, D. Kale and B. Wallace, J. Wiens) 301–318 (PMLR, Los Angeles, CA, 2016)
33. Suresh, H. et al. Clinical intervention prediction and understanding using deep networks. arXiv [cs.LG] (PMLR, Los Angeles, CA, USA, 2017)
34. Razavian, N., Marcus, J. & Sontag, D. Multi-task prediction of disease onsets from longitudinal laboratory tests. In Proceedings of the 1st Machine Learning for Healthcare Conference, (eds F. Doshi-Velez, J. Fackler, D. Kale and B. Wallace, J. Wiens) Vol. 56, pp 73–100 (PMLR, Los Angeles, CA, 2016)
35. Che, Z., Purushotham, S., Cho, K., Sontag, D. & Liu, Y. Recurrent neural networks for multivariate time series with missing values. arXiv [cs.LG] (2016)
36. Johnson, A. E. W. et al. MIMIC-III, a freely accessible critical care database. Sci. Data 3, 160035 (2016)
37. Harutyunyan, H., Khachatrian, H., Kale, D. C. & Galstyan, A. Multitask learning and benchmarking with clinical time series data. arXiv [stat.ML] (2017)
38. Society of Critical Care Medicine. Critical care statistics. Available at: http://www.sccm.org/Communications/Pages/CriticalCareStats.aspx (Accessed 25 Jan 2018)
39. American Hospital Association. Fast facts on U.S. Hospitals, 2018. Available at: https://www.aha.org/statistics/fast-facts-us-hospitals (Accessed 25 Jan 2018)
40. Shickel, B., Tighe, P., Bihorac, A. & Rashidi, P. Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. arXiv [cs.LG] (2017)
41. Bergstrom, N., Braden, B. J., Laguzza, A. & Holman, V. The braden scale for predicting pressure sore risk. Nurs. Res. 36, 205–210 (1987)
42. Tabak, Y. P., Sun, X., Nunez, C. M. & Johannes, R. S. Using electronic health record data to develop inpatient mortality predictive model: Acute Laboratory Risk of Mortality Score (ALaRMS). J. Am. Med. Inform. Assoc. 21, 455–463 (2014)
43. Nguyen, O. K. et al. Predicting all-cause readmissions using electronic health record data from the entire hospitalization: Model development and comparison. J. Hosp. Med. 11, 473–480 (2016)
44. Liu, V., Kipnis, P., Gould, M. K. & Escobar, G. J. Length of stay predictions: improvements through the use of automated laboratory and comorbidity variables. Med. Care 48, 739–744 (2010)
45. Walsh, C. & Hripcsak, G. The effects of data sources, cohort selection, and outcome definition on a predictive model of risk of thirty-day hospital readmissions. J. Biomed. Inform. 52, 418–426 (2014)
46. TM Early Warning Score (ViEWS) in 75,419 consecutive admissions to a Canadian regional hospital. Resuscitation 83, 297–302 (2012)
47. Escobar, G. J. et al. Risk-adjusting hospital inpatient mortality using automated inpatient, outpatient, and laboratory databases. Med. Care. 46, 232–239 (2008)
48. van Walraven, C. et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ 182, 551–557 (2010)
49. Yamana, H., Matsui, H., Fushimi, K. & Yasunaga, H. Procedure-based severity index for inpatients: development and validation using administrative database. BMC Health Serv. Res. 15, 261 (2015)
50. Pine, M. et al. Modifying ICD-9-CM coding of secondary diagnoses to improve risk-adjustment of inpatient mortality rates. Med. Decis. Making 29, 69–81 (2009)
51. Smith, G. B., Prytherch, D. R., Meredith, P., Schmidt, P. E. & Featherstone, P. I. The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation 84, 465–470 (2013)
52. Khurana, H. S. et al. Real-time automated sampling of electronic medical records predicts hospital mortality. Am. J. Med. 129, 688–698.e2 (2016)
53. Rothman, M. J., Rothman, S. I. & Beals, J. 4th Development and validation of a continuous measure of patient condition using the electronic medical record. J. Biomed. Inform. 46, 837–848 (2013)
54. Finlay, G. D., Rothman, M. J. & Smith, R. A. Measuring the modified early warning score and the Rothman index: advantages of utilizing the electronic medical record in an early warning system. J. Hosp. Med. 9, 116–119 (2014)
55. Zapatero, A. et al. Predictive model of readmission to internal medicine wards. Eur. J. Intern. Med. 23, 451–456 (2012)
56. Shams, I., Ajorlou, S. & Yang, K. A predictive analytics approach to reducing 30-day avoidable readmissions among patients with heart failure, acute myocardial infarction, pneumonia, or COPD. Health Care. Manag. Sci. 18, 19–34 (2015)
57. Tsui, E., Au, S. Y., Wong, C. P., Cheung, A. & Lam, P. Development of an automated model to predict the risk of elderly emergency medical admissions within a month following an index hospital visit: A Hong Kong experience. Health Inform. J. 21, 46–56 (2013)
58. Choudhry, S. A. et al. A public-private partnership develops and externally validates a 30-day hospital readmission risk prediction model. Online J. Public Health Inform. 5, 219 (2013)
59. Caruana, R. et al. Intelligible models for healthcare: predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 1721–1730. http://doi.acm.org/10.1145/2783258.2788613 (ACM, Sydney, NSW, Australia, 2015)
60. Tonkikh, O. et al. Functional status before and during acute hospitalization and readmission risk identification. J. Hosp. Med. 11, 636–641 (2016)
61. Betihavas, V. et al. An absolute risk prediction model to determine unplanned cardiovascular readmissions for adults with chronic heart failure. Heart Lung Circ. 24, 1068–1073 (2015)
62. Whitlock, T. L. et al. A scoring system to predict readmission of patients with acute pancreatitis to the hospital within thirty days of discharge. Clin. Gastroenterol. Hepatol. 9, 175–180 (2011). quiz e18
63. Coleman, E. A., Min, S.-J., Chomiak, A. & Kramer, A. M. Posthospital care transitions: patterns, complications, and risk identification. Health Serv. Res. 39, 1449–1465 (2004)
64. Graboyes, E. M., Liou, T.-N., Kallogjeri, D., Nussenbaum, B. & Diaz, J. A. Risk factors for unplanned hospital readmission in otolaryngology patients. Otolaryngol. Head Neck Surg. 149, 562–571 (2013)
65. He, D., Mathews, S. C., Kalloo, A. N. & Hutfless, S. Mining high-dimensional administrative claims data to predict early hospital readmissions. J. Am. Med. Inform. Assoc. 21, 272–279 (2014)
66. Futoma, J., Morris, J. & Lucas, J. A comparison of models for predicting early hospital readmissions. J. Biomed. Inform. 56, 229–238 (2015)
67. Donzé, J., Aujesky, D., Williams, D. & Schnipper, J. L. Potentially avoidable 30-day hospital readmissions in medical patients: derivation and validation of a prediction model. JAMA Intern. Med. 173, 632–638 (2013)
68. Perotte, A. et al. Diagnosis code assignment: models and evaluation metrics. J. Am. Med. Inform. Assoc. 21, 231–237 (2014)
69. Krumholz, H. M., Terry, S. F. & Waldstreicher, J. Data acquisition, curation, and use for a continuously learning health system. JAMA 316, 1669–1670 (2016)
70. Grumbach, K., Lucey, C. R. & Claiborne Johnston, S. Transforming from centers of learning to learning health systems: the challenge for academic health centers. JAMA 311, 1109–1110 (2014)
71. Halamka, J. D. & Tripathi, M. The HITECH era in retrospect. N. Engl. J. Med. 377, 907–909 (2017)
72. Bates, D. W. et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J. Am. Med. Inform. Assoc. 10, 523–530 (2003)
73. Obermeyer, Z. & Emanuel, E. J. Predicting the future --- big data, machine learning, and clinical medicine. N. Engl. J. Med. 375, 1216–1219 (2016)
74. Avati, A. et al. Improving palliative care with deep learning. arXiv [cs.CY] (2017)
75. Health Level 7. FHIR Specification Home Page (2017). Available at: http://hl7.org/fhir/ (Accessed 3 Aug 2017)
76. Escobar, G. J. et al. Nonelective rehospitalizations and postdischarge mortality: predictive models suitable for use in real time. Med. Care. 53, 916–923 (2015)
77. 2016 Measure updates and specifications report: hospital-wide all-cause unplanned readmission --- version 5.0. Yale--New Haven Health Services Corporation/Center for Outcomes Research & Evaluation (New Haven, CT, 2016)
78. Knaus, W. A., Draper, E. A., Wagner, D. P. & Zimmerman, J. E. APACHE II: a severity of disease classification system. Crit. Care Med. 13, 818–829 (1985)
79. Kansagara, D. et al. Risk prediction models for hospital readmission: a systematic review. JAMA 306, 1688–1698 (2011)
80. Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997)
81. Rokach, L. Ensemble-based Classifiers. Artif. Intell. Rev. 33, 1–39 (2010)
82. Cabitza, F., Rasoini, R. & Gensini, G. F. Unintended consequences of machine learning in medicine. JAMA 18, 517–518 (2017)
83. Bahdanau, D., Cho, K. & Bengio, Y. Neural machine translation by jointly learning to align and translate. arXiv [cs.CL] (2014)
84. Pedregosa, F. et al. Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
85. Pencina, M. J. & D’Agostino, R. B. Sr. Evaluating discrimination of risk prediction models: the C statistic. JAMA 314, 1063–1064 (2015)
86. Kramer, A. A. & Zimmerman, J. E. Assessing the calibration of mortality benchmarks in critical care: the Hosmer-Lemeshow test revisited. Crit. Care Med. 35, 2052–2056 (2007)
87. Romero-Brufau, S., Huddleston, J. M., Escobar, G. J. & Liebow, M. Why the C-statistic is not informative to evaluate early warning scores and what metrics to use. Crit. Care 19, 285 (2015)
88. SciKit Learn. SciKit learn documentation on area under the curve scores. Available at: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html (Accessed 3 Aug 2017)
89. SciKit Learn. SciKit learn documentation on F1 score. Available at: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html (Accessed 3 Aug 2017)