Benchmarking deep learning models on large healthcare datasets

Journal of Biomedical Informatics

Volumn 83, Issue , 2018, Pages 112-134

  Purushotham, Sanjay ^a   Meng, Chuizheng ^b   Che, Zhengping ^a   Liu, Yan ^a  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^b TSINGHUA UNIVERSITY   (China)

Author keywords

Deep learning models; ICD 9 code group prediction; Length of stay; Mortality prediction; Super learner algorithm

Indexed keywords

ARTIFICIAL INTELLIGENCE; BENCHMARKING; DEEP NEURAL NETWORKS; FORECASTING; HEALTH CARE; INTENSIVE CARE UNITS; MEDICAL COMPUTING; NATURAL LANGUAGE PROCESSING SYSTEMS; SPEECH RECOGNITION;

CLINICAL TIME SERIES; HEALTH CARE APPLICATION; LEARNING MODELS; LENGTH OF STAY; MACHINE LEARNING MODELS; MEDICAL INFORMATION; PREDICTION TASKS; STATE OF THE ART;

LEARNING ALGORITHMS;

ADULT; AGED; ARTICLE; BENCHMARKING; CODING ALGORITHM; FEMALE; HEALTH CARE; HOSPITAL MORTALITY; HUMAN; ICD-9; INFORMATION PROCESSING; INTENSIVE CARE UNIT; LENGTH OF STAY; MACHINE LEARNING; MALE; SCORING SYSTEM; SEQUENTIAL ORGAN FAILURE ASSESSMENT SCORE; SIMPLIFIED ACUTE PHYSIOLOGY SCORE; STATISTICAL MODEL; ALGORITHM; ARTIFICIAL NEURAL NETWORK; FACTUAL DATABASE; FORECASTING; INTERNATIONAL CLASSIFICATION OF DISEASES; MIDDLE AGED; VERY ELDERLY;

AGED; AGED, 80 AND OVER; ALGORITHMS; BENCHMARKING; DATABASES, FACTUAL; DEEP LEARNING; FEMALE; FORECASTING; HOSPITAL MORTALITY; HUMANS; INTENSIVE CARE UNITS; INTERNATIONAL CLASSIFICATION OF DISEASES; LENGTH OF STAY; MALE; MIDDLE AGED; NEURAL NETWORKS (COMPUTER);

EID: 85048265375     PISSN: 15320464     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jbi.2018.04.007     Document Type: Article

References (68)

1. Z. Che, S. Purushotham, K. Cho, D. Sontag, Y. Liu, Recurrent Neural Networks for Multivariate Time Series with Missing Values, 2016, arXiv preprint arXiv:1606.01865.
2. H. Harutyunyan, H. Khachatrian, D.C. Kale, A. Galstyan, Multitask Learning and Benchmarking with Clinical Time Series Data, 2017, arXiv preprint arXiv:1703.07771.
3. Le Gall, J.-R., Lemeshow, S., Saulnier, F., A new simplified acute physiology score (saps ii) based on a european/north american multicenter study. Jama 270 (1993), 2957–2963.
4. Lee, J., Scott, D.J., Villarroel, M., Clifford, G.D., Saeed, M., Mark, R.G., Open-access mimic-ii database for intensive care research. 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBC, 2011, IEEE, 8315–8318.
5. Johnson, A.E., Pollard, T.J., Shen, L., Lehman, L.-W.H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Celi, L.A., Mark, R.G., Mimic-iii, a freely accessible critical care database. Scient. Data, 3, 2016.
6. A.E. Johnson, T.J. Pollard, R.G. Mark, Reproducibility in critical care: a mortality prediction case study, in: Machine Learning for Healthcare Conference, 2017, 361–376.
7. Che, Z., Kale, D., Li, W., Bahadori, M.T., Liu, Y., Deep computational phenotyping. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015, ACM, 507–516.
8. Z. Che, S. Purushotham, R. Khemani, Y. Liu, Interpretable deep models for icu outcome prediction, in: AMIA Annual Symposium Proceedings, vol. 2016, American Medical Informatics Association, 2016, p. 371.
9. Caballero Barajas, K.L., Akella, R., Dynamically modeling patient's health state from electronic medical records: a time series approach. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015, ACM, 69–78.
10. Calvert, J., Mao, Q., Rogers, A.J., Barton, C., Jay, M., Desautels, T., Mohamadlou, H., Jan, J., Das, R., A computational approach to mortality prediction of alcohol use disorder inpatients. Comp Biol Med 75 (2016), 74–79.
11. Celi, L.A., Galvin, S., Davidzon, G., Lee, J., Scott, D., Mark, R., A database-driven decision support system: customized mortality prediction. J Personal Med 2 (2012), 138–148.
12. M. Ghassemi, M.A. Pimentel, T. Naumann, T. Brennan, D.A. Clifton, P. Szolovits, M. Feng, A multivariate timeseries modeling approach to severity of illness assessment and forecasting in ICU with sparse, heterogeneous clinical data, in: AAAI, 2015, pp. 446–453.
13. Ghassemi, M., Naumann, T., Doshi-Velez, F., Brimmer, N., Joshi, R., Rumshisky, A., Szolovits, P., Unfolding physiological state: Mortality modelling in intensive care units. Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2014, ACM, 75–84.
14. Hoogendoorn, M., el Hassouni, A., Mok, K., Ghassemi, M., Szolovits, P., Prediction using patient comparison vs. modeling: a case study for mortality prediction. 2016 IEEE 38th Annual International Conference of the Engineering in Medicine and Biology Society (EMBC), 2016, IEEE, 2464–2467.
15. R. Joshi, P. Szolovits, Prognostic physiology: modeling patient severity in intensive care units using radial domain folding, in: AMIA Annual Symposium Proceedings, vol. 2012, American Medical Informatics Association, 2012, p. 1276.
16. Lee, J., Maslove, D.M., Customization of a severity of illness score using local electronic medical record data. J. Intens. Care Med. 32 (2017), 38–47.
17. L.-W. Lehman, M. Saeed, W. Long, J. Lee, R. Mark, Risk stratification of icu patients using topic models inferred from unstructured progress notes, in: AMIA Annual Symposium Proceedings, vol. 2012, American Medical Informatics Association, 2012, p. 505.
18. Y. Luo, Y. Xin, R. Joshi, L.A. Celi, P. Szolovits, Predicting icu mortality risk by grouping temporal trends from a multivariate panel of physiologic measurements, in: AAAI, 2016, pp. 42–50.
19. S. Joshi, S. Gunasekar, D. Sontag, G. Joydeep, Identifiable phenotyping using constrained non-negative matrix factorization, in: Machine Learning for Healthcare Conference, 2016, pp. 17–41.
20. Lee, J., Maslove, D.M., Dubin, J.A., Personalized mortality prediction driven by electronic medical data and a patient similarity metric. PloS One, 10, 2015, e0127428.
21. Lee, J., Patient-specific predictive modeling using random forests: an observational study for the critically ill. JMIR Med. Inf., 5, 2017.
22. Y.-F. Luo, A. Rumshisky, Interpretable topic features for post-ICU mortality prediction, in: AMIA Annual Symposium Proceedings, vol. 2016, American Medical Informatics Association, 2016, p. 827.
23. Vincent, J.-L., Moreno, R., Takala, J., Willatts, S., De Mendonça, A., Bruining, H., Reinhart, C., Suter, P., Thijs, L., The sofa (sepsis-related organ failure assessment) score to describe organ dysfunction/failure. Intens. Care Med. 22 (1996), 707–710.
24. Knaus, W.A., Zimmerman, J.E., Wagner, D.P., Draper, E.A., Lawrence, D.E., Apache-acute physiology and chronic health evaluation: a physiologically based classification system. Crit. Care Med. 9 (1981), 591–597.
25. Dybowski, R., Gant, V., Weller, P., Chang, R., Prediction of outcome in critically ill patients using artificial neural network synthesised by genetic algorithm. The Lancet 347 (1996), 1146–1150.
26. Kim, S., Kim, W., Park, R.W., A comparison of intensive care unit mortality prediction models through the use of data mining techniques. Health. Inf. Res. 17 (2011), 232–243.
27. Tu, J.V., Guerriere, M.R., Use of a neural network as a predictive instrument for length of stay in the intensive care unit following cardiac surgery. Comp. Biomed. Res. 26 (1993), 220–229.
28. G. Doig, K. Inman, W. Sibbald, C. Martin, J. Robertson, Modeling mortality in the intensive care unit: comparing the performance of a back-propagation, associative-learning neural network with multivariate logistic regression, in: Proceedings of the Annual Symposium on Computer Application in Medical Care, American Medical Informatics Association, 1993, p. 361.
29. Hanson, C.W. III, Marshall, B.E., Artificial intelligence applications in the intensive care unit. Crit. Care Med. 29 (2001), 427–435.
30. Pirracchio, R., Mortality prediction in the ICU based on mimic-ii results from the super ICU learner algorithm (SICULA) project. Secondary Analysis of Electronic Health Records, 2016, Springer, 295–313.
31. E.C. Polley, M.J. Van der Laan, Super Learner in Prediction, U.C. Berkeley Division of Biostatistics Working Paper Series, 2010.
32. R. Caruana, S. Baluja, T. Mitchell, Using the future to sort out the present: Rankprop and multitask learning for medical risk evaluation, in: Advances in Neural Information Processing Systems, 1996, pp. 959–965.
33. Cooper, G.F., Aliferis, C.F., Ambrosino, R., Aronis, J., Buchanan, B.G., Caruana, R., Fine, M.J., Glymour, C., Gordon, G., Hanusa, B.H., et al. An evaluation of machine-learning methods for predicting pneumonia mortality. Artif. Intell. Med. 9 (1997), 107–138.
34. Lasko, T.A., Denny, J.C., Levy, M.A., Computational phenotype discovery using unsupervised feature learning over noisy, sparse, and irregular clinical data. PloS One, 8, 2013, e66341.
35. Oellrich, A., Collier, N., Groza, T., Rebholz-Schuhmann, D., Shah, N., Bodenreider, O., Boland, M.R., Georgiev, I., Liu, H., Livingston, K., et al. The digital revolution in phenotyping. Brief. Bioinf., 2015, bbv083.
36. Z. Che, S. Purushotham, R. Khemani, Y. Liu, Distilling Knowledge From Deep Networks with Applications to Healthcare Domain, 2015 arXiv preprint arXiv:1512.03542.
37. Dabek, F., Caban, J.J., A neural network based model for predicting psychological conditions. Brain Informatics and Health, 2015, Springer, 252–261.
38. N.Y. Hammerla, J.M. Fisher, P. Andras, L. Rochester, R. Walker, T. Plötz, Pd disease state assessment in naturalistic environments using deep learning, in: Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015, pp. 1742–1748.
39. Z.C. Lipton, D.C. Kale, C. Elkan, R. Wetzell, Learning to Diagnose with LSTM Recurrent Neural Networks, 2015 arXiv preprint arXiv:1511.03677.
40. S. Purushotham, W. Carvalho, T. Nilanon, Y. Liu, Variational recurrent adversarial deep domain adaptation, in: International Conference on Learning Representations (ICLR), 2017.
41. Li-wei, H.L., Adams, R.P., Mayaud, L., Moody, G.B., Malhotra, A., Mark, R.G., Nemati, S., A physiological time series dynamics-based approach to patient monitoring and outcome prediction. IEEE J. Biomed. Health Inf. 19 (2015), 1068–1076.
42. Lee, J., Patient-specific predictive modeling using random forests: an observational study for the critically ill. JMIR Med. Inf., 5, 2017.
43. Hochreiter, S., Schmidhuber, J., Long short-term memory. Neural Comput. 9 (1997), 1735–1780.
44. I. Silva, G. Moody, D.J. Scott, L.A. Celi, R.G. Mark, Predicting in-hospital mortality of ICU patients: the physionet/computing in cardiology challenge 2012, in: Computing in Cardiology (CinC), 2012, IEEE, 2012, pp. 245–248.
45. Van der Laan, M.J., Polley, E.C., Hubbard, A.E., Super learner. Statist. Appl. Genet. Molec. Biol., 6, 2007.
46. LeCun, Y., Bengio, Y., Hinton, G., Deep learning. Nature 521 (2015), 436–444.
47. Larochelle, H., Bengio, Y., Louradour, J., Lamblin, P., Exploring strategies for training deep neural networks. J. Mach. Learn. Res. 10 (2009), 1–40.
48. Bengio, Y., Courville, A., Vincent, P., Representation learning: a review and new perspectives. IEEE Trans. Patt. Anal. Mach. Intell. 35 (2013), 1798–1828.
49. G. Dahl, A.-r. Mohamed, G.E. Hinton, et al, Phone recognition with the mean-covariance restricted boltzmann machine, in: Advances in Neural Information Processing Systems, 2010, pp. 469–477.
50. Dahl, G.E., Yu, D., Deng, L., Acero, A., Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Trans. Audio, Speech, Lang. Process. 20 (2012), 30–42.
51. A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep convolutional neural networks, in: Advances in Neural Information Processing Systems, 2012, pp. 1097–1105.
52. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T., Caffe: convolutional architecture for fast feature embedding. Proceedings of the 22nd ACM International Conference on Multimedia, 2014, ACM, 675–678.
53. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1–9.
54. T. Mikolov, A. Deoras, S. Kombrink, L. Burget, J. Černockỳ Empirical evaluation and combination of advanced language modeling techniques, in: Twelfth Annual Conference of the International Speech Communication Association, 2011, pp. 605–608.
55. A. Bordes, X. Glorot, J. Weston, Y. Bengio, Joint learning of words and meaning representations for open-text semantic parsing, in: Artificial Intelligence and Statistics, 2012, pp. 127–135.
56. T. Mikolov, W.-T. Yih, G. Zweig, Linguistic regularities in continuous space word representations, in: hlt-Naacl, vol. 13, 2013, pp. 746–751.
57. T. Mikolov, I. Sutskever, K. Chen, G.S. Corrado, J. Dean, Distributed representations of words and phrases and their compositionality, in: Advances in Neural Information Processing Systems, 2013, pp. 3111–3119.
58. Hornik, K., Stinchcombe, M., White, H., Multilayer feedforward networks are universal approximators. Neural Netw. 2 (1989), 359–366.
59. V. Nair, G.E. Hinton, Rectified linear units improve restricted boltzmann machines, in: Proceedings of the 27th International Conference on Machine Learning (ICML), 2010, pp. 807–814.
60. Williams, R.J., Zipser, D., A learning algorithm for continually running fully recurrent neural networks. Neural Comput. 1 (1989), 270–280.
61. K. Cho, B. van Merriënboer, D. Bahdanau, Y. Bengio, On the Properties of Neural Machine Translation: Encoder-Decoder Approaches, 2014, arXiv preprint arXiv:1409.1259.
62. J. Chung, C. Gulcehre, K. Cho, Y. Bengio, Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling, 2014 arXiv preprint arXiv:1412.3555.
63. N. Srivastava, R.R. Salakhutdinov, Multimodal learning with deep boltzmann machines, in: Advances in Neural Information Processing Systems, 2012, pp. 2222–2230.
64. F. Bastien, P. Lamblin, R. Pascanu, J. Bergstra, I. Goodfellow, A. Bergeron, N. Bouchard, D. Warde-Farley, Y. Bengio, Theano: New Features and Speed Improvements, 2012, arXiv preprint arXiv:1211.5590.
65. F. Chollet, Keras: Theano-Based Deep Learning Library, 2015 Code: https://github.com/fchollet. Documentation: http://keras.io.
66. B. Xu, N. Wang, T. Chen, M. Li, Empirical Evaluation of Rectified Activations in Convolutional Network, 2015 arXiv preprint arXiv:1505.00853.
67. Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R., Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15 (2014), 1929–1958.
68. Luengo, J., García, S., Herrera, F., A study on the use of statistical tests for experimentation with neural networks: analysis of parametric test conditions and non-parametric tests. Expert Syst. Appl. 36 (2009), 7798–7808.