Feature Robustness in Non-stationary Health Records: Caveats to Deployable Model Performance in Common Clinical Machine Learning Tasks

Proceedings of Machine Learning Research

Volumn 106, Issue , 2019, Pages 381-405

  Nestor, Bret ^a   McDermott, Matthew B A ^b   Boag, Willie ^b   Berner, Gabriela ^c   Naumann, Tristan ^d   Hughes, Michael C ^e   Goldenberg, Anna ^f   Ghassemi, Marzyeh ^a  

^a UNIVERSITY OF TORONTO   (Canada)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^c HARVARD UNIVERSITY   (United States)

^d MICROSOFT RESEARCH   (United States)

^e TUFTS UNIVERSITY   (United States)

^f HOSPITAL FOR SICK CHILDREN   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

DROPS; MACHINE LEARNING; RECORDS MANAGEMENT; STATISTICAL TESTS;

'CURRENT; ELECTRONIC HEALTH; HEALTH RECORDS; LEARNING TASKS; LENGTH OF STAY; MACHINE-LEARNING; MODELING ABILITIES; MODELING PERFORMANCE; NONSTATIONARY; PREDICTION MODELLING;

FORECASTING;

EID: 85161283968     PISSN: None     EISSN: 26403498     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (47)

1. Hila Becker and Marta Arias. Real-time ranking with concept drift using expert advice. In KDD, volume 7, pages 86-94, 2007.
2. Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35 (8):1798-1828, 2013.
3. James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb):281-305, 2012.
4. Ayne A. Beyene, Tewelle Welemariam, Marie Persson, and Niklas Lavesson. Improved concept drift handling in surgery prediction and other applications. Knowledge and Information Systems, 44(1):177-196, July 2015. ISSN 0219-3116. doi: 10.1007/s10115-014-0756-9. URL https://doi.org/10.1007/s10115-014-0756-9.
5. Olivier Bodenreider. The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Res, 32(suppl 1):D267-D270, 2004.
6. Robert Challen, Joshua Denny, Martin Pitt, Luke Gompels, Tom Edwards, and Krasimira Tsaneva-Atanasova. Artificial intelligence, bias and clinical safety. BMJ Qual Saf, 28(3): 231-237, March 2019. ISSN 2044-5415, 2044-5423. doi: 10.1136/bmjqs-2018-008370. URL https://qualitysafety.bmj.com/content/28/3/231.
7. Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent Neural Networks for Multivariate Time Series with Missing Values. Scientific Reports, 8 (1), 2018. URL http://www.nature.com/articles/s41598-018-24271-9.
8. Edward Choi, Mohammad Taha Bahadori, Le Song, Walter F Stewart, and Jimeng Sun. Gram: graph-based attention model for healthcare representation learning. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 787-795. ACM, 2017.
9. Zhiyong Cui (UW). Gated Recurrent Unit with a Decay mechanism for Multivariate Time Series with Missing Values, March 2019. URL https://github.com/zhiyongc/GRU-D. original-date: 2018-05-13T00:07:42Z.
10. Sharon E Davis, Thomas A Lasko, Guanhua Chen, Edward D Siew, and Michael E Matheny. Calibration drift in regression and machine learning models for acute kidney injury. Journal of the American Medical Informatics Association: JAMIA, 24(6):1052-1061, November 2017. ISSN 1067-5027. doi: 10.1093/jamia/ocx030. URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6080675/.
11. Thomas G Dietterich. Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, 10(7):1895-1923, 1998.
12. Guy Divita, Qing T Zeng, Adi V Gundlapalli, Scott Duvall, Jonathan Nebeker, and Matthew H Samore. Sophia: a expedient UMLS concept extraction annotator. In Proc. of AMIA Annual Symposium, volume 2014, page 467. American Medical Informatics Association, 2014.
13. Marzyeh Ghassemi, Mike Wu, Michael Hughes, and Finale Doshi-Velez. Predicting intervention onset in the icu with switching state space models. In Proceedings of the AMIA Summit on Clinical Research Informatics (CRI), volume 2017. American Medical Informatics Association, 2017.
14. Marzyeh Ghassemi, Tristan Naumann, Peter Schulam, Andrew L Beam, and Rajesh Ranganath. Opportunities in machine learning for healthcare. arXiv preprint arXiv:1806.00388, 2018.
15. Jen J Gong, Tristan Naumann, Peter Szolovits, and John V Guttag. Predicting Clinical Outcomes Across Changing Electronic Health Record Systems. In the 23rd ACM SIGKDD International Conference, pages 1497-1505, New York, New York, USA, 2017. ACM Press.
16. Han-JD. inspired by 'Recurrent Neural Networks for Multivariate Time Series with Missing Values', March 2019. URL https://github.com/Han-JD/GRU-D. original-date: 2018-09-17T19:24:26Z.
17. Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. arXiv preprint arXiv:1703.07771, 2017.
18. Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Comput., 9 (8):1735-1780, November 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735. URL http://dx.doi.org/10.1162/neco.1997.9.8.1735.
19. Harold Hotelling. Analysis of a complex of statistical variables into principal components. Journal of educational psychology, 24(6):417, 1933.
20. George Hripcsak, David J Albers, and Adler Perotte. Parameterizing time in electronic health record studies. Journal of the American Medical Informatics Association, 22(4): 794-804, 2015.
21. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
22. Kristel J.M. Janssen, A. Rogier T. Donders, Frank E. Harrell, Yvonne Vergouwe, Qingxia Chen, Diederick E. Grobbee, and Karel G.M. Moons. Missing covariate data in medical research: To impute is better than to ignore. Journal of Clinical Epidemiology, 63(7): 721-727, 2010.
23. Alistair E. W. Johnson, Tom J. Pollard, and Tristan Naumann. Generalizability of predictive models for intensive care unit patients. arXiv:1812.02275 [cs, stat], December 2018. URL http://arxiv.org/abs/1812.02275. arXiv: 1812.02275.
24. Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3:160035, 2016.
25. I.T. Jolliffe. Principal Component Analysis. Springer Verlag, second edition, 2002.
26. Kenneth Jung and Nigam H Shah. Implications of non-stationarity on predictive modeling using ehrs. Journal of biomedical informatics, 58:168-174, 2015.
27. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
28. Robert Kocher, Ezekiel J Emanuel, and Nancy-Ann M DeParle. The affordable care act and the future of clinical medicine: the opportunities and challenges. Annals of internal medicine, 153(8):536-539, 2010.
29. Thomas A Lasko, Joshua C Denny, and Mia A Levy. Computational phenotype discovery using unsupervised feature learning over noisy, sparse, and irregular clinical data. PloS one, 8(6):e66341, 2013.
30. David Lazer, Ryan Kennedy, Gary King, and Alessandro Vespignani. The parable of google flu: traps in big data analysis. Science, 343(6176):1203-1205, 2014.
31. Bryan Lim and Mihaela van der Schaar. Disease-atlas: Navigating disease trajectories with deep learning. arXiv preprint arXiv:1803.10254, 2018.
32. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Technical report, TensorFlow, 2015. URL https://www.tensorflow.org/.
33. M.B.A. McDermott, T. Yan, T. Naumann, N. Hunt, H. Suresh, P. Szolovits, and M. Ghassemi. Semi-supervised Biomedical Translation with Cycle Wasserstein Regression GANs. In Association for the Advancement of Artificial Intelligence, New Orleans, LA, 2018.
34. L. McInnes and J. Healy. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. ArXiv e-prints, February 2018.
35. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS 2017 Autodiff Workshop, 2017. URL https://openreview.net/forum?id=BJJsrmfCZ¬eId=BJJsrmfCZ.
36. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12:2825-2830, October 2011. ISSN 1533-7928. URL http://jmlr.csail.mit.edu/papers/v12/pedregosa11a.html.
37. Tom J. Pollard, Alistair E. W. Johnson, Jesse D. Raffa, Leo A. Celi, Roger G. Mark, and Omar Badawi. The eICU Collaborative Research Database, a freely available multicenter database for critical care research. Scientific Data, 5:180178, September 2018. doi: 10.1038/sdata.2018.178. URL https://www.nature.com/articles/sdata2018178.
38. Sanjay Purushotham, Chuizheng Meng, Zhengping Che, and Yan Liu. Benchmarking deep learning models on large healthcare datasets. Journal of biomedical informatics, 83:112-134, 2018.
39. A. Raghu, M. Komorowski, L.A. Celi, P. Szolovits, and M. Ghassemi. Continuous state-space models for optimal sepsis treatment: a deep reinforcement learning approach. In Machine Learning for Healthcare Conference (MLHC), pages 147-163, 2017.
40. Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning with applications to medical imaging. arXiv preprint arXiv:1902.07208, 2019.
41. Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M Dai, Nissan Hajaj, Michaela Hardt, Peter J Liu, Xiaobing Liu, Jake Marcus, Mimi Sun, et al. Scalable and accurate deep learning with electronic health records. NPJ Digital Medicine, 1(1):18, 2018.
42. Guergana K Savova, James J Masanz, Philip V Ogren, Jiaping Zheng, Sunghwan Sohn, Karin C Kipper-Schuler, and Christopher G Chute. Mayo clinical text analysis and knowledge extraction system (ctakes): architecture, component evaluation and applications. J Am Med Inform Assoc, 17(5):507-513, 2010.
43. Adarsh Subbaswamy and Suchi Saria. Counterfactual normalization: Proactively addressing dataset shift using causal mechanisms. In UAI, pages 947-957, 2018.
44. Adarsh Subbaswamy, Peter Schulam, and Suchi Saria. Preventing failures due to dataset shift: Learning predictive models that transport. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3118-3127, 2019.
45. Harini Suresh, Nathan Hunt, Alistair Johnson, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Clinical intervention prediction and understanding with deep neural networks. In Proceedings of the 2nd Machine Learning for Healthcare Conference, volume 68 of Proceedings of Machine Learning Research, pages 322-337, Boston, Massachusetts, 18-19 Aug 2017. PMLR.
46. Shirly Wang, Matthew McDermott, Geeticka Chauhan, Michael C Hughes, Tristan Naumann, and Marzyeh Ghassemi. Mimic-extract: A data extraction, preprocessing, and representation pipeline for mimic-iii. arXiv preprint arXiv:1907.08322, 2019.
47. I. Žliobaite. Learning under Concept Drift: an Overview. ArXiv e-prints, October 2010.