A new insight into missing data in intensive care unit patient profiles: Observational study

JMIR Medical Informatics

Volumn 7, Issue 1, 2019, Pages

  Sharafoddini, Anis ^a   Dubin, Joel A ^a   Maslove, David M ^b   Lee, Joon ^a,c  

^a UNIVERSITY OF WATERLOO   (Canada)

^b QUEEN S UNIVERSITY   (Canada)

^c UNIVERSITY OF CALGARY   (Canada)

Author keywords

Clinical laboratory tests; Electronic health records; Hospital mortality; Machine learning

Indexed keywords

EID: 85097194114     PISSN: None     EISSN: 22919694     Source Type: Journal    
DOI: 10.2196/11605     Document Type: Article

References (83)

1. Weiskopf NG, Weng C. Methods and dimensions of electronic health record data quality assessment: Enabling reuse for clinical research. J Am Med Inform Assoc 2013 Jan 1; 20(1):144-151 [FREE Full text] [doi: 10.1136/amiajnl-2011-000681] [Medline: 22733976]
2. Chan KS, Fowles JB, Weiner JP. Review: Electronic health records and the reliability and validity of quality measures: A review of the literature. Med Care Res Rev 2010 Oct; 67(5):503-527. [doi: 10.1177/1077558709359007] [Medline: 20150441]
3. Weiskopf NG, Hripcsak G, Swaminathan S, Weng C. Defining and measuring completeness of electronic health records for secondary use. J Biomed Inform 2013 Oct; 46(5):830-836 [FREE Full text] [doi: 10.1016/j.jbi.2013.06.010] [Medline: 23820016]
4. Wells BJ, Chagin KM, Nowacki AS, Kattan MW. Strategies for handling missing data in electronic health record derived data. EGEMS (Wash DC) 2013; 1(3):1035 [FREE Full text] [doi: 10.13063/2327-9214.1035] [Medline: 25848578]
5. Little RJ, D'Agostino R, Cohen ML, Dickersin K, Emerson SS, Farrar JT, et al. The prevention and treatment of missing data in clinical trials. N Engl J Med 2012 Oct 4; 367(14):1355-1360 [FREE Full text] [doi: 10.1056/NEJMsr1203730] [Medline: 23034025]
6. Sterne JA, White IR, Carlin JB, Spratt M, Royston P, Kenward MG, et al. Multiple imputation for missing data in epidemiological and clinical research: Potential and pitfalls. Br Med J 2009; 338:b2393 [FREE Full text] [Medline: 19564179]
7. Haukoos JS, Newgard CD. Advanced statistics: Missing data in clinical research-part 1: An introduction and conceptual framework. Acad Emerg Med 2007 Jul; 14(7):662-668 [FREE Full text] [doi: 10.1197/j.aem.2006.11.037] [Medline: 17538078]
8. Newgard CD, Haukoos JS. Advanced statistics: Missing data in clinical research-part 2: Multiple imputation. Acad Emerg Med 2007 Jul; 14(7):669-678 [FREE Full text] [doi: 10.1197/j.aem.2006.11.038] [Medline: 17595237]
9. Pringle M, Ward P, Chilvers C. Assessment of the completeness and accuracy of computer medical records in four practices committed to recording data on computer. Br J Gen Pract 1995 Oct; 45(399):537-541 [FREE Full text] [Medline: 7492423]
10. Thiru K, de Lusignan S, Hague N. Have the completeness and accuracy of computer medical records in general practice improved in the last five years? The report of a two-practice pilot study. Health Informatics J 2016 Jul 25; 5(4):224-232. [doi: 10.1177/146045829900500410]
11. Forster M, Bailey C, Brinkhof MW, Graber C, Boulle A, Spohr M, ART-LINC collaboration of International Epidemiological Databases to Evaluate AIDS. Electronic medical record systems, data quality and loss to follow-up: Survey of antiretroviral therapy programmes in resource-limited settings. Bull World Health Organ 2008 Dec; 86(12):939-947 [FREE Full text] [Medline: 19142294]
12. Jones RB, Hedley AJ. A computer in the diabetic clinic. Completeness of data in a clinical information system for diabetes. Pract Diab Int 1986 Nov; 3(6):295-296. [doi: 10.1002/pdi.1960030610]
13. Porcheret M, Hughes R, Evans D, Jordan K, Whitehurst T, Ogden H, North Staffordshire General Practice Research Network. Data quality of general practice electronic health records: The impact of a program of assessments, feedback, and training. J Am Med Inform Assoc 2004; 11(1):78-86 [FREE Full text] [doi: 10.1197/jamia.M1362] [Medline: 14527973]
14. Soto CM, Kleinman KP, Simon SR. Quality and correlates of medical record documentation in the ambulatory care setting. BMC Health Serv Res 2002 Dec 10; 2(1):22 [FREE Full text] [Medline: 12473161]
15. Tang PC, LaRosa MP, Gorden SM. Use of computer-based records, completeness of documentation, and appropriateness of documented clinical decisions. J Am Med Inform Assoc 1999 May 01; 6(3):245-251. [doi: 10.1136/jamia.1999.0060245]
16. Jensen RE, Chan KS, Weiner JP, Fowles JB, Neale SM. Implementing electronic health record-based quality measures for developmental screening. Pediatrics 2009 Oct; 124(4):e648-e654. [doi: 10.1542/peds.2008-3091] [Medline: 19786425]
17. Botsis T, Hartvigsen G, Chen F, Weng C. Secondary use of EHR: Data quality issues and informatics opportunities. AMIA Jt Summits Transl Sci Proc 2010 Mar 01; 2010:1-5 [FREE Full text] [Medline: 21347133]
18. Sharafoddini A, Dubin JA, Lee J. Patient similarity in prediction models based on health data: A scoping review. JMIR Med Inform 2017 Mar 03; 5(1):e7 [FREE Full text] [doi: 10.2196/medinform.6730] [Medline: 28258046]
19. Rusanov A, Weiskopf NG, Wang S, Weng C. Hidden in plain sight: Bias towards sick patients when sampling patients with sufficient electronic health record data for research. BMC Med Inform Decis Mak 2014 Jun 11; 14:51. [doi: 10.1186/1472-6947-14-51] [Medline: 24916006]
20. Weiskopf NG, Rusanov A, Weng C. Sick patients have more data: The non-random completeness of electronic health records. AMIA Annu Symp Proc 2013; 2013:1472-1477 [FREE Full text] [Medline: 24551421]
21. Agniel D, Kohane I, Weber G. Biases in electronic health record data due to processes within the healthcare system: Retrospective observational study. Br Med J 2018 Dec 30; 361:k1479 [FREE Full text] [doi: 10.1136/bmj.k1479] [Medline: 29712648]
22. Kuhn M, Johnson K. Applied Predictive Modeling. New York: Springer; 2013.
23. Lee J, Maslove DM. Using information theory to identify redundancy in common laboratory tests in the intensive care unit. BMC Med Inform Decis Mak 2015 Jul 31; 15:59 [FREE Full text] [doi: 10.1186/s12911-015-0187-x] [Medline: 26227625]
24. Oliveira AM, Oliveira MV, Souza CL. Prevalence of unnecessary laboratory tests and related avoidable costs in intensive care unit. J Bras Patol Med Lab 2014; 50:410-416. [doi: 10.5935/1676-2444.20140049]
25. Cismondi F, Celi LA, Fialho AS, Vieira SM, Reti SR, Sousa JM, et al. Reducing unnecessary lab testing in the ICU with artificial intelligence. Int J Med Inform 2013 May; 82(5):345-358 [FREE Full text] [doi: 10.1016/j.ijmedinf.2012.11.017] [Medline: 23273628]
26. Rothwell PM. External validity of randomised controlled trials: "to whom do the results of this trial apply?". Lancet 2005; 365(9453):82-93. [doi: 10.1016/S0140-6736(04)17670-8] [Medline: 15639683]
27. Doyle JD, Garmon EH. American Society of Anesthesiologists Classification (ASA Class). StatPearls 2018. [Medline: 28722969]
28. Vesin A, Azoulay E, Ruckly S, Vignoud L, Rusinovà K, Benoit D, et al. Reporting and handling missing values in clinical studies in intensive care units. Intensive Care Med 2013 Aug; 39(8):1396-1404. [doi: 10.1007/s00134-013-2949-1] [Medline: 23685609]
29. Fitzmaurice GM, Laird NM, Ware JH. Applied Longitudinal Analysis. Hoboken, New Jersey: Wiley; 2011.
30. Baraldi AN, Enders CK. An introduction to modern missing data analyses. J Sch Psychol 2010 Feb; 48(1):5-37. [doi: 10.1016/j.jsp.2009.10.001] [Medline: 20006986]
31. Stockdale M, Royal K. Missing data as a validity threat for medical and healthcare education research: Problems and solutions. Int J Health Care 2016 Jun 23; 2(2). [doi: 10.5430/ijh.v2n2p67]
32. Myers TA. Goodbye, listwise deletion: Presenting hot deck imputation as an easy and effective tool for handling missing data. Communication Methods and Measures 2011 Oct; 5(4):297-310. [doi: 10.1080/19312458.2011.624490]
33. Pigott TD. A review of methods for missing data. Educ Res Eval 2001 Dec 1; 7(4):353-383. [doi: 10.1076/edre.7.4.353.8937]
34. Roth PL. Missing data - a conceptual review for applied psychologists. Pers Psychol 1994; 47(3):537-560. [doi: 10.1111/j.1744-6570.1994.tb01736.x]
35. Gorelick MH. Bias arising from missing data in predictive models. J Clin Epidemiol 2006 Oct; 59(10):1115-1123. [doi: 10.1016/j.jclinepi.2004.11.029] [Medline: 16980153]
36. Schulman CS, Staul L. Standards for frequency of measurement and documentation of vital signs and physical assessments. Crit Care Nurse 2010 Jun; 30(3):74-76 [FREE Full text] [doi: 10.4037/ccn2010406] [Medline: 20515885]
37. Asiimwe SB, Okello S, Moore CC. Frequency of vital signs monitoring and its association with mortality among adults with severe sepsis admitted to a general medical ward in Uganda. PLoS One 2014; 9(2):e89879 [FREE Full text] [doi: 10.1371/journal.pone.0089879] [Medline: 24587094]
38. Cardona-Morrell M, Nicholson M, Hillman K. Vital Signs: From Monitoring to Prevention of Deterioration in General Wards. In: Vincent JL, editor. Annual Update In Intensive Care And Emergency Medicine 2015. Cham: Springer; 2018.
39. Frassica JJ. Frequency of laboratory test utilization in the intensive care unit and its implications for large scale data collection efforts. AMIA Annu Symp Proc 2003:844 [FREE Full text] [Medline: 14728349]
40. Johnson AE, Pollard TJ, Shen L, Lehman LH, Feng M, Ghassemi M, et al. MIMIC-III, a freely accessible critical care database. Sci Data 2016; 3:160035 [FREE Full text] [doi: 10.1038/sdata.2016.35] [Medline: 27219127]
41. Harrell M, Fabbri D, Levy M. Evaluating EHR Data Availability for Cohort Selection in Retrospective Studies. 2016 Presented at: IEEE International Conference on Healthcare Informatics (ICHI); 2016; Chicago, IL, USA p. 4-7. [doi: 10.1109/ICHI.2016.68]
42. Little RJ. Missing-data adjustments in large surveys. J Bus Econ Stat 1988 Jul; 6(3):287-296. [doi: 10.1080/07350015.1988.10509663]
43. Zhou XH, Eckert GJ, Tierney WM. Multiple imputation in public health research. Stat Med 2001; 20(9-10):1541-1549. [Medline: 11343373]
44. Buuren SV. Flexible Imputation Of Missing Data. New York: Chapman and Hall/CRC; 2018.
45. Andridge RR, Little RJ. A review of hot deck imputation for survey non-response. Int Stat Rev 2010 Apr; 78(1):40-64 [FREE Full text] [doi: 10.1111/j.1751-5823.2010.00103.x] [Medline: 21743766]
46. Abraham W, Russell D. Missing data: A review of current methods and applications in epidemiological research. Curr Opin Psychiatr 2004 Jul; 17(4):315-321. [doi: 10.1097/01.yco.0000133836.34543.7e]
47. Groenwold RH, White IR, Donders AR, Carpenter JR, Altman DG, Moons KG. Missing covariate data in clinical research: when and when not to use the missing-indicator method for analysis. Can Med Assoc J 2012 Aug 07; 184(11):1265-1269 [FREE Full text] [doi: 10.1503/cmaj.110977] [Medline: 22371511]
48. Knol MJ, Janssen KJ, Donders AR, Egberts AC, Heerdink ER, Grobbee DE, et al. Unpredictable bias when using the missing indicator method or complete case analysis for missing confounder values: An empirical example. J Clin Epidemiol 2010 Jul; 63(7):728-736. [doi: 10.1016/j.jclinepi.2009.08.028] [Medline: 20346625]
49. Steyerberg E. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. New York: Springer-Verlag; 2008.
50. Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care 1998 Jan; 36(1):8-27. [Medline: 9431328]
51. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: Development and validation. J Chronic Dis 1987; 40(5):373-383. [Medline: 3558716]
52. Menendez ME, Neuhaus V, van Dijk CN, Ring D. The Elixhauser comorbidity method outperforms the Charlson index in predicting inpatient death after orthopaedic surgery. Clin Orthop Relat Res 2014 Sep; 472(9):2878-2886 [FREE Full text] [doi: 10.1007/s11999-014-3686-7] [Medline: 24867450]
53. Southern DA, Quan H, Ghali WA. Comparison of the Elixhauser and Charlson/Deyo methods of comorbidity measurement in administrative data. Med Care 2004 Apr; 42(4):355-360. [Medline: 15076812]
54. Farley JF, Harley CR, Devine JW. A comparison of comorbidity measurements to predict healthcare expenditures. Am J Manag Care 2006 Feb; 12(2):110-119 [FREE Full text] [Medline: 16464140]
55. Sharabiani MT, Aylin P, Bottle A. Systematic review of comorbidity indices for administrative data. Med Care 2012 Dec; 50(12):1109-1118. [doi: 10.1097/MLR.0b013e31825f64d0] [Medline: 22929993]
56. Le Gall JR, Lemeshow S, Saulnier F. A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. J Am Med Assoc 1993; 270(24):2957-2963. [Medline: 8254858]
57. Saeys Y, Inza I, Larrañaga P. A review of feature selection techniques in bioinformatics. Bioinformatics 2007 Oct 1; 23(19):2507-2517 [FREE Full text] [doi: 10.1093/bioinformatics/btm344] [Medline: 17720704]
58. Robnik-Sikonja M, Kononenko I. Theoretical and empirical analysis of ReliefF and RReliefF. Mach Learn 2003; 53(1-2):23-69. [doi: 10.1023/A:1025667309714]
59. Peng H, Long F, Ding C. Feature selection based on mutual information: Criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans Pattern Anal Mach Intell 2005 Aug; 27(8):1226-1238. [doi: 10.1109/TPAMI.2005.159] [Medline: 16119262]
60. Mitchell T. Machine Learning. New York: McGraw-Hill Education; 1997
61. Aggarwal CC. Data Classification: Algorithms And Applications. New York: Chapman and Hall/CRC; 2018.
62. Martens FK, Tonk EC, Kers JG, Janssens AC. Small improvement in the area under the receiver operating characteristic curve indicated small changes in predicted risks. J Clin Epidemiol 2016 Nov; 79:159-164. [doi: 10.1016/j.jclinepi.2016.07.002] [Medline: 27430730]
63. Cook NR. Response to letters regarding article, "use and misuse of the receiver operating characteristic curve in risk prediction". Circulation 2007 Aug 07; 116(6):e134-e134. [doi: 10.1161/Circulationaha.107.715359]
64. Pencina MJ, D'Agostino RB, Pencina KM, Janssens AC, Greenland P. Interpreting incremental value of markers added to risk prediction models. Am J Epidemiol 2012 Sep 15; 176(6):473-481 [FREE Full text] [doi: 10.1093/aje/kws207] [Medline: 22875755]
65. Gowda S, Desai PB, Hull VV, Math AA, Vernekar SN, Kulkarni SS. A review on laboratory liver function tests. Pan Afr Med J 2009 Nov 22; 3:17 [FREE Full text] [Medline: 21532726]
66. Wennberg JE. Dealing with medical practice variations: A proposal for action. Health Aff (Millwood) 1984; 3(2):6-32. [Medline: 6432667]
67. Daniels M, Schroeder SA. Variation among Physicians in Use of Laboratory Tests II. Relation to Clinical Productivity and Outcomes of Care. Medical Care 1977; 15(6):482-487. [doi: 10.1097/00005650-197706000-00004]
68. Solomon DH, Hashimoto H, Daltroy L, Liang MH. Techniques to improve physicians' use of diagnostic tests: A new conceptual framework. J Am Med Assoc 1998 Dec 16; 280(23):2020-2027. [Medline: 9863854]
69. Sharafoddini A, Dubin JA, Lee J. Finding Similar Patient Subpopulations in the ICU Using Laboratory Test Ordering Patterns. In: Proceedings of the 2018 7th International Conference on Bioinformatics and Biomedical Science. 2018 Presented at: 2018 7th International Conferenceon Bioinformatics and Biomedical Science; 2018; Shenzhen, China. [doi: 10.1145/3239264.3239277]
70. Zhang Z, Xu X. Lactate clearance is a useful biomarker for the prediction of all-cause mortality in critically ill patients: A systematic review and meta-analysis*. Crit Care Med 2014 Sep; 42(9):2118-2125. [doi: 10.1097/CCM.0000000000000405] [Medline: 24797375]
71. Levinson AT, Casserly BP, Levy MM. Reducing mortality in severe sepsis and septic shock. Semin Respir Crit Care Med 2011 Apr; 32(2):195-205. [doi: 10.1055/s-0031-1275532] [Medline: 21506056]
72. Beier K, Eppanapally S, Bazick HS, Chang D, Mahadevappa K, Gibbons FK, et al. Elevation of blood urea nitrogen is predictive of long-term mortality in critically ill patients independent of "normal" creatinine. Crit Care Med 2011 Feb; 39(2):305-313 [FREE Full text] [doi: 10.1097/CCM.0b013e3181ffe22a] [Medline: 21099426]
73. Cauthen CA, Lipinski MJ, Abbate A, Appleton D, Nusca A, Varma A, et al. Relation of blood urea nitrogen to long-term mortality in patients with heart failure. Am J Cardiol 2008 Jun 01; 101(11):1643-1647. [doi: 10.1016/j.amjcard.2008.01.047] [Medline: 18489944]
74. Kajimoto K, Sato N, Takano T, Investigators of the Acute Decompensated Heart Failure Syndromes (ATTEND) registry. Relation between elevated blood urea nitrogen, clinical features or comorbidities, and clinical outcome in patients hospitalized for acute heart failure syndromes. Int J Cardiol 2015 Dec 15; 201:311-314. [doi: 10.1016/j.ijcard.2015.08.061] [Medline: 26301667]
75. Bazick HS, Chang D, Mahadevappa K, Gibbons FK, Christopher KB. Red cell distribution width and all-cause mortality in critically ill patients. Crit Care Med 2011 Aug; 39(8):1913-1921 [FREE Full text] [doi: 10.1097/CCM.0b013e31821b85c6] [Medline: 21532476]
76. Hunziker S, Celi LA, Lee J, Howell MD. Red cell distribution width improves the simplified acute physiology score for risk prediction in unselected critically ill patients. Crit Care 2012 May 18; 16(3):R89 [FREE Full text] [doi: 10.1186/cc11351] [Medline: 22607685]
77. Patel KV, Semba RD, Ferrucci L, Newman AB, Fried LP, Wallace RB, et al. Red cell distribution width and mortality in older adults: A meta-analysis. J Gerontol A Biol Sci Med Sci 2010 Mar; 65(3):258-265 [FREE Full text] [doi: 10.1093/gerona/glp163] [Medline: 19880817]
78. Purtle SW, Moromizato T, McKane CK, Gibbons FK, Christopher KB. The association of red cell distribution width at hospital discharge and out-of-hospital mortality following critical illness*. Crit Care Med 2014 Apr; 42(4):918-929. [doi: 10.1097/CCM.0000000000000118] [Medline: 24448196]
79. Şenol K, Saylam B, Kocaay F, Tez M. Red cell distribution width as a predictor of mortality in acute pancreatitis. Am J Emerg Med 2013 Apr; 31(4):687-689. [doi: 10.1016/j.ajem.2012.12.015] [Medline: 23399348]
80. Ahn SY, Ryu J, Baek SH, Han JW, Lee JH, Ahn S, et al. Serum anion gap is predictive of mortality in an elderly population. Exp Gerontol 2014 Feb; 50:122-127. [doi: 10.1016/j.exger.2013.12.002] [Medline: 24333141]
81. Kim MJ, Kim YH, Sol IS, Kim SY, Kim JD, Kim HY, et al. Serum anion gap at admission as a predictor of mortality in the pediatric intensive care unit. Sci Rep 2017 May 03; 7(1):1456 [FREE Full text] [doi: 10.1038/s41598-017-01681-9] [Medline: 28469150]
82. Lee SW, Kim S, Na KY, Cha R, Kang SW, Park CW, et al. Serum anion gap predicts all-cause mortality in patients with advanced chronic kidney disease: A retrospective analysis of a randomized controlled study. PLoS One 2016; 11(6):e0156381 [FREE Full text] [doi: 10.1371/journal.pone.0156381] [Medline: 27249416]
83. Sahu A, Cooper HA, Panza JA. The initial anion gap is a predictor of mortality in acute myocardial infarction. Coron Artery Dis 2006 Aug; 17(5):409-412. [Medline: 16845247]