Calibration drift in regression and machine learning models for acute kidney injury

Journal of the American Medical Informatics Association

Volumn 24, Issue 6, 2017, Pages 1052-1061

  Davis, Sharon E ^a   Lasko, Thomas A ^a   Chen, Guanhua ^a   Siew, Edward D ^a,b   Matheny, Michael E ^a,b  

^a VANDERBILT UNIVERSITY SCHOOL OF MEDICINE   (United States)

^b VETERANS AFFAIRS MEDICAL CENTER   (United States)

Author keywords

Acute kidney injury; Calibration; Clinical decision support; Clinical prediction; Discrimination; Machine learning

Indexed keywords

ACUTE KIDNEY FAILURE; AGED; ARTICLE; CALIBRATION; CLINICAL DECISION SUPPORT SYSTEM; FEMALE; HUMAN; MACHINE LEARNING; MAJOR CLINICAL STUDY; MALE; MEDICAL DECISION MAKING; OUTCOME ASSESSMENT; PROBABILITY; RANDOM FOREST; REGRESSION ANALYSIS; RISK ASSESSMENT; SAFETY; STATISTICAL MODEL; VALIDATION STUDY; BAYES THEOREM; CLINICAL DECISION MAKING; DECISION SUPPORT SYSTEM; MIDDLE AGED; PUBLIC HOSPITAL; UNITED STATES;

ACUTE KIDNEY INJURY; AGED; BAYES THEOREM; CLINICAL DECISION-MAKING; DECISION SUPPORT TECHNIQUES; FEMALE; HOSPITALS, VETERANS; HUMANS; LOGISTIC MODELS; MACHINE LEARNING; MALE; MIDDLE AGED; UNITED STATES;

EID: 85032453950     PISSN: 10675027     EISSN: 1527974X     Source Type: Journal    
DOI: 10.1093/jamia/ocx030     Document Type: Article

References (64)

1. Amarasingham R, Patzer RE, Huesch M, Nguyen NQ, Xie B. Implementing electronic health care predictive analytics: considerations and challenges. Health Affairs. 2014;33(7):1148-54.
2. Ohno-Machado L, Resnic FS, Matheny ME. Prognosis in critical care. Ann Rev Biomed Engineering. 2006;8:567-99.
3. Moons KG, Kengne AP, Grobbee DE, et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart. 2012;98(9):691-98.
4. Matheny ME, Miller RA, Ikizler TA, et al. Development of inpatient risk stratification models of acute kidney injury for use in electronic health records. Med Decision Making. 2010;30(6):639-50.
5. Kansagara D, Englander H, Salanitro A, et al. Risk prediction models for hospital readmission: a systematic review. JAMA. 2011;306(15): 1688-98.
6. Steyerberg EW, Moons KG, van der Windt DA, et al. Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10(2):e1001381.
7. Toll DB, Janssen KJ, Vergouwe Y, Moons KG. Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol. 2008;61(11):1085-94.
8. Hickey GL, Grant SW, Murphy GJ, et al. Dynamic trends in cardiac surgery: why the logistic euroscore is no longer suitable for contemporary cardiac surgery and implications for future risk models. Eur J Cardiothoracic Surg. 2013;43(6):1146-52.
9. Minne L, Eslami S, De Keizer N, De Jonge E, De Rooij SE, Abu-Hanna A. Effect of changes over time in the performance of a customized SAPS-II model on the quality of care assessment. Intensive Care Med. 2012;38(1):40-46.
10. Minne L, Eslami S, de Keizer N, de Jonge E, de Rooij SE, Abu-Hanna A. Statistical process control for monitoring standardized mortality ratios of a classification tree model. Methods Inf Med. 2012;51(4):353-58.
11. Moons KG, Altman DG, Vergouwe Y, Royston P. Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ. 2009;338:b606.
12. Harrison DA, Lone NI, Haddow C, et al. External validation of the Intensive Care National Audit & Research Centre (ICNARC) risk prediction model in critical care units in Scotland. BMC Anesthesiol. 2014;14:116.
13. Paul E, Bailey M, Van Lint A, Pilcher V. Performance of APACHE III over time in Australia and New Zealand: a retrospective cohort study. Anaesthesia Intensive Care. 2012;40(6):980-94.
14. Madan P, Elayda MA, Lee VV, Wilson JM. Risk-prediction models for mortality after coronary artery bypass surgery: application to individual patients. Int J Cardiol. 2011;149(2):227-31.
15. Amarasingham R, Audet AJ, Bates DW, et al. Consensus statement on electronic health predictive analytics: a guiding framework to address challenges. eGEMs. 2016;4(1):1-11.
16. Parikh RB, Kakad M, Bates DW. Integrating predictive analytics into high-value care: the dawn of precision delivery. JAMA. 2016;315(7):651-52.
17. Pencina MJ, Peterson ED. Moving from clinical trials to precision medicine: the role for predictive modeling. JAMA. 2016;315(16):1713-14.
18. Sajda P. Machine learning for detection and diagnosis of disease. Ann Rev Biomed Engineering. 2006;8:537-65.
19. Steyerberg EW, van der Ploeg T, Van Calster B. Risk prediction with machine learning and regression methods. Biometrical J. Biometrische Zeitschrift. 2014;56(4):601-06.
20. Uchino S, Kellum JA, Bellomo R, et al. Acute renal failure in critically ill patients: a multinational, multicenter study. JAMA. 2005;294(7):813-18.
21. Brivet FG, Kleinknecht DJ, Loirat P, Landais PJ. Acute renal failure in intensive care units - causes, outcome, and prognostic factors of hospital mortality: a prospective, multicenter study. French Study Group on Acute Renal Failure. Crit Care Med. 1996;24(2):192-98
22. Coca SG, Yusuf B, Shlipak MG, Garg AX, Parikh CR. Long-term risk of mortality and other adverse outcomes after acute kidney injury: a systematic review and meta-analysis. Am J Kidney Dis. 2009;53(6):961-73
23. Liano F, Junco E, Pascual J, Madero R, Verde E. The spectrum of acute renal failure in the intensive care unit compared with that seen in other settings. The Madrid Acute Renal Failure Study Group. Kidney Int Suppl. 1998;66:S16-24.
24. Cronin RM, VanHouten JP, Siew ED, et al. National Veterans Health Administration Inpatient Risk Stratification Models for Hospital-Acquired Acute Kidney Injury. J AmMed Inform Assoc. 2015;22(5):1054-71.
25. Breidthardt T, Christ-Crain M, Stolz D, et al. A combined cardiorenal assessment for the prediction of acute kidney injury in lower respiratory tract infections. Am J Med. 2012;125(2):168-75.
26. Kim WH, Lee SM, Choi JW, et al. Simplified clinical risk score to predict acute kidney injury after aortic surgery. J Cardiothorac Vasc Anesth. 2013;27(6):1158-66.
27. Kristovic D, Horvatic I, Husedzinovic I, et al. Cardiac surgery-associated acute kidney injury: risk factors analysis and comparison of prediction models. Interact Cardiovasc Thorac Surg. 2015;21(3):366-73.
28. McMahon GM, Zeng X, Waikar SS. A risk prediction score for kidney failure or mortality in rhabdomyolysis. JAMA Int Med. 2013;173(19):1821-28.
29. Ng SY, Sanagou M, Wolfe R, Cochrane A, Smith JA, Reid CM. Prediction of acute kidney injury within 30 days of cardiac surgery. J Thoracic Cardiovasc Surgery. 2014;147(6):1875-83, 83 e1.
30. Park MH, Shim HS, Kim WH, et al. Clinical risk scoring models for prediction of acute kidney injury after living donor liver transplantation: a retrospective observational study. PloS One. 2015;10(8):e0136230.
31. Slankamenac K, Beck-Schimmer B, Breitenstein S, Puhan MA, Clavien PA. Novel prediction score including pre- and intraoperative parameters best predicts acute kidney injury after liver surgery. World J Surgery. 2013;37(11):2618-28.
32. Wang YN, Cheng H, Yue T, Chen YP. Derivation and validation of a prediction score for acute kidney injury in patients hospitalized with acute heart failure in a Chinese cohort. Nephrology. 2013;18(7):489-96.
33. Rodriguez E, Soler MJ, Rap O, Barrios C, Orfila MA, Pascual J. Risk factors for acute kidney injury in severe rhabdomyolysis. PloS One. 2013;8(12):e82992.
34. Schneider DF, Dobrowolsky A, Shakir IA, Sinacore JM, Mosier MJ, Gamelli RL. Predicting acute kidney injury among burn patients in the 21st century: a classification and regression tree analysis. J Burn Care Res. 2012;33(2):242-51.
35. Legrand M, Pirracchio R, Rosa A, et al. Incidence, risk factors and prediction of post-operative acute kidney injury following cardiac surgery for active infective endocarditis: an observational study. Crit Care. 2013;17(5):R220.
36. Brown JR, MacKenzie TA, Maddox TM, et al. Acute kidney injury risk prediction in patients undergoing coronary angiography in a national Veterans Health Administration cohort with external validation. J Am Heart Assoc. 2015;4(12):e002136.
37. Gurm HS, Seth M, Kooiman J, Share D. A novel tool for reliable and accurate prediction of renal complications in patients undergoing percutaneous coronary intervention. J Am Coll Cardiol. 2013;61(22):2242-48.
38. Matheny ME, Ohno-Machado L, Resnic FS. Discrimination and calibration of mortality risk prediction models in interventional cardiology. J Biomed Inform. 2005;38(5):367-75.
39. Jiang X, Osl M, Kim J, Ohno-Machado L. Calibrating predictive model estimates to support personalized medicine. J Am Med Inform Assoc, 2012;19(2):263-74.
40. Van Calster B, Vickers AJ. Calibration of risk prediction models: impact on decision-analytic performance. Med Decis Mak. 2015;35(2):162-69.
41. Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. New York, NY: Springer; 2009.
42. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128-38.
43. Bouwmeester W, Zuithoff NP, Mallett S, et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med. 2012;9(5):1-12.
44. Collins GS, de Groot JA, Dutton S, et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol. 2014;14:40.
45. Mallett S, Royston P, Waters R, Dutton S, Altman DG. Reporting performance of prognostic models in cancer: a review. BMC Med. 2010;8:21.
46. Cook DA, Joyce CJ, Barnett RJ, et al. Prospective independent validation of APACHE III models in an Australian tertiary adult intensive care unit. Anaesth Intensive Care. 2002;30(3):308-15.
47. Malley JD, Kruppa J, Dasgupta A, Malley KG, Ziegler A. Probability machines: consistent probability estimation using nonparametric learning machines. Methods Inform Med. 2012;51(1):74-81.
48. Breiman L. Statistical modeling: the two cultures. Statistical Science. 2001;16(3):199-231 49. Perlin JB, Kolodner RM, Roswell RH. The Veterans Health Administration: quality, value, accountability, and information as transforming strategies for patient-centered care. AmJManaged Care. 2004;10(11 Pt 2):828-36.
49. Perlin JB, Kolodner RM, Roswell RH. The Veterans Health Administration: quality, value, accountability, and information as transforming strategies for patient-centered care. AmJManaged Care. 2004;10(11 Pt 2):828-36
50. Khwaja A. KDIGO clinical practice guidelines for acute kidney injury. Nephron Clin Pract. 2012;120(4):c179-84.
51. Tibshirani R. Regression shrinkage and selection via the lasso. J Royal Stat Soc Series B. 1996;58(1):267-88.
52. Hoerl AE, Kennard RW. Ridge regression: biased estimation for nonorthogonal problems. Technometrics. 1970;12(1):55-67.
53. Zou H, Hastie T. Regularization and variable selection via the elastic net. J Royal Stat Soc Series B. 2005;67(2):301-20.
54. Breiman L. Random forests. Machine Learning. 2001;45(1):5-32.
55. Bishop CM. Neural Networks for Pattern Recognition. New York: Oxford University Press; 1995.
56. Hand DJ. Naive Bayes. In: Wu X, Kumar V, eds. The Top Ten Algorithms in Data Mining. Chapman&Hall/CRC; 2009:163-78.
57. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29-36.
58. Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol. 2016;74:167-76.
59. Van Hoorde K, Van Huffel S, Timmerman D, Bourne T, Van Calster B. A spline-based tool to assess and visualize the calibration of multiclass risk predictions. J Biomed Inform. 2015;54:283-93.
60. Debray TP, Vergouwe Y, Koffijberg H, Nieboer D, Steyerberg EW, Moons KG. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol. 2015;68(3):279-89.
61. Meinshausen N, Bühlmann P. Stability selection. J Royal Stat Soc Series B (Statistical Methodology). 2010;72(4):417-73.
62. Steyerberg EW, Borsboom GJ, van Houwelingen HC, Eijkemans MJ, Habbema JD. Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat Med. 2004;23(16):2567-86.
63. Kappen TH, Vergouwe Y, van Klei WA, van Wolfswinkel L, Kalkman CJ, Moons KG. Adaptation of clinical prediction models for application in local settings. Med Decis Mak. 2012;32(3):E1-10.
64. Janssen KJ, Moons KG, Kalkman CJ, Grobbee DE, Vergouwe Y. Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol. 2008;61(1):76-86.