A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models

Journal of Clinical Epidemiology

Volumn 110, Issue , 2019, Pages 12-22

  Christodoulou, Evangelia ^a   Ma, Jie ^b   Collins, Gary S ^b,c   Steyerberg, Ewout W ^d   Verbakel, Jan Y ^a,b   Van Calster, Ben ^a,d  

^a UNIVERSITY OF LEUVEN   (Belgium)

^b UNIVERSITY OF OXFORD   (United Kingdom)

^c OXFORD UNIVERSITY HOSPITALS NHS FOUNDATION TRUST   (United Kingdom)

^d LEIDEN UNIVERSITY MEDICAL CENTER   (Netherlands)

Author keywords

AUC; Calibration; Clinical prediction models; Logistic regression; Machine learning; Reporting

Indexed keywords

AREA UNDER THE CURVE; ARTIFICIAL NEURAL NETWORK; CALIBRATION; HUMAN; MEDLINE; OUTCOME ASSESSMENT; PREDICTION; RANDOM FOREST; RECEIVER OPERATING CHARACTERISTIC; REVIEW; SAMPLE SIZE; SUPPORT VECTOR MACHINE; SYSTEMATIC REVIEW; VALIDATION PROCESS; ALGORITHM; COMPARATIVE STUDY; META ANALYSIS; PREDICTIVE VALUE; SENSITIVITY AND SPECIFICITY; STATISTICAL MODEL; SUPERVISED MACHINE LEARNING; THEORETICAL MODEL;

ALGORITHMS; AREA UNDER CURVE; HUMANS; LOGISTIC MODELS; MODELS, THEORETICAL; OUTCOME ASSESSMENT, HEALTH CARE; PREDICTIVE VALUE OF TESTS; SENSITIVITY AND SPECIFICITY; SUPERVISED MACHINE LEARNING;

EID: 85062005742     PISSN: 08954356     EISSN: 18785921     Source Type: Journal    
DOI: 10.1016/j.jclinepi.2019.02.004     Document Type: Review

References (113)

1. Steyerberg, E.W., Clinical prediction models. 2009, Springer, New York, NY.
2. Hastie, T., Tibshirani, R., Friedman, J., The elements of statistical learning: data mining, inference, and prediction. 2nd ed., 2009, Springer, New York, NY.
3. Kononenko, I., Machine learning for medical diagnosis: history, state of the art and perspective. Artif Intell Med 23 (2001), 89–109.
4. Lisboa, P.J., Taktak, A.F.G., The use of artificial neural networks in decision support in cancer: a systematic review. Neural Netw 19 (2006), 408–415.
5. Beam, A.L., Kohane, I.S., Big data and machine learning in health care. JAMA 319 (2018), 1317–1318.
6. Chen, J.H., Asch, S.M., Machine learning and prediction in medicine — beyond the peak of inflated expectations. N Engl J Med 376 (2017), 2507–2509.
7. Goldstein, B.A., Navar, A.M., Carter, R.E., Moving beyond regression techniques in cardiovascular risk prediction: applying machine learning to address analytic challenges. Eur Heart J 38 (2017), 1805–1814.
8. Breiman, L., Statistical modeling: the two cultures (with comments and a rejoinder by the author). Stat Sci 16 (2001), 199–231.
9. Moons, K.G.M., de Groot, J.A.H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D.G., et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med, 11, 2014, e1001744.
10. Mitchell, T.M., Machine learning. 1997, McGraw Hill, New York, NY.
11. Boulesteix, A.L., Schmid, M., Machine learning versus statistical modeling. Biom J 56 (2014), 588–593.
12. Deo, R.C., Nallamothu, B.K., Learning about machine learning: the promise and pitfalls of big data and the electronic health record. Circ Cardiovasc Qual Outcomes 9 (2016), 618–620.
13. He, H., Garcia, E.A., Learning from imbalanced data. IEEE Trans Knowl Data Eng 21 (2008), 1263–1284.
14. Pochet, N.L.M.M., Suykens, J.A.K., Support vector machines versus logistic regression: improving prospective performance in clinical decision-making. Ultrasound Obstet Gynecol 27 (2006), 607–608.
15. Rajkomar, A., Oren, E., Chen, K., Dai, A.M., Hajaj, N., Liu, P.J., et al. Scalable and accurate deep learning for electronic health records. NPJ Digit Med 1 (2018), 1–10.
16. Luo, W., Phung, D., Tran, T., Gupta, S., Rana, S., Karmakar, C., et al. Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view. J Med Internet Res, 18, 2016, e323.
17. van der Ploeg, T., Austin, P.C., Steyerberg, E.W., Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints. BMC Med Res Methodol, 14, 2014, 137.
18. Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M.J., Steyerberg, E.W., A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol 74 (2016), 167–176.
19. Collins, G.S., Reitsma, J.B., Altman, D.G., Moons, K.G.M., Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. J Clin Epidemiol 68 (2015), 134–143.
20. Boulesteix, A.L., Lauer, S., Eugster, M.J.A., A plea for neutral comparison studies in computational sciences. PLoS One, 8, 2013, e61562.
21. Hand, D.J., Classifier technology and the illusion of progress. Stat Sci 1 (2006), 1–14.
22. Whiting, P.F., Rutjes, A.W.S., Westwood, M.E., Mallett, S., Deeks, J.J., Reitsma, J.B., et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 155 (2011), 529–536.
23. Probst, P., Bischl, B., Boulesteix, A.-L., Tunability: importance of hyperparameters of machine learning algorithms. 2018 ArXiv Prepr ArXiv180209596.
24. Collins, G.S., Ogundimu, E.O., Cook, J.A., Manach, Y.L., Altman, D.G., Quantifying the impact of different approaches for handling continuous predictors on the performance of a prognostic model. Stat Med 35 (2016), 4124–4135.
25. Steyerberg, E.W., Harrell, F.E. Jr., Borsboom, G.J.J.M., Eijkemans, M.J.C., Vergouwe, Y., Habbema, J.D.F., Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 54 (2001), 774–781.
26. Pepe, M.S., The statistical evaluation of medical tests for classification and prediction. 2003, Oxford University Press, New York.
27. Adavi, M., Salehi, M., Roudbari, M., Artificial neural networks versus bivariate logistic regression in prediction diagnosis of patients with hypertension and diabetes. Med J Islam Repub Iran 30 (2016), 2–6.
28. Anderson, A.E., Kerr, W.T., Thames, A., Li, T., Xiao, J., Cohen, M.S., Electronic health record phenotyping improves detection and screening of type 2 diabetes in the general United States population: a cross-sectional, unselected, retrospective study. J Biomed Inform 60 (2016), 162–168.
29. Habibi, Z., Ertiaei, A., Nikdad, M.S., Mirmohseni, A.S., Afarideh, M., Heidari, V., et al. Predicting ventriculoperitoneal shunt infection in children with hydrocephalus using artificial neural network. Childs Nerv Syst 32 (2016), 2143–2151.
30. Ichikawa, D., Saito, T., Ujita, W., Oyama, H., How can machine-learning methods assist in virtual screening for hyperuricemia? A healthcare machine-learning approach. J Biomed Inform 64 (2016), 20–24.
31. Jahani, M., Mahdavi, M., Comparison of predictive models for the early diagnosis of diabetes. Healthc Inform Res 22 (2016), 95–100.
32. Kabeshova, A., Launay, C.P., Gromov, V.A., Fantino, B., Levinoff, E.J., Allali, G., et al. Falling in the elderly: do statistical models matter for performance criteria of fall prediction? Results from two large population-based studies. Eur J Intern Med 27 (2016), 48–56.
33. Kate, R.J., Perez, R.M., Mazumdar, D., Pasupathy, K.S., Nilakantan, V., Prediction and detection models for acute kidney injury in hospitalized older adults. BMC Med Inform Decis Mak, 16, 2016, 39.
34. Kulkarni, P., Smith, L.D., Woeltje, K.F., Assessing risk of hospital readmissions for improving medical practice. Health Care Manag Sci 19 (2016), 291–299.
35. Lu, T., Hu, Y.H., Tsai, C.F., Liu, S.P., Chen, P.L., Applying machine learning techniques to the identification of late-onset hypogonadism in elderly men. Springerplus, 5, 2016, 729.
36. Mahajan, S., Burman, P., Hogarth, M., Analyzing 30-day readmission rate for heart failure using different predictive models. Stud Health Technol Inform 225 (2016), 143–147.
37. Malik, S., Khadgawat, R., Anand, S., Gupta, S., Non-invasive detection of fasting blood glucose level via electrochemical measurement of saliva. Springerplus, 5, 2016, 701.
38. Matis, G.K., Chrysou, O.I., Silva, D., Karanikas, M.A., Baltsavias, G., Lyratzopoulos, N., et al. Prediction of lumbar disc herniation patients’ satisfaction with the aid of an artificial neural network. Turk Neurosurg 26 (2016), 253–259.
39. Belliveau, T., Jette, A.M., Seetharama, S., Axt, J., Rosenblum, D., Larose, D., et al. Developing artificial neural network models to predict functioning one year after traumatic spinal cord injury. Arch Phys Med Rehabil 97 (2016), 1663–1668.e3.
40. Mortazavi, B.J., Downing, N.S., Bucholz, E.M., Dharmarajan, K., Manhapra, A., Li, S.X., et al. Analysis of machine learning techniques for heart failure readmissions. Circ Cardiovasc Qual Outcomes 9 (2016), 629–640.
41. Nakas, C.T., Schütz, N., Werners, M., Leichtle, A.B.L., Accuracy and calibration of computational approaches for inpatient mortality predictive modeling. PLoS One, 11, 2016, e0159046.
42. Ratliff, J.K., Balise, R., Veeravagu, A., Cole, T.S., Cheng, I., Olshen, R.A., et al. Predicting occurrence of spine surgery complications using big data modeling of an administrative claims database. J Bone Joint Surg Am 98 (2016), 824–834.
43. Rau, H.H., Hsu, C.Y., Lin, Y.A., Atique, S., Fuad, A., Wei, L.M., et al. Development of a web-based liver cancer prediction model for type II diabetes patients by using an artificial neural network. Comput Methods Programs Biomed 125 (2016), 58–65.
44. Ross, E.G., Shah, N.H., Dalman, R.L., Nead, K.T., Cooke, J.P., Leeper, N.J., The use of machine learning for the identification of peripheral artery disease and future mortality risk. J Vasc Surg 64 (2016), 1515–1522.e3.
45. Taylor, R.A., Pare, J.R., Venkatesh, A.K., Mowafi, H., Melnick, E.R., Fleischman, W., et al. Prediction of in-hospital mortality in emergency department patients with sepsis: a local big data-driven, machine learning approach. Acad Emerg Med 23 (2016), 269–278.
46. Thottakkara, P., Ozrazgat-Baslanti, T., Hupf, B.B., Rashidi, P., Pardalos, P., Momcilovic, P., et al. Application of machine learning techniques to high-dimensional clinical data to forecast postoperative complications. PLoS One, 11, 2016, e0155705.
47. Tong, L., Erdmann, C., Daldalian, M., Li, J., Esposito, T., Comparison of predictive modeling approaches for 30-day all-cause non-elective readmission risk. BMC Med Res Methodol, 16, 2016, 26.
48. van der Ploeg, T., Nieboer, D., Steyerberg, E.W., Modern modeling techniques had limited external validity in predicting mortality from traumatic brain injury. J Clin Epidemiol 78 (2016), 83–89.
49. Wang, H.Y., Hsieh, C.H., Wen, C.N., Wen, Y.H., Chen, C.H., Lu, J.J., Cancers screening in an asymptomatic population by using multiple tumour markers. PLoS One, 11, 2016, e0158285.
50. Berchialla, P., Scarinzi, C., Snidero, S., Gregori, D., Lawson, A.B., Lee, D., et al. Comparing models for quantitative risk assessment: an application to the European Registry of foreign body injuries in children. Stat Methods Med Res 25 (2016), 1244–1259.
51. Wang, Z., Wen, X., Lu, Y., Yao, Y., Zhao, H., Exploiting machine learning for predicting skeletal-related events in cancer patients with bone metastases. Oncotarget 7 (2016), 12612–12622.
52. Wu, H.Y., Gong, C.S.A., Lin, S.P., Chang, K.Y., Tsou, M.Y., Ting, C.K., Predicting postoperative vomiting among orthopedic patients receiving patient-controlled epidural analgesia using SVM and LR. Sci Rep 6 (2016), 1–7.
53. Yahya, N., Ebert, M.A., Bulsara, M., House, M.J., Kennedy, A., Joseph, D.J., et al. Statistical-learning strategies generate only modestly performing predictive models for urinary symptoms following external beam radiotherapy of the prostate: a comparison of conventional and machine-learning methods. Med Phys, 43, 2016, 2040.
54. Zhang, Y.-D., Wang, J., Wu, C.-J., Bao, M.-L., Li, H., Wang, X.-N., et al. An imaging-based approach predicts clinical outcomes in prostate cancer through a novel support vector machine classification. Oncotarget, 7, 2016, 78140.
55. Zhou, Z., Folkert, M., Cannon, N., Iyengar, P., Westover, K., Zhang, Y., et al. Predicting distant failure in early stage NSCLC treated with SBRT using clinical parameters Predicting distant failure in lung SBRT. Radiother Oncol 119 (2016), 501–504.
56. Acion, L., Kelmansky, D., Van der Laan, M., Sahker, E., Jones, D.S., Arndt, S., Use of a machine learning framework to predict substance use disorder treatment success. PLoS One, 12, 2017, e0175383.
57. Alghamdi, M., Al-Mallah, M., Keteyian, S., Brawner, C., Ehrman, J., Sakr, S., Predicting diabetes mellitus using SMOTE and ensemble machine learning approach: the Henry Ford ExercIse Testing (FIT) project. PLoS One, 12, 2017, e0179805.
58. Allyn, J., Allou, N., Augustin, P., Philip, I., Martinet, O., Belghiti, M., et al. A comparison of a machine learning model with EuroSCORE II in predicting mortality after elective cardiac surgery: a decision curve analysis. PLoS One, 12, 2017, e0169772.
59. Amini, P., Maroufizadeh, S., Samani, R.O., Hamidi, O., Sepidarkish, M., Prevalence and determinants of preterm birth in Tehran, Iran: a comparison between logistic regression and decision tree methods. Osong Public Health Res Perspect 8 (2017), 195–200.
60. Asaoka, R., Hirasawa, K., Iwase, A., Fujino, Y., Murata, H., Shoji, N., et al. Validating the usefulness of the “random forests” classifier to diagnose early glaucoma with optical coherence tomography. Am J Ophthalmol 174 (2017), 95–103.
61. Berikol, G.B., Yildiz, O., Özcan, T., Diagnosis of acute coronary syndrome with a support vector machine. J Med Syst, 40, 2016, 84.
62. Batterham, M., Neale, E., Martin, A., Tapsell, L., Data mining: potential applications in research on nutrition and health. Nutr Diet 74 (2017), 3–10.
63. Batterham, M., Tapsell, L., Charlton, K., O'Shea, J., Thorne, R., Using data mining to predict success in a weight loss trial. J Hum Nutr Diet 30 (2017), 471–478.
64. Cheng, F.W., Gao, X., Bao, L., Mitchell, D.C., Wood, C., Sliwinski, M.J., et al. Obesity as a risk factor for developing functional limitation among older adults: a conditional inference tree analysis. Obesity 25 (2017), 1263–1269.
65. Chiriac, A.M., Wang, Y., Schrijvers, R., Bousquet, P.J., Mura, T., Molinari, N., et al. Designing predictive models for beta-lactam allergy using the drug allergy and hypersensitivity database. J Allergy Clin Immunol Pract 6 (2018), 139–148.e2.
66. Dean, J.A., Welsh, L.C., Wong, K.H., Aleksic, A., Dunne, E., Islam, M.R., et al. Normal tissue complication probability (NTCP) modelling of severe acute mucositis using a novel oral mucosal surface organ at risk. Clin Oncol 29 (2017), 263–273.
67. Deng, X., Predicting the risk for hospital-acquired pressure ulcers in critical care patients. Crit Care Nurse 37 (2017), e1–e11.
68. Ebell, M.H., Hansen, J.G., Proposed clinical decision rules to diagnose acute rhinosinusitis among adults in primary care. Ann Fam Med 15 (2017), 347–354.
69. Fei, Y., Hu, J., Gao, K., Tu, J., qin, Li W., Wang, W., Predicting risk for portal vein thrombosis in acute pancreatitis patients: a comparison of radical basis function artificial neural network and logistic regression models. J Crit Care 39 (2017), 115–123.
70. Fei, Y., Hu, J., Li, W.Q., Wang, W., Zong, G.Q., Artificial neural networks predict the incidence of portosplenomesenteric venous thrombosis in patients with acute pancreatitis. J Thromb Haemost 15 (2017), 439–445.
71. Fei, Y., Gao, K., Hu, J., Tu, J., qin, Li W., Wang, W., et al. Predicting the incidence of portosplenomesenteric vein thrombosis in patients with acute pancreatitis using classification and regression tree algorithm. J Crit Care 39 (2017), 124–130.
72. Casanova, R., Saldana, S., Simpson, S.L., Lacy, M.E., Subauste, A.R., Blackshear, C., et al. Prediction of incident diabetes in the jackson heart study using high-dimensional machine learning. PLoS One, 11, 2016, e0163942.
73. Frizzell, J.D., Liang, L., Schulte, P.J., Yancy, C.W., Heidenreich, P.A., Hernandez, A.F., et al. Prediction of 30-day all-cause readmissions in patients hospitalized for heart failure: comparison of machine learning and other statistical approaches. JAMA Cardiol 2 (2017), 204–209.
74. Hettige, N.C., Nguyen, T.B., Yuan, C., Rajakulendran, T., Baddour, J., Bhagwat, N., et al. Classification of suicide attempters in schizophrenia using sociocultural and clinical features: a machine learning approach. Gen Hosp Psychiatry 47 (2017), 20–28.
75. Hu, Y.H., Tai, C.T., Chen, S.C.C., Lee, H.W., Sung, S.F., Predicting return visits to the emergency department for pediatric patients: applying supervised learning techniques to the Taiwan National Health Insurance Research Database. Comput Methods Programs Biomed 144 (2017), 105–112.
76. Huang, S.H., Loh, J.K., Tsai, J.T., Houg, M.F., Shi, H.Y., Predictive model for 5-year mortality after breast cancer surgery in Taiwan residents. Chin J Cancer, 36, 2017, 23.
77. Imai, S., Yamada, T., Kasashi, K., Kobayashi, M., Iseki, K., Usefulness of a decision tree model for the analysis of adverse drug reactions: evaluation of a risk prediction model of vancomycin-associated nephrotoxicity constructed using a data mining procedure. J Eval Clin Pract 23 (2017), 1240–1246.
78. Kessler, R.C., Hwang, I., Hoffmire, C.A., McCarthy, J.F., Petukhova, M.V., Rosellini, A.J., et al. Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans health Administration. Int J Methods Psychiatr Res, 26, 2017, e1575.
79. Kim, S.M., Kim, Y., Jeong, K., Jeong, H., Kim, J., Logistic LASSO regression for the diagnosis of breast cancer using clinical demographic data and the BI-RADS lexicon for ultrasonography. Ultrasonography 37 (2018), 36–42.
80. Luo, Y., Li, Z., Guo, H., Cao, H., Song, C., Guo, X., et al. Predicting congenital heart defects: a comparison of three data mining methods. PLoS One, 12, 2017, e0177811.
81. Nuutinen, M., Leskelä R.L., Suojalehto, E., Tirronen, A., Komssi, V., Development and validation of classifiers and variable subsets for predicting nursing home admission. BMC Med Inform Decis Mak, 17, 2017, e0177811.
82. Shi, K.Q., Zhou, Y.Y., Yan, H.D., Li, H., Wu, F.L., Xie, Y.Y., et al. Classification and regression tree analysis of acute-on-chronic hepatitis B liver failure: seeing the forest for the trees. J Viral Hepat 24 (2017), 132–140.
83. Churpek, M.M., Yuen, T.C., Winslow, C., Meltzer, D.O., Kattan, M.W., Edelson, D.P., Multicenter comparison of machine learning methods and conventional regression for predicting clinical deterioration on the wards. Crit Care Med 44 (2016), 368–374.
84. Shneider, B.L., Moore, J., Kerkar, N., Magee, J.C., Ye, W., Karpen, S.J., et al. Initial assessment of the infant with neonatal cholestasis-Is this biliary atresia?. PLoS One, 12, 2017, e0176275.
85. Tighe, D.F., Thomas, A.J., Sassoon, I., Kinsman, R., McGurk, M., Developing a risk stratification tool for audit of outcome after surgery for head and neck squamous cell carcinoma. Head Neck 39 (2017), 1357–1363.
86. Wallert, J., Tomasoni, M., Madison, G., Held, C., Predicting two-year survival versus non-survival after first myocardial infarction using machine learning and Swedish national register data. BMC Med Inform Decis Mak, 17, 2017, 99.
87. Weng, S.F., Reps, J., Kai, J., Garibaldi, J.M., Qureshi, N., Can machine-learning improve cardiovascular risk prediction using routine clinical data?. PLoS One, 12, 2017, e0174944.
88. Yip, T.C.F., Ma, A.J., Wong, V.W.S., Tse, Y.K., Chan, H.L.Y., Yuen, P.C., et al. Laboratory parameter-based machine learning model for excluding non-alcoholic fatty liver disease (NAFLD) in the general population. Aliment Pharmacol Ther 46 (2017), 447–456.
89. Zhang, C., Garrard, L., Keighley, J., Carlson, S., Gajewski, B., Subgroup identification of early preterm birth (ePTB): informing a future prospective enrichment clinical trial design. BMC Pregnancy Childbirth, 17, 2017, 18.
90. Zhao, Y., Healy, B.C., Rotstein, D., Guttmann, C.R.G., Bakshi, R., Weiner, H.L., et al. Exploration of machine learning techniques in predicting multiple sclerosis disease course. PLoS One, 12, 2017, e0174866.
91. Zhao, Y., Xiong, P., McCullough, L.E., Miller, E.E., Li, H., Huang, Y., et al. Comparison of breast cancer risk predictive models and screening strategies for Chinese women. J Womens Health (Larchmt) 26 (2017), 294–302.
92. Arslan, A.K., Colak, C., Sarihan, M.E., Different medical data mining approaches based prediction of ischemic stroke. Comput Methods Programs Biomed 130 (2016), 87–92.
93. Chen, W., Sun, C., Wei, R., Zhang, Y., Ye, H., Chi, R., et al. Establishing decision trees for predicting successful postpyloric nasoenteric tube placement in critically ill patients. JPEN J Parenter Enteral Nutr 42 (2018), 132–138.
94. Souza Filho JB, de O., de Seixas, J.M., Galliez, R., de Bragança Pereira, B., de Q Mello, F.C., dos Santos, A.M., et al. A screening system for smear-negative pulmonary tuberculosis using artificial neural networks. Int J Infect Dis 49 (2016), 33–39.
95. Olivera, A.R., Roesler, V., Iochpe, C., Schmidt, M.I., Vigo, A., Barreto, S.M., et al. Comparison of machine-learning algorithms to build a predictive model for detecting undiagnosed diabetes – ELSA-Brasil: accuracy study. Sao Paulo Med J 135 (2017), 234–246.
96. Dean, J.A., Wong, K.H., Welsh, L.C., Jones, A.B., Schick, U., Newbold, K.L., et al. Normal tissue complication probability (NTCP) modelling using spatial dose metrics and machine learning methods for severe acute oral mucositis resulting from head and neck radiotherapy. Radiother Oncol 120 (2016), 21–27.
97. Eigentler, T., Assi, Z., Hassel, J.C., Heinzerling, L., Starz, H., Berneburg, M., et al. Which melanoma patient carries a BRAF-mutation? A comparison of predictive models. Oncotarget, 7, 2016, 36130.
98. Austin, P.C., Steyerberg, E.W., Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med 33 (2014), 517–535.
99. Harrell, F.E. Jr., Regression modeling strategies. 2015, Springer, New York, NY.
100. Van Calster, B., Wynants, L., Verbeek, J.F.M., Verbakel, J.Y., Christodoulou, E., Vickers, A.J., et al. Reporting and interpreting decision curve analysis: a guide for investigators. Eur Urol 74 (2018), 796–804.
101. Chen, W., Sahiner, B., Samuelson, F., Pezeshk, A., Petrick, N., Calibration of medical diagnostic classifier scores to the probability of disease. Stat Methods Med Res 27 (2016), 1394–1409.
102. Drummond, C., Holte, R.C., Cost curves: an improved method for visualizing classifier performance. Mach Learn 65 (2006), 95–130.
103. van Smeden, M., Moons, K.G.M., de Groot, J.A.H., Collins, G.S., Altman, D.G., Eijkemans, M.J.C., et al. Sample size for binary logistic prediction models: beyond events per variable criteria. Stat Methods Med Res, 2018, 10.1177/0962280218784726 In press.
104. Deo, R.C., Machine learning in medicine. Circulation 132 (2015), 1920–1930.
105. Fernández-Delgado, M., Cernadas, E., Barro, S., Amorim, D., Do we need hundreds of classifiers to solve real world classification problems?. J Mach Learn Res 15 (2014), 3133–3181.
106. Couronné R., Probst, P., Boulesteix, A.L., Random forest versus logistic regression: a large-scale benchmark experiment. BMC Bioinformatics, 19, 2018, 270.
107. Ennis, M., Hinton, G., Naylor, D., Revow, M., Tibshirani, R., A comparison of statistical learning methods on the Gusto database. Stat Med 17 (1998), 2501–2508.
108. Mitchell, T., Does machine learning really work?. AI Mag, 18, 1997, 11.
109. Steyerberg, E.W., Uno, H., Ioannidis, J.P.A., Van Calster, B., Poor performance of clinical prediction models: the harm of commonly applied methods. J Clin Epidemiol 98 (2018), 133–143.
110. Pouwels, K.B., Widyakusuma, N.N., Groenwold, R.H.H., Hak, E., Quality of reporting of confounding remained suboptimal after the STROBE guideline. J Clin Epidemiol 69 (2016), 217–224.
111. Michelessi, M., Lucenteforte, E., Miele, A., Oddone, F., Crescioli, G., Fameli, V., et al. Diagnostic accuracy research in glaucoma is still incompletely reported: an application of Standards for Reporting of Diagnostic Accuracy Studies (STARD) 2015. PLoS One, 12, 2017, e0189716.
112. Kim, D.Y., Park, H.S., Cho, S., Yoon, H.S., The quality of reporting randomized controlled trials in the dermatology literature in an era where the CONSORT statement is a standard. Br J Dermatol, 2018, 10.1111/bjd.17432 In press.
113. Boulesteix, A.L., Ten simple rules for reducing overoptimistic reporting in methodological computational research. PLoS Comput Biol, 11, 2015, e1004191.