Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker

Heart

Volumn 98, Issue 9, 2012, Pages 683-690

  Moons, Karel G M ^a   Kengne, Andre Pascal ^a,b,c   Woodward, Mark ^b,d   Royston, Patrick ^e   Vergouwe, Yvonne ^a   Altman, Douglas G ^f   Grobbee, Diederick E ^a  

^a UNIVERSITY MEDICAL CENTER UTRECHT   (Netherlands)

^b UNIVERSITY OF SYDNEY   (Australia)

^c UNIVERSITY OF CAPE TOWN   (South Africa)

^d JOHNS HOPKINS UNIVERSITY   (United States)

^e MRC CLINICAL TRIALS UNIT   (United Kingdom)

^f UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ALBUMIN; BIOLOGICAL MARKER; CREATININE; EPIDERMAL GROWTH FACTOR RECEPTOR; TRIACYLGLYCEROL;

CARDIOVASCULAR DISEASE; CARDIOVASCULAR RISK; CLINICAL DECISION MAKING; COHORT ANALYSIS; CREATININE BLOOD LEVEL; DATA ANALYSIS; HIGH RISK POPULATION; HUMAN; INTERNAL VALIDITY; MICROALBUMINURIA; MODEL; OUTCOME ASSESSMENT; PHYSICIAN ATTITUDE; PRIORITY JOURNAL; PROGNOSIS; QUALITY CONTROL; REVIEW; RISK ASSESSMENT; RISK PREDICTION MODEL; STATISTICAL ANALYSIS; TRIACYLGLYCEROL BLOOD LEVEL; VALIDATION PROCESS;

BIOLOGICAL MARKERS; CARDIOVASCULAR DISEASES; COST-BENEFIT ANALYSIS; DECISION MAKING; DECISION SUPPORT TECHNIQUES; HUMANS; MODELS, THEORETICAL; PREDICTIVE VALUE OF TESTS; REPRODUCIBILITY OF RESULTS; RISK;

EID: 84860113852     PISSN: 13556037     EISSN: 1468201X     Source Type: Journal    
DOI: 10.1136/heartjnl-2011-301246     Document Type: Review

References (88)

1. Laupacis A, Sekar N, Stiell IG. Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA 1997;277:488-94.
2. Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. Rotterdam: Springer, 2009.
3. Toll DB, Janssen KJ, Vergouwe Y, et al. Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol 2008;61:1085-94.
4. Moons KG, Grobbee DE. Clinical epidemiology: an introduction. In: Vaccaro AR, ed. Orthop Knowledge Update 8. Rosemont: American Academy of Orthopaedic Surgeons, 2005:109-18.
5. Grobbee DE, Hoes AW. Clinical Epidemiology - Principles, Methods and Applications for Clinical Research. London: Jones and Bartlett Publishers, 2009.
6. Wilson PW, D'Agostino RB, Levy D, et al. Prediction of coronary heart disease using risk factor categories. Circulation 1998;97:1837-47. (Pubitemid 28213628)
7. Conroy RM, Pyörälä K, Fitzgerald AP, et al. Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project. Eur Heart J 2003;24:987-1003. (Pubitemid 36712275)
8. Woodward M, Brindle P, Tunstall-Pedoe H. Adding social deprivation and family history to cardiovascular risk assessment: the ASSIGN score from the Scottish Heart Health Extended Cohort (SHHEC). Heart 2007;93:172-6. (Pubitemid 46180289)
9. Nashef SA, Roques F, Michel P, et al. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg 1999;16:9-13. (Pubitemid 29336425)
10. Assmann G, Cullen P, Schulte H. Simple scoring scheme for calculating the risk of acute coronary events based on the 10-year follow-up of the Prospective Cardiovascular Munster (PROCAM) study. Circulation 2002;105:310-15. (Pubitemid 34106178)
11. Wells PS, Anderson DR, Bormanis J, et al. Value of assessment of pretest probability of deep-vein thrombosis in clinical management. Lancet 1997;350:1795-8. (Pubitemid 28036513)
12. Wells PS, Ginsberg JS, Anderson DR, et al. A simple clinical model to categorize pretest probability in patients with suspected pulmonary embolism. Blood 1997;90:424.
13. Moons KG, Royston P, Vergouwe Y, et al. Prognosis and prognostic research: what, why, and how? BMJ 2009;338:1317-20.
14. Christensen E. Prognostic models including the Child-Pugh, MELD and Mayo risk scores - where are we and where should we go? J Hepatol 2004;41:344-50. (Pubitemid 39177643)
15. Feinstein AR. 'Clinical judgment' revisited: the distraction of quantitative models. Ann Intern Med 1994;120:799-805. (Pubitemid 24125302)
16. Concato J, Feinstein AR, Holford TR. The risk of determining risk with multivariable models. Ann Intern Med 1993;118:201-10. (Pubitemid 23034807)
17. Braitman LE, Davidoff F. Predicting clinical states in individual patients. Ann Intern Med 1996;125:406-12. (Pubitemid 126449710)
18. Lloyd-Jones DM. Cardiovascular risk prediction: basic concepts, current status, and future directions. Circulation 2010;121:1768-77.
19. Cui J. Overview of risk prediction models in cardiovascular disease research. Ann Epidemiol 2009;19:711-17.
20. Tzoulaki I, Liberopoulos G, Ioannidis JP. Assessment of claims of improved prediction beyond the Framingham risk score. JAMA 2009;302:2345-52.
21. de Kleijn DP, Moll FL, Hellings WE, et al. Local atherosclerotic plaques are a source of prognostic biomarkers for adverse cardiovascular events. Arterioscler Thromb Vasc Biol 2010;30:612-19.
22. Nozaki T, Sugiyama S, Koga H, et al. Significance of a multiple biomarkers strategy including endothelial dysfunction to improve risk stratification for cardiovascular events in patients at high risk for coronary heart disease. J Am Coll Cardiol 2009;54:601-8.
23. Paynter NP, Chasman DI, Buring JE, et al. Cardiovascular disease risk prediction with and without knowledge of genetic variation at chromosome 9p21.3. Ann Intern Med 2009;150:65-72.
24. Hlatky MA, Greenland P, Arnett DK, et al. Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association. Circulation 2009;119:2408-16.
25. Moons KG. Criteria for scientific evaluation of novel markers: a perspective. Clin Chem 2010;56:537-41.
26. Ransohoff DF. How to improve reliability and efficiency of research about molecular markers: roles of phases, guidelines, and study design. J Clin Epidemiol 2007;60:1205-19. (Pubitemid 350061148)
27. Ioannidis JP. Prediction of cardiovascular disease outcomes and established cardiovascular risk factors by genome-wide association markers. Circ Cardiovasc Genet 2009;2:7-15.
28. Ioannidis JP. Why most published research findings are false. PLoS Med 2005;2:e124.
29. Moons KG, Biesheuvel CJ, Grobbee DE. Test research versus diagnostic research. Clin Chem 2004;50:473-6. (Pubitemid 38297294)
30. Counsell C, Dennis M. Systematic review of prognostic models in patients with acute stroke. Cerebrovasc Dis 2001;12:159-70. (Pubitemid 32973945)
31. Perel P, Edwards P, Wentz R, et al. Systematic review of prognostic models in traumatic brain injury. BMC Med Inform Decis Mak 2006;6:38.
32. Altman DG. Prognostic models: a methodological framework and review of models for breast cancer. In: Lyman GH, Burstein HJ, eds. Breast Cancer Translational Therapeutic Strategies. New York: Informa Healthcare, 2007:11-25.
33. Leushuis E, van der Steeg JW, Steures P, et al. Prediction models in reproductive medicine: a critical appraisal. Hum Reprod Update 2009;15:537-52.
34. Mallett S, Royston P, Dutton S, et al. Reporting methods in studies developing prognostic models in cancer: a review. BMC Med 2010;8:20.
35. Mallett S, Royston P, Waters R, et al. Reporting performance of prognostic models in cancer: a review. BMC Med 2010;8:21.
36. Janssens AC, Ioannidis JP, Bedrosian S, et al. Strengthening the reporting of genetic risk prediction studies (GRIPS): explanation and elaboration. Eur J Epidemiol 2011;26:313-37.
37. Janssens AC, Ioannidis JP, van Duijn CM, et al. Strengthening the reporting of genetic risk prediction studies: the GRIPS statement. Eur J Epidemiol 2011;26:255-9.
38. Royston P, Moons KG, Altman DG, et al. Prognosis and prognostic research: developing a prognostic model. BMJ 2009;338:b604.
39. Altman DG, Vergouwe Y, Royston P, et al. Prognosis and prognostic research: validating a prognostic model. BMJ 2009;338:b605.
40. Moons KG, Altman DG, Vergouwe Y, et al. Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ 2009;338:1487-90.
41. Moons KGM, Kengne AP, Grobbee DE, et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart 2012;98:691-698.
42. Kengne AP, Patel A, Marre M, et al. Contemporary model for cardiovascular risk prediction in people with type 2 diabetes. Eur J Cardiovasc Prev Rehabil 2011;18:393-8.
43. Biesheuvel CJ, Vergouwe Y, Oudega R, et al. Advantages of the nested case-control design in diagnostic research. BMC Med Res Methodol 2008;8:48.
44. Peduzzi P, Concato J, Feinstein AR, et al. Importance of events per independent variable in proportional hazards regression analysis. II. Accuracy and precision of regression estimates. J Clin Epidemiol 1995;48:1503-10. (Pubitemid 26035533)
45. Peduzzi P, Concato J, Kemper E, et al. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996;49:1373-9. (Pubitemid 27028925)
46. Harrell FE. Regression Modeling Strategies. New York: Springer-Verlag, 2001.
47. Simon R, Altman DG. Statistical aspects of prognostic factor studies in oncology. Br J Cancer 1994;69:979-85. (Pubitemid 24161844)
48. Moons KG, Grobbee DE. When should we remain blind and when should our eyes remain open in diagnostic studies? J Clin Epidemiol 2002;55:633-6. (Pubitemid 34831054)
49. Janssen KJ, Donders AR, Harrell FE Jr, et al. Missing covariate data in medical research: to impute is better than to ignore. J Clin Epidemiol 2010;63:721-7.
50. Janssen KJ, Vergouwe Y, Donders AR, et al. Dealing with missing predictor values when applying clinical prediction models. Clin Chem 2009;55:994-1001.
51. Rubin DB. Multiple Imputation for Nonresponse in Surveys. New York: Wiley, 1987.
52. Little RA, Rubin DB. Statistical Analysis with Missing Data. New York: Wiley, 1987.
53. Donders AR, van der Heijden GJ, Stijnen T, et al. Review: a gentle introduction to imputation of missing values. J Clin Epidemiol 2006;59:1087-91. (Pubitemid 44374314)
54. Schafer JL. Analysis of Incomplete Multivariate Data. London: Chapman & Hill, 1997.
55. Moons KG, Donders RA, Stijnen T, et al. Using the outcome for imputation of missing predictor values was preferred. J Clin Epidemiol 2006;59:1092-101. (Pubitemid 44389818)
56. de Groot JA, Janssen KJ, Zwinderman AH, et al. Multiple imputation to correct for partial verification bias revisited. Stat Med 2008;27:5880-9.
57. de Groot JA, Janssen KJ, Zwinderman AH, et al. Multiple imputation to correct for partial verification bias revisited. Stat Med 2009;28:899.
58. Altman DG. Categorising continuous covariates. Br J Cancer 1991;64:975.
59. Altman DG, Royston P. The cost of dichotomising continuous variables. BMJ 2006;332:1080. (Pubitemid 44365491)
60. Royston P, Altman DG, Sauerbrei W. Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med 2006;25:127-41. (Pubitemid 43136953)
61. Royston P, Sauerbrei W, Altman DG. Modeling the effects of continuous risk factors. J Clin Epidemiol 2000;53:219-21. (Pubitemid 30151228)
62. Steyerberg EW. Clinical Prediction Models - A Practical Approach to Development, Validation, and Updating. Springer, New York 2009.
63. Mantel N. Why stepdown procedures in variable selection? Technometrics 1970;12:621-5.
64. Derksen S, Kesselman HJ. Backward, forward and stepwise automated subset selection algorithms: frequency of obtaining authentic and noise variables. Br J Math Stat Psychol 1992;45:265-82.
65. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996;15:361-87. (Pubitemid 26072233)
66. Steyerberg EW, Eijkemans MJ, Harrell FE Jr, et al. Prognostic modeling with logistic regression analysis: in search of a sensible strategy in small data sets. Med Decis Making 2001;21:45-56. (Pubitemid 32245030)
67. Sun GW, Shook TL, Kay GL. Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. J Clin Epidemiol 1996;49:907-16. (Pubitemid 26283404)
68. Hosmer DW, Lemeshow S. Applied Logistic Regression. New York: Willey, 2000.
69. D'Agostino RB, Nam BH. Evaluation of the performance of survival analysis models: discrimination and calibration measures. In: Balakrishnan N, Rao CR, eds. Handbook Statistics. Amsterdam: Elsevier, 2004:1-25.
70. May S, Hosmer DW. A simplified method of calculating an overall goodness-of-fit test for the Cox proportional hazards model. Lifetime Data Anal 1998;4:109-20. (Pubitemid 128480552)
71. Hosmer DW, Lemeshow S. Applied Survival Analysis: Regression Modelling of Time to Event Data. New York: Wiley, 1999.
72. Harrell FE Jr, Lee KL, Califf RM, et al. Regression modelling strategies for improved prognostic prediction. Stat Med 1984;3:143-52. (Pubitemid 14052973)
73. Chambless LE, Diao G. Estimation of time-dependent area under the ROC curve for long-term risk prediction. Stat Med 2006;25:3474-86. (Pubitemid 44575164)
74. Gönen M, Heller S. Concordance probability and discriminatory power in proportional hazards regression. Biometrika 2005;92:965-70. (Pubitemid 41681247)
75. Moons KG, Donders AR, Steyerberg EW, et al. Penalized maximum likelihood estimation to directly adjust diagnostic and prognostic prediction models for overoptimism: a clinical example. J Clin Epidemiol 2004;57:1262-70. (Pubitemid 40023146)
76. Efron B, Tibshirani R. An Introduction to the Bootstrap. Monographs on Statistics and Applied Probability. New York: Chapman & Hall, 1993.
77. Sullivan LM, Massaro JM, D'Agostino RB, et al. Presentation of multivariate data for clinical use: the Framingham Study risk score functions. Stat Med 2004;23:1631-60. (Pubitemid 38660414)
78. Patel A, MacMahon S, Chalmers J, et al. Effects of a fixed combination of perindopril and indapamide on macrovascular and microvascular outcomes in patients with type 2 diabetes mellitus (the ADVANCE trial): a randomised controlled trial. Lancet 2007;370:829-40. (Pubitemid 47362321)
79. Collett D. Modelling Survival Data in Medical Research. New York: Chapman & Hall/ CRC, 2003.
80. Moons KG, van Es GA, Michel BC, et al. Redundancy of single diagnostic test evaluation. Epidemiology 1999;10:276-81. (Pubitemid 29187763)
81. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 2007;115:928-35. (Pubitemid 46294733)
82. Cook NR. Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clin Chem 2008;54:17-23. (Pubitemid 351182592)
83. Pepe MS, Feng Z, Huang Y, et al. Integrating the predictiveness of a marker with its performance as a classifier. Am J Epidemiol 2008;167:362-8. (Pubitemid 351197592)
84. Pencina MJ, D'Agostino RB, D'Agostino RB Jr, et al. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008;27:157-72. discussion 207-12.
85. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making 2006;26:565-74. (Pubitemid 44728772)
86. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010;21:128-38.
87. Pencina MJ, D'Agostino RB, Steyerberg EW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med 2011;30:11-21.
88. Levey AS, Coresh J, Greene T, et al. Using standardized serum creatinine values in the modification of diet in renal disease study equation for estimating glomerular filtration rate. Ann Intern Med 2006;145:247-54. (Pubitemid 46768608)