A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta-analysis

Statistics in Medicine

Volumn 32, Issue 18, 2013, Pages 3158-3180

  Debray, Thomas P A ^a   Moons, Karel G M ^a   Ahmed, Ikhlaaq ^b   Koffijberg, Hendrik ^a   Riley, Richard David ^b  

^a UNIVERSITY MEDICAL CENTER UTRECHT   (Netherlands)

^b UNIVERSITY OF BIRMINGHAM   (United Kingdom)

Author keywords

Individual participant data (IPD); Internal external validation; Logistic regression; Meta analysis; Multivariable; Prediction research; Risk prediction models

Indexed keywords

ARTICLE; CONTROLLED STUDY; HUMAN; INDIVIDUAL PARTICIPANT DATA; LOGISTIC REGRESSION ANALYSIS; MATHEMATICAL COMPUTING; MATHEMATICAL MODEL; MEDICAL DOCUMENTATION; META ANALYSIS (TOPIC); MULTIVARIATE ANALYSIS; POISSON DISTRIBUTION; PREDICTION; PROBABILITY; PROCESS DEVELOPMENT; RELIABILITY; RISK ASSESSMENT; SAMPLE SIZE; VALIDATION PROCESS;

INDIVIDUAL PARTICIPANT DATA (IPD); INTERNAL-EXTERNAL VALIDATION; LOGISTIC REGRESSION; META-ANALYSIS; MULTIVARIABLE; PREDICTION RESEARCH; RISK PREDICTION MODELS;

CLINICAL TRIALS AS TOPIC; DATA INTERPRETATION, STATISTICAL; FORECASTING; HUMANS; META-ANALYSIS AS TOPIC; MODELS, STATISTICAL; VENOUS THROMBOSIS;

EID: 84880044696     PISSN: 02776715     EISSN: 10970258     Source Type: Journal    
DOI: 10.1002/sim.5732     Document Type: Article

References (63)

1. Reilly BM, Evans AT. Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Annals of Internal Medicine 2006; 144(3):201-209.
2. Moons KGM, Kengne AP, Woodward M, Royston P, Vergouwe Y, Altman DG, Grobbee DE. Risk prediction models: I.Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart 2012; 98(9):683-690. DOI: 10.1136/heartjnl-2011-301246.
3. Adams ST, Leveson SH. Clinical prediction rules. British Medical Journal 2012; 344:d8312. DOI: 10.1136/bmj.d8312.
4. Wilson PW, D'Agostino RB, Levy D, Belanger AM, Silbershatz H, Kannel WB. Prediction of coronary heart disease using risk factor categories. Circulation 1998; 97(18):1837-1847. DOI: 10.1161/01.CIR.97.18.1837.
5. Conroy RM, Pyörälä K, Fitzgerald AP, Sans S, Menotti A, DeBacker G, DeBacquer D, Ducimetière P, Jousilahti P, Keil U, Njølstad I, Oganov RG, Thomsen T, Tunstall-Pedoe H, Tverdal A, Wedel H, Whincup P, Wilhelmsen L, Graham IM. Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project. European Heart Journal 2003; 24(11):987-1003. DOI: 10.1016/S0195-668X(03)00114-3.
6. Oudega R, Moons KGM, Hoes AW. Ruling out deep venous thrombosis in primary care. A simple diagnostic algorithm including D-dimer testing. Thrombosis and Haemostasis 2005; 94(1):200-205. DOI: 10.1160/TH04-12-0829.
7. Schuit E, Kwee A, Westerhuis MEMH, VanDessel HJHM, Graziosi GCM, VanLith JMM, Nijhuis JG, Oei SG, Oosterbaan HP, Schuitemaker NW, Wouters MGAJ, Visser GHA, Mol BWJ, Moons KGM, Groenwold RHH. A clinical prediction model to assess the risk of operative delivery. BJOG : An International Journal of Obstetrics and Gynaecology 2012; 119(8):915-923. DOI: 10.1111/j.1471-0528.2012.03334.x.
8. Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. Springer: New York, 2009.
9. Royston P, Moons KGM, Altman DG, Vergouwe Y. Prognosis and prognostic research: developing a prognostic model. British Medical Journal 2009; 338:b604. DOI: 10.1136/bmj.b604.
10. Riley RD, Simmonds MC, Look MP. Evidence synthesis combining individual patient data and aggregate data: a systematic review identified current practice and possible methods. Journal of Clinical Epidemiology 2007; 60(5):431-439. DOI: 10.1016/j.jclinepi.2006.09.009.
11. Riley RD, Lambert PC, Abo-Zaid G. Meta-analysis of individual participant data: rationale, conduct, and reporting. British Medical Journal 2010; 340:c221. DOI: 10.1136/bmj.c221.
12. Royston P, Parmar MKB, Sylvester R. Construction and validation of a prognostic model across several studies, with an application in superficial bladder cancer. Statistics in Medicine 2004; 23(6):907-926. DOI: 10.1002/sim.1691.
13. Steyerberg EW, Mushkudiani N, Perel P, Butcher I, Lu J, McHugh GS, Murray GD, Marmarou A, Roberts I, Habbema JDF, Maas AIR. Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Medicine 2008; 5(8):e165. DOI: 10.1371/journal.pmed.0050165.
14. Phillips RS, Sutton AJ, Riley RD, Chisholm JC, Picton SV, Stewart LA, the PICNICC Collaboration. Predicting infectious complications in neutropenic children and young people with cancer (IPD protocol). Systematic Reviews 2012; 1(1):8. DOI: 10.1186/2046-4053-1-8.
15. Ioannidis JP, Cappelleri JC, Schmid CH, Lau J. Impact of epidemic and individual heterogeneity on the population distribution of disease progression rates. An example from patient populations in trials of human immunodeficiency virus infection. American Journal of Epidemiology 1996; 144(11):1074-1085.
16. Ioannidis JP, Lau J. Heterogeneity of the baseline risk within patient populations of clinical trials: a proposed evaluation algorithm. American Journal of Epidemiology 1998; 148(11):1117-1126.
17. Moons KGM, Kengne AP, Grobbee DE, Royston P, Vergouwe Y, Altman DG, Woodward M. Risk prediction models: II. External validation, model updating, and impact assessment. Heart 2012; 98(9):691-698. DOI: 10.1136/heartjnl-2011-301247.
18. Walter SD. Variation in baseline risk as an explanation of heterogeneity in meta-analysis. Statistics in Medicine 1997; 16(24):2883-2900. DOI: 10.1002/(SICI)1097-0258(19971230)16:24(2883::AID-SIM825)3.0.CO;2-B.
19. Hailpern SM, Visintainer PF. Odds ratios and logistic regression: further examples of their use and interpretation. TheStata Journal 2003; 3(3):213-225.
20. Janssen KJM, Vergouwe Y, Kalkman CJ, Grobbee DE, Moons KGM. A simple method to adjust clinical prediction models to local circumstances. Canadian Journal of Anaesthesia 2009; 56(3):194-201. DOI: 10.1007/s12630-009-9041-x.
21. Greenland S. Principles of multilevel modelling. International Journal of Epidemiology 2000; 29(1):158-167. DOI: 10.1093/ije/29.1.158.
22. Riley RD, Higgins JPT, Deeks JJ. Interpretation of random effects meta-analyses. British Medical Journal 2011; 342:d549. DOI: 10.1136/bmj.d549.
23. AlKhalaf MM, Thalib L, Doi SAR. Combining heterogenous studies using the random-effects model is a mistake and leads to inconclusive meta-analyses. Journal of Clinical Epidemiology 2011; 64(2):119-123. DOI: 10.1016/j.jclinepi.2010.01.009.
24. Debray TPA, Koffijberg H, Vergouwe Y, Moons KG, Steyerberg EW. Aggregating published prediction models with individual participant data: a comparison of different approaches. Statistics in Medicine 2012; 31(23):2697-2712. DOI: 10.1002/sim.5412.
25. van Houwelingen HC. Validation, calibration, revision and combination of prognostic survival models. Statistics in Medicine 2000; 19(24):3401-3415. DOI: 10.1002/1097-0258(20001230)19:24(3401::AID-SIM554)3.0.CO;2-2.
26. Steyerberg EW, Borsboom GJJM, van Houwelingen HC, Eijkemans MJC, Habbema JDF. Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Statistics in Medicine 2004; 23(16):2567-2586. DOI: 10.1002/sim.1844.
27. Harrell FE Jr. Regression Modeling Strategies with Applications to Linear Models, Logistic Regression and Survival Analysis, Springer Series in Statistics. Springer: New York, 2001.
28. Abo-Zaid GMA, Guo B, Deeks JJ, Debray TPA, Steyerberg EW, Moons KGM, Riley RD. Individual participant data meta-analyses should not ignore clustering. Journal of Clinical Epidemiology 2012; Accepted.
29. Böhning D, Sarol J Jr. Estimating risk difference in multicenter studies under baseline-risk heterogeneity. Biometrics 2000; 56(1):304-308. DOI: 10.1111/j.0006-341X.2000.00304.x.
30. Trikalinos TA, Ioannidis JP. Predictive modeling and heterogeneity of baseline risk in meta-analysis of individual patient data. Journal of Clinical Epidemiology 2001; 54(3):245-252. DOI: 10.1016/j.cct.2007.08.002.
31. Steenbergen MR, Jones BS. Modeling multilevel data structures. American Journal of Political Science 2002; 46(1):218-237.
32. Lee KJ, Thompson SG. Flexible parametric models for random-effects distributions. Statistics in Medicine 2008; 27(3):418-434. DOI: 10.1002/sim.2897.
33. Toll DB, Janssen KJM, Vergouwe Y, Moons KGM. Validation, updating and impact of clinical prediction rules: a review. Journal of Clinical Epidemiology 2008; 61(11):1085-1094. DOI: 10.1016/j.jclinepi.2008.04.008.
34. Altman DG, Royston P. What do we mean by validating a prognostic model? Statistics in Medicine 2000; 19(4):453-473. DOI: 10.1002/(SICI)1097-0258(20000229)19:4(453::AID-SIM350)3.0.CO;2-5.
35. Altman DG, Vergouwe Y, Royston P, Moons KGM. Prognosis and prognostic research: validating a prognostic model. British Medical Journal 2009; 338:b605. DOI: 10.1136/bmj.b605.
36. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010; 21(1):128-138. DOI: 10.1097/EDE.0b013e3181c30fb2.
37. Cook NR. Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clinical Chemistry 2008; 54(1):17-23. DOI: 10.1373/clinchem.2007.096529.
38. Cox DR. Two further applications of a model for binary regression. Biometrika 1958; 45(3):562-565.
39. Miller ME, Langefeld CD, Tierney WM, Hui SL, McDonald CJ. Validation of probabilistic predictions. Medical Decision Making 1993; 13(1):49-58.
40. Spiegelhalter DJ. Probabilistic prediction in patient management and clinical trials. Statistics in Medicine 1986; 5(5):421-433.
41. Harrell FJ, Lee K, Califf R, Pryor D, Rosati R. Regression modelling strategies for improved prognostic prediction. Statistics in Medicine 1984; 3:143-152. DOI: 10.1002/sim.4780030207.
42. Royston P, Altman DG. Visualizing and assessing discrimination in the logistic regression model. Statistics in Medicine 2010; 29(24):2508-2520. DOI: 10.1002/sim.3994.
43. DerSimonian R, Laird N. Meta-analysis in clinical trials. Controlled Clinical Trials 1986; 7(3):177-188. DOI: 10.1016/0197-2456(86)90046-2.
44. Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. British Medical Journal 2003; 327(7414):557-560. DOI: 10.1136/bmj.327.7414.557.
45. Rücker G, Schwarzer G, Carpenter JR, Schumacher M. Undue reliance on I(2) in assessing heterogeneity may mislead. BMC Medical Research Methodology 2008; 8:79. DOI: 10.1186/1471-2288-8-79.
46. Steyerberg EW, Bleeker SE, Moll HA, Grobbee DE, Moons KGM. Internal and external validation of predictive models: a simulation study of bias and precision in small samples. Journal of Clinical Epidemiology 2003; 56(5):441-447. DOI: 10.1016/S0895-4356(03)00047-7.
47. Vergouwe Y, Steyerberg EW, Eijkemans MJC, Habbema JDF. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. Journal of Clinical Epidemiology 2005; 58(5):475-483. DOI: 10.1016/j.jclinepi.2004.06.017.
48. Gerds TA, Cai T, Schumacher M. The performance of risk prediction models. Biometrical Journal 2008; 50(4):457-479. DOI: 10.1002/bimj.200810443.
49. Vergouwe Y, Steyerberg EW, Eijkemans MJC, Habbema JDF. Validity of prognostic models: when is a model clinically useful? Seminars in Urologic Oncology 2002; 20(2):96-107.
50. Vickers AJ, Cronin AM. Traditional statistical methods for evaluating prediction models are uninformative as to clinical value: towards a decision analytic framework. Seminars in Oncology 2010; 37(1):31-38. DOI: 10.1053/j.seminoncol.2009.12.004.
51. Breslow NE, Day NE. Statistical methods in cancer research. Volume II - the design and analysis of cohort studies. IARC Scientific Publications 1987; (82):1-406.
52. Callas PW, Pastides H, Hosmer DW. Empirical comparisons of proportional hazards, poisson, and logistic regression modeling of occupational cohort data. American Journal Of Industrial Medicine 1998; 33(1):33-47. DOI: 10.1002/(SICI)1097-0274(199801)33:1(33::AID-AJIM5)3.0.CO;2-X.
53. Ma R, Krewski D, Burnett RT. Random effects Cox models: a Poisson modelling approach. Biometrika 2003; 90(1):157-169.
54. Crowther MJ, Riley RD, Staessen JA, Wang J, Gueyffier F, Lambert PC. Individual patient data meta-analysis of survival data using poisson regression models. BMC Medical Research Methodology 2012; 12:34. DOI: 10.1186/1471-2288-12-34.
55. Royston P. Estimating a smooth baseline hazard function for the Cox model. Technical Report, University College London, 2011.
56. Bender R, Augustin T, Blettner M. Generating survival times to simulate Cox proportional hazards models. Statistics in Medicine 2005; 24(11):1713-1723. DOI: 10.1002/sim.2059.
57. Glidden DV, Vittinghoff E. Modelling clustered survival data from multicentre clinical trials. Statistics in Medicine 2004; 23(3):369-388. DOI: 10.1002/sim.1599.
58. Royston P, Parmar MKB. Flexible parametric proportional-hazards and proportional-odds models for censored survival data, with application to prognostic modelling and estimation of treatment effects. Statistics in Medicine 2002; 21(15):2175-2197. DOI: 10.1002/sim.1203.
59. Jackson D, Riley R, White IR. Multivariate meta-analysis: potential and promise. Statistics in Medicine 2011; 30(20):2481-2498. DOI: 10.1002/sim.4172.
60. Copas JB. Using regression models for prediction: shrinkage and regression to the mean. Statistical Methods in Medical Research 1997; 6(2):167-183. DOI: 10.1177/096228029700600206.
61. Steyerberg EW, Eijkemans MJC, Habbema JDF. Application of shrinkage techniques in logistic regression analysis: a case study. Statistica Neerlandica 2001; 55(1):76-88. DOI: 10.1111/1467-9574.00157.
62. Maddala GS, Li H, Srivastava VK. A comparative study of different shrinkage estimators for panel data models. Annals of Economics and Finance 2001; 2(1):1-30.
63. Baltagi BH, Bresson G, Griffin JM, Pirotte A. Homogeneous, heterogeneous or shrinkage estimators? Some empirical evidence from French regional gasoline consumption. Empirical Economics 2003; 28:795-811. DOI: 10.1007/s00181-003-0161-9.