Internal validation of risk models in clustered data: A comparison of bootstrap schemes

American Journal of Epidemiology

Volumn 177, Issue 11, 2013, Pages 1209-1217

  Bouwmeester, W ^b   Moons, K G M ^a   Kappen, T H ^b   Van Klei, W A ^b   Twisk, J W R ^b   Eijkemans, M J C ^b   Vergouwe, Y ^b  

^a UNIVERSITY MEDICAL CENTER UTRECHT   (Netherlands)

^b NONE

Author keywords

bootstrapping; clustered data; internal validation; model performance; multilevel analysis; risk models

Indexed keywords

ANTIEMETIC AGENT;

ACCURACY ASSESSMENT; DATA ACQUISITION; DISEASE INCIDENCE; EMPIRICAL ANALYSIS; HEALTH RISK; NUMERICAL MODEL; OPTIMIZATION; PERFORMANCE ASSESSMENT; SAMPLING; SIMULATION;

ACCURACY; ADULT; ARTICLE; BOOTSTRAPPING; CLUSTER ANALYSIS; COMPARATIVE STUDY; CONTROLLED STUDY; CORRELATION COEFFICIENT; HUMAN; INTERNAL VALIDITY; MAJOR CLINICAL STUDY; MULTILEVEL ANALYSIS; POSTOPERATIVE NAUSEA AND VOMITING; RISK; RISK ASSESSMENT; SIMULATION;

BOOTSTRAPPING; CLUSTERED DATA; INTERNAL VALIDATION; MODEL PERFORMANCE; MULTILEVEL ANALYSIS; RISK MODELS;

COMPUTER SIMULATION; HUMANS; MODELS, STATISTICAL; POSTOPERATIVE NAUSEA AND VOMITING; REGRESSION ANALYSIS; RISK ASSESSMENT; VALIDATION STUDIES AS TOPIC;

EID: 84878536272     PISSN: 00029262     EISSN: 14766256     Source Type: Journal    
DOI: 10.1093/aje/kws396     Document Type: Article

References (33)

1. Oudega R, Moons KG, Hoes AW. Ruling out deep venous thrombosis in primary care. A simple diagnostic algorithm including D-dimer testing. Thromb Haemost. 2005;94(1): 200-205.
2. Steyerberg EW, Mushkudiani N, Perel P, et al. Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Med. 2008;5(8):e165.
3. Steyerberg EW, Harrell FE Jr, Borsboom GJ, et al. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. 2001; 54(8):774-781.
4. Efron B, Tibshirani R. An Introduction to the Bootstrap. New York: Chapman and Hall; 1993:chap 6-7.
5. Hox J. Multilevel Analysis: Techniques and Applications. Hove, UK: Routledge, Taylor & Francis; 2010:chap 11.
6. Scott AJ, Holt D. The effect of two-stage sampling on ordinary least squares methods. J Am Stat Assoc. 1982;77(380):848-854.
7. Senn S. Some controversies in planning and analysing multicentre trials. Stat Med. 1998;17(15-16):1753-1765.
8. Goldstein H. Extensions to the basic multilevel model. In: Multilevel Statistical Models. London, UK: Edward Arnold; 1995:51-71.
9. Xiao Y, Abrahamowicz M. Bootstrap-based methods for estimating standard errors in Cox's regression analyses of clustered event times. Stat Med. 2010;29(7-8):915-923.
10. Massonnet G, Burzykowski T, Janssen P. Resampling plans for frailty models. Commun Stat Simul Comput. 2006; 35(2):497-514.
11. Twisk JWR. Applied Multilevel Analysis. New York: Cambridge University Press; 2006:chap 2.
12. Moineddin R, Matheson FI, Glazier RH. A simulation study of sample size for multilevel logistic regression models. BMC Med Res Methodol. 2007;7:34.
13. Adams G, Gulliford MC, Ukoumunne OC, et al. Patterns of intra-cluster correlation from primary care research to inform study design and analysis. J Clin Epidemiol. 2004;57(8): 785-794.
14. Skrondal A, Rabe-Hesketh S. Prediction in multilevel generalized linear models. J R Stat Soc Ser A Stat Soc. 2009;172(3):659-687.
15. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15(4):361-387.
16. Miller ME, Hui SL, Tierney WM. Validation techniques for logistic regression models. Stat Med. 1991;10(8):1213-1226.
17. Van Houwelingen HC, Thorogood J. Construction, validation and updating of a prognostic model for kidney graft survival. Stat Med. 1995;14(18):1999-2008.
18. van Oirbeek R, Lesaffre E. An application of Harrell's C-index to PH frailty models. Stat Med. 2010;29(30):3160-3171.
19. van Houwelingen JC, le Cessie S. Predictive value of statistical models. Stat Med. 1990;9(11):1303-1325.
20. Copas JB. Regression, prediction and shrinkage. J R Stat Soc Series B. 1983;45(3):311-354.
21. Raudenbush SW, Yang M-L, Yosef M. Maximum likelihood for generalized linear models with nested random effects via high-order, multivariate Laplace approximation. J Comput Graph Stat. 2000;9(1):141-157.
22. Joe H. Accuracy of Laplace approximation for discrete response mixed models. Comput Stat Data Anal. 2008; 52(12):5066-5074.
23. The R Development Core Team. R: A Language and Environment for Statistical Computing: Reference Index. Version 2.11.1. Vienna, Austria: R Foundation for Statistical Computing; 2010.
24. Bates D, Maechler M, Dai B. The lme4 package: linear mixedeffects models using S4 classes. Version 0.999375-28. 2009. (http://lme4.r-forge.r-project.org/ ).
25. Kappen TH, Vergouwe Y, van KleiWA, et al. Implementing a prediction rule for postoperative nausea and vomiting: a cluster-randomized trial [abstract]. Presented at the American Society of Anesthesiologists 2008 Meeting, Orlando, FL. Abstracts Anesthesiology, October 2008; 109: A1266.
26. Van den Bosch JE, Moons KG, Bonsel GJ, et al. Does measurement of preoperative anxiety have added value for predicting postoperative nausea and vomiting? Anesth Analg. 2005;100(5):1525-1532.
27. Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. New York: Springer; 2009:chap 5.
28. Peduzzi P, Concato J, Kemper E, et al. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol. 1996;49(12):1373-1379.
29. Courvoisier DS, Combescure C, Agoritsas T, et al. Performance of logistic regression modeling: beyond the number of events per variable, the role of data structure. J Clin Epidemiol. 2011;64(9):993-1000.
30. Harrell FE. Regression Modeling Strategies With Applications to Linear Models, Logistic Regression and Survival Analysis. New York, NY: Springer Verlag; 2001:chap 4.
31. Steyerberg EW, Eijkemans MJ, Habbema JD. Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. J Clin Epidemiol. 1999;52(10): 935-942.
32. Bates D. Mixed models in R using the lme4 package, part 1: linear mixed models with simple, scalar random effects. Presented at the 8th International Amsterdam Conference on Multilevel Analysis, Amsterdam, March 16, 2011.
33. Bates D. Fitting linear mixed models in R, using the lme4 package. R News. 2005;5(1):27-30.