Using simulation studies to evaluate statistical methods

Statistics in Medicine

Volumn 38, Issue 11, 2019, Pages 2074-2102

  Morris, Tim P ^a   White, Ian R ^a   Crowther, Michael J ^b  

^a MRC CLINICAL TRIALS UNIT   (United Kingdom)

^b UNIVERSITY OF LEICESTER   (United Kingdom)

Author keywords

graphics for simulation; Monte Carlo; simulation design; simulation reporting; simulation studies

Indexed keywords

ARTICLE; NOMENCLATURE; SAMPLING; SIMULATION; STATISTICAL ANALYSIS; STATISTICS; BIOSTATISTICS; COMPUTER SIMULATION; METHODOLOGY; MONTE CARLO METHOD; PRACTICE GUIDELINE; PROCEDURES; STATISTICAL BIAS; STATISTICAL MODEL;

BIAS; BIOSTATISTICS; COMPUTER SIMULATION; GUIDELINES AS TOPIC; MODELS, STATISTICAL; MONTE CARLO METHOD; RESEARCH DESIGN;

EID: 85060135966     PISSN: 02776715     EISSN: 10970258     Source Type: Journal    
DOI: 10.1002/sim.8086     Document Type: Article

References (76)

1. Feiveson AH. Power by simulation. Stata J. 2002;2(2):107-124.
2. Rubin DB. Bayesianly justifiable and relevant frequency calculations for the applies statistician. Ann Stat. 1984;12(4):1151-1172.
3. Grieve AP. Idle thoughts of a ‘well-calibrated’ Bayesian in clinical drug development. Pharm Stat. 2016;15(2):96-108.
4. Hoaglin DC, Andrews DF. The reporting of computation-based results in statistics. Am Stat. 1975;29(3):122-126.
5. Hauck WW, Anderson S. A survey regarding the reporting of simulation studies. Am Stat. 1984;38(3):214-216.
6. Ripley BD. Stochastic Simulation. Hoboken, NJ: John Wiley & Sons; 1987.
7. Burton A, Altman DG, Royston P, Holder RL. The design of simulation studies in medical statistics. Statist Med. 2006;25(24):4279-4292.
8. Koehler E, Brown E, Haneuse SJPA. On the assessment of Monte Carlo error in simulation-based statistical analyses. Am Stat. 2009;63(2):155-162.
9. Morgan BJT. Elements of Simulation. Boca Raton, FL: Chapman & Hall/CRC; 1995.
10. Chang M. Monte Carlo Simulation for the Pharmaceutical Industry: Concepts, Algorithms, and Case Studies. Boca Raton, FL: CRC Press; 2011.
11. Díaz-Emparanza I. Is a small Monte Carlo analysis a good analysis? Stat Pap. 2002;43(4):567-577.
12. White IR. simsum: analyses of simulation studies including Monte Carlo error. Stata J. 2010;10(3):369-385.
13. Smith MK, Marshall A. Importance of protocols for simulation studies in clinical drug development. Stat Methods Med Res. 2011;20(6):613-622.
14. Crowther MJ, Lambert PC. Simulating biologically plausible complex survival data. Statist Med. 2013;32(23):4118-4134.
15. Holford NHG, Hale M, Ko HC, et al. Simulation in drug development: good practices. Draft Publication of the Center for Drug Development Science (CDDS). 1999.
16. O'Kelly M, Anisimov V, Campbell C, Hamilton S. Proposed best practice for projects that involve modelling and simulation. Pharm Stat. 2017;16(2):107-113.
17. Haramoto H, Matsumoto M, Nishimura T, Panneton F, L'Ecuyer P. Efficient jump ahead for F2-linear random number generators. INFORMS J Comput. 2008;20(3):385-390.
18. Kenward MG, Roger JH. Small sample inference for fixed effects from restricted maximum likelihood. Biometrics. 1997;53(3):983-997.
19. White IR. Letter to the editor: survival analysis of randomized clinical trials adjusted for patients who switch treatments by M. G. Law and J. M. Kaldor, Statistics in Medicine, 15, 2069–2076 (1996). Statist Med. 1997;16(22):2619-2620.
20. White IR, Thompson SG. Adjusting for partially missing baseline measurements in randomized trials. Statist Med. 2005;24(7):993-1007.
21. Hughes RA, Sterne JAC, Tilling K. Comparison of imputation variance estimators. Stat Methods Med Res. 2016;25(6):2541-2557.
22. Morris TP, White IR, Royston P. Tuning multiple imputation by predictive mean matching and local residual draws. BMC Med Res Methodol. 2014;14(1):75.
23. Skrondal A. Design and analysis of Monte Carlo experiments: attacking the conventional wisdom. Multivar Behav Res. 2000;35(2):137-167.
24. Graves N, Barnett AG, Burn E, Cook D. Smaller clinical trials for dec,on making; using p-values could be costly. London, UK: F1000Research; 2018;7.
25. Senn S, Chambless LE, Roeback JR. Methods for assessing difference between groups in change when initial measurement is subject to intra-individual variation. Statist Med. 1994;13(21):2280-2285.
26. Kuss O. Statistical methods for meta-analyses including information from studies without any events—add nothing to nothing and succeed nevertheless. Statist Med. 2015;34(7):1097-1116.
27. Chaurasia A, Harel O. Partial F-tests with multiply imputed data in the linear regression framework via coefficient of determination. Statist Med. 2015;34(3):432-443.
28. Wu C, Shi X, Cui Y, Ma S. A penalized robust semiparametric approach for gene–environment interactions. Statist Med. 2015;34(30):4016-4030.
29. Ferrante L, Skrami E, Gesuita R, Cameriere R. Bayesian calibration for forensic age estimation. Statist Med. 2015;34(10):1779-1790.
30. Zhang Z, Wang C, Troendle JF. Optimizing the order of hypotheses in serial testing of multiple endpoints in clinical trials. Statist Med. 2015;34(9):1467-1482.
31. Kahan BC. Bias in randomised factorial trials. Statist Med. 2013;32(26):4540-4549.
32. Campbell H, Dean CB. The consequences of proportional hazards based model selection. Statist Med. 2014;33(6):1042-1056.
33. Robins JM, Wang N. Inference for imputation estimators. Biometrika. 2000;87(1):113-124.
34. Reiter JP. Multiple imputation when records used for imputation are not used or disseminated for analysis. Biometrika. 2008;95(4):933-946.
35. Hauck WW, Anderson S, Marcus SM. Should we adjust for covariates in nonlinear regression analyses of randomized trials? Control Clin Trials. 1998;19(3):249-256.
36. Zhang Z. Estimating a marginal causal odds ratio subject to confounding. Commun Stat Theory Method. 2008;38(3):309-321.
37. van Smeden M, de Groot JAH, Moons KGM, et al. No rationale for 1 variable per 10 events criterion for binary logistic regression analysis. BMC Med Res Methodol. 2016;16(1).
38. Crowther MJ, Look MP, Riley RD. Multilevel mixed effects parametric survival models using adaptive Gauss–Hermite quadrature with application to recurrent events and individual participant data meta-analysis. Statist Med. 2014;33(22):3844-3858.
39. Royston P, Barthel FMS, Parmar MKB, Choodari-Oskooei B, Isham V. Designs for clinical trials with time-to-event outcomes based on stopping guidelines for lack of benefit. Trials. 2011;12(1):81.
40. Bartlett JW. Combining bootstrapping with multiple imputation. 2016. http://thestatsgeek.com/2016/03/12/combining-bootstrapping-with-multiple-imputation/
41. Rubin DB. Inference and missing data. Biometrika. 1976;63(3):581-592.
42. White IR, Carlin JB. Bias and efficiency of multiple imputation compared with complete-case analysis for missing covariate values. Statist Med. 2010;29(28):2920-2931.
43. White IR, Royston P. Imputing missing covariate values for the Cox model. Statist Med. 2009;28(15):1982-1998.
44. Neyman J. On the two different aspects of the representative method: the method of stratified sampling and the method of purposive selection. J R Stat Soc. 1934;97(4):558-625.
45. Meng XL. Multiple-imputation inferences with uncongenial sources of input. Stat Sci. 1994;9(4):538-558.
46. Rubin DB. Multiple imputation after 18+ years. J Am Stat Assoc. 1996;91(434):473-489.
47. Morris TP. Rank minimization with a two-step analysis should not replace randomization in clinical trials. J Clin Epidemiol. 2012;65(7):810-811.
48. Kimani PK, Todd S, Stallard N. Estimation after subpopulation selection in adaptive seamless trials. Statist Med. 2015;34(18):2581-2601.
49. Carreras M, Gutjahr G, Brannath W. Adaptive seamless designs with interim treatment selection: a case study in oncology. Statist Med. 2015;34(8):1317-1333.
50. Efron B, Hastie T. Computer Age Statistical Inference. Cambridge, UK: Cambridge University Press; 2016.
51. Gutman R, Rubin DB. Robust estimation of causal effects of binary treatments in unconfounded studies with dichotomous outcomes. Statist Med. 2013;32(11):1795-1814.
52. Taguri M, Chiba Y. A principal stratification approach for evaluating natural direct and indirect effects in the presence of treatment-induced intermediate confounding. Statist Med. 2015;34(1):131-144.
53. Li P, Redden DT. Small sample performance of bias-corrected sandwich estimators for cluster-randomized trials with binary outcomes. Statist Med. 2015;34(2):281-296.
54. Marozzi M. Multivariate multidistance tests for high-dimensional low sample size case-control studies. Statist Med. 2015;34(9):1511-1526.
55. Keogh RH, Morris TP. Multiple imputation in Cox regression when there are time-varying effects of covariates. Statist Med. 2018;37(25):3661-3678.
56. Morris TP, Fisher DJ, Kenward MG, Carpenter JR. Meta-analysis of Gaussian individual patient data: two stage or not two stage? Statist Med. 2018;37(9):1419-1438.
57. Cramér H. Mathematical Methods of Statistics. Princeton, NJ: Princeton University Press; 1946.
58. Rao CR. Information and accuracy attainable in the estimation of statistical parameters. Bull Calcutta Math Soc. 1945;37:81-91.
59. Williamson EJ, Forbes A, White IR. Variance reduction in randomised trials by inverse probability weighting using the propensity score. Statist Med. 2014;33(5):721-737.
60. He X, Whitmore GA, Loo GY, Hochberg MC, Lee MLT. A model for time to fracture with a shock stream superimposed on progressive degradation: the study of osteoporotic fractures. Statist Med. 2015;34(4):652-663.
61. Chen Y, Hong C, Riley RD. An alternative pseudolikelihood method for multivariate random-effects meta-analysis. Statist Med. 2015;34(3):361-380.
62. Hsu CH, Taylor JMG, Hu C. Analysis of accelerated failure time data with dependent censoring using auxiliary variables via nonparametric multiple imputation. Statist Med. 2015;34(19):2768-2780.
63. Alonso A, Milanzi E, Molenberghs G, Buyck C, Bijnens L. A new modeling approach for quantifying expert opinion in the drug discovery process. Statist Med. 2015;34(9):1590-1604.
64. Seaman SR, Bartlett JW, White IR. Multiple imputation of missing covariates with non-linear effects and interactions: an evaluation of statistical methods. BMC Med Res Methodol. 2012;12(1):46.
65. Lambert PC, Dickman PW, Rutherford MJ. Comparison of different approaches to estimating age standardized net survival. BMC Med Res Methodol. 2015;15(1):64.
66. Rücker G, Schwarzer G. Presenting simulation results in a nested loop plot. BMC Med Res Methodol. 2014;14(1):129.
67. Gasparini A, White IR. rsimsum: analysis of simulation studies including Monte Carlo error. R package version 0.3.1. 2018.
68. Wicklin R. A zipper plot for visualizing coverage probability in simulation studies. Cary, NC: SAS Institute Inc; 2018.
69. Austin PC, Stuart EA. Optimal full matching for survival outcomes: a method that merits more widespread use. Statist Med. 2015;34(30):3949-3967.
70. Thompson JA, Fielding KL, Davey C, Aiken AM, Hargreaves JR, Hayes RJ. Bias and inference from misspecified mixed-effect models in stepped wedge trial analysis. Statist Med. 2017;36(23):3670-3682.
71. Sayers A, Crowther MJ, Judge A, Whitehouse MR, Blom AW. Determining the sample size required to establish whether a medical device is non-inferior to an external benchmark. BMJ Open. 2017;7(8):e015397.
72. Pham TM, Carpenter JR, Morris TP, Wood AM, Petersen I. Population-calibrated multiple imputation for a binary/categorical covariate in categorical regression models. Statist Med. 2018.
73. Boos DD, Osborne JA. Assessing variability of complex descriptive statistics in Monte Carlo studies using resampling methods. Int Stat Rev. 2015;83(2):228-238.
74. Audigier V, White IR, Jolani S, et al. Multiple imputation for multilevel data with continuous and binary variables. Stat Sci. 2018;33(2):160-183.
75. Langan D, Higgins JPT, Simmonds M. Comparative performance of heterogeneity variance estimators in meta-analysis: a review of simulation studies. Res Synth Methods. 2017;8(2):181-198.
76. Boulesteix AL, Wilson R, Hapfelmeier A. Towards evidence-based computational statistics: lessons from clinical research on the role and design of real-data benchmark studies. BMC Med Res Methodol. 2017;17(1).