Using Cluster Bootstrapping to Analyze Nested Data With a Few Clusters

Educational and Psychological Measurement

Volumn 78, Issue 2, 2018, Pages 297-318

  Huang, Francis L ^a  

^a UNIVERSITY OF MISSOURI   (United States)

Author keywords

cluster bootstrapping; cluster randomized trials; clustered data; low number of clusters

Indexed keywords

EID: 85041022610     PISSN: 00131644     EISSN: 15523888     Source Type: Journal    
DOI: 10.1177/0013164416678980     Document Type: Article

References (70)

1. Angrist J. D., Pischke J.-S., (2014). Mastering ‘metrics: The path from cause to effect. Princeton, NJ: Princeton University Press.
2. Arceneaux K., Nickerson D. W., (2009). Modeling certainty with clustered data: A comparison of methods. Political Analysis, 17, 177-190. doi:10.1093/pan/mpp004
3. Bates D., Mächler M., Bolker B., Walker S., (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67, 1-48. doi:10.18637/jss.v067.i01
4. Bell B. A., Morgan G. B., Schoeneberger J. A., Kromrey J. D., Ferron J. M., (2014). How low can you go?Methodology, 10, 1-11. doi:10.1027/1614-2241/a000062
5. Bell R., McCaffrey D., (2002). Bias reduction in standard errors for linear regression with multi-stage samples. Survey Methodology, 28, 169-182.
6. Cameron A. C., Gelbach J. B., Miller D. L., (2008). Bootstrap-based improvements for inference with clustered errors. Review of Economics and Statistics, 90, 414-427. doi:10.1162/rest.90.3.414
7. Cameron A. C., Miller D. L., (2015). A practitioner’s guide to cluster-robust inference. Journal of Human Resources, 50, 317-372. doi:10.3368/jhr.50.2.317
8. Chernick M. R., LaBudde R. A., (2011). An introduction to bootstrap methods with applications to R. Hoboken, NJ: Wiley.
9. Clarke P., (2008). When can group level clustering be ignored? Multilevel models versus single-level models with sparse data. Journal of Epidemiology and Community Health, 62, 752-758. doi:10.1136/jech.2007.060798
10. Cohen J., Cohen P., West S. G., Aiken L. S., (2003). Applied multiple regression/correlation analysis for the behavioral sciences. Mahwah, NJ: Lawrence Erlbaum.
11. Coyne M. D., McCoach D. B., Loftus S., Zipoli R., Jr. Ruby M., Crevecoeur Y. C., Kapp S., (2010). Direct and extended vocabulary instruction in kindergarten: Investigating transfer effects. Journal of Research on Educational Effectiveness, 3, 93-120.
12. Curtis J. R., Nielsen E. L., Treece P. D., Downey L., Dotolo D., Shannon S. E... Engelberg R. A., (2011). Effect of a quality-improvement intervention on end-of-life care in the intensive care unit. American Journal of Respiratory and Critical Care Medicine, 183, 348-355. doi:10.1164/rccm.201006-1004OC
13. Davison A. C., Hinkley D. V., (1997). Bootstrap methods and their application (Vol. 1). New York, NY: Cambridge University Press.
14. Dedrick R. F., Ferron J. M., Hess M. R., Hogarty K. Y., Kromrey J. D., Lang T. R... Lee R. S., (2009). Multilevel modeling: A review of methodological issues and applications. Review of Educational Research, 79, 69-102.
15. Dong N., Maynard R., (2013). PowerUp!: A tool for calculating minimum detectable effect sizes and minimum required sample sizes for experimental and quasi-experimental design studies. Journal of Research on Educational Effectiveness, 6, 24-67. doi:10.1080/19345747.2012.673143
16. Dong Y., Peng C.-Y. J., (2013). Principled missing data methods for researchers. SpringerPlus, 2, 222. doi:10.1186/2193-1801-2-222
17. Donner A., Klar N., (2004). Pitfalls of and controversies in cluster randomization trials. American Journal of Public Health, 94, 416-422.
18. Efron B., (1979). Bootstrap methods: Another look at the jackknife. Annals of Statistics, 7, 1-26. doi:10.1214/aos/1176344552
19. Efron B., Tibshirani R. J., (1994). An introduction to the bootstrap. Boca Raton, FL: CRC press.
20. Enders C. K., (2010). Applied missing data analysis. New York, NY: Guilford Press.
21. Enders C. K., Tofighi D., (2007). Centering predictor variables in cross-sectional multilevel models: A new look at an old issue. Psychological Methods, 12, 121-138. doi:10.1037/1082-989X.12.2.121
22. Field C. A., Welsh A. H., (2007). Bootstrapping clustered data. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69, 369-390.
23. Flynn T. N., Peters T. J., (2004). Use of the bootstrap in analysing cost data from cluster randomised trials: Some simulation results. BMC Health Services Research, 4, 33. doi:10.1186/1472-6963-4-33
24. Fox J., (2016). Applied regression analysis and generalized linear models (3rd ed.). Thousand Oaks, CA: Sage.
25. Gehlbach H., Brinkworth M. E., King A. M., Hsu L. M., McIntyre J., Rogers T., (2016). Creating birds of similar feathers: Leveraging similarity to improve teacher–student relationships and academic achievement. Journal of Educational Psychology, 108, 342-352. doi:10.1037/edu0000042
26. Goldstein H., (2011). Multilevel statistical models (Vol. 922). New York, NY: Wiley.
27. Hahs-Vaughn D. L., (2005). A primer for using and understanding weights with national datasets. Journal of Experimental Education, 73, 221-248. doi:10.3200/JEXE.73.3.221-248
28. Halekoh U., Højsgaard S., (2014). A Kenward-Roger approximation and parametric bootstrap methods for tests in linear mixed models: The R package pbkrtest. Journal of Statistical Software, 59, 1-32. doi:10.18637/jss.v059.i09
29. Hanks A. S., Just D. R., Brumberg A., (2016). Marketing vegetables in elementary school cafeterias to increase uptake. Pediatrics, 138. doi:10.1542/peds.2015-1720
30. Harden J. J., (2011). A bootstrap method for conducting statistical inference with clustered data. State Politics & Policy Quarterly, 11, 223-246. doi:10.1177/1532440011406233
31. Harrell F., (2016). rms: Regression modeling strategies. Retrieved from https://CRAN.R-project.org/package=rms
32. Heck R. H., Thomas S. L., (2008). An introduction to multilevel modeling techniques (2nd ed.). New York, NY: Routledge.
33. Hedges L. V., Hedberg E. C., (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29, 60-87. doi:10.3102/0162373707299706
34. Hox J. J., Kreft I. G., (1994). Multilevel analysis methods. Sociological Methods & Research, 22, 283-299.
35. Huang F., (2016). Alternatives to multilevel modeling for the analysis of clustered data. Journal of Experimental Education, 84, 175-196. doi:10.1080/00220973.2014.952397
36. Ifcher J., Zarghamee H., (2014). Affect and overconfidence: A laboratory investigation. Journal of Neuroscience, Psychology, and Economics, 7, 125-150. doi:10.1037/npe0000022
37. Ivers N. M., Taljaard M., Dixon S., Bennett C., McRae A., Taleban J... Donner A., (2011). Impact of CONSORT extension for cluster randomised trials on quality of reporting and study methodology: Review of random sample of 300 trials, 2000-8. BMJ, 343, d5886-d5886. doi:10.1136/bmj.d5886
38. Kenward M. G., Roger J. H., (1997). Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53, 983-997.
39. Kish L., (1965). Survey sampling. New York, NY: Wiley.
40. Kizilcec R. F., Bailenson J. N., Gomez C. J., (2015). The instructor’s face in video instruction: Evidence from two large-scale field studies. Journal of Educational Psychology, 107, 724-739. doi:10.1037/edu0000013
41. Kreft I., (1996). Are multilevel techniques necessary? An overview, including simulation studies. Unpublished manuscript.
42. Kreft I. G. G., Yoon B., (1994). Are multilevel techniques necessary? An attempt at demystification. Retrieved from http://eric.ed.gov/?id=ED371033
43. Lai M. H. C., Kwok O., (2015). Examining the rule of thumb of not using multilevel modeling: The “design effect smaller than two” rule. Journal of Experimental Education, 83, 423-438. doi:10.1080/00220973.2014.907229
44. Maas C., Hox J., (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1, 86-92.
45. McCoach D. B., Adelson J. L., (2010). Dealing with dependence (part I): Understanding the effects of clustered data. Gifted Child Quarterly, 54, 152-155. doi:10.1177/0016986210363076
46. McNeish D. M., Stapleton L. M., (2016). The effect of small sample size on two-level model estimates: A review and illustration. Educational Psychology Review, 28, 295-314.
47. Meijer E., Busing F., Van der Leeden R., (1998). Estimating bootstrap confidence intervals for two-level models. In Hox J. J., De Leeuw E. D., (Eds.), Assumptions, robustness, and estimation methods in multivariate modeling (pp. 35-48). Amsterdam, Netherlands: TT-Publikaties.
48. Moerbeek M., van Schie S., (2016). How large are the consequences of covariate imbalance in cluster randomized trials: A simulation study with a continuous outcome and a binary covariate at the cluster level. BMC Medical Research Methodology, 16, 79. doi:10.1186/s12874-016-0182-7
49. Moulton B. R., (1986). Random group effects and the precision of regression estimates. Journal of Econometrics, 32, 385-397. doi:10.1016/0304-4076(86)90021-7
50. Moulton B. R., (1990). An illustration of a pitfall in estimating the effects of aggregate variables on micro units. Review of Economics and Statistics, 72, 334-338. doi:10.2307/2109724
51. Mundfrom D. J., Schultz M. R., (2001). A comparison between hierarchical linear modeling and multiple linear regression in selected data sets. Multiple Linear Regression Viewpoints, 27, 3-11.
52. Murnane R. J., Willett J. B., (2011). Methods matter: Improving causal inference in educational and social science research. New York, NY: Oxford University Press.
53. Musca S. C., Kamiejski R., Nugier A., Méot A., Er-Rafiy A., Brauer M., (2011). Data with hierarchical structure: Impact of intraclass correlation and sample size on type-I error. Frontiers in Psychology, 2, 74. doi:10.3389/fpsyg.2011.00074
54. Muthén L. K., Muthén B. O., (2002). How to use a Monte Carlo study to decide on sample size and determine power. Structural Equation Modeling: A Multidisciplinary Journal, 9, 599-620. doi:10.1207/S15328007SEM0904_8
55. Poi B. P., (2004). From the help desk: Some bootstrapping techniques. Stata Journal, 4, 312-328.
56. Preacher K. J., Hayes A. F., (2004). SPSS and SAS procedures for estimating indirect effects in simple mediation models. Behavior Research Methods, Instruments, & Computers, 36, 717-731.
57. R Core Team. (2016). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org/
58. Raudenbush S., Bryk A., (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
59. Roberts J., Fan X., (2004). Bootstrapping within the multilevel/hierarchical linear modeling framework: A primer for use with SAS and SPLUS. Multiple Linear Regression Viewpoints, 30, 23-33.
60. Rosseel Y., (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48, 1-36.
61. Scherbaum C. A., Ferreter J. M., (2009). Estimating statistical power and required sample sizes for organizational research using multilevel modeling. Organizational Research Methods, 12, 347-367.
62. Snijders T., Bosker R., (2012). Multilevel analysis (2nd ed.). Thousand Oaks, CA: Sage.
63. Spybrook J., Bloom H., Congdon R., Hill C., Martinez A., Raudenbush S., (2011). Optimal design plus empirical evidence: Documentation for the “Optimal Design” software. New York, NY: William T. Grant Foundation.
64. StataCorp. (2015). Stata statistical software: Release 14. College Station, TX: Stata Press.
65. Taljaard M., Teerenstra S., Ivers N. M., Fergusson D. A., (2016). Substantial risks associated with few clusters in cluster randomized and stepped wedge designs. Clinical Trials, 13, 459-463. doi:10.1177/1740774516634316
66. Thai H.-T., Mentré F., Holford N. H., Veyrat-Follet C., Comets E., (2013). A comparison of bootstrap approaches for estimating uncertainty of parameters in linear mixed-effects models. Pharmaceutical Statistics, 12, 129-140.
67. Tonidandel S., Williams E. B., LeBreton J. M., (2014). Size matters.. just not in the way that you think. In Lance C., Vandenberg R., (Eds.), More statistical and methodological myths and urban legends (pp. 162-183). New York, NY: Routledge.
68. Vallejo Seco G., Ato García M., Fernández García M. P., Livacic Rojas P. E., (2013). Multilevel bootstrap analysis with assumptions violated. Psicothema, 25, 520-528. doi:10.7334/psicothema2013.58
69. van den Bos E., van Duijvenvoorde A. C. K., Westenberg P. M., (2016). Effects of adolescent sociocognitive development on the cortisol response to social evaluation. Developmental Psychology, 52, 1151-1163. doi:10.1037/dev0000133
70. Van der Leeden R., Meijer E., Busing F. M., (2008). Resampling multilevel models. In Handbook of multilevel analysis (pp. 401-433). New York, NY: Springer. Retrieved from http://link.springer.com/chapter/10.1007/978-0-387-73186-5_11