Multiple imputation of missing data in multilevel designs: A comparison of different strategies

Psychological Methods

Volumn 22, Issue 1, 2017, Pages 141-165

  Lüdtke, Oliver ^a   Robitzsch, Alexander ^a   Grund, Simon ^a  

^a LEIBNIZ INSTITUTE FOR SCIENCE AND MATHEMATICS EDUCATION   (Germany)

Author keywords

Intraclass correlation; Missing data; Multilevel data; Multilevel modeling; Multiple imputation

Indexed keywords

COMPUTER SIMULATION; HUMAN; MULTILEVEL ANALYSIS; PSYCHOLOGY; PSYCINFO; STATISTICAL MODEL; COMPARATIVE STUDY; METHODOLOGY; STATISTICAL ANALYSIS; STATISTICAL BIAS;

BIAS; COMPUTER SIMULATION; DATA INTERPRETATION, STATISTICAL; HUMANS; LINEAR MODELS; MULTILEVEL ANALYSIS; RESEARCH DESIGN;

EID: 84986005436     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/met0000096     Document Type: Article

References (82)

1. Allison, P. D. (2009). Fixed effects regression models. Thousand Oaks, CA: Sage. http://dx.doi.org/10.4135/9781412993869.
2. Andridge, R. R. (2011). Quantifying the impact of fixed effects modeling of clusters in multiple imputation for cluster randomized trials. Biometrical Journal, 53, 57-74. http://dx.doi.org/10.1002/bimj.201000140.
3. Asparouhov, T., & Muthén, B. O. (2010). Multiple imputation with Mplus. Retrieved from http://www.statmodel.com/download/Imputations7.pdf.
4. Black, A. C., Harel, O., & McCoach, D. B. (2011). Missing data techniques for multilevel data: Implications of model misspecification. Journal of Applied Statistics, 38, 1845-1865. http://dx.doi.org/10.1080/02664763 .2010.529882.
5. Bliese, P. D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations (pp. 349-381). San Francisco, CA: Jossey-Bass.
6. Bodner, T. E. (2008). What improves with increased missing data imputations? Structural Equation Modeling, 15, 651-675. http://dx.doi.org/ 10.1080/10705510802339072.
7. Bos, W., Lankes, E.-M., Prenzel, M., Schwippert, K., Walther, G., & Valtin, R. (Eds.). (2003). Erste Ergebnisse aus IGLU. Schülerleistungen am Ende der vierten Jahrgangsstufe im internationalen Vergleich [The first results from IGLU-An international comparison of student achievement at the end of fourth grade]. Münster, Germany: Waxmann.
8. Carpenter, J. R., Goldstein, H., & Kenward, M. G. (2011). REALCOMEIMPUTE software for multilevel multiple imputation with mixed response types. Journal of Statistical Software, 45, 1-14. http://dx.doi.org/ 10.18637/jss.v045.i05.
9. Carpenter, J. R., & Kenward, M. G. (2013). Multiple imputation and its application. Chichester, UK: Wiley. http://dx.doi.org/10.1002/ 9781119942283.
10. Cohen, J. (1988). Statistical power for the behavioral sciences. Hillsdale, NJ: Erlbaum.
11. Collins, L. M., Schafer, J. L., & Kam, C.-M. (2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6, 330-351. http://dx.doi.org/10.1037/1082-989X.6.4.330.
12. Cronbach, L. J. (1976). Research on classrooms and schools: Formulations of questions, design and analysis. Stanford, CA: Stanford Evaluation Consortium.
13. Croon, M. A., & van Veldhoven, M. J. P. M. (2007). Predicting group-level outcome variables from variables measured at the individual level: A latent variable multilevel model. Psychological Methods, 12, 45-57. http://dx.doi.org/10.1037/1082-989X.12.1.45.
14. Demirtas, H., Freels, S. A., & Yucel, R. M. (2008). Plausibility of multivariate normality assumption when multiply imputing non-Gaussian continuous outcomes: A simulation assessment. Journal of Statistical Computation and Simulation, 78, 69-84. http://dx.doi.org/10.1080/ 10629360600903866.
15. Dettmers, S., Trautwein, U., Lüdtke, O., Kunter, M., & Baumert, J. (2010). Homework works if homework quality is high: Using multilevel modeling to predict the development of achievement in mathematics. Journal of Educational Psychology, 102, 467-482. http://dx.doi.org/10.1037/ a0018453.
16. Díaz-Ordaz, K., Kenward, M. G., Cohen, A., Coleman, C. L., & Eldridge, S. (2014). Are missing data adequately handled in cluster randomised trials? A systematic review and guidelines. Clinical Trials, 11, 590-600. http://dx.doi.org/10.1177/1740774514537136.
17. Drechsler, J. (2015). Multiple imputation of multilevel missing data-Rigor versus simplicity. Journal of Educational and Behavioral Statistics, 40, 69-95. http://dx.doi.org/10.3102/1076998614563393.
18. Enders, C. K. (2010). Applied missing data analysis. New York, NY: Guilford Press.
19. Enders, C. K., Baraldi, A. N., & Cham, H. (2014). Estimating interaction effects with incomplete predictor variables. Psychological Methods, 19, 39-55. http://dx.doi.org/10.1037/a0035314.
20. Enders, C. K., Mistler, S. A., & Keller, B. T. (2016). Multilevel multiple imputation: A review and evaluation of joint modeling and chained equations imputation. Psychological Methods, 21, 222-240. http://dx .doi.org/10.1037/met0000063.
21. Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003). Bayesian data analysis. London, UK: CRC Press.
22. Gibson, N. M., & Olejnik, S. (2003). Treatment of missing data at the second level of hierarchical linear models. Educational and Psychological Measurement, 63, 204-238. http://dx.doi.org/10.1177/ 0013164402250987.
23. Goldstein, H. (2010). Multilevel statistical models. London, UK: Edward Arnold. http://dx.doi.org/10.1002/9780470973394.
24. Goldstein, H., Carpenter, J. R., & Browne, W. J. (2014). Fitting multilevel multivariate models with missing data in responses and covariates that may include interactions and non-linear terms. Journal of the Royal Statistical Society: Series A (Statistics in Society), 177, 553-564. http:// dx.doi.org/10.1111/rssa.12022.
25. Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60, 549-576. http://dx.doi.org/10 .1146/annurev.psych.58.110405.085530.
26. Graham, J. W. (2012). Missing data: Analysis and design. New York, NY: Springer. http://dx.doi.org/10.1007/978-1-4614-4018-5.
27. Graham, J. W., Taylor, B. J., Olchowski, A. E., & Cumsille, P. E. (2006). Planned missing data designs in psychological research. Psychological Methods, 11, 323-343. http://dx.doi.org/10.1037/1082-989X.11.4.323.
28. Grund, S., Lüdtke, O., & Robitzsch, A. (2016). Multiple imputation of missing covariate values in multilevel models with random slopes: A cautionary note. Behavior Research Methods, 48, 640-649. http://dx .doi.org/10.3758/s13428-015-0590-3.
29. Hedges, L. V., & Hedberg, E. C. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29, 60-87. http://dx.doi.org/10.3102/ 0162373707299706.
30. Henry, K. L., Stanley, L. R., Edwards, R. W., Harkabus, L. C., & Chapin, L. A. (2009). Individual and contextual effects of school adjustment on adolescent alcohol use. Prevention Science, 10, 236-247. http://dx.doi .org/10.1007/s11121-009-0124-2.
31. Hoff, P. D. (2009). A first course in Bayesian statistical methods. New York, NY: Springer. http://dx.doi.org/10.1007/978-0-387-92407-6.
32. Hox, J. J. (2010). Multilevel analysis: Techniques and applications. Mahwah, NJ: Erlbaum.
33. Hox, J., van Buuren, S., & Jolani, S. (2016). Incomplete multilevel data. In J. Harring, L. M. Stapleton, & S. N. Beretvas (Eds.), Multilevel modeling for educational research: Addressing practical issues found in realworld applications (pp. 39-61). New York, NY: Information Age.
34. Jelicić, H., Phelps, E., & Lerner, R. M. (2009). Use of missing data methods in longitudinal studies: The persistence of bad practices in developmental psychology. Developmental Psychology, 45, 1195-1199. http://dx.doi.org/10.1037/a0015665.
35. Kenny, D. A., Kashy, D. A., & Cook, W. L. (2006). The analysis of dyadic data. New York, NY: Guilford Press.
36. Kenny, D. A., Mannetti, L., Pierro, A., Livi, S., & Kashy, D. A. (2002). The statistical analysis of data from small groups. Journal of Personality and Social Psychology, 83, 126-137. http://dx.doi.org/10.1037/0022-3514.83.1.126.
37. Kounin, J. S. (1970). Discipline and group management in classrooms. New York, NY: Holt, Rinehart & Winston.
38. Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing data. New York, NY: Wiley. http://dx.doi.org/10.1002/9781119013563.
39. Lüdtke, O., Marsh, H. W., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthén, B. (2008). The multilevel latent covariate model: A new, more reliable approach to group-level effects in contextual studies. Psychological Methods, 13, 203-229. http://dx.doi.org/10.1037/ a0012869.
40. Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 1, 86-92. http://dx.doi.org/10.1027/ 1614-2241.1.3.86.
41. Magnus, J. R., & Neudecker, H. (1988). Matrix differential calculus with applications in statistics and econometrics. Hoboken, NJ: Wiley.
42. Mathieu, J. E., Aguinis, H., Culpepper, S. A., & Chen, G. (2012). Understanding and estimating the power to detect cross-level interaction effects in multilevel modeling. Journal of Applied Psychology, 97, 951-966. http://dx.doi.org/10.1037/a0028380.
43. Mehta, P. D., & Neale, M. C. (2005). People are variables too: Multilevel structural equations modeling. Psychological Methods, 10, 259-284. http://dx.doi.org/10.1037/1082-989X.10.3.259.
44. Mistler, S. A. (2013). A SAS macro for applying multiple imputation to multilevel data. In Proceedings of the SAS Global Forum 2013. Contributed paper (Statistics and Data Analysis) 438-2013. San Francisco, CA. Retrieved from https://support.sas.com/resources/papers/ proceedings13/438-2013.pdf.
45. Mistler, S. A. (2015). Multilevel multiple imputation: An examination of competing methods (Unpublished doctoral dissertation). Arizona State University, Tempe, AZ.
46. Mullis, I. V. S., Martin, M. O., Gonzales, E. J., & Kennedy, A. M. (2003). PIRLS 2001. international report: IEA's study of reading literacy achievement in primary schools in 35. countries. Boston, MA: International Study Center, Lynch School of Education, Boston College.
47. Muthén, B. O., & Asparouhov, T. (2011) Beyond multilevel regression modeling: Multilevel analysis in a general latent variable framework. In J. J. Hox & J. K. Roberts, The handbook of advanced multilevel analysis (pp. 15-40). Milton Park, UK: Routledge.
48. Muthén, L. K., & Muthén, B. O. (1998-2010). Mplus user's guide (6th ed.). Los Angeles, CA: Author.
49. Ostroff, C. (1993). Comparing correlations based on individual-level and aggregated data. Journal of Applied Psychology, 78, 569-582. http://dx .doi.org/10.1037/0021-9010.78.4.569.
50. Peugh, J. L., & Enders, C. K. (2004). Missing data in educational research: A review of reporting practices and suggestions for improvement. Review of Educational Research, 74, 525-556. http://dx.doi.org/10.3102/ 00346543074004525.
51. Preacher, K. J., Zhang, Z., & Zyphur, M. J. (2016). Multilevel structural equation models for assessing moderation within and across levels of analysis. Psychological Methods, 21, 189-205. http://dx.doi.org/10 .1037/met0000052.
52. Preacher, K. J., Zyphur, M. J., & Zhang, Z. (2010). A general multilevel SEM framework for assessing multilevel mediation. Psychological Methods, 15, 209-233. http://dx.doi.org/10.1037/a0020141.
53. Quartagno, M., & Carpenter, J. (2016). jomo: A package for multilevel joint modelling multiple imputation. R package version 2.1-2. Retrieved from http://CRAN.R-project.org/package=jomo.
54. Raghunathan, T. E., Lepkowski, J. E., Hoewyk, J. V., & Solenberger, P. (2001). A multivariate technique for multiply imputing missing values using a sequence of regression models. Survey Methodology, 27, 85-96.
55. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models (2nd ed.). Thousand Oaks, CA: Sage.
56. Rubin, D. B. (1976). Inference and missing data. Biometrika, 63, 581-592. http://dx.doi.org/10.1093/biomet/63.3.581.
57. Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys. Hoboken, NJ: Wiley. http://dx.doi.org/10.1002/9780470316696.
58. Savalei, V., & Falk, C. (2014). Robust two-stage approach outperforms robust full information maximum likelihood with incomplete nonnormal data. Structural Equation Modeling, 21, 280-302. http://dx.doi.org/10 .1080/10705511.2014.882692.
59. Schafer, J. L. (1997). Analysis of incomplete multivariate data. Boca Raton, FL: CRC Press.
60. Schafer, J. L. (2001). Multiple imputation with PAN. In L. M. Collins & A. G. Sayer (Eds.), New methods for the analysis of change (pp. 357-377). Washington, DC: American Psychological Association. http://dx.doi.org/10.1037/10409-012.
61. Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147-177.
62. Schafer, J. L., & Olsen, M. K. (1998). Multiple imputation for multivariate missing-data problems: A data analyst's perspective. Multivariate Behavioral Research, 33, 545-571. http://dx.doi.org/10.1207/ s15327906mbr3304_5.
63. Schafer, J. L., & Yucel, R. M. (2002). Computational strategies for multivariate linear mixed-effects models with missing values. Journal of Computational and Graphical Statistics, 11, 437-457. http://dx.doi.org/ 10.1198/106186002760180608.
64. Schafer, J. L., & Zhao, J. H. (2013). pan: Multiple imputation for multivariate panel or clustered data (R package version 0.9) [Computer program]. Retrieved from http://CRAN.R-project.org/package=pan.
65. Shin, Y. (2013). Efficient handling of predictors and outcomes having missing values. In L. Rutkowski, M. von Davier, & D. Rutkowski (Eds.), A handbook of international large-scale assessment data analysis: Background, technical issues, and methods of data analysis (pp. 451-479). London, UK: Chapman & Hall/CRC Press.
66. Shin, Y., & Raudenbush, S. W. (2010). A latent cluster-mean approach to the contextual effects model with missing data. Journal of Educational and Behavioral Statistics, 35, 26-53. http://dx.doi.org/10.3102/ 1076998609345252.
67. Snijders, T. A. B., & Bosker, R. J. (2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London, UK: Sage.
68. Taljaard, M., Donner, A., & Klar, N. (2008). Imputation strategies for missing continuous outcomes in cluster randomized trials. Biometrical Journal, 50, 329-345. http://dx.doi.org/10.1002/bimj.200710423.
69. Tanner, M. A., & Wong, W. H. (1987). The calculation of posterior distributions by data augmentation. Journal of the American Statistical Association, 82, 528-540. http://dx.doi.org/10.1080/01621459.1987 .10478458.
70. van Buuren, S. (2011). Multiple imputation of multilevel data. In J. J. Hox & J. K. Roberts, The handbook of advanced multilevel analysis (pp. 173-196). Milton Park, UK: Routledge.
71. van Buuren, S. (2012). Flexible imputation of missing data. Boca Raton, FL: Chapman & Hall/CRC Press. http://dx.doi.org/10.1201/b11826.
72. van Buuren, S., & Groothuis-Oudshoorn, K. (2011). MICE: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45, 1-67.
73. von Hippel, P. T. (2009). How to impute square, interactions, and other transformed variables. Sociological Methodology, 39, 265-291. http:// dx.doi.org/10.1111/j.1467-9531.2009.01215.x.
74. Walsh, B. M., Matthews, R. A., Tuller, M. D., Parks, K. M., & McDonald, D. P. (2010). A multilevel model of the effects of equal opportunity climate on job satisfaction in the military. Journal of Occupational Health Psychology, 15, 191-207. http://dx.doi.org/10 .1037/a0018756.
75. West, S. G. (2001). New approaches to missing data in psychological research: Introduction to the special section. Psychological Methods, 6, 315-316. http://dx.doi.org/10.1037/1082-989X.6.4.315.
76. White, I. R., Royston, P., & Wood, A. M. (2011). Multiple imputation using chained equations: Issues and guidance for practice. Statistics in Medicine, 30, 377-399. http://dx.doi.org/10.1002/sim.4067.
77. Yuan, K.-H., & Bentler, P. M. (2007). Multilevel covariance structure analysis by fitting multiple single-level models. Sociological Methodology, 37, 53-82. http://dx.doi.org/10.1111/j.1467-9531.2007. 00182.x.
78. Yuan, K.-H., Tong, X., & Zhang, Z. (2015). Bias and efficiency for SEM with missing data and auxiliary variables: Two-stage robust method versus two-stage ML. Structural Equation Modeling, 22, 178-192. http://dx.doi.org/10.1080/10705511.2014.935750.
79. Yuan, K.-H., Yang-Wallentin, F., & Bentler, P. M. (2012). ML versus MI for missing data with violation of distribution conditions. Sociological Methods & Research, 41, 598-629. http://dx.doi.org/10.1177/0049 124112460373.
80. Yucel, R. M. (2008). Multiple imputation inference for multivariate multilevel continuous data with ignorable non-response. Philosophical Transactions. Series A, Mathematical, Physical, and Engineering Sciences, 366, 2389-2403. http://dx.doi.org/10.1098/rsta.2008.0038.
81. Yucel, R. M. (2011). Random-covariances and mixed-effects models for imputing multivariate multilevel continuous data. Statistical Modelling, 11, 351-370. http://dx.doi.org/10.1177/1471082X1001 100404.
82. Yucel, R. M., & Demirtas, H. (2010). Impact of non-normal random effects on inference by multiple imputation: A simulation assessment. Computational Statistics & Data Analysis, 54, 790-801. http://dx.doi.org/10 .1016/j.csda.2009.01.016.