Accommodating Small Sample Sizes in Three-Level Models When the Third Level is Incidental

Multivariate Behavioral Research

Volumn 52, Issue 2, 2017, Pages 200-215

  McNeish, Daniel ^a   Wentzel, Kathryn R ^b  

^a UTRECHT UNIVERSITY   (Netherlands)

^b UNIVERSITY OF MARYLAND   (United States)

Author keywords

HLM; mixed effects model; multilevel model; small sample; Three level

Indexed keywords

HUMAN; MODEL; PSYCHOLOGY; SAMPLE SIZE; ADOLESCENT; ADOLESCENT BEHAVIOR; ALGORITHM; CHILD; CLUSTER ANALYSIS; COMPUTER SIMULATION; EDUCATIONAL STATUS; MONTE CARLO METHOD; MOTIVATION; MULTIVARIATE ANALYSIS; REGRESSION ANALYSIS; SCHOOL TEACHER; SOCIAL SUPPORT; STATISTICAL ANALYSIS; STUDENT; UNITED STATES;

ADOLESCENT; ADOLESCENT BEHAVIOR; ALGORITHMS; CHILD; CLUSTER ANALYSIS; COMPUTER SIMULATION; DATA INTERPRETATION, STATISTICAL; EDUCATIONAL STATUS; HUMANS; MONTE CARLO METHOD; MOTIVATION; MULTIVARIATE ANALYSIS; REGRESSION ANALYSIS; SCHOOL TEACHERS; SOCIAL SUPPORT; STUDENTS; UNITED STATES;

EID: 85007258930     PISSN: 00273171     EISSN: None     Source Type: Journal    
DOI: 10.1080/00273171.2016.1262236     Document Type: Article

References (69)

1. Allison, P. D., (2005). Fixed effects regression methods for longitudinal data using SAS. Raleigh, NC:SAS Institute.
2. Allison, P. D., (2009). Fixed effects regression models. London, UK:Sage.
3. Austin, P. C., (2010). Estimating multilevel logistic regression models when the number of clusters is low:a comparison of different statistical software procedures. The International Journal of Biostatistics, 6, Article 16. doi:10.2202/1557-4679.1195
4. Bauer, D. J., & Hussong, A. M., (2009). Psychometric approaches for developing commensurate measures across independent studies:traditional and new models. Psychological Methods, 14, 101–125. doi:10.1037/a0017642
5. Bauer, D. J., & Sterba, S. K., (2011). Fitting multilevel models with ordinal outcomes:Performance of alternative specifications and methods of estimation. Psychological Methods, 16, 373–390. doi:10.1037/a0025813
6. Bell, A., & Jones, K., (2015). Explaining fixed effects:Random effects modeling of time-series cross-sectional and panel data. Political Science Research and Methods, 3, 133–153. doi:10.1017/psrm.2014.7
7. Bell, B. A., Morgan, G. B., Schoeneberger, J. A., Kromrey, J. D., & Ferron, J. M., (2014). How low can you go? An investigation of the influence of sample size and model complexity on point and interval estimates in two-level linear models. Methodology, 10, 1–11. doi:10.1027/1614-2241/a000062
8. Blozis, S. A., & Harring, J. R., (2016). Understanding individual-level change through the basis functions of a latent curve model. Sociological Methods & Research. Advance online publication. doi:10.1177/0049124115605341
9. Bradley, J. V., (1978). Robustness?. British Journal of Mathematical and Statistical Psychology, 31, 144–152. doi:10.1111/j.2044-8317.1978.tb00581.x
10. Browne, W. J., & Draper, D., (2006). A comparison of Bayesian and likelihood-based methods for fitting multilevel models. Bayesian Analysis, 1, 473–514. doi:10.1214/06-ba117
11. Bryk, A. S., & Raudenbush, S. W., (1988). Toward a more appropriate conceptualization of research on school effects:A three-level hierarchical linear model. American Journal of Education, 97, 65–108. doi:10.1086/443913
12. Cameron, A. C., Gelbach, J. B., & Miller, D. L., (2008). Bootstrap-based improvements for inference with clustered errors. The Review of Economics and Statistics, 90, 414–427. doi:10.1162/rest.90.3.414
13. Clarke, P., (2008). When can group level clustering be ignored? Multilevel models versus single-level models with sparse data. Journal of Epidemiology and Community Health, 62, 752–758. doi:10.1136/jech.2007.060798
14. Cohen, J., (1988). Statistical power analysis for the behavioral sciences. San Diego, CA:Academic Press.
15. Curran, P. J., & Hussong, A. M., (2009). Integrative data analysis:the simultaneous analysis of multiple data sets. Psychological Methods, 14, 81–100. doi:10.1037/a0015914
16. Curran, P. J., Obeidat, K., & Losardo, D., (2010). Twelve frequently asked questions about growth curve modeling. Journal of Cognition and Development, 11, 121–136. doi:10.1080/15248371003699969
17. Fay, M. P., & Graubard, B. I., (2001). Small‐sample adjustments for Wald‐type tests using sandwich estimators. Biometrics, 57, 1198–1206. doi:10.1111/j.0006-341x.2001.01198.x
18. Ferron, J. M., Bell, B. A., Hess, M. R., Rendina-Gobioff, G., & Hibbard, S. T., (2009). Making treatment effect inferences from multiple-baseline data:The utility of multilevel modeling approaches. Behavior Research Methods, 41, 372–384. doi:10.3758/brm.41.2.372
19. Fitzmaurice, G. M., Laird, N. M., & Ware, J. H., (2011) Applied longitudinal analysis. Hoboken, NJ:Wiley.
20. Flora, D. B., & Curran, P. J., (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9, 466–491. doi:10.1037/1082-989x.9.4.466
21. Galbraith, S., Daniel, J. A., & Vissel, B., (2010). A study of clustered data and approaches to its analysis. The Journal of Neuroscience, 30, 10601–10608. doi:10.1523/jneurosci.0362-10.2010
22. Gardiner, J. C., Luo, Z., & Roman, L. A., (2009). Fixed effects, random effects and GEE:What are the differences?. Statistics in Medicine, 28, 221–239. doi:10.1002/sim.3478
23. Gelman A, & Hill J. (2006). Data analysis using regression and multilevel/hierarchical models. Cambridge, MA:Cambridge University Press.
24. Greene, W., (2002). Econometric analysis. Englewood Cliffs, NJ:Prentice-Hall.
25. Greene, W., (2011). Fixed effects vector decomposition:a magical solution to the problem of time-invariant variables in fixed effects models?. Political Analysis, 19, 135–146. doi:10.1093/pan/mpq034
26. Harville, D. A., (1977). Maximum likelihood approaches to variance component estimation and to related problems. Journal of the American Statistical Association, 72, 320–338. doi:10.2307/2286796
27. Hedges, L. V., & Hedberg, E. C., (2007). Intraclass correlation values for planning group randomized trials in education. Educational Evaluation and Policy Analysis, 29, 60–87. doi:10.3102/0162373707299706
28. Hox J. J., (2010). Multilevel analysis:Techniques and applications. Mahwah, NJ:Lawrence Erlbaum.
29. Huang, F. L., (2016). Alternatives to multilevel modeling for the analysis of clustered data. The Journal of Experimental Education, 84, 175–196. doi:10.1080/00220973.2014.952397
30. Hussong, A. M., Curran, P. J., & Bauer, D. J., (2013). Integrative data analysis in clinical psychology research. Annual Review of Clinical Psychology, 9, 61–89. doi:10.1146/annurev-clinpsy-050212-185522
31. Jacqmin-Gadda, H., Sibillot, S., Proust, C., Molina, J. M., & Thiébaut, R., (2007). Robustness of the linear mixed model to misspecified error distribution. Computational Statistics & Data Analysis, 51, 5142–5154. doi:10.1016/j.csda.2006.05.021
32. Johnson, D. W., Johnson, R. T., Buckman, L. A., & Richards, P. S., (1985). The effect of prolonged implementation of cooperative learning on social support within the classroom. The Journal of Psychology, 119, 405–411. doi:10.1080/00223980.1985.10542911
33. Kauermann, G., & Carroll, R. J., (2001). A note on the efficiency of sandwich covariance matrix estimation. Journal of the American Statistical Association, 96, 1387–1396. doi:10.1198/016214501753382309
34. Kenward, M. G., & Roger, J. H., (1997). Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53, 983–997. doi:10.2307/2533558
35. Kenward, M. G., & Roger, J. H., (2009). An improved approximation to the precision of fixed effects from restricted maximum likelihood. Computational Statistics & Data Analysis, 53, 2583–2595. doi:10.1016/j.csda.2008.12.013
36. Lai, M. H., & Kwok, O. M., (2015). Examining the rule of thumb of not using multilevel modeling:The “design effect smaller than two” rule. The Journal of Experimental Education, 83, 423–438. doi:10.1080/00220973.2014.907229
37. Laird, N. M., & Ware, J. H., (1982). Random-effects models for longitudinal data. Biometrics, 38, 963–974. doi:10.2307/2529876
38. Lu, B., Preisser, J. S., Qaqish, B. F., Suchindran, C., Bangdiwala, S. I., & Wolfson, M., (2007). A comparison of two bias-corrected covariance estimators for generalized estimating equations. Biometrics, 63, 935–941. doi:10.1111/j.1541-0420.2007.00764.x
39. Maas, C. J., & Hox, J. J., (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1, 86–92. doi:10.1027/1614-2241.1.3.85
40. Mancl, L. A., & DeRouen, T. A., (2001). A covariance estimator for GEE with improved small sample properties. Biometrics, 57, 126–134. doi:10.1111/j.0006-341x.2001.00126.x
41. McNeish, D., (2014). Modeling sparsely clustered data:Design-based, model-based, and single-level methods. Psychological Methods, 19, 552–563. doi:10.1037/met0000024
42. McNeish, D., (2016). On using Bayesian methods to address small sample problems. Structural Equation Modeling, 23, 750–773. doi:10.1080/10705511.2016.1186549
43. McNeish, D. M., & Harring, J. R., (2015). Clustered data with small sample sizes:Comparing the performance of model-based and design-based approaches. Communications in Statistics-Simulation and Computation. Advance online publication. doi:10.1080/03610918.2014.983648
44. McNeish, D., & Stapleton, L. M., (2016a). The effect of small sample size on two level model estimates:A review and illustration. Educational Psychology Review, 28, 295–314. doi:10.1007/s10648-014-9287-x
45. McNeish, D., & Stapleton, L.M., (2016b). Modeling clustered data with very few clusters. Multivariate Behavioral Research, 51, 495–518. doi:10.1080/00273171.2016.1167008
46. McNeish, D., Stapleton, L. M., & Silverman, R. D., (2016). On the unnecessary ubiquity of hierarchical linear modeling. Psychological Methods. Advance online publication. doi:10.1037/met0000078
47. Moerbeek, M., (2004). The consequence of ignoring a level of nesting in multilevel analysis. Multivariate Behavioral Research, 39, 129–149. doi:10.1207/s15327906mbr3901_5
48. Moineddin, R., Matheson, F. I., & Glazier, R. H., (2007). A simulation study of sample size for multilevel logistic regression models. BMC Medical Research Methodology, 7, 34. doi:10.1186/1471-2288-7-34
49. Morel, J. G., Bokossa, M. C., & Neerchal, N. K., (2003). Small sample correction for the variance of GEE estimators. Biometrical Journal, 45, 395–409. doi:10.1002/bimj.200390021
50. Murnane, R. J., & Willett, J. B., (2010). Methods matter:Improving causal inference in educational and social science research. New York, NY:Oxford University Press.
51. Murray, D. M., (2001). Statistical models appropriate for designs often used in group‐randomized trials. Statistics in Medicine, 20, 1373–1385. doi:10.1002/sim.675
52. Murray, D. M., Varnell, S. P., & Blitstein, J. L., (2004). Design and analysis of group-randomized trials:a review of recent methodological developments. American Journal of Public Health, 94, 423–432. doi:10.2105/ajph.94.3.423
53. Muthén, B. O., & Satorra, A., (1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267–316. doi:10.2307/271070
54. Opdenakker, M. C., & Van Damme, J., (2000). Effects of schools, teaching staff and classes on achievement and well-being in secondary education:Similarities and differences between school outcomes. School Effectiveness and School Improvement, 11, 165–196. doi:10.1076/0924-3453(200006)11:2;1-q;ft165
55. Pan, W., & Wall, M. M., (2002). Small‐sample adjustments in using the sandwich variance estimator in generalized estimating equations. Statistics in Medicine, 21, 1429–1441. doi:10.1002/sim.1142
56. Plümper, T., & Troeger, V. E., (2007). Efficient estimation of time-invariant and rarely changing variables in finite sample panel analyses with unit fixed effects. Political Analysis, 15, 124–139. doi:10.1093/pan/mpm002
57. Pfeffermann, D., Skinner, C. J., Holmes, D. J., Goldstein, H., & Rasbash, J., (1998). Weighting for unequal selection probabilities in multilevel models. Journal of the Royal Statistical Society:Series B, 60, 23–40. doi:10.1111/1467-9868.00106
58. Pornprasertmanit, S., Lee, J., & Preacher, K. J., (2014). Ignoring clustering in confirmatory factor analysis:some consequences for model fit and standardized parameter estimates. Multivariate Behavioral Research, 49, 518–543. doi:10.1080/00273171.2014.933762
59. Rabe‐Hesketh, S., & Skrondal, A., (2006). Multilevel modelling of complex survey data. Journal of the Royal Statistical Society:Series A, 169, 805–827. doi:10.1111/j.1467-985x.2006.00426.x
60. Raudenbush, S. W., & Bryk, A. S., (2002). Hierarchical linear models:Applications and data analysis methods (2nd ed.). Thousand Oaks, CA:Sage.
61. Schaalje, G. B., McBride, J. B., & Fellingham, G. W., (2002). Adequacy of approximations to distributions of test statistics in complex mixed linear models. Journal of Agricultural, Biological, and Environmental Statistics, 7, 512–524. doi:10.1198/108571102726
62. Schoeneberger, J. A., (2015). The impact of sample size and other factors when estimating multilevel logistic models. The Journal of Experimental Education, 84, 373–397. doi:10.1080/00220973.2015.1027805
63. Selya, A. S., Rose, J. S., Dierker, L. C., Hedeker, D., & Mermelstein, R. J., (2012). A practical guide to calculating Cohen's f2, a measure of local effect size, from PROC MIXED. Frontiers in Psychology, 3, 1–6. doi:10.3389/fpsyg.2012.00111
64. Sullivan, L. M., Dukes, K. A., & Losina, E., (1999). Tutorial in biostatistics. An introduction to hierarchical linear modelling. Statistics in Medicine, 18, 855–888. doi:10.1002/0470023724.ch1b(i)
65. van den Noortgate, W., Opdenakker, M. C., & Onghena, P., (2005). The effects of ignoring a level in multilevel analysis. School Effectiveness and School Improvement, 16, 281–303. doi:10.1080/09243450500114850
66. Verbeke, G., & Lesaffre, E., (1997). The effect of misspecifying the random-effects distribution in linear mixed models for longitudinal data. Computational Statistics & Data Analysis, 23, 541–556. doi:10.1016/s0167-9473(96)00047-3
67. Westgate, P. M., (2013). A bias correction for covariance estimators to improve inference with generalized estimating equations that use an unstructured correlation matrix. Statistics in Medicine, 32, 2850–2858. doi:10.1002/sim.5709
68. White, H., (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica, 48, 817–838.
69. White, H., (1984). Asymptotic theory for econometricians. San Diego, CA:Academic Press. doi:10.2307/1912934