Modeling Clustered Data with Very Few Clusters

Multivariate Behavioral Research

Volumn 51, Issue 4, 2016, Pages 495-518

  McNeish, Daniel ^a,b   Stapleton, Laura M ^a  

^a UNIVERSITY OF MARYLAND   (United States)

^b UTRECHT UNIVERSITY   (Netherlands)

Author keywords

Bayesian; cluster randomized trial; fixed effect model; GEE; HLM; multilevel model; small sample

Indexed keywords

CONTROLLED CLINICAL TRIAL; HUMAN; MODEL; PSYCHOLOGY; RANDOMIZED CONTROLLED TRIAL; BAYES THEOREM; CLUSTER ANALYSIS; COMPUTER SIMULATION; LANGUAGE TEST; MULTILEVEL ANALYSIS; PROCEDURES; RANDOMIZED CONTROLLED TRIAL (TOPIC); SAMPLE SIZE; SOFTWARE; STATISTICAL MODEL;

BAYES THEOREM; CLUSTER ANALYSIS; COMPUTER SIMULATION; HUMANS; LANGUAGE TESTS; MODELS, STATISTICAL; MULTILEVEL ANALYSIS; PSYCHOLOGY, EDUCATIONAL; RANDOMIZED CONTROLLED TRIALS AS TOPIC; SAMPLE SIZE; SOFTWARE;

EID: 84973596359     PISSN: 00273171     EISSN: None     Source Type: Journal    
DOI: 10.1080/00273171.2016.1167008     Document Type: Article

References (83)

1. P.D.Allison, (2005). Fixed effects regression methods for longitudinal data using SAS. Raleigh, NC:SAS Institute.
2. J.D.Angrist, & J.S.Pischke, (2008). Mostly harmless econometrics:An empiricist's companion. Princeton, NJ:Princeton University Press.
3. S.A.Baldwin, & G.W.Fellingham, (2013). Bayesian methods for the analysis of small sample multilevel data with a complex variance structure. Psychological Methods, 18, 151–164. doi:10.1037/a0030642
4. G.A.Ballinger, (2004). Using generalized estimating equations for longitudinal data analysis. Organizational Research Methods, 7, 127–150. doi:10.1177/1094428104263672
5. D.J.Bauer, & S.K.Sterba, (2011). Fitting multilevel models with ordinal outcomes:Performance of alternative specifications and methods of estimation. Psychological Methods, 16, 373–390. doi:10.1037/a0025813
6. M.D.Begg, & M.K.Parides, (2003). Separation of individual-level and cluster-level covariate effects in regression analysis of correlated data. Statistics in Medicine, 22, 2591–2602. doi:10.1002/sim.1524
7. B.A.Bell, G.B.Morgan, J.A.Schoeneberger, J.D.Kromrey, & J.M.Ferron, (2014). How low can you go? An investigation of the influence of sample size and model complexity on point and interval estimates in two-level linear models. Methodology, 10, 1–11. doi:10.1002/sim.1524
8. J.Berkhof, & J.K.Kampen, (2004). Asymptotic effect of misspecification in the random part of the multilevel model. Journal of Educational and Behavioral Statistics, 29, 201–218. doi:10.3102/10769986029002201
9. J.V.Bradley, (1978). Robustness?. British Journal of Mathematical and Statistical Psychology, 31, 144–152. doi:10.1111/j.2044-8317.1978.tb00581.x
10. W.J.Browne, & D.Draper, (2006). A comparison of Bayesian and likelihood-based methods for fitting multilevel models. Bayesian Analysis, 1, 473–514. doi:10.1214/06-BA117
11. A.C.Cameron, J.B.Gelbach, & D.L.Miller, (2011). Robust inference with multiway clustering. Journal of Business & Economic Statistics, 29, 238–249. doi:10.1198/jbes.2010.07136
12. M.W.L.Cheung, (2013). Implementing restricted maximum likelihood estimation in structural equation models. Structural Equation Modeling, 20, 157–167. doi:10.1080/10705511.2013.742404
13. M.J.Daniels, (1999). A prior for the variance in hierarchical models. The Canadian Journal of Statistics, 27, 567–578. doi:10.2307/3316112
14. S.Depaoli, & R.van de Schoot, (2016). Improving transparency and replication in Bayesian statistics:The WAMBS-checklist. Psychological Methods. Advance online publication. doi:10.1037/met0000065
15. L.J.Emrich, & M.R.Piedmonte, (1992). On some small sample properties of generalized estimating equation estimates for multivariate dichotomous outcomes. Journal of Statistical Computation and Simulation, 41, 19–29. doi:10.1080/00949659208811388
16. C.K.Enders, & D.Tofighi, (2007). Centering predictor variables in cross-sectional multilevel models:a new look at an old issue. Psychological Methods, 12, 121–138. doi:10.1037/1082-989X.12.2.121
17. M.P.Fay, & B.I.Graubard, (2001). Small-sample adjustments for Wald-type tests using sandwich estimators. Biometrics, 57, 1198–1206. doi:10.1111/j.0006-341X.2001.01198.x
18. J.M.Ferron, B.A.Bell, M.R.Hess, G.Rendina-Gobioff, & S.T.Hibbard, (2009). Making treatment effect inferences from multiple-baseline data:The utility of multilevel modeling approaches. Behavior Research Methods, 41, 372–384. doi:10.3758/BRM.41.2.372
19. G.M.Fitzmaurice, N.M.Laird, & J.H.Ware, (2004) Applied longitudinal analysis. Hoboken, NJ:Wiley.
20. D.B.Flora, & P.J.Curran, (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9, 466–491. doi:10.1037/1082-989X.9.4.466
21. C.O.Fritz, P.E.Morris, & J.J.Richler, (2012). Effect size estimates:current use, calculations, and interpretation. Journal of Experimental Psychology:General, 141, 2–18. doi:10.1037/a0024338
22. S.Galbraith, J.A.Daniel, & B.Vissel, (2010). A study of clustered data and approaches to its analysis. The Journal of Neuroscience, 30, 10601–10608. doi:10.1523/JNEUROSCI.0362-10.2010
23. J.C.Gardiner, Z.Luo, & L.A.Roman, (2009). Fixed effects, random effects and GEE:What are the differences? Statistics in Medicine, 28, 221–239. doi:10.1002/sim.3478
24. A.Gelman, (2006). Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper). Bayesian Analysis, 1, 515–534.
25. A.Gelman, J.B.Carlin, H.S.Stern, & D.B.Rubin, (2003). Bayesian data analysis. Boca Raton, FL:CRC press. doi:10.1214/06-BA117A
26. J.C.Gunsolley, C.Getchell, & V.M.Chinchilli, (1995). Small sample characteristics of generalized estimating equations. Communications in Statistics-Simulation and Computation, 24, 869–878. doi/10.1080/03610919508813280
27. D.L.Hahs-Vaughn, (2005). A primer for using and understanding weights with national datasets. The Journal of Experimental Education, 73, 221–248. doi:10.3200/JEXE.73.3.221-248
28. J.A.Hausman, (1978). Specification tests in econometrics. Econometrica, 46, 1251–1271.
29. L.V.Hedges, & E.C.Hedberg, (2007). Intraclass correlation values for planning group randomized trials in education. Educational Evaluation and Policy Analysis, 29, 60–87. doi:10.3102/0162373707299706
30. N.J.Horton, & S.R.Lipsitz, (1999). Review of software to fit generalized estimating equation regression models. The American Statistician, 53, 160–169. doi:10.2307/2685737
31. J.Hox, (2002). Multilevel analysis:Techniques and applications. Mahwah, NJ:Erlbaum.
32. J.Hox, R.van de Schoot, & S.Matthijsse, (2012). How few countries will do? Comparative survey analysis from a Bayesian perspective. Survey Research Methods, 6, 87–93. doi:10.18148/srm/2012.v6i2.5033
33. F.L.Huang, (2016) Alternatives to multilevel modeling for the analysis of clustered data. The Journal of Experimental Education, 84, 175–196. doi:10.1080/00220973.2014.952397
34. P.J.Huber, (1967, June). The behavior of maximum likelihood estimates under nonstandard conditions. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability (Vol. 1, No. 1, pp. 221–233).
35. H.Jacqmin-Gadda, S.Sibillot, C.Proust, J.M.Molina, & R.Thiébaut, (2008). Robustness of the linear mixed model to misspecified error distribution. Computational Statistics & Data Analysis, 51, 5142–5154. doi:10.1016/j.csda.2006.05.021
36. R.N.Kackar, & D.A.Harville, (1984). Approximations for standard errors of estimators of fixed and random effects in mixed linear models. Journal of the American Statistical Association, 79, 853–862. doi:10.2307/2288715
37. G.Kauermann, & R.J.Carroll, (2001). A note on the efficiency of sandwich covariance matrix estimation. Journal of the American Statistical Association, 96, 1387–1396. doi:10.1198/016214501753382309
38. M.G.Kenward, & J.H.Roger, (1997). Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53, 983–997. doi:10.2307/2533558
39. M.G.Kenward, & J.H.Roger, (2009). An improved approximation to the precision of fixed effects from restricted maximum likelihood. Computational Statistics & Data Analysis, 53, 2583–2595. doi:10.1016/j.csda.2008.12.013
40. R.K.Kowalchuk, H.J.Keselman, J.Algina, & R.D.Wolfinger, (2004). The analysis of repeated measurements with mixed-model adjusted F tests. Educational and Psychological Measurement, 64, 224–242. doi:10.1177/0013164403260196
41. I.G.G.Kreft, (1996). Are multilevel techniques necessary? An overview, including simulation studies. Unpublished manuscript, Los Angeles, CA:California State University
42. J.K.Kruschke, H.Aguinis, & H.Joo, (2012). The time has come Bayesian methods for data analysis in the organizational sciences. Organizational Research Methods, 15, 722–752. doi:10.1177/1094428112457829
43. K.Y.Liang, & S.L.Zeger, (1986). Longitudinal data analysis using generalized linear models. Biometrika, 73, 13–22. doi:10.1093/biomet/73.1.13
44. S.L.Lohr, (2014). Design effects for a regression slope in a cluster sample. Journal of Survey Statistics and Methodology, 2, 97–125. doi:10.1093/jssam/smu003
45. B.Lu, J.S.Preisser, B.F.Qaqish, C.Suchindran, S.I.Bangdiwala, & M.Wolfson, (2007). A comparison of two bias-corrected covariance estimators for generalized estimating equations. Biometrics, 63, 935–941. doi:10.1111/j.1541-0420.2007.00764.x
46. C.J.Maas, & J.J.Hox, (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1, 86–92. doi:10.1027/1614-2241.1.3.85
47. C.Maas, & J.J.Hox, (2004). Robustness issues in multilevel regression analysis. Statistica Neerlandica, 58, 127–137. doi:10.1027/1614-2241.1.3.85
48. O.Manor, & D.M.Zucker, (2004). Small sample inference for the fixed effects in the mixed linear model. Computational Statistics & Data Analysis, 46, 801–817. doi:10.1016/j.csda.2003.10.005
49. L.A.Mancl, & T.A.DeRouen, (2001). A covariance estimator for GEE with improved small sample properties. Biometrics, 57, 126–134. doi:10.1111/j.0006-341X.2001.00126.x
50. P.McCullagh, & J.A.Nelder, (1989). Generalized linear models. London, UK:Chapman and Hall.
51. C.E.McCulloch, & S.R.Searle, (2001). Generalized, linear, and mixed models. New York, NY:Wiley.
52. D.M.McNeish, (2016). Using data-dependent priors to mitigate small sample bias in latent growth models:A discussion and illustration using Mplus. Journal of Educational and Behavioral Statistics, 41, 27–56. doi:10.3102/1076998615621299
53. D.M.McNeish, & J.R.Harring, (2015). Clustered data with small sample sizes:Comparing the performance of model-based and design-based approaches. Communications in Statistics-Simulation and Computation. Advance online publication. http://dx.doi.org/10.100/03610917.2014.983648
54. D.M.McNeish, & L.M.Stapleton, (2014). The effect of small sample size on two level model estimates:A review and illustration. Educational Psychology Review. Advance online publication. http://dx.doi.org/10.1007/s10648-014-9287-x.
55. D.M.McNeish, L.M.Stapleton, & R.D.Silverman, (2016). On the unnecessary ubiquity of hierarchical linear modeling. Psychological Methods. Advance online publication. doi:10.1037/met0000078
56. J.G.Morel, M.C.Bokossa, & N.K.Neerchal, (2003). Small sample correction for the variance of GEE estimators. Biometrical Journal, 45, 395–409. doi:10.1002/bimj.200390021
57. R.J.Murnane, & J.B.Willett, (2010). Methods matter:Improving causal inference in educational and social science research. New York, NY:Oxford University Press.
58. W.Pan, & M.M.Wall, (2002). Small‐sample adjustments in using the sandwich variance estimator in generalized estimating equations. Statistics in Medicine, 21, 1429–1441. doi:10.1002/sim.1142
59. M.A.Petersen, (2009). Estimating standard errors in finance panel data sets:Comparing approaches. Review of Financial Studies, 22, 435–480. doi:10.1093/rfs/hhn053
60. N.G.Polson, & J.G.Scott, (2012). On the half-Cauchy prior for a global scale parameter. Bayesian Analysis, 7, 887–902. doi:10.1214/12-BA730
61. S.W.Raudenbush, & A.S.Bryk, (2002). Hierarchical linear models:Applications and data analysis methods (2nd ed.). Thousand Oaks, CA:Sage.
62. G.B.Schaalje, J.B.McBride, & G.W.Fellingham, (2002). Adequacy of approximations to distributions of test statistics in complex mixed linear models. Journal of Agricultural, Biological, and Environmental Statistics, 7, 512–524. doi:10.1198/108571102726
63. P.Z.Schochet, (2015). Statistical theory for the RCT-YES software:Design-based causal inference for RCTs (NCEE 2015–4011). Washington, DC:U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Analytic Technical Assistance and Development. Retrieved from http://ies.ed.gov/ncee/edlabs.
64. S.Schreiber, (2008). The Hausman test statistic can be negative even asymptotically. Journal of Economics and Statistics, 228, 394–405. doi:10.1515/jbnst-2008-0407
65. T.Sheytanova, (2014). Accuracy of the Hausman test in panel data:A Monte Carlo study. (Unpublished thesis) Öreboro University, Sweden. http://oru.diva-portal.org/smash/get/diva2:805823/FULLTEXT01.pdf
66. S.S.Skene, & M.G.Kenward, (2010a). The analysis of very small samples of repeatedmeasurements I:An adjusted sandwich estimator. Statistics in Medicine, 29, 2825–2837. doi:10.1002/sim.4073
67. S.S.Skene, & M.G.Kenward, (2010b). The analysis of very small samples of repeated measurements II:A modified Box correction. Statistics in Medicine, 29, 2838–2856. doi:10.1002/sim.4072
68. J.Spilke, H.P.Piepho, & X.Hu, (2005). A simulation study on tests of hypotheses and confidence intervals for fixed effects in mixed models for blocked experiments with missing data. Journal of Agricultural, Biological, and Environmental Statistics, 10, 374–389. doi:10.1198/108571105X58199
69. D.Stegmueller, (2013). How many countries for multilevel modeling? A Comparison of frequentist and Bayesian approaches. American Journal of Political Science, 57, 748–761. doi:10.1111/ajps.12001
70. S.L.Thomas, & R.H.Heck, (2001). Analysis of large-scale secondary data in higher education research:Potential perils associated with complex sampling designs. Research in Higher Education, 42, 517–540. doi:10.1023/A:1011098109834
71. S.L.Thomas, R.H.Heck, & K.W.Bauer, (2005). Weighting and adjusting for design effects in secondary data analyses. New Directions for Institutional Research, 127, 51–72.
72. J.W.Twisk, (2004). Longitudinal data analysis. A comparison between generalized estimating equations and random coefficient analysis. European Journal of Epidemiology, 19, 769–776. doi:10.1023/B:EJEP.0000036572.00663.f2
73. G.Vallejo, & P.Livacic-Rojas, (2005). Comparison of two procedures for analyzing small sets of repeated measures data. Multivariate Behavioral Research, 40, 179–205. doi:10.1207/s15327906mbr4002_2
74. R.van de Schoot, J.J.Broere, K.H.Perryck, M.Zondervan-Zwijnenburg, & N.E.Van Loey, (2015). Analyzing small data sets using Bayesian estimation:the case of posttraumatic stress symptoms following mechanical ventilation in burn survivors. European Journal of Psychotraumatology, 6, 1–13. doi:10.3402/ejpt.v6.25216
75. R.van de Schoot, D.Kaplan, J.Denissen, J.B.Asendorpf, F.J.Neyer, & M.A.Aken, (2014). A gentle introduction to Bayesian analysis:applications to developmental research. Child Development, 85, 842–860. doi:10.1111/cdev.12169
76. M.P.Wand, J.T.Ormerod, S.A.Padoan, & R.Fuhrwirth, (2011). Mean field variational Bayes for elaborate distributions. Bayesian Analysis, 6, 847–900. doi:10.1214/11-BA631
77. P.M.Westgate, (2013). A bias correction for covariance estimators to improve inference with generalized estimating equations that use an unstructured correlation matrix. Statistics in Medicine, 32, 2850–2858. doi:10.1002/sim.5709
78. H.White, (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica, 48, 817–838. doi:10.2307/1912934
79. S.Wu, C.M.Crespi, & W.K.Wong, (2012). Comparison of methods for estimating the intraclass correlation coefficient for binary responses in cancer prevention cluster randomized trials. Contemporary Clinical Trials, 33, 869–880. doi:10.1016/j.cct.2012.05.004
80. S.L.Zeger, & K.Y.Liang, (1986). Longitudinal data analysis for discrete and continuous outcomes. Biometrics, 42, 121–130. doi:10.2307/2531248
81. S.L.Zeger, K.Y.Liang, & P.S.Albert, (1988). Models for longitudinal data:A generalized estimating equation approach. Biometrics, 44, 1049–1060. doi:10.2307/2531734
82. D.M.Zucker, O.Lieberman, & O.Manor, (2000). Improved small sample inference in the mixed linear model:Bartlett correction and adjusted likelihood. Journal of the Royal Statistical Society:Series B, 62, 827–838. doi:10.1111/1467-9868.00267
83. M.J.Zyphur, F.L.Oswald, & D.E.Rupp, (2015). Rendezvous overdue:Bayes analysis meets organizational research. Journal of Management, 41, 387–389. doi:10.1177/0149206314549252