Examining the rule of thumb of not using multilevel modeling: The "design effect smaller than two" rule

Journal of Experimental Education

Volumn 83, Issue 3, 2015, Pages 423-438

  Lai, Mark H C ^a   Kwok, Oi Man ^a  

^a TEXAS A AND M UNIVERSITY   (United States)

Author keywords

clustering; design effects; intraclass correlation; multilevel; simulation studies

Indexed keywords

EID: 84929045689     PISSN: 00220973     EISSN: 19400683     Source Type: Journal    
DOI: 10.1080/00220973.2014.907229     Document Type: Article

References (40)

1. Beretvas, S.N. (2010). Cross-classified and multiple-membership models. In J.J. Hox. & J.K. Roberts. (Eds.), Handbook of advanced multilevel analysis (pp. 313-334). New York, NY: Routledge.
2. Bonnet, M., Goossens, F.A., Schuengel, C. (2011). Parental strategies and trajectories of peer victimization in 4 to 5 year olds. Journal of School Psychology, 49, 385-398. doi: 10.1016/j.jsp.2011.04.002
3. Bouman, T., van der Meulen, M., Goossens, F.A., Olthof, T., Vermande, M.M., Aleva, E.A. (2012). Peer and self-reports of victimization and bullying: Their differential association with internalizing problems and social adjustment. Journal of School Psychology, 50, 759-774. doi: 10.1016/j.jsp.2012.08.004
4. Browne, W.J., Draper, D. (2000). Implementation and performance issues in the Bayesian and likelihood fitting of multilevel models. Computational Statistics, 15, 391-420. doi: 10.1007/s001800000041
5. Clarke, P. (2008). When can group level clustering be ignored? Multilevel models versus single-level models with sparse data. Journal of Epidemiology and Community Health, 62, 752-758. doi: 10.1136/jech.2007.060798
6. Corte, C., Zucker, R.A. (2008). Self-concept disturbances: Cognitive vulnerability for early drinking and early drunkenness in adolescents at high risk for alcohol problems. Addictive Behaviors, 33, 1282-1290. doi: 10.1016/j.addbeh.2008.06.002
7. De Los Reyes, A., Youngstrom, E.A., Swan, A.J., Youngstrom, J.K., Feeny, N.C., Findling, R.L. (2011). Informant discrepancies in clinical reports of youths and interviewers impressions of the reliability of informants. Journal of Child and Adolescent Psychopharmacology, 21, 417-24. doi: 10.1089/cap.2011.0011
8. Deng, S., Lopez, V., Roosa, M.W., Ryu, E., Burrell, G.L., Tein, J.-Y., Crowder, S. (2006). Family processes mediating the relationship of neighborhood disadvantage to early adolescent internalizing problems. The Journal of Early Adolescence, 26, 206-231. doi: 10.1177/0272431605285720
9. Enders, C.K., Tofighi, D. (2007). Centering predictor variables in cross-sectional multilevel models: A new look at an old issue. Psychological Methods, 12, 121-138. doi: 10.1037/1082-989X.12.2.121
10. Fuentes, M., Hart-Johnson, T., Green, C.R. (2007). The association among neighborhood socioeconomic status, race and chronic pain in Black and White older adults. Journal of the National Medical Association, 99, 1160-1169.
11. Goldstein, H., Spiegelhalter, D.J. (1996). League tables and their limitations: Statistical issues in comparisons of institutional performance. Journal of the Royal Statistical Society. Series A (Statistics in Society), 159, 385-443. doi: 10.2307/2983325
12. Greene, W.H. (2003). Econometric analysis. (5th ed.). Upper Saddle River, NJ: Prentice Hall.
13. Hong, S., You, S. (2012). Understanding Latino children's heterogeneous academic growth trajectories: Latent growth mixture modeling approach. The Journal of Educational Research, 105, 235-244. doi: 10.1080/00220671.2011.584921
14. Hoogland, J.J., Boomsma, A. (1998). Robustness studies in covariance structure modeling: An overview and a meta-analysis. Sociological Methods & Research, 26, 329-367. doi: 10.1177/0049124198026003003
15. Hox, J.J. (2010). Multilevel analysis: Techniques and applications. (2nd ed.). New York, NY: Routledge.
16. Hox, J.J., Maas, C.J. M. (2002). Sample sizes for multilevel modeling. In J. Blasius., J.J. Hox., E. de Leeuw., P. Schmidt. (Eds.), Social science methodology in the new millennium: Proceedings of the fifth international conference on logic and methodology. (2nd expanded ed.). Opladen, RG: Leske + Budrich.
17. Jester, J.M., Nigg, J.T., Buu, A., Puttler, L.I., Glass, J.M., Heitzeg, M.M., ., Zucker, R.A. (2008). Trajectories of childhood aggression and inattention/hyperactivity: Differential effects on substance abuse in adolescence. Journal of the American Academy of Child and Adolescent Psychiatry, 47, 1158-1165. doi: 10.1097/CHI.0b013e3181825a4e
18. Kilian, B., Hofer, M., Kuhnle, C. (2010). Value orientations as determinants and outcomes of conflicts between on-task and off-task actions in the classroom. Learning and Individual Differences, 20, 501-506. doi: 10.1016/j.lindif.2010.03.003
19. Kish, L. (1965). Survey sampling. New York, NY: Wiley.
20. Kreft, G.G., de Leeuw, E.D. (1988). The see-saw effect: A multilevel problem? Quality and Quantity, 22, 127-137. doi: 10.1007/bf00223037
21. Kwok, O., West, S.G., Green, S.B. (2007). The impact of misspecifying the within-subject covariance structure in multiwave longitudinal multilevel models: A Monte Carlo study. Multivariate Behavioral Research, 42, 557-592. doi: 10.1080/00273170701540537
22. LaVange, L.M., Stearns, S.C., Lafata, J.E., Koch, G.G., Shah, B.V. (1996). Innovative strategies using SUDAAN for analysis of health surveys with complex samples. Statistical Methods in Medical Research, 5, 311-329. doi: 096228029600500306
23. Linnenbrink-Garcia, L., Rogat, T.K., Koskey, K.L. K. (2011). Affect and engagement during small group instruction. Contemporary Educational Psychology, 36, 13-24. doi: 10.1016/j.cedpsych.2010.09.001
24. Ly, J., Zhou, Q., Chu, K., Chen, S.H. (2012). Teacher-child relationship quality and academic achievement of Chinese American children in immigrant families. Journal of School Psychology, 50, 535-553.
25. Maas, C.J. M., Hox, J.J. (2004). The influence of violations of assumptions on multilevel parameter estimates and their standard errors. Computational Statistics & Data Analysis, 46, 427-440. doi: 10.1016/j.csda.2003.08.006
26. Maas, C.J. M., Hox, J.J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1, 86-92. doi: 10.1027/1614-1881.1.3.86
27. Muthén, B.O. (1994). Multilevel covariance structure analysis. Sociological Methods & Research, 22, 376-398. doi: 10.1177/0049124194022003006
28. Muthén, B.O., Satorra, A. (1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267-316. doi: 10.2307/271070
29. Muthén, L.K., Muthén, B.O. (1998-2012). Mplus user's guide. (7th ed.). Los Angeles, CA: Authors.
30. Peugh, J.L. (2010). A practical guide to multilevel modeling. Journal of School Psychology, 48, 85-112. doi: 10.1016/j.jsp.2009.09.002
31. Qureshi, I., Fang, Y. (2011). Socialization in open source software projects: A growth mixture modeling approach. Organizational Research Methods, 14, 208-238.
32. Raudenbush, S.W., Bryk, A.S. (2002). Hierarchical linear models: Applications and data analysis methods. (2nd ed.). Thousand Oaks, CA: Sage.
33. Rust, K.F., Rao, J.N. K. (1996). Variance estimation for complex surveys using replication techniques. Statistical Methods in Medical Research, 5, 283-310. doi: 096228029600500305
34. Semke, C.A., Garbacz, S.A., Kwon, K., Sheridan, S.M., Woods, K.E. (2010). Family involvement for children with disruptive behaviors: The role of parenting stress and motivational beliefs. Journal of School Psychology, 48, 293-312. doi: 10.1016/j.jsp.2010.04.001
35. Snijders, T.A. B., Bosker, R.J. (2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling. (2nd ed.). London, UK: Sage.
36. Thomas, S.L., Heck, R.H. (2001). Analysis of large-scale secondary data in higher education research: Potential perils associated with complex sampling designs. Research in Higher Education, 42, 517-540. doi: 10.1023/A:1011098109834
37. Turner, J.C., Midgley, C., Meyer, D.K., Gheen, M., Anderman, E.M., Kang, Y., Patrick, H. (2002). The classroom environment and students reports of avoidance strategies in mathematics: A multimethod study. Journal of Educational Psychology, 94, 88-106. doi: 10.1037//0022-0663.94.1.88
38. Von Grünigen, R., Kochenderfer-Ladd, B., Perren, S., Alsaker, F.D. (2012). Links between local language competence and peer relations among Swiss and immigrant children: The mediating role of social behavior. Journal of School Psychology, 50, 195-213. doi: 10.1016/j.jsp.2011.09.005
39. Wagner, U., Christ, O., Pettigrew, T.F., Stellmacher, J., Wolf, C. (2006). Prejudice and minority proportion: Contact instead of threat effects. Social Psychology Quarterly, 69, 380-390. doi: 10.1177/019027250606900406
40. Wong, M.M., Nigg, J.T., Zucker, R.A., Puttler, L.I., Fitzgerald, H.E., Jester, J.M., ., Adams, K. (2006). Behavioral control and resiliency in the onset of alcohol and illicit drug use: A prospective study from preschool to adolescence. Child Development, 77, 1016-1033. doi: 10.1111/j.1467-8624.2006.00916.x