Confidence interval estimation for standardized effect sizes in multilevel and latent growth modeling

Journal of Consulting and Clinical Psychology

Volumn 83, Issue 1, 2015, Pages 157-168

  Feingold, Alan ^a  

^a OREGON SOCIAL LEARNING CENTER   (United States)

Author keywords

Confidence intervals; Effect sizes; Hierarchical linear models; Latent growth models; Multilevel analysis

Indexed keywords

ARTICLE; CONFIDENCE INTERVAL; EFFECT SIZE; MONTE CARLO METHOD; MULTILEVEL ANALYSIS; SAMPLE SIZE; SAMPLING ERROR; SEX DIFFERENCE; STATISTICAL BIAS; THERAPY EFFECT; ADOLESCENT; CLINICAL TRIAL (TOPIC); HUMAN; PROCEDURES; REGRESSION ANALYSIS; STATISTICAL MODEL; STATISTICS AND NUMERICAL DATA;

ADOLESCENT; CLINICAL TRIALS AS TOPIC; CONFIDENCE INTERVALS; HUMANS; MODELS, STATISTICAL; MONTE CARLO METHOD; MULTILEVEL ANALYSIS; REGRESSION ANALYSIS; SAMPLE SIZE;

EID: 84925794253     PISSN: 0022006X     EISSN: 19392117     Source Type: Journal    
DOI: 10.1037/a0037721     Document Type: Article

References (71)

1. Aderka, I. M., Gillihan, S. J., McLean, C. P., & Foa, E. B. (2013). The relationship between posttraumatic and depressive symptoms during prolonged exposure with and without cognitive restructuring for the treatment of posttraumatic stress disorder. Journal of Consulting and Clinical Psychology, 81, 375-382. doi:10.1037/a0031523
2. Agresti, A. (2002). Categorical data analysis (2nd ed.). New York, NY: Wiley. doi:10.1002/0471249688
3. Algina, J., & Kesselman, H. J. (2008). Approximate confidence intervals for effect sizes. Educational and Psychological Measurement, 68, 233-244.
4. American Psychological Association. (2009). Publication manual of the American Psychological Association (6th ed.) Washington, DC: Author.
5. APA Publications and Communications Board Working Group on Journal Article Reporting Standards. (2008). Reporting standards for research in psychology: Why do we need them? What might they be? American Psychologist, 63, 839-851. doi:10.1037/0003-066X.63.9.839
6. Arch, J. J., Eifert, G. H., Davies, C., Vilardaga, J. C. P., Rose, R. D., & Craske, M. G. (2012). Randomized clinical trial of cognitive behavioral therapy (CBT) versus acceptance and commitment therapy (ACT) for mixed anxiety disorders. Journal of Consulting and Clinical Psychology, 80, 750-765. doi:10.1037/a0028310
7. Baguley, T. (2009). Standardized or simple effect size: What should be reported? British Journal of Psychology, 100, 603-617. doi:10.1348/000712608X377117
8. Becker, B. J. (1988). Synthesizing standardized mean-change scores. British Journal of Mathematical and Statistical Psychology, 41, 257-278. doi:10.1111/j.2044-8317.1988.tb00901.x
9. Blanton, H., & Jaccard, J. (2006). Arbitrary metrics in psychology. American Psychologist, 61, 27-41. doi:10.1037/0003-066X.61.1.27
10. Bollen, K. A., & Curran, P. J. (2006). Latent curve models: A structural equation perspective. Hoboken, NJ: Wiley.
11. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. New York, NY: Wiley. doi:10.1002/9780470743386
12. Chaffin, M., Funderburk, B., Bard, D., Valle, L. A., & Gurwitch, R. (2011). A combined motivation and parent-child interaction therapy package reduces child welfare recidivism in a randomized dismantling field trial. Journal of Consulting and Clinical Psychology, 79, 84-95. doi:10.1037/a0021227
13. Chen, H., Cohen, P., & Chen, S. (2010). How big is a big odds ratio? Interpreting the magnitudes of odds ratios in epidemiological studies. Communications in Statistics-Simulation and Computation, 39, 860-864. doi:10.1080/03610911003650383
14. Cheung, M. W. (2009). Comparison of methods for constructing confidence intervals of standardized indirect effects. Behavior Research Methods, 41, 425-438. doi:10.3758/BRM.41.2.425
15. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
16. Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral analysis (3rd ed.). Mahwah, NJ: Erlbaum.
17. Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.). (2009). The handbook of research synthesis (2nd ed.). New York, NY: Russell Sage Foundation.
18. Cumming, G. (2013). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. New York, NY: Routledge.
19. Curran, P. J. (2003). Have multilevel models been structural equation models all along? Multivariate Behavioral Research, 38, 529-569. doi:10.1207/s15327906mbr3804-5
20. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B, 39, 1-38.
21. Dunlap, W. P., Cortina, J. M., Vaslow, J. B., & Burke, M. J. (1996). Meta-analysis of experiments with matched groups or repeated measures designs. Psychological Methods, 1, 170-177. doi:10.1037/1082-989X.1.2.170
22. Efron, B., & Tibshirani, R. (1993). An introduction to the bootstrap. Boca Raton, FL: Chapman & Hall. doi:10.1007/978-1-4899-4541-9
23. Elliott, D. S., Huizinga, D., & Menard, S. (1989). Multiple problem youth: Delinquency, substance use, and mental health problems. New York, NY: Springer-Verlag. doi:10.1007/978-1-4613-9637-6
24. Feingold, A. (1995). The additive effects of differences in central tendency and variability are important in comparisons between groups. American Psychologist, 50, 5-13. doi:10.1037/0003-066X.50.1.5
25. Feingold, A. (2009). Effect sizes for growth-modeling analysis for controlled clinical trials in the same metric as for classical analysis. Psychological Methods, 14, 43-53. doi:10.1037/a0014699
26. Feingold, A. (2013). A regression framework for effect size assessments in longitudinal modeling of group differences. Review of General Psychology, 17, 111-121. doi:10.1037/a0030048
27. Feingold, A., Oliveto, A., Schottenfeld, R., & Kosten, T. R. (2002). Utility of crossover designs in clinical trials: Efficacy of desipramine vs. placebo in opioid-dependent cocaine abusers. American Journal on Addictions, 11, 111-123. doi:10.1080/10550490290087884
28. Feingold, A., Tiberio, S., & Capaldi, D. M. (2014). New approaches for examining associations with latent categorical variables: Applications to substance abuse and aggression. Psychology of Addictive Behaviors, 28, 257-267. doi:10.1037/a0031487
29. Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3-8. doi:10.3102/0013189X005010003
30. Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Thousand Oaks, CA: Sage.
31. Goldstein, H. (2011). Multilevel statistical models (4th ed.). Hoboken, NJ: Wiley.
32. Grissom, R. J. (2000). Homogeneity of variance in clinical data. Journal of Consulting and Clinical Psychology, 68, 155-165. doi:10.1037/0022-006X.68.1.155
33. Grissom, R. J., & Kim, J. J. (2012). Effect sizes for research: Univariate and multivariate applications (2nd ed.) New York, NY: Routledge.
34. Gueorguieva, R., & Krystal, J. H. (2004). Move over ANOVA: Progress in analyzing repeated-measures data and its reflection in papers published in the Archives of General Psychiatry. Archives of General Psychiatry, 61, 310-317. doi:10.1001/archpsyc.61.3.310
35. Hedeker, D., & Gibbons, R. D. (2006). Longitudinal data analysis. Hoboken, NJ: Wiley.
36. Hedges, L. V. (2007). Effect sizes in cluster-randomized designs. Journal of Educational and Behavioral Statistics, 32, 341-370. doi:10.3102/1076998606298043
37. Hedges, L. V. (2009). Effect sizes in nested designs. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis (2nd ed., pp. 337-356). New York, NY: Russell Sage Foundation.
38. Hedges, L. V., & Olkin, L. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
39. Hedges, L. V., Pustejovskya, J. E., & Shadish, W. R. (2012). A standardized mean difference effect size for single case designs. Research Synthesis Methods, 3, 224-239. doi:10.1002/jrsm.1052
40. Hodges, J. L., & Lehmann, E. L. (2005). Basic concepts of probability and statistics (2nd ed.). Philadelphia, PA: Society for Industrial and Applied Mathematics. doi:10.1137/1.9780898719123
41. Hosmer, D. W., & Lemeshow, S. (2000). Applied logistic regression (2nd ed.) New York, NY: Wiley. doi:10.1002/0471722146
42. Kelley, K., & Preacher, K. J. (2012). On effect size. Psychological Methods, 17, 137-152. doi:10.1037/a0028086
43. Kelley, K., & Rausch, J. R. (2011). Sample size planning for longitudinal models: Accuracy in parameter estimation for polynomial change parameters. Psychological Methods, 16, 391-405. doi:10.1037/a0023352
44. Kerr, D. C. R., DeGarmo, D. S., Leve, L. D., & Chamberlain, P. (2014). Juvenile justice girls' depressive symptoms and suicidal ideation 9 years after multidimensional treatment foster care. Journal of Consulting and Clinical Psychology, 82, 684-693. doi:10.1037/a0036521
45. Lau, R. S., & Cheung, G. W. (2012). Estimating and comparing specific mediation models in complex latent variable models. Organizational Research Methods, 15, 3-16. doi:10.1177/1094428110391673
46. Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
47. Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing data (2nd ed.). Hoboken, NJ: Wiley.
48. Ljótsson, B., Hesser, H., Andersson, E., Lindfors, P., Hursti, T., Rück, C.,. . . Hedman, E. (2013). Mechanisms of change in an exposure-based treatment for irritable bowel syndrome. Journal of Consulting and Clinical Psychology, 81, 1113-1126. doi:10.1037/a0033439
49. Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1, 86-92.
50. MacKinnon, D. P., Lockwood, C. M., & Williams, J. (2004). Confidence limits for the indirect effect: Distribution of the product and resampling methods. Multivariate Behavioral Research, 39, 99 -128. doi: 10.1207/s15327906mbr3901-4
51. McCulloch, C. E., & Searle, S. R. (2001). Generalized, linear, and mixed models. New York, NY: Wiley.
52. Morris, S. B. (2008). Estimating effect sizes from pretest-posttest-control group designs. Organizational Research Methods, 11, 364-386. doi: 10.1177/1094428106291059
53. Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7, 105-125. doi:10.1037/1082-989X.7.1.105
54. Muthén, B. O., & Muthén, L. K. (2000). The development of heavy drinking and alcohol-related problems from ages 18 to 37 in a U.S. national sample. Journal of Studies on Alcohol, 61, 290-300.
55. Muthén, L. K., & Muthén, B. O. (2002). How to use a Monte Carlo study to decide on sample size and determine power. Structural Equation Modeling, 9, 599-620. doi:10.1207/S15328007SEM0904-8
56. Muthén, L. K., & Muthén, B. O. (2012). Mplus user's guide (7th ed.). Los Angeles, CA: Muthén and Muthén.
57. Nunnally, J. (1978). Psychometric methods. New York, NY: McGraw-Hill.
58. Odgaard, E. C., & Fowler, R. L. (2010). Confidence intervals for effect sizes: Compliance and clinical significance in the Journal of Consulting and Clinical Psychology. Journal of Consulting and Clinical Psychology, 78, 287-297. doi:10.1037/a0019294
59. Olejnik, S., & Algina, J. (2000). Measures of effect size for comparative studies: Applications, interpretations, and limitations. Contemporary Educational Psychology, 25, 241-286. doi:10.1006/ceps.2000.1040
60. Olsen, M. K., & Schafer, J. L. (2001). A two-part random-effects model for semicontinuous longitudinal data. Journal of the American Statistical Association, 96, 730-745. doi:10.1198/016214501753168389
61. Preacher, K. J., Wichman, A. L., MacCallum, R. C., & Briggs, N. E. (2008). Latent growth modeling. Los Angeles, CA: Sage.
62. Raudenbush, S. W. (1995). Hierarchical linear models to study the effects of social context on development. In J. M. Gottman (Ed.) The analysis of change (pp. 165-201). Mahwah, NJ: Erlbaum.
63. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
64. Raudenbush, S. W., & Liu, X. (2001). Effects of study duration, frequency of observation, and sample size on power in studies of group differences in polynomial change. Psychological Methods, 6, 387- 401. doi: 10.1037/1082-989X.6.4.387
65. Rosenthal, J. A. (1996). Qualitative descriptors of strength of association and effect size. Journal of Social Service Research, 21, 37-59.
66. Rosenthal, R., Rosnow, R. L., & Rubin, D. B. (2000). Contrasts and effect sizes in behavioral research: A correlational approach. Cambridge, England: Cambridge University Press.
67. Safren, S. A., O'Cleirigh, C. M., Bullis, J. R., Otto, M. W., Stein, M. D., & Pollack, M. H. (2012). Cognitive behavioral therapy for adherence and depression (CBT-AD) in HIV-infected injection drug users: A randomized controlled trial. Journal of Consulting and Clinical Psychology, 80, 404-415. doi:10.1037/a0028208
68. Shadish, W. R., & Haddock, C. K. (2009). Combining estimates of effect sizes. In H. Cooper, L. V. Hedges, & J. C. Valentine (eds.), The handbook of research synthesis (2nd ed., pp. 257-277). New York, NY: Russell Sage.
69. Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. New York, NY: Oxford University Press.
70. Steiger, J. H. (2004). Beyond the F test: Effect size confidence intervals and tests of close fit in the analysis of variance and contrast analysis. Psychological Methods, 9, 164 -182. doi:10.1037/1082-989X.9.2.164
71. Twohig, M. P., Hayes, S. C., Plumb, J. C., Pruitt, L. D., Collins, A. B., Hazlett-Stevens, H., & Woidneck, M. R. (2010). A randomized clinical trial of acceptance and commitment therapy versus progressive relaxation training for obsessive-compulsive disorder. Journal of Consulting and Clinical Psychology, 78, 705-716. doi:10.1037/a0020508