Beyond cohen's d: Alternative effect size measures for between-subject designs

Journal of Experimental Education

Volumn 82, Issue 1, 2014, Pages 22-50

  Peng, Chao Ying Joanne ^a   Chen, Li Ting ^a  

^a INDIANA UNIVERSITY   (United States)

Author keywords

effect size; meta analysis; robust; statistical assumptions; typology

Indexed keywords

EID: 84887826735     PISSN: 00220973     EISSN: 19400683     Source Type: Journal    
DOI: 10.1080/00220973.2012.745471     Document Type: Article

References (102)

1. Algina, J., Keselman, H. J., and, Penfield, R. D. 2005. An alternative to Cohen's standardized mean difference effect size: A robust parameter and confidence interval in the two independent groups case. Psychological Methods, 10: 317-328. doi: 10.1037/1082-989X.10.3.317 (Pubitemid 41527952)
2. Alhija, F. N.-A., and, Levy, A. 2009. Effect size reporting practices in published articles. Educational and Psychological Measurement, 69: 245-265. doi: 10.1177/0013164408315266
3. American Educational Research Association. 2006. Standards for reporting on empirical social science research in AERA publications. Educational Researcher, 35: 33-40. doi: 10.3102/0013189X035006033
4. American Psychological Association. 2010. Publication manual of the American Psychological Association (6th ed.), Washington, DC: Author.
5. Andersen, M. B., McCullagh, P., and, Wilson, G. J. 2007. But what do the numbers really tell us?: Arbitrary metrics and effect size reporting in sport psychology research. Journal of Sport & Exercise Psychology, 29: 664-672. Retrieved from http://journals.humankinetics.com/jsep
6. APA Publications and Communications Board Working Group on Journal Articles Reporting Standards. 2008. Reporting standards for research in psychology: Why do we need them? What might they be?. American Psychologist, 63: 839-851. doi: 10.1037/0003-066X.63.9.839
7. Bond, C. F. Jr., Wiitala, W. L., and, Richard, F. D. 2003. Meta-analysis of raw mean differences. Psychological Methods, 8: 406-418. doi: 10.1037/1082-989X.8.4.406 (Pubitemid 38061515)
8. Bonett, D. G. 2008. Confidence intervals for standardized linear contrasts of means. Psychological Methods, 13: 99-109. doi: 10.1037/1082-989X. 13.2.99
9. Brunner, E., and, Puri, M. L. 2001. Nonparametric methods in factorial designs. Statistical Papers, 42: 1-52. doi: 10.1007/s003620000039
10. Burr, I. W., and, Cislak, P. J. 1968. On a general system of distributions: I. Its curve-shape characteristics: II. The sample median. Journal of American Statistical Association, 63: 627-635. Retrieved from http://www.amstat.org/publications/jasa.cfm
11. Cliff, N. 1993. Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological Bulletin, 114: 494-509. doi: 10.1037/0033-2909.114.3. 494
12. Cliff, N. 1996. Ordinal methods for behavioral data analysis, Mahwah, NJ: Erlbaum.
13. Cohen, J. 1969. Statistic power analysis in the behavioral sciences, New York, NY: Academic Press.
14. Cohen, J. 1988. Statistic power analysis in the behavioral sciences, Hillsdale, NJ: Erlbaum.
15. Cohen, J. 1994. The earth is round (p <.05). American Psychologist, 49: 997-1003. doi: 10.1037/0003-066X.49.12.997
16. Cooper, H., Hedges, L. V., and, Valentine, J. C. 2009. The handbook of research synthesis and meta-analysis (2nd ed.), New York, NY: Russell Sage Foundation.
17. Delaney, H. D., and, Vargha, A. 2002. Comparing several robust tests of stochastic equality with ordinally scaled variables and small to moderate sized samples. Psychological Methods, 7: 485-503. doi: 10.1037/1082-989X.7.4.485
18. Dunleavy, E. M., Barr, C. D., Glenn, D. M., and, Miller, K. R. 2006. Effect size reporting in applied psychology: How are we doing?. The Industrial-Organizational Psychologist, 43: 29-37. Retrieved from http://www.openj-gate.com/browse/Archive.aspx?year =2009&Journal-id=102632
19. Efron, B., and, Tibshirani, R. J. 1993. An introduction to the bootstrap, New York, NY: Chapman & Hall.
20. Ellison, B. E. 1964. Two theorems for inferences about the normal distribution with applications in acceptance sampling. Journal of American Statistical Association, 59: 89-95. Retrieved from http://www.amstat.org/ publications/jasa.cfm
21. Fidler, F., Cumming, G., Thomason, N., Pannuzzo, D., Smith, J., Fyffe, P., and, Schmitt, R. 2005. Evaluating the effectiveness of editorial policy to improve statistical practice: The case of the Journal of Consulting and Clinical Psychology. Journal of Consulting and Clinical Psychology, 73: 136-143. doi: 10.1037/0022-006X.73.1.136 (Pubitemid 40223901)
22. Glass, G. V. 1976. Primary, secondary, and meta-analysis research. Educational Researcher, 5: 3-8. doi: 10.3102/0013189X005010003
23. Glass, G. V., McGaw, B., and, Smith, M. L. 1981. Meta-analysis in social research, Newbury Park, CA: Sage.
24. Grissom, R. J. 1994. Probability of the superior outcome of one treatment over another. Journal of Applied Psychology, 79: 314-316. doi: 10.1037/0021-9010.79.2.31
25. Grissom, R. J. 2000. Heterogeneity of variance in clinical data. Journal of Consulting and Clinical Psychology, 68: 155-165. doi: 10.1037/0022-006X.68.1. 155 (Pubitemid 30120960)
26. Grissom, R. J., and, Kim, J. J. 2001. Review of assumptions and problems in the appropriate conceptualization of effect size. Psychological Methods, 6: 135-146. doi: 10.1037/1082-989x.6.2.135 (Pubitemid 33604035)
27. Grissom, R. J., and, Kim, J. J. 2005. Effect sizes for research: A broad practical approach, Mahwah, NJ: Erlbaum.
28. Grissom, R. J., and, Kim, J. J. 2012. Effect sizes for research: Univariate and multivariate applications (2nd ed.), New York, NY: Routledge.
29. Harrison, J., Thompson, B., and, Vannest, K. J. 2009. Interpreting the evidence for effective interventions to increase the academic performance of students with ADHD: Relevance of the statistical significance controversy. Review of Educational Research, 79: 740-775. doi: 10.3102/0034654309331516
30. Hedges, L. V. 1981. Distributional theory for Glass's estimator of effect size and related estimators. Journal of Educational Statistics, 6: 107-128. doi: 10.2307/1164588
31. Hedges, L. V. 1982. Estimation of effect size from a series of independent experiments. Psychological Bulletin, 92: 490-499. doi: 10.1037/0033-2909.92.2.490
32. Hedges, L. V., and, Olkin, I. 1984. Nonparametric estimators of effect size in meta-analysis. Psychological Bulletin, 96: 573-580. doi: 10.1037/0033-2909.96.3.573
33. Hedges, L. V., and, Olkin, I. 1985. Statistical methods for meta-analysis, Orlando, FL: Academic Press.
34. Hess, M. R., and, Kromrey, J. D. Robust confidence intervals for effect sizes: A comparative study of Cohen's d and Cliff's delta under non-normality and heterogeneous variance. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA
35. Hess, B., Olejnik, S., and, Huberty, C. J. 2001. The efficacy of two improvement-over-chance effect sizes for two-group univariate comparisons under variance heterogeneity and nonnormality. Educational and Psychological Measurement, 61: 909-936. doi: 10.1177/00131640121971572
36. Hinkle, D. E. 2003. Applied Statistics for the Behavioral Sciences, Boston, MA: Houghton Mifflin.
37. Hogarty, K. Y., and, Kromrey, J. D. We've been reporting some effect sizes: Can you guess what they mean?. Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA
38. Huberty, C. J. 2002. A history of effect size indices. Educational and Psychological Measurement, 62: 227-240. doi: 10.1177/0013164402062002002 (Pubitemid 36688798)
39. Huberty, C. J., and, Lowman, L. L. 2000. Group overlap as a basis for effect size. Educational and Psychological Measurement, 60: 543-563. doi: 10.1177/0013164400604004
40. Huynh, C. L. A unified approach to the estimation of effect size in meta-analysis. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco. (ERIC Document Reproduction Service No. 306 248)
41. Jacobson, N., Follette, W. C., and, Revenstorf, D. 1984. Psychotherapy outcome research: Methods for reporting variability and evaluating clinical significance. Behavior Therapy, 15: 336-352. doi: 10.1016/S0005-7894(84)80002-7 (Pubitemid 14034170)
42. Jacobson, N., and, Truax, P. 1991. Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59: 12-19. doi: 10.1037//0022-006X.59.1.12
43. James, G. S. 1951. The comparison of several groups of observations when the ratios of the population variances are unknown. Biometrika, 38: 234-329. Retrieved from http://biomet.oxfordjournals.org
44. James, G. S. 1954. Tests of linear hypothesis in univariate and multivariate analysis when the ratios of the populations variances are unknown. Biometrika, 41: 19-43. Retrieved from http://biomet.oxfordjournals.org
45. Kelley, K. 2005. The effects of nonnormal distributions on confidence intervals around the standardized mean difference: Bootstrap and parametric confidence intervals. Educational and Psychological Measurement, 65: 51-69. doi: 10.1177/0013164404264850 (Pubitemid 40176606)
46. Kelley, K., and, Preacher, K. J. 2012. On effect size. Psychological Methods, 17: 137-152. doi: 10.1037/a0028086
47. Keselman, H. J., Algina, J., Lix, L. M., Wilcox, R. R., and, Deering, K. N. 2008. A generally robust approach for testing hypotheses and setting confidence intervals for effect sizes. Psychological Methods, 13: 110-129. doi: 10.1037/1082-989X.13.2.110
48. Keselman, H. J., Huberty, C. J., Lix, L. M., Olejnik, S., Cribbie, R. A., Donahue, B., and,.Levin, J. R. 1998. Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analyses. Review of Educational Research, 68: 350-386. doi: 10.3102/00346543068003350 (Pubitemid 128376783)
49. Kirk, R. E. 1996. Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56: 746-759. doi: 10.1177/0013164496056005002
50. Kirk, R. E. 2007. Effect magnitude: A different focus. Journal of Statistical Planning and Inference, 137: 1634-1646. doi: 10.1016/j.jspi.2006.09. 011
51. Kline, R. B. 2004. " Beyond significant testing: Reforming data analysis methods in behavioral research ". Washington, DC: American Psychological Association.
52. Kraemer, H. C., and, Andrews, G. 1982. A nonparametric technique for meta-analysis effect size calculation. Psychological Bulletin, 91: 404-412. doi: 10.1037/0033-2909.91.2.404
53. Kraemer, H. C., and, Kupfer, D. J. 2006. Size of treatment effects and their importance to clinical research and practice. Biological Psychiatry, 59: 990-996. doi: 10.1016/j.biopsych.2005.09.014 (Pubitemid 43849650)
54. Kramer, S., and, Rosenthal, R. 1999. " Effect sizes and significance levels in small-sample research ". In Statistical strategies for small sample research, Edited by: Hoyle, R. 59-79. Thousand Oaks, CA: Sage.
55. Kromrey, J. D., and, Coughlin, K. B. ROBUST-ES: A SAS macro for computing robust estimates of effect size. Paper presented at the annual meeting of the SouthEast SAS Users Group, Hilton Head, SC,. Retrieved from http://analytics. ncsu.edu/sesug/2007/PO19.pdf
56. Levy, P. 1967. Substantive significance of significant differences between groups. Psychological Bulletin, 67: 37-40. doi: 10.1037/h0020415
57. MacGrath, R. E., and, Meyer, G. J. 2006. When effect sizes disagree: The case of r and d. Psychological Methods, 11: 386-401. doi: 10.1037/1082-989X.11. 4.386 (Pubitemid 44950900)
58. Matthews, M. S., Gentry, M., McCoach, D. B., Worrell, F. C., Matthews, D., and, Dixon, F. 2008. Evaluating the state of a field: Effect size reporting in gifted education. The Journal of Experimental Education, 77: 55-65. doi: 10.3200/JEXE.77.1.55-68
59. Maxwell, S. E., Camp, C. J., and, Arvey, R. D. 1981. Measures of strength of association: A comparative examination. Journal of Applied Psychology, 66: 525-534. doi: 10.1037/0021-9010.66.5.525
60. McGraw, K. O., and, Wong, S. P. 1992. A common language effect size statistic. Psychological Bulletin, 111: 361-365. doi: 10.1037/0033-2909.111.2. 361
61. Meline, T., and, Wang, B. 2004. Effect reporting practices in AJSLP and other ASHA journals, 1999-2003. American Journal of Speech-Language Pathology, 13: 202-207. Retrieved from http://ajslp.asha.org (Pubitemid 39330077)
62. Meyers, L. S., Gamst, G., and, Guarino, A. J. 2006. Applied multivariate research: Design and interpretation, Thousand Oaks, CA: Sage.
63. Murphy, B. P. 1976. Comparison of some two sample means tests by simulation. Communications in Statistics B: Simulations and Computation, 5: 23-32. doi: 10.1080/03610917608812004
64. Nakagawa, S., and, Cuthill, I. C. 2007. Effect size, confidence interval, and statistical significance: A practical guide for biologists. Biological Reviews, 82: 591-605. doi: 10.1111/j.1469-185X.2007.00027.x (Pubitemid 47609922)
65. Natesan, P., and, Thompson, B. 2007. Extending improvement-over change I-index effect size simulation studies to cover some small-sample cases. Educational and Psychological Measurement, 67: 59-72. doi: 10.1177/ 001364406292028
66. Oakes, M. 1986. Statistical inference: A commentary for the social and behavioral sciences, New York, NY: Wiley.
67. Odgaard, E. C., and, Fowler, R. L. 2010. Confidence intervals for effect sizes: Compliance and clinical significance in the Journal of Consulting and Clinical Psychology. Journal of Consulting and Clinical Psychology, 78: 287-297. doi: 10.1037/a0019294
68. Olejnik, S., and, Algina, J. 2000. Measures of effect size for comparative studies: Applications, interpretations, and limitations. Contemporary Educational Psychology, 25: 241-286. doi: 10.1006/ceps.2000.1040
69. Osborne, J. W. 2008. Sweating the small stuff in educational psychology: How effect size and power reporting failed to change from 1969 to 1999, and what that means for the future of changing practices. Educational Psychology, 28: 151-160. doi: 10.1080/01443410701491718
70. Peng, C.-Y. J., Chen, L.-T., Chiang, H.-M., and, Chiang, Y.-C. 2013. The impact of APA and AERA guidelines an effect size reporting. Educational Psychology Review, 25: 157-209. doi: 10.1007/s10648-013-9218-2
71. Pratt, J. W. 1964. Obustness [sic] of some procedures for the two-sample location problem. Journal of American Statistical Association, 59: 665-680. Retrieved from http://www.amstat.org/publications/jasa.cfm
72. Pratt, J. W., and, Gibbons, J. D. 1981. Concepts of nonparametric theory, New York, NY: Springer-Verlag.
73. Rosenthal, R. 1994. " Parametric measures of effect size ". In The handbook of research synthesis, Edited by: Cooper, H. and Hedges, L. V. 231-244. New York, NY: Russell Sage Foundation.
74. Ruscio, J. 2008. A probability-based measure of effect size: Robustness to base rates and other factors. Psychological Methods, 13: 19-30. doi: 10.1037/1082-989X.13.1.19
75. Sarhan, A. S., and, Greenberg, B. G. 1962. Contributions to order statistics, New York, NY: Wiley.
76. Smith, M. L., and, Glass, G. V. 1977. Meta-analysis of psychotherapy outcome studies. American Psychologist, 32: 752-760. doi: 10.1037/0003-066X.32. 9.752
77. Smith, M. L., and, Honoré, H. H. 2008. Effect size reporting in current health education literautre. American Journal of Health Studies, 23: 130-135. Retrieved from http://www.va-ajhs.com
78. Snyder, P. A., and, Thompson, B. 1998. Use of tests of statistical significance and other analytic choices in a school psychology journal: Review of practices and suggested alternatives. School Psychology Quarterly, 13: 335-348. doi: 10.1037/h0088990 (Pubitemid 128656762)
79. Somers, R. H. 1968. An approach to the multivariate analysis of ordinal data. American Sociological Review,: 971-977. doi: 10.2307/2092687
80. Staudte, R. G., and, Sheather, S. J. 1990. Robust estimation and testing, New York, NY: Wiley.
81. Steiger, J. H., and, Fouladi, R. T. 1997. " Noncentrality interval estimation and the evaluation of statistical models ". In What if there were no significance tests?, Edited by: Harlow, L., Mulaik, S. and Steiger, J. H. 221-257. Hillsdale, NJ: Erlbaum.
82. Sun, S. Y., Pan, W., and, Wang, L. L. 2010. A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology. Journal of Educational Psychology, 102: 989-1004. doi: 10.1037/a0019507
83. Tabachnick, B. G., and, Fidell, L. S. 2007. Using multivariate statistics, Boston, MA: Pearson/Allyn & Bacon.
84. Thompson, B. 1988. A note on statistical significance testing. Measurement and Evaluation in Counseling and Development, 20: 146-148.
85. Thompson, B. 1989. Statistical significance, result importance, and result generalizability: Three noteworthy but somewhat different issues. Measurement and Evaluation in Counseling and Development, 22: 2-5.
86. Thompson, B. 1993. Statistical significance testing in contemporary practice (theme issue). The Journal of Experimental Education, 61: 285-393. (Guest Ed.)
87. Thompson, B. 1994. Guidelines for authors. Educational and Psychological Measurement, 54: 837-847.
88. Thompson, B. 2002. What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31: 25-32. doi: 10.3102/0013189X031003025
89. Thompson, B. 2008. Foundations of behavioral statistics: An insight-based approach, New York, NY: Guilford Press.
90. Thompson, B., and, Snyder, P. A. 1997. Statistical significance testing practices. The Journal of Experimental Education, 66: 75-83. doi: 10.1080/00220979709601396
91. Vacha-Haase, T., Nilsson, J. E., Reetz, D. R., Lance, T. S., and, Thompson, B. 2000. Reporting practices and APA editorial policies regarding statistical significance and effect size. Theory and Psychology, 10: 413-425. doi: 10.1177/0959354300103006
92. Vargha, A., and, Delaney, H. D. 1998. The Kruskal-Wallis test and stochastic homogeneity. Journal of Educational and Behavioral Statistics, 23: 170-192. doi: 10.2307/1165320
93. Vargha, A., and, Delaney, H. D. 2000. A critique and improvement of the CL common language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics, 25: 101-132. doi: 10.2307/1165329
94. Welch, B. L. 1951. On the comparison of several mean values: An alternative approach. Biometrika, 38: 330-336.
95. Wilcox, R. R. 2005. Introduction to robust estimation and hypothesis testing (2nd ed.), San Diego, CA: Elsevier Academic Press.
96. Wilkinson, L., and the APA Task Force on Statistical Inference. 1999. Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54: 594-604. doi: 10.1037/0003-066X.54.8.594
97. 2 shrinkage in multiple regression: A comparison of different analytical methods. The Journal of Experimental Education, 69: 203-224. doi: 10.1080/00220970109600656
98. Yuen, K. K. 1974. The two-sample trimmed t for unequal population variances. Biometrika, 61: 165-170. doi: 10.2307/2334299
99. Yuen, K. K., and, Dixon, W. J. 1973. The approximate behavior and performance of the two-sample trimmed t. Biometrika, 60: 369-374. doi: 10.2307/2334550
100. Zhang, Z., and, Schoeps, N. 1997. On robust estimation of effect size under semiparametric models. Psychometrika, 62: 201-214. doi: 10.1007/BF02295275
101. Zientek, L. R., Capraro, M. M., and, Capraro, R. M. 2008. Reporting practices in quantitative teacher education research: one look at the evidence cited in the AERA Panel Report. Educational Researcher, 37: 208-216. doi: 10.3102/0013189X08319762
102. Zimmerman, D. M., and, Zumbo, B. D. 1993. Rank transformations and the power of the Student t test and Welch t test for non-normal populations with unequal variances. Canadian Journal of Experimental Psychology, 47: 523-539. doi: 10.1037/h0078850