A Comprehensive Review of Effect Size Reporting and Interpreting Practices in Academic Journals in Education and Psychology

Journal of Educational Psychology

Volumn 102, Issue 4, 2010, Pages 989-1004

  Sun, Shuyan ^a   Pan, Wei ^a   Wang, Lihshing Leigh ^a  

^a UNIVERSITY OF CINCINNATI   (United States)

Author keywords

Confidence intervals; Effect size; NHST; Practical significance; Statistical significance

Indexed keywords

EID: 78649692113     PISSN: 00220663     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0019507     Document Type: Article

References (116)

1. Alhija, F. N., & Levy, A. (2007, April). Effect size reporting practices in published articles. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
2. Standards on reporting on empirical social science research in AERA publications. Educational Researcher 2006, 35:33-40. American Educational Research Association.
3. Publication manual of the American Psychological Association 1994, American Psychological Association, Author, Washington, DC. 4th ed.
4. Publication manual of the American Psychological Association 2001, American Psychological Association, Author, Washington, DC. 5th ed.
5. Publication manual of the American Psychological Association 2010, American Psychological Association, Author, Washington, DC. 6th ed.
6. Anderson D.R., Burnham K.P., Thompson W.L. Null hypothesis testing: Problems, prevalence, and an alternative. Journal of Wildlife Management 2000, 64:912-923.
7. Berkson J. Some difficulties of interpretation encountered in the application of the chi-squared test. Journal of the American Statistical Association 1938, 33:526-536.
8. Biskin B. Comment on significance testing. Measurement and Evaluation in Counseling and Development 1998, 31:58-62.
9. Capraro M.M. An introduction to confidence intervals for both statistical estimates and effect sizes. Research in the Schools 2005, 12:22-32.
10. Capraro R.M., Capraro M.M. Treatments of effect sizes and statistical significance tests in textbooks. Educational and Psychological Measurement 2002, 62:771-782.
11. Carver R.P. The case against statistical significance testing. Harvard Educational Review 1978, 48:378-399.
12. Chow S.L. Significance test or effect size?. Psychological Bulletin 1988, 103:105-110.
13. Chow S.L. Statistical significance 1996, Sage, Thousand Oaks, CA.
14. Cohen J. Statistical power analysis for the behavioral sciences 1988, Erlbaum, Hillsdale, NJ. 2nd ed.
15. Cohen J. Things I have learned so far. American Psychologist 1990, 45:1304-1312.
16. Cohen J. The earth is round (p < .05). American Psychologist 1994, 49:997-1003.
17. Colegravea N., Ruxton G.D. Confidence intervals are a more useful complement to nonsignificant tests than are power calculations. Behavioral Ecology 2003, 14:446-450.
18. Cortina J.M., Dunlap W.P. Logic and purpose of significance testing. Psychological Methods 1997, 2:161-172.
19. Cromwell, S. (2001, February). An introductory summary of various effect size choices. Paper presented at the annual meeting of the Southwest Educational Research Association, New Orleans, LA.
20. Cumming G., Finch S. A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. Educational and Psychological Measurement 2001, 61:532-574.
21. Cumming G., Finch S. Inference by eye: Confidence intervals and how to read pictures of data. American Psychologist 2002, 60:170-180.
22. Curtis, D. A., & Araki, C. J. (2002, April). Effect size statistics: An analysis of statistics textbooks used in psychology and education. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
23. Dar R., Serlin R.C., Omer H. Misuse of statistical tests in three decades of psychotherapy research. Journal of Consulting and Clinical Psychology 1994, 62:75-82.
24. Dunleavy E.M., Barr C.D., Glenn D.M., Miller K.R. Effect size reporting in applied psychology: How are we doing?. The Industrial-Organizational Psychologist 2006, 43:29-37.
25. Falk R., Greenbaum C.W. Significance tests die hard: The amazing persistence of a probabilistic misconception. Theory & Psychology 1995, 5:75-98.
26. Fan X. Statistical significance and effect size in education research: Two sides of a coin. Journal of Educational Research 2001, 94:275-283.
27. Fleiss J.L. Measures of effect size for categorical data. The handbook of research synthesis 1994, 245-260. Russell Sage Foundations, New York, NY. H. Cooper, L.V. Hedges (Eds.).
28. Fuchs L.S., Fuchs D., Finelli R., Courey S.J., Hamlett C.L. Expanding schema-based transfer instruction to help third graders solve real-life mathematical problems. American Educational Research Journal 2004, 41:419-445.
29. Fuchs L.S., Fuchs D., Prentice K., Hamlett C.L., Finelli R., Courey S.J. Enhancing mathematical problem solving among third grade students with schema-based instruction. Journal of Educational Psychology 2004, 96:635-647.
30. Glass G.V. Primary, secondary, and meta-analysis of research. Educational Researcher 1976, 5:3-8.
31. Glass G.V. Policy for the unpredictable (uncertainty research and policy). Educational Researcher 1979, 8:12-14.
32. Glass G.V., McGaw B., Smith M.L. Meta-analysis in social research 1981, Sage, Beverly Hills, CA.
33. Gravetter F.J., Wallnau L.B. Statistics for the behavior sciences 2007, Wadsworth, Belmont, CA. 7th ed.
34. Grissom R.J., Kim J.J. Effect size for research: A broad practical approach 2005, Erlbaum, Mahwah, NJ.
35. Harlow L.L., Mulaik S.A., Steiger J.H. What if there were no significance tests? 1997, Erlbaum, Mahwah, NJ.
36. Hedges L.V. Distributional theory for Glass's estimator of effect size and related estimators. Journal of Educational Statistics 1981, 6:107-128.
37. Hedges L.V., Olkin I. Statistical methods for meta-analysis 1985, Academic Press, Orlando, FL.
38. Henson R.K. Effect size measures and meta-analytic thinking in counseling psychology research. The Counseling Psychologist 2006, 34:601-629.
39. Henson R.K., Smith A.D. State of the art in statistical significance and effect size reporting: A review of the APA Task Force report and current trends. Journal of Research and Development in Education 2000, 33:285-296.
40. Hoenig J.M., Heisey D.M. The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician 2001, 55:19-24.
41. Hojat M., Xu G. Statistical significance versus practical (clinical) importance of research findings. Advances in Health Sciences Education 2004, 9:241-249.
42. Huberty C.J., Lowman L.L. Group overlap as a basis for effect size. Educational and Psychological Measurement 2000, 60:543-563.
43. Hutchins, H. M., & Henson, R. K. (2002, February). In search of OZ: Effect size reporting and interpretation in communication research. Paper presented at the annual meeting of the Southwest Educational Research Association, Austin, TX.
44. Ives B. Effect size use in studies of learning disabilities. Journal of Learning Disabilities 2003, 36:490-504.
45. Jitendra A.K., Griffin C.C., Haria P., Leh J., Adams A., Kaduvettoor A. A comparison of single and multiple strategy instruction on third-grade students' mathematical problem solving. Journal of Educational Psychology 2007, 99:115-127.
46. Johnson D.H. Statistical sirens: The allure of nonparametrics. Ecology 1995, 76:1998-2000.
47. Kampenes V.B., Dybå T., Hannay J.E., Sjøberg D.I. A systematic review of effect size in software engineering experiments. Information and Software Technology 2007, 49:1073-1086.
48. Katzer J., Sodt J. An analysis of the use of statistical testing in communication research. Journal of Communication 1973, 23:251-266.
49. Keselman H.J., Huberty C.J., Lix L.M., Olejnik S., Cribbie R.A., Donahue B., Levin J.R. Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analyses. Review of Educational Research 1998, 68:350-386.
50. Kirk R.E. Practical significance: A concept whose time has come. Educational and Psychological Measurement 1996, 56:746-759.
51. Kirk R.E. Promoting good statistical practice: Some suggestions. Educational and Psychological Measurement 2001, 61:213-218.
52. Kline R.B. Beyond significance testing: Reforming data analysis methods in behavioral research 2004, American Psychological Association, Washington, DC.
53. Lakshmi, M. (2000, January). The effect size statistics: Overview of various choices. Paper presented at the annual meeting of the Southwest Educational Research Association, Dallas, TX.
54. Lance, T. S., & Vacha-Haase, T. (1998, August). The counseling psychologist: Trends and usages of statistical significance testing. Paper presented at the annual meeting of the American Psychological Association, San Francisco, CA.
55. Mahadevan, L. (2000, January). The effect size statistic: Overview of various choices. Paper presented at the annual meeting of the Southwest Educational Research Association, Dallas, TX.
56. Maxwell S.E., Delaney H.D. Designing experiments and analyzing data: A model comparison perspective 1990, Wadsworth, Belmont, CA.
57. May H., Supovitz J.A. Capturing the cumulative effects of school reform: An 11-year study of the impacts of America's choice on student achievement. Educational Evaluation and Policy Analysis 2006, 28:231-257.
58. McLean E.J., Ernest J.M. The role of statistical significance testing in educational research. Research in the Schools 1998, 5:15-22.
59. McMillan, J. H., Lawson, S., Lewis, K., & Synder, A. (2002, April). Reporting effect size: The road less traveled. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
60. Meehl P.E. The problem is epistemology, not statistics: Replace significance tests by confidence intervals and quantify accuracy of risky numerical predictions. What if there were no significance tests? 1997, 391-423. Erlbaum, Mahwah, NJ. L.L. Harlow, S.A. Mulaik, J.H. Steiger (Eds.).
61. Meline T., Schmitt J.F. Case studies for evaluating statistical significance in group designs. American Journal of Speech-Language Pathology 1997, 6:33-41.
62. Meline T., Wang B. Effect-size reporting practices in AJSLP and other ASHA journals, 1999-2003. American Journal of Speech-Language Pathology 2004, 13:202-207.
63. Nakagawa S., Cuthill I.C. Effect size, confidence interval and statistical significance: A practical guide for biologists. Biological Review 2007, 82:591-605.
64. Olejnik S.F. Planning educational research: Determining the necessary sample size. Journal of Experimental Education 1984, 53:40-48.
65. Ottenbacher K.J., Barrett K.A. Measures of effect size in the reporting of rehabilitation research. American Journal of Physical Medicine & Rehabilitation 1989, 68:52-58.
66. Patton M.Q. Qualitative evaluation and research methods 1990, Sage, Thousand Oaks, CA.
67. Patton M.Q. Enhancing the quality and credibility of qualitative analysis. Health Services Research 1999, 34:1189-1208.
68. Paul K.M., Plucker J.A. Two steps forward, one step back: Effect size reporting in gifted education research from 1995-2000. Roeper Review 2003, 26:68-72.
69. Plucker J.A. Debunking the myth of the " highly significant" result: Effect sizes in gifted education research. Roeper Review 1997, 2:122-126.
70. Robey, R. R. (2004, November). Effect sizes in research manuscripts: Selecting, calculating, reporting and interpreting. Seminar presented before the annual conference of the American Speech-Language-Hearing Association, Philadelphia, PA.
71. Robinson D.H., Levin J.R. Reflections on statistical and substantive significance, with a slice of replication. Educational Researcher 1997, 26:21-27.
72. Robinson D.H., Wainer H. On the past and future of null hypothesis significance testing. Journal of Wildlife Management 2002, 66:263-271.
73. Rosenthal R., Rosnow R.L., Rubin D.B. Contrasts and effect sizes in behavioral research: A correlational approach 2000, Cambridge University Press, New York, NY.
74. Schmidt F. Statistical significance testing and cumulative knowledge in psychology: Implications for the training of researchers. Psychological Methods 1996, 1:115-129.
75. Schmidt F., Hunter J.E. The impact of data-analysis methods on cumulative research knowledge: Statistical significance testing, confidence intervals, and meta-analysis. Evaluation and the Health Professions 1995, 18:408-427.
76. Seaman M.A. Categorical data [Electronic resource] 2001, Retrieved from http://www.ed.sc.edu/seaman/edrm711/questions/categorical.htm.
77. Serlin R.C. Confidence intervals and the scientific method: A case for Holm on the range. Journal of Experimental Education 1993, 61:350-360.
78. Shaver J.P. What statistical significance testing is, and what it is not. Journal of Experimental Education 1993, 61:293-316.
79. Sim J., Reid N. Statistical inference by confidence intervals. Physical Therapy 1999, 79:186-196.
80. Simard V., Nielsen T.A. Sleep paralysis-associated sensed presence as a possible manifestation of social anxiety. Dreaming 2005, 15:245-260.
81. Smithson M. Correct confidence intervals for various regression effect sizes and parameters: The importance of noncentral distributions in computing intervals. Educational and Psychological Measurement 2001, 61:605-632.
82. Snyder P.A., Lawson S. Evaluating results using corrected and uncorrected effect size estimates. Journal of Experimental Education 1993, 61:334-349.
83. Snyder P.A., Thompson B. Use of tests of statistical significance and other analytic choices in a school psychology journal. School Psychology Quarterly 1998, 13:335-348.
84. Steiger J.H., Fouladi R.T. Noncentrality interval estimation and the evaluation of statistical models. What if there were no significance tests? 1997, 221-257. Erlbaum, Mahwah, NJ. L.L. Harlow, S.A. Mulaik, J.H. Steiger (Eds.).
85. Sun, S., Pan, W., & Wang, L. (2009, April). Rethinking observed power: Concept, practice, and implications. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
86. Thomas L. Retrospective power analysis. Conservation Biology 1997, 11:276-280.
87. Thompson B. The use of statistical significance tests in research: Bootstrap and other alternatives. Journal of Experimental Education 1993, 61:361-377.
88. Thompson B. Guidelines for authors. Educational and Psychological Measurement 1994, 54:837-847.
89. Thompson B. AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher 1996, 25:26-30.
90. Thompson B. Editorial policies regarding statistical significance tests: Further comments. Educational Researcher 1997, 26:29-32.
91. Thompson B. In praise of brilliance: Where that praise really belongs. American Psychologist 1998, 53:799-800.
92. Thompson B. Statistical significance and effect size reporting: Portrait of possible future. Research in the Schools 1998, 5:33-38.
93. Thompson B. If statistical significance tests are broken/misused, what practices should supplement or replace them?. Theory and Psychology 1999, 82:165-181.
94. Thompson B. Improving research clarity and usefulness with effect size indices as supplements to statistical significance tests. Exceptional Children 1999, 65:329-337.
95. Thompson B. Statistical significance tests, effect size reporting and the vain pursuit of pseudo-objectivity. Theory and Psychology 1999, 9:191-196.
96. Thompson B. Why " encouraging" effect size reporting is not working: The etiology of researcher resistance to change practices. The Journal of Psychology 1999, 133:133-140.
97. Thompson B. " Statistical," " practical," and " clinical": How many kinds of significance do counselors need to consider?. Journal of Counseling & Development 2002, 80:64-71.
98. Thompson B. What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher 2002, 31:25-32.
99. Thompson B. Effect sizes, confidence intervals, and confidence intervals for effect sizes. Psychology in the Schools 2007, 45:423-432.
100. Thompson B. Computing and interpreting effect sizes, confidence intervals, and confidence intervals for effect sizes. Best practices in quantitative methods 2008, 246-262. Sage, Thousand Oaks, CA. J.W. Osborne (Ed.).
101. Thompson B., Snyder P.A. Statistical significance testing practices in the Journal of Experimental Education. Journal of Experimental Education 1997, 66:75-83.
102. Trusty J., Thompson B., Petrocelli J.V. Practical guide for reporting effect size in quantitative research in the Journal of Counseling & Development. Journal of Counseling & Development 2004, 82:107-112.
103. Vacha-Haase T., Ness C.M. Statistical significance testing as it relates to practice: Use within professional psychology: Research and practice. Professional Psychology: Research and Practice 1999, 30:104-105.
104. Vacha-Haase T., Nilsson J.E. Statistical significance reporting: Current trends and uses in MECD. Measurement and Evaluation in Counseling and Development 1998, 31:46-57.
105. Vacha-Haase T., Nilsson J.E., Reetz D.R., Lance T.S., Thompson B. Reporting practices and APA editorial policies regarding statistical significance and effect size. Theory & Psychology 2000, 10:413-425.
106. Vacha-Haase T., Thompson B. How to estimate and interpret various effect sizes. Journal of Counseling Psychology 2004, 51:473-481.
107. Vachon F., Tremblay S., Jones D.M. Task-set reconfiguration suspends perceptual processing: Evidence from semantic priming during the attentional blink. Journal of Experimental Psychology: Human Perception and Performance 2007, 33:330-347.
108. Vaske J.J., Gliner J.A., Morgan G.A. Communicating judgments about practical significance: Effect size, confidence intervals and odds ratios. Human Dimensions of Wildlife 2002, 7:287-300.
109. Volker M.A. Reporting effect size estimates in school psychology research. Psychology in the Schools 2006, 43:653-672.
110. Ward, R. M. (2002). Highly significant findings in psychology: A power and effect size survey. Unpublished doctoral dissertation, University of Rhode Island.
111. Wilkinson L. Statistical methods in psychology journals: Guidelines and explanations. American Psychologist 1999, 54:594-604. APA Task Force on Statistical Inference.
112. Yates F. The influence of " statistical methods for research workers" on the development of the science of statistics. Journal of the American Statistical Association 1951, 46:19-34.
113. Yetkiner, Z. E., Capraro, R. M., Zientek, L. R., & Tompson, B. (2008, July). Effect size and confidence interval reporting practices in mathematics education. Paper presented at the 11th International Congress on Mathematical Education, Monterrey, Mexico.
114. Young M.A. Supplementing tests of statistical significance: Variation accounted for. Journal of Speech and Hearing Research 1993, 36:644-656.
115. Yuan K.-H., Maxwell S. On the post hoc power in testing mean differences. Journal of Educational and Behavioral Statistics 2005, 30:141-167.
116. Zientek L.R., Capraro M.M., Capraro R.M. Reporting practices in quantitative teacher education research: One look at the evidence cited in the AERA panel report. Educational Researcher 2008, 37:208-216.