Effect sizes and F ratios < 1.0. Sense or nonsense?

Methodology

Volumn 3, Issue 1, 2007, Pages 35-46

  Voelkle, Manuel C ^a   Ackerman, Phillip L ^b   Wittmann, Werner W ^a  

^a UNIVERSITY OF MANNHEIM   (Germany)

^b GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Adjusted R2; ANOVA; Effect size; F ratio

Indexed keywords

EID: 36148960367     PISSN: 16141881     EISSN: 16142241     Source Type: Journal    
DOI: 10.1027/1614-2241.3.1.35     Document Type: Review

References (65)

1. American Psychological Association. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.
2. Balluerka, N., Gómez, J., & Hidalgo, D. (2005). The controversy over null hypothesis significance testing revisited. Methodology, 1, 55-70.
3. 2. Educational and Psychological Measurement, 35, 541-554.
4. Cohen, J. (1962). The statistical power of abnormal-social psychological research. Journal of Abnormal and Social Psychology, 65, 145-153.
5. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
6. Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159.
7. Cohen, J. (1994). The earth is round (p <.05). American Psychologist, 49, 997-1003.
8. Cohen, J., & Cohen, P. (1983). Applied multiple regression/ correlation analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
9. Cumming, G., & Finch, S. (2001). A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. Educational and Psychological Measurement, 61, 532-574.
10. Edgington, E.S. (1974). A new tabulation of statistical procedures used in APA journals. American Psychologist, 29, 25-26.
11. Elmore, P.B., & Woehlke, P.L. (1998, April). Twenty years of research methods employed in American Educational Research Journal, Educational Researcher, and Review of Educational Research. Paper presented at the Annual meeting of the American Educational Research Association, San Diego, CA (ERIC Document Reproduction Service No. 420 701).
12. Ezekiel, M. (1930). Methods of correlational analysis. New York: Wiley.
13. Finch, S., Cumming, G., & Thomason, N. (2001). Reporting of statistical evidence in the Journal of Applied Psychology: Little evidence of reform. Educational and Psychological Measurement, 61, 281-210.
14. Fleischman, A.I. (1980). Confidence intervals for correlation ratios. Educational and Psychological Measurement, 40, 659-670.
15. Gigerenzer, G. (2000). Adaptive thinking: Rationality in the real world. Oxford: Oxford University Press.
16. Goodwin, L.D., & Goodwin, W.L. (1985). An analysis of statistical techniques used in the Journal of Educational Psychology. Educational Psychologist, 20, 13-21.
17. Harlow, L.L., Mulaik, S.A., & Steiger, J.H. (Eds.). (1997). What if there were no significance tests? Mahwah, NJ: Erlbaum.
18. Hays, W.L. (1994). Statistics (5th ed.). Orlando: Harcourt Brace.
19. Herzberg, P.A. (1969). The parameters of cross-validation. Psychometrika Monograph Supplement, 16, 1-67.
20. Huberty, C.J., & Mourad, S.A. (1980). Estimation in multiple correlation/prediction. Educational and Psychological Measurement, 40, 101-112.
21. Hunter, J.E., & Schmidt, F.L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.
22. Hyde, J.S. (2001). Reporting effect sizes: The roles of editors, textbook authors, and publication manuals. Educational and Psychological Measurement, 61, 225-228.
23. Kelley, T.L. (1935). An unbiased correlation ratio measure. Proceedings of the National Academy of Sciences, 21, 554-559.
24. Kellow, T.J. (1998). Beyond statistical significant tests: The importance of using other estimates of treatment effects to interpret evaluation results. The American Journal of Evaluation, 19(1), 123-134.
25. Keselman, H.J., Huberty, C.J., Lix, L.M., Olejnik, S., Cribbie, R.A., Donahue, B., et al. (1998). Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analyses. Review of Educational Research, 68, 350-386.
26. Kieffer, K.M., Reese, R.J., & Thompson, B. (2001). Statistical techniques employed in AERJ and JCP articles from 1988 to 1997: A methodological review. Journal of Experimental Education, 69, 280-309.
27. Kirk, R.E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.
28. Kline, R.B. (2004). Beyond significance testing: Performing data analysis methods in behavioral research. Washington, DC: American Psychological Association.
29. Kromrey, J.D., & Hines, C.V. (1996). Estimating the coefficient of cross-validity in multiple regression: A comparison of analytical and empirical methods. Journal of Experimental Education, 64, 240-266.
30. Larson, S.C. (1931). The shrinkage of the coefficient of multiple correlation. Journal of Educational Psychology, 22, 45-55.
31. Lord, F.M. (1950). Efficiency of prediction when a regression equation from one sample is used in a new sample. (Research Bulletin 50-110). Princeton, NJ: Educational Testing Service.
32. Maxwell, S.E. (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147-163.
33. Maxwell, S.E., Camp, C.J., & Avery, R.D. (1981). Measures of strength of association: A comparative examination. Journal of Applied Psychology, 66, 525-534.
34. Maxwell, S.E., & Delaney, H.D. (1990). Designing experiments and analyzing data: A model comparison perspective. Belmont, CA: Wadsworth.
35. McLean, J.E., & Ernest, J.M. (1998). The role of statistical significance testing in educational research. Research in the Schools, 5, 15-22.
36. Meehl, P.E. (1990). Appraising and amending theories: The strategy of Lakatosian defence and two principles that warrant it. Psychological Inquiry, 1(2), 108-141.
37. 2. Journal of Finance, 28, 1009-1013.
38. Moore, D.S., & McCabe, G.P. (1999). Introduction to the practice of statistics (3rd ed.). New York: Freeman.
39. O'Grady, K.E. (1982). Measures of explained variance: Cautions and limitations. Psychological Bulletin, 92, 766-777.
40. Olejnik, S., & Algina, J. (2000). Measures of effect size for comparative studies: Applications, interpretations, and limitations. Contemporary Educational Psychology, 25, 241-286.
41. 2 values from multifactor ANOVA designs. Educational and Psychological Measurement, 64, 916-924.
42. Popper, K.R. (1989). Logik der Forschung [The logic of scientific discovery] (9th ed.). Tübingen, Germany: Mohr.
43. R Development Core Team. (2004). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing.
44. Roberts, J.K., & Henson, R.K. (2002). Correction for bias in estimating effect sizes. Educational and Psychological Measurement, 62, 241-253.
45. Rosenthal, R., Rosnow, R.L., & Rubin, D.B. (2000). Contrasts and effect sizes in behavioral research: A correlational approach. New York: Cambridge University Press.
46. Rosnow, R.L., & Rosenthal, R. (1996). Computing contrasts, effect sizes, and counternulls on other people's published data: General procedures for research consumers. Psychological Methods, 1, 331-340.
47. Schmidt, F.L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for the training of researchers. Psychological Methods, 1, 115-129.
48. Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies. Psychological Bulletin, 105, 309-316.
49. Smithson, M. (2001). Correct confidence intervals for various regression effect sizes and parameters: The importance of non-central distributions in computing intervals. Educational and Psychological Measurement, 61, 605-632.
50. Smithson, M. (2002). Confidence intervals. Thousand Oaks, CA: Sage.
51. Snyder, P.A., & Lawson, S. (1993). Evaluating results using corrected and uncorrected effect size estimates. Journal of Experimental Education, 61, 334-349.
52. Snyder, P.A., & Thompson, B. (1998). Use of tests of statistical significance and other analytic choices in a school psychology journal: Review of practices and suggested alternatives. School Psychology Quarterly, 13, 335-348.
53. Steiger, J.H. (2004). Beyond the F test: Effect size confidence intervals and tests of close fit in the analysis of variance and contrast analysis. Psychological Methods, 9, 164-182.
54. Steiger, J.H., & Fouladi, R.T. (1997). Noncentrality interval estimation and the evaluation of statistical models. In L.L. Harlow, S.A. Mulaik, & J.H. Steiger (Eds.), What if there were no significance tests? (pp. 221-257). Mahwah, NJ: Erlbaum.
55. Thompson, B. (1999a). Journal editorial policies regarding statistical significance tests: Heat is to fire as p is to importance. Educational Psychology Review, 11, 157-169.
56. Thompson, B. (1999b). Why "encouraging" effect size reporting is not working: The etiology of researcher resistance to changing practices. The Journal of Psychology, 133, 133-140.
57. Thompson, B. (2002a). "Statistical," "practical," and "clinical": How many kinds of significance do counselors need to consider? Journal of Counseling and Development, 80(1), 64-71.
58. Thompson, B. (2002b). What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31(3), 24-31.
59. Thompson, B., & Snyder, P.A. (1997). Statistical significance testing practices in the Journal of Experimental Education. Journal of Experimental Education, 66, 75-83.
60. Vacha-Haase, T., & Nilsson, J.E. (1998). Statistical significance reporting: Current trends and usages within MECD. Measurement and Evaluation in Counseling and Development, 31, 46-57.
61. Vacha-Haase, T., Nilsson, J.E., Reetz, D.R., Lance, T.S., & Thompson, B. (2000). Reporting practices and APA editorial policies regarding statistical significance and effect size. Theory and Psychology, 10, 413-425.
62. Vacha-Haase, T., & Thompson, B. (2004). How to estimate and interpret various effect sizes. Journal of Counseling Psychology, 51, 473-481.
63. Wherry, R.J. (1931). A new formula for predicting the shrinkage of the coefficient of multiple correlation. Annals of Mathematical Statistics, 2, 440-451.
64. Wilkinson, L., & APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.
65. 2 shrinkage in multiple regression: A comparison of different analytical methods. Journal of Experimental Education, 69, 203-224.