Bias and precision of some classical ANOVA effect sizes when assumptions are violated

Behavior Research Methods

Volumn 45, Issue 2, 2013, Pages 536-546

  Troncoso Skidmore, Susan ^a   Thompson, Bruce ^b  

^a SAM HOUSTON STATE UNIVERSITY   (United States)

^b TEXAS A AND M UNIVERSITY   (United States)

Author keywords

Analysis of variance; Effect size; Epsilon squared; Eta squared; Homogeneity of variance; Omega squared; Power; Practical significance; Type I error

Indexed keywords

ANALYSIS OF VARIANCE; ARTICLE; COMPUTER PROGRAM; EPIDEMIOLOGY; HUMAN; METHODOLOGY; MONTE CARLO METHOD; REPRODUCIBILITY; SAMPLE SIZE; STATISTICAL MODEL; THEORETICAL MODEL;

ANALYSIS OF VARIANCE; BIAS (EPIDEMIOLOGY); HUMANS; MODELS, STATISTICAL; MODELS, THEORETICAL; MONTE CARLO METHOD; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; SAMPLE SIZE; SOFTWARE;

EID: 84878255000     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-012-0257-2     Document Type: Article

References (59)

1. Aiken, L. S., West, S. G., & Millsap, R. E. (2008). Doctoral training in statistics, measurement and methodology in psychology. American Psychologist, 63, 32-50. doi: 10. 1037/0003-066X. 63. 1. 32.
2. Algina, J., Keselman, H. J., & Penfield, R. D. (2005). An alternative to Cohen's standardized mean difference effect size: A robust parameter and confidence interval in the two independent groups case. Psychological Methods, 10, 317-328. doi: 10. 1037/1082-989X. 10. 3. 317.
3. American Educational Research Association. (2006). Standards for reporting on empirical social science research in AERA publications. Educational Researcher, 35(6), 33-40. doi: 10. 3102/0013189X035006033.
4. American Psychological Association. (1994). Publication manual of the American Psychological Association (4th ed.). Washington, DC: American Psychological Association.
5. American Psychological Association. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: American Psychological Association.
6. American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: American Psychological Association.
7. Box, G. E. P. (1954). Some theorems on quadratic forms applied in the study of analysis of variance problems. II. Effects of inequality of variance and of correlation between errors in the two-way classification. Annals of Mathematical Statistics, 25, 484-498. doi: 10. 1214/aoms/1177728717.
8. Capraro, R. M., & Thompson, B. (2008). The educational researcher defined: What will future researchers be trained to do? The Journal of Educational Research, 101, 247-253.
9. 2. Educational and Psychological Measurement, 35, 541-554. doi: 10. 1177/001316447503500304.
10. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
11. David, H. A. (1995). First (?) occurrence of common terms in mathematical statistics. The American Statistician, 49, 121-133. doi: 10. 2307/2684625.
12. Donaldson, T. S. (1968). Robustness of the F-test to errors of both kinds and the correlation between the numerator and denominator of the F-ratio. Journal of the American Statistical Association, 63, 660-676. doi: 10. 2307/2284037.
13. Edginton, E. S. (1964). A tabulation of inferential statistics used in psychology journals. American Psychologist, 19, 202-203. doi: 10. 1037/h0039177.
14. Edginton, E. S. (1974). A new tabulation of statistical procedures used in APA journals. American Psychologist, 29, 25-26. doi: 10. 1037/h0035846.
15. Elmore, P. B., & Woehlke, P. L. (1996, April). Research methods employed in American Educational Research Journal, Educational Researcher, and Review of Educational Research from 1978-1995. Paper presented at the Annual meeting of the American Educational Research Association, New York.
16. Fan, X., Felsovalyi, A., Sivo, S. A., & Keenan, S. C. (2001). SAS for Monte Carlo studies: A guide for quantitative researchers. Cary, NC: SAS Institute.
17. Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral and biomedical sciences. Behavior Research Methods, 39, 175-191.
18. Fidler, F. (2002). The fifth edition of the APA Publication Manual: Why its statistics recommendations are so controversial. Educational and Psychological Measurement, 62, 749-770. doi: 10. 1177/001316402236876.
19. Fidler, F. (2005). From statistical significance to effect estimation: Statistical reform in psychology, medicine, and ecology. Doctoral dissertation, University of Melbourne. www. botany. unimelb. edu. au/envisci/docs/fidler/fidlerphd_aug06. pdf.
20. Fisher, R. A. (1918). The causes of human variability. The Eugenics Review, 10, 213-220.
21. Fleishman, A. I. (1978). A method for simulating non-normal distributions. Psychometrika, 43, 521-532. doi: 10. 1007/BF02293811.
22. Gamst, G., Meyers, L. S., & Guarino, A. J. (2008). Analysis of variance designs: A conceptual and computational approach with SPSS and SAS. Cambridge: Cambridge University Press.
23. Glass, G. V, Peckham, P. D., & Sanders, J. R. (1972). Consequences of failure to meet assumption underlying the fixed effects analysis of variance and covariance. Review of Educational Research, 42, 237-288.
24. Grissom, R. J., & Kim, J. J. (2005). Effect sizes for research: A broad practical approach. New York: Psychology Press.
25. Harwell, M. R., Rubinstein, E. N., Hayes, W. S., & Olds, C. C. (1992). Summarizing Monte Carlo results in methodological research: The one- and two-factor fixed effects ANOVA cases. Journal of Educational Statistics, 17, 315-339. doi: 10. 2307/1165127.
26. Hays, W. L. (1981). Statistics (3rd ed.). New York: Holt, Rinehart and Winston.
27. Henson, R. K., & Williams, C. (2006, April). Doctoral training in research methodology: A national survey of education-related degrees. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
28. Hsu, P. L. (1938). Contribution to the theory of "Student's" t-test as applied to the problem of two samples. Statistical Research Memoirs, 2, 1-24.
29. Kelley, T. L. (1935). An unbiased correlation ratio measure. Proceedings of the National Academy of Sciences, 21, 554-559. doi: 10. 1073/pnas. 21. 9. 554.
30. Keselman, H. J. (1975). A Monte Carlo investigation of three estimates of treatment magnitude: Epsilon squared, eta squared, and omega squared. Canadian Psychological Review, 16, 44-48. doi: 10. 1037/h0081789.
31. Keselman, H. J., Algina, J., Lix, L. M., Wilcox, R. R., & Deering, K. N. (2008a). A generally robust approach for testing hypotheses and setting confidence intervals for effect sizes. Psychological Methods, 13, 110-129. doi: 10. 1037/1082-989X. 13. 2. 110.
32. Keselman, H. J., Algina, J., Lix, L. M., Wilcox, R. R., & Deering, K. (2008b). Supplemental materials to 129a. A SAS program to implement a general approximate degrees of freedom solution for inference and estimation. http://dx. doi. org/10. 1037/ 1082-989X. 13. 2. 110. supp.
33. Keselman, H. J., Huberty, C. J., Lix, L. M., Olejnik, S., Cribbie, R. A., Donahue, B., Kowalchuk, R. K., Lowman, L. L., Petoskey, M. D., Keselman, J. C., & Levin, J. R. (1998). Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analyses. Review of Educational Research, 68, 350-386.
34. Kieffer, K. M., Reese, R. J., & Thompson, B. (2001). Statistical techniques employed in "AERJ" and "JCP" articles from 1988 to 1997: A methodological review. The Journal of Experimental Education, 69, 280-309. doi: 10. 1080/00220970109599489.
35. Kirk, R. E. (1995). Experimental design: Procedures for the behavioral sciences (3rd ed.). New York: Brooks/Cole.
36. Kromrey, J. D., & Hines, C. V. (1996). Estimating the coefficient of cross-validity in multiple regression: A comparison of analytical and empirical methods. The Journal of Experimental Education, 64, 240-266.
37. Lix, L. M., Keselman, J. C., & Keselman, H. J. (1996). Consequences of assumption violations revisited: A quantitative review of alternatives to the one-way analysis of variance "F" test. Review of Educational Research, 66, 579-619.
38. Pierce, C. A., Block, R. A., & Aguinis, H. (2004). Cautionary note on reporting eta-squared values from multifactor ANOVA designs. Educational and Psychological Measurement, 64, 916-924. doi: 10. 1177/0013164404264848.
39. Robey, R. R., & Barcikowski, R. S. (1992). Type I error and the number of iterations in Monte Carlo studies of robustness. British Journal of Mathematical and Statistical Psychology, 45, 283-288.
40. Skidmore, S. T., & Thompson, B. (2010). Statistical techniques used in published articles: A historical review of reviews. Educational and Psychological Measurement, 70, 777-795. doi: 10. 1177/0013164410379320.
41. 2 effect size. The Journal of Experimental Education, 79, 257-278. doi: 10. 1080/00220973. 2010. 484437.
42. Snyder, P. A., & Thompson, B. (1998). Use of tests of statistical significance and other analytic choices in a school psychology journal: Review of practices and suggested alternatives. School Psychology Quarterly, 13, 335-348. doi: 10. 1037/h0088990 [Named the 1999 APA Division 16 Fellows' Article of the Year].
43. Thompson, B. (1999). Improving research clarity and usefulness with effect size indices as supplements to statistical significance tests. Exceptional Children, 65, 329-337.
44. Thompson, B. (2002). What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31(3), 24-31. doi: 10. 3102/0013189X031003025.
45. Thompson, B. (2006a). Foundations of behavioral statistics: An insight-based approach. New York, NY: Guilford.
46. Thompson, B. (2006b). Research synthesis: Effect sizes. In J. Green, G. Camilli, & P. B. Elmore (Eds.), Handbook of complementary methods in education research (pp. 583-603). Washington, DC: American Educational Research Association.
47. Thompson, B., & Snyder, P. A. (1998). Statistical significance and reliability analyses in recent JCD research articles. Journal of Counseling and Development, 76, 436-441.
48. Tukey, J. W. (1960). A survey of sampling from contaminated distributions. In S. G. Olkin, W. Hoeffding, W. Madow, & H. Mann (Eds.), Contributions to probability and statistics: Essays in honor of Harold Hotelling. Stanford, CA: Stanford University Press.
49. Vale, C. D., & Maurelli, V. A. (1983). Simulating multivariate nonnormal distributions. Psychometrika, 48, 465-471. doi: 10. 1007/BF02293687.
50. 2 biased, and if so, what is the best correction formula? The Journal of Experimental Education, 75, 109-125. doi: 10. 3200/ JEXE. 75. 2. 109-125.
51. Wilcox, R. R. (1987). New designs in analysis of variance. Annual Review of Psychology, 38, 29-60. doi: 10. 1146/annurev. ps. 38. 020187. 000333.
52. Wilcox, R. R. (1993). Robustness in ANOVA. In L. K. Edwards (Ed.), Applied analysis of variance in behavioral science (pp. 345-374). New York: Marcel Dekker.
53. Wilcox, R. R. (1995). ANOVA: A paradigm for low power and misleading measures of effect size. Review of Educational Research, 65, 51-77.
54. Wilcox, R. R. (2006). Graphical methods for assessing effect size. The Journal of Experimental Education, 74, 353-367. doi: 10. 3200/JEXE. 74. 4. 351-367.
55. Wilcox, R. R., Charlin, V., & Thompson, K. L. (1986). New Monte Carlo results on the robustness of the ANOVA F, W, F* statistics. Communications in Statistics: Simulation and Computation, 15, 933-944. doi: 10. 1080/03610918608812553.
56. Wilcox, R. R., & Keselman, H. J. (2003). Modern robust data analysis methods: Measures of central tendency. Psychological Methods, 8, 254-274. doi: 10. 1037/1082-989X. 8. 3. 254.
57. Wilkinson, L., & APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604. doi: 10. 1037/0003-066X. 54. 8. 594.
58. 2 shrinkage in multiple regression: A comparison of different analytical methods. The Journal of Experimental Education, 69, 203-224. doi: 10. 1080/00220970109600656.
59. Zhang, Z., & Schoeps, N. (1997). On robust estimation of effect size under semiparametric models. Psychometrika, 62, 201-214. doi: 10. 1007/BF02295275.