Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analyses

Review of Educational Research

Volumn 68, Issue 3, 1998, Pages 350-386

  Keselman, Harvey J ^b   Huberty, Carl J ^c   Cribbie, Robert A ^b   Lowman, Laureen L ^c   Lix, Lisa M ^d   Donahue, Barbara ^c   Petoskey, Martha D ^c   Olejnik, Stephen ^c   Kowalchuk, Rhonda K ^b   Keselman, Joanne C ^b   Levin, Joel R ^a  

^a UNIVERSITY OF WISCONSIN MADISON   (United States)

^b UNIVERSITY OF MANITOBA   (Canada)

^c UNIVERSITY OF GEORGIA   (United States)

^d Saskatchewan Health   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0032272447     PISSN: 00346543     EISSN: None     Source Type: Journal    
DOI: 10.3102/00346543068003350     Document Type: Article

References (115)

1. Algina, J. (1982). Remarks on the analysis of covariance in repeated measures designs. Multivariate Behavioral Research, 17, 117-130.
2. Algina, J. (1994). Some alternative approximate tests for a split plot design. Multivariate Behavioral Research, 29, 365-384.
3. Algina, J., & Oshima, T. C. (1994). Type I error rates for Huynh's general approximation and improved general approximation tests. British Journal of Mathematical and Statistical Psychology, 47, 151-165.
4. American Psychological Association. (1994). Publication manual of the American Psychological Association (4th ed.). Washington, DC: Author.
5. Behrens, J. T. (1997). Principles and procedures of exploratory data analysis. Psychological Methods, 2, 131-160.
6. Boik, R. J. (1988). Scheffe's mixed model for multivariate repeated measures: A relative efficiency evaluation. Communications in Statistics, Theory and Methods, 20, 1233-1255.
7. Boik, R. J. (1993). The analysis of two-factor interactions in fixed effects linear models. Journal of Educational Statistics, 18, 1-40.
8. Boik, R. J. (1997). Analysis of repeated measures under second-stage sphericity: An empirical Bayes approach. Journal of Educational and Behavioral Statistics, 22, 155-192.
9. Bryant, J. L., & Paulson, A. S. (1976). An extension of Tukey's method of multiple comparisons to experimental designs with random concomitant variables. Biometrika, 63, 631-638.
10. Bryk, A., Raudenbush, S., Seltzer, M. & Congdon, R., Jr. (1989). An introduction to HLM: Computer program and users' guide. Chicago: University of Chicago Press.
11. Carlson, J. E., & Timm, N. H. (1974). Analyses of nonorthogonal fixed effects designs. Psychological Bulletin, 81, 563-570.
12. Carver, R. P. (1993). The case against statistical significance testing, revisited. Journal of Experimental Education, 61, 287-292.
13. Ceurvorst, R. W., & Stock, W. A. (1978). Comments on the analysis of covariance with repeated measures designs. Multivariate Behavioral Research, 13, 509-513.
14. Christensen, W. F., & Rencher, A. C. (1997). A comparison of Type I error rates and power levels for seven solutions to the multivariate Behrens-Fisher problem. Communications in Statistics-Simulation and Computation, 26, 1251-1273.
15. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
16. Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304-1312.
17. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159.
18. Cohen, J., & Cohen, P. (1983). Applied multiple regression/correlation for the behavioral sciences. Hillsdale, NJ: Erlbaum.
19. Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally.
20. Coombs, W. T., & Algina, J. (1996). On sample size requirements for Johansen's test. Journal of Educational and Behavioral Statistics, 21, 169-178.
21. Coombs, W. T., & Algina, J. (1996). New test statistics for MANOVA/descriptive discriminant analysis. Educational and Psychological Measurement, 56, 382-402.
22. Coombs, W. T., Algina, J., & Oltman, D. O. (1996). Univariate and multivariate omnibus hypothesis tests selected to control Type I error rates when population variances are not necessarily equal. Review of Educational Research, 66, 137-179.
23. Coursol, A., & Wagner, E. E. (1986). Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias. Professional Psychology, 17, 136-137.
24. Davidson, M. L. (1972). Univariate versus multivariate tests in repeated measures experiments. Psychological Bulletin, 77, 446-452.
25. Delaney, H. D., & Maxwell, S. E. (1981). On using analysis of covariance in repeated measures design. Multivariate Behavioral Research, 16, 105-124.
26. Edgington, E. S. (1964). A tabulation of inferential statistics used in psychology journals. American Psychologist, 19, 202-203.
27. Ekstrom, D., Quade, D., & Golden, R. N. (1990). Statistical analysis of repeated measures in psychiatric research. Archives of General Psychiatry, 47, 770-772.
28. Elmore, P. B., & Woehlke, P. L. (1988). Statistical methods employed in American Educational Research Journal, Educational Researcher, and Review of Educational Research from 1978 to 1987. Educational Researcher, 17(9), 19-20.
29. Elmore, P. B., & Woehlke, P. L. (1998, April). Twenty years of research methods employed in American Educational Research Journal, Educational Researcher, and Review of Educational Research. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
30. Games, P. A. (1980). Alternative analyses of repeated-measure designs by ANOVA and MANOVA. In A. Von Eye (Ed.), Statistical methods in longitudinal research: Vol. 1. Principles and structuring change (pp. 81-121). San Diego, CA: Academic Press.
31. Glass, G. V, Peckham P. D., & Sanders, J. R. (1972). Consequences of failure to meet assumptions underlying the fixed effects analyses of variance and covariance. Review of Educational Research, 42, 237-288.
32. Goodwin, L. D., & Goodwin, W. L. (1985a). An analysis of statistical techniques used in the Journal of Educational Psychology, 1979-1983. Educational Psychologist, 20, 13-21.
33. Goodwin, L. D., & Goodwin, W. L. (1985b). Statistical techniques in AERJ articles, 1979-1983: The preparation of graduate students to read the educational research literature. Educational Researcher, 14(2), 5-11.
34. Hamilton, B. L. (1977). An empirical investigation of the effects of heterogeneous regression slopes in analysis of covariance. Educational and Psychological Measurement, 37, 701-712.
35. Harlow, L. L., Muliak, S. A., & Steiger, J. H. (Eds.). (1997). What if there were no significance tests? Hillsdale, NJ: Erlbaum.
36. Harwell, M. R., Rubinstein, E. N., Hayes, W. S., & Olds, C. C. (1992). Summarizing Monte Carlo results in methodological research: The one-and two-factor fixed effects ANOVA cases. Journal of Educational Statistics, 17, 315-339.
37. Hsiung, T., & Olejnik, S. (1994). Contrast analysis for additive non-orthogonal two-factor designs in unequal variance cases. British Journal of Mathematical and Statistical Psychology, 47, 337-354.
38. Huberty, C. J (1994). Applied discriminant analysis. New York: Wiley.
39. Huberty, C. J (1996, August). Some issues and problems in discriminant analysis. Paper presented at the Joint Statistical Meetings, Chicago, IL.
40. Huberty, C. J, & Lowman, L. (1996, June). The analysis of multivariate designs. Paper presented at the annual meeting of the Psychometric Society, Banff, Canada.
41. Huberty, C. J, & Morris, J. D. (1989). Multivariate analysis versus multiple univariate analyses. Psychological Bulletin, 105, 302-308.
42. Huberty, C. J, & Petoskey, M. D. (in press). Multivariate analysis of variance and covariance. In H. E. A. Tinsley & S. Brown (Eds.), Handbook of multivariate statistics and mathematical modeling. San Diego, CA: Academic Press.
43. James, G. S. (1954). Tests of linear hypotheses in univariate and multivariate analysis when the ratios of the population variances are unknown. Biometrika, 41, 19-43.
44. Keppel, G. (1991). Design and analysis: A researcher's handbook (3rd ed.). Englewood Cliffs, NJ: Prentice Hall.
45. Keselman, H. J. (1982). Multiple comparisons for repeated measures means. Multivariate Behavioral Research, 17, 87-92.
46. Keselman, H. J. (1994). Stepwise and simultaneous multiple comparison procedures of repeated measures means. Journal of Educational Statistics, 19, 127-162.
47. Keselman, H. J., & Algina, J. (1997). The analysis of higher-order repeated measures designs. In B. Thompson (Ed.), Advances in social science methodology. Greenwich, CT: JAI Press.
48. Keselman, H. J., Algina, J., Kowalchuk, R. K., & Wolfinger, R. D. (1997). The analysis of repeated measurements with mixed-model Satterthwaite F tests. Unpublished manuscript.
49. Keselman, H. J., Algina, J., Kowalchuk, R. K., & Wolfinger, R. D. (1998). A comparison of two approaches for selecting covariance structures in the analysis of repeated measurements. Communications in Statistics-Simulation and Computation, 27, 591-604.
50. Keselman, H. J., Carriere, K. C., & Lix, L. M. (1993). Testing repeated measures hypotheses when covariance matrices are heterogeneous. Journal of Educational Statistics, 18, 305-319.
51. Keselman, H. J., Carriere, K. C., & Lix, L. M. (1995). Robust and powerful nonorthogonal analyses. Psychometrika, 60, 395-418.
52. Keselman, H. J., & Keselman, J. C. (1993). Analysis of repeated measurements. In L. K. Edwards (Ed.), Applied analysis of variance in behavioral science (pp. 105-145). New York: Marcel Dekker.
53. Keselman, H. J., Keselman, J. C., & Games, P. A. (1991). Maximum familywise Type I error rate: The least significant difference, Newman-Keuls, and other multiple comparison procedures. Psychological Bulletin, 110, 155-161.
54. Keselman, H. J., Keselman, J. C., & Shaffer, J. P. (1991). Multiple pairwise comparisons of repeated measures means under violation of multisample sphericity. Psychological Bulletin, 110, 162-170.
55. Keselman, H. J., Kowalchuk, R. K., & Lix, L. M. (1998). Robust nonorthogonal analyses revisited: An update based on trimmed means. Psychometrika, 63, 145-163.
56. Keselman, H. J., & Lix, L. M. (1997). Analyzing multivariate repeated measures designs when covariance matrices are heterogeneous. British Journal of Mathematical and Statistical Psychology, 50, 319-338.
57. Keselman, H. J., Lix, L. M., & Kowalchuk, R. K. (1998). Multiple comparison procedures for trimmed means. Psychological Methods, 3, 123-141.
58. Keselman, H. J., Rogan, J. C., Mendoza, J. L., & Breen, L. J. (1980). Testing the validity conditions of repeated measures F tests. Psychological Bulletin, 87, 479-481.
59. Kirk, R. E. (1982). Experimental design: Procedures for the behavioral sciences (2nd ed.). Belmont, CA: Brooks/Cole.
60. Kirk, R. E. (1995). Experimental design: Procedures for the behavioral sciences (3rd ed.). Belmont, CA: Brooks/Cole.
61. Kirk, R. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.
62. Kowalchuk, R. K., Lix, L. M., & Keselman. (1996, June). The analysis of repeated measures design. Paper presented at the annual meeting of the Psychometric Society, Banff, Canada.
63. Kruskal, W. H., & Wallis, W. A. (1952). Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association, 47, 583-621.
64. Levene, H. (1960). Robust tests for equality of variances. In I. Olkin (Ed.), Contributions to probability and statistics. Stanford, CA: Stanford University Press.
65. Levin, J. R. (1997). Overcoming feelings of powerlessness in "aging" researchers: A primer on statistical power in analysis of variance designs. Psychology and Aging, 12, 84-106.
66. 0? Educational and Psychological Measurement, 58, 313-333.
67. Levin, J. R., Serlin, R. C., & Seaman, M. A. (1994). A controlled, powerful multiple-comparison strategy for several situations. Psychological Bulletin, 115, 153-159.
68. Levy, K. J. (1980). A Monte Carlo study of analysis of covariance under violations of the assumptions of normality and equal regression slopes. Educational and Psychological Measurement, 40, 835-840.
69. Lix, L. M., Cribbie, R. A., & Keselman, H. J. (1996, June). The analysis of between-subjects univariate designs. Paper presented at the annual meeting of the Psychometric Society, Banff, Canada.
70. Lix, L. M., & Keselman, H. J. (1995). Approximate degrees of freedom tests: A unified perspective on testing for mean equality. Psychological Bulletin, 117, 547-560.
71. Lix, L. M., & Keselman, H. J. (1996). Interaction contrasts in repeated measures designs. British Journal of Mathematical and Statistical Psychology, 49, 147-162.
72. Lix, L. M., & Keselman, H. J. (1998). To trim or not to trim: Tests of location equality under heteroscedasticity and nonnormality. Educational and Psychological Measurement, 58, 409-429.
73. Lix, L. M., Keselman, J. C., & Keselman, H. J. (1996). Consequences of assumption violations revisited: A quantitative review of alternatives to the one-way analysis of variance F test. Review of Educational Research, 66, 579-620.
74. Marascuilo, L. A., & Levin, J. R. (1970). Appropriate post hoc comparisons for interaction and nested hypotheses in analysis of variance designs: The elimination of Type IV errors. American Educational Research Journal, 7, 397-421.
75. Mauchly, J. W. (1940). Significance test for sphericity of a normal n-variate distribution. Annals of Mathematical Statistics, 29, 204-209.
76. Maxwell, S. E. (1980). Pairwise multiple comparisons in repeated measures designs. Journal of Educational Statistics, 5, 269-287.
77. Maxwell, S. E., & Delaney, H. D. (1990). Designing experiments and analyzing data: A model comparison perspective. Belmont, CA: Wadsworth.
78. Maxwell, S. E., O'Callaghan, M. F., & Delaney, H. D. (1993). Analysis of covariance. In L. K. Edwards (Ed.), Applied analysis of variance in behavioral science. New York: Marcel Dekker.
79. McAuliffe, T. J., & Dembo, M. H. (1994). Status rules of behavior in scenarios of peer learning. Journal of Educational Psychology, 86, 163-172.
80. McCall, R. B., & Appelbaum, M. I. (1973). Bias in the analysis of repeated-measures designs: Some alternative approaches. Child Development, 44, 401-415.
81. Mendoza, J. L. (1980). A significance test for multisample sphericity. Psychometrika, 45, 495-498.
82. Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, 156-166.
83. Milligan, G. W., Wong, D. S., & Thompson, P. A. (1987). Robustness properties of nonorthogonal analysis of variance. Psychological Bulletin, 101, 464-470.
84. Norušis, M. J. (1993). SPSS for Windows advanced statistics Release 6. Chicago: SPSS.
85. O'Brien, R. G., & Kaiser, M. K. (1985). MANOVA method for analyzing repeated measures designs: An extensive primer. Psychological Bulletin, 97, 316-333.
86. O'Brien, R. G., & Muller, K. E. (1993). Unified power analysis for t-tests through multivariate hypotheses. In L. K. Edwards (Ed.), Applied analysis of variance in behavioral science (pp. 297-344). New York: Marcel Dekker.
87. Olejnik, S., & Donahue (1996, June). The analysis ofcovariate designs. Paper presented at the annual meeting of the Psychometric Society, Banff, Canada.
88. Olejnik, S., & Hess, B. (1997). Top ten reasons why most omnibus ANOVA F-tests should be abandoned. Journal of Vocational Education Research, 22, 219-232.
89. Olejnik, S., & Huberty, C. J (1993, April). Preliminary statistical tests. Paper presented at the annual meeting of the American Educational Research Association, Atlanta, GA.
90. Olejnik, S., & Lee, J. L. (1990). Multiple comparison procedures when population variances differ. Athens: University of Georgia. (ERIC Document Reproduction Service No. ED 319 754)
91. Orr, J. M., Sackett, P. R., & Dubois, C. L. Z. (1991). Outlier detection and treatment in "I/O psychology": A survey of researcher beliefs and an empirical illustration. Personnel Psychology, 44, 473-486.
92. Ridgeway, V. G., Dunston, P. J., & Qian, G. (1993). A methodological analysis of teaching and learning strategy research at the secondary school level. Reading Research Quarterly, 28, 335-349.
93. Robinson, D. H., & Levin, J. R. (1997). Reflections on statistical and substantive significance, with a slice of replication. Educational Researcher, 26, 21-26.
94. Rogosa, D. (1980). Comparing non-parallel regression lines. Psychological Bulletin, 88, 307-321.
95. Romaniuk, J. G., Levin, J. R., & Hubert, L. J. (1977). Hypothesis-testing procedures in repeated measures designs: On the road map not taken. Child Development, 48, 1757-1760.
96. Roth, P. L. (1994). Missing data: A conceptual review for applied psychologists. Personnel Psychology, 47, 537-560.
97. SAS. (1990). SAS/STAT user's guide. Cary, NC: Author.
98. Schmidt, F. L. (1992). What do data really mean? American Psychologist, 47, 1173-1181.
99. Seaman, M. A., Levin, J. R., & Serlin, R. C. (1991). New developments in pairwise multiple comparisons: Some powerful and practicable procedures. Psychological Bulletin, 110, 577-586.
100. Seidman, E., Allen, L., Aber, J. L., Mitchell, C., & Feinman, J. (1994). The impact of school transitions in early adolescence on the self-esteem and perceived social context of poor urban youth. Child Development, 65, 507-522.
101. Simpson, M. L., Olejnik, S., Tam, A. Y., & Supattathum, S. (1994). Elaborative verbal rehearsals and college students' cognitive performance. Journal of Educational Psychology, 86, 267-278.
102. Steinberg, L., Lamborn, S. D., Darling, N., Mounts, N. S., & Dornbusch, S. M. (1994). Over-time changes in adjustment and competence among adolescents from authoritative, authoritarian, indulgent, and neglectful families. Child Development, 65, 754-770.
103. Stevens, J. (1996). Applied multivariate statistics for the social sciences. Mahwah, NJ: Erlbaum.
104. Tabachnick, B. G., & Fidell, L. S. (1996). Using multivariate statistics (3rd ed.). New York: Harper Collins.
105. Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher, 25, 26-30.
106. West, C. K., Carmody, C., & Stallings, W. M. (1983). The quality of research articles in the Journal of Educational Research, 1970 and 1980. Journal of Educational Research, 77, 70-76.
107. Wilcox, R. R. (1987). New designs in analysis of variance. Annual Review of Psychology, 38, 29-60.
108. Wilcox, R. R. (1993). Analysing repeated measures or randomized block designs using trimmed means. British Journal of Mathematical and Statistical Psychology, 46, 63-76.
109. Wilcox, R. R. (1995). ANOVA: A paradigm for low power and misleading measures of effect size? Review of Educational Research, 65, 51-77.
110. Wilcox, R. R. (1996). Statistics for the social sciences. New York: Academic Press.
111. Wilcox, R. R. (1998). How many discoveries have been lost by ignoring modern statistical methods? American Psychologist, 53, 300-314.
112. Wilcox, R. R., Charlin, V. L., & Thompson, K. L. (1986). New Monte Carlo results on the robustness of the ANOVA F, W, and F* statistics. Communications in Statistics-Simulation and Computation, 15, 933-943.
113. Wilkinson, L. (1988). SYSTAT: The system for statistics. Evanston, IL: SYSTAT.
114. Winer, B. J. (1971). Statistical principles in experimental design (2nd ed.). New York: McGraw-Hill.
115. Winer, B. J., Brown, D. R., & Michels, K. M. (1991). Statistical principles in experimental design. New York: McGraw-Hill.