A Generally Robust Approach for Testing Hypotheses and Setting Confidence Intervals for Effect Sizes

Psychological Methods

Volumn 13, Issue 2, 2008, Pages 110-129

  Keselman, H J ^a   Algina, James ^b   Lix, Lisa M ^a   Wilcox, Rand R ^c   Deering, Kathleen N ^d  

^a UNIVERSITY OF MANITOBA   (Canada)

^b UNIVERSITY OF FLORIDA   (United States)

^c UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^d UNIVERSITY OF BRITISH COLUMBIA   (Canada)

Author keywords

classical robust confidence intervals; classical robust effect size statistics; classical robust test statistics; online computer program; online numerical examples

Indexed keywords

ARTICLE; CONFIDENCE INTERVAL; HUMAN; METHODOLOGY; PSYCHOLOGICAL MODEL; PSYCHOLOGY; SAMPLE SIZE; STATISTICS;

CONFIDENCE INTERVALS; HUMANS; MODELS, PSYCHOLOGICAL; PSYCHOLOGY; SAMPLE SIZE;

EID: 45449109015     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/1082-989X.13.2.110     Document Type: Article

References (80)

1. Algina J., and Keselman H.J. Testing repeated measures hypotheses when covariance matrices are heterogeneous: Revisiting the robustness of the Welch-James test. Multivariate Behavioral Research 32 (1997) 255-274
2. Algina J., and Keselman H.J. Approximate confidence intervals for effect sizes. Educational and Psychological Measurement 63 (2003) 537-553
3. Algina J., Keselman H.J., and Penfield R.D. An alternative to Cohen's standardized mean difference effect size: A robust parameter and confidence interval in the two independent group case. Psychological Methods 10 (2005) 317-328
4. Algina J., Keselman H.J., and Penfield R.D. Effect sizes and their intervals: The two-level repeated measures case. Educational and Psychological Measurement 65 (2005) 241-258
5. Algina J., Keselman H.J., and Penfield R.D. Confidence intervals for an effect size when variances are not equal. Journal of Modern Applied Statistical Methods 5 (2006) 2-13
6. Algina J., and Olejnik S.F. Implementing the Welch-James procedure with factorial designs. Educational and Psychological Measurement 44 (1984) 39-48
7. American Psychological Association. Publication manual of the American Psychological Association. 5th ed. (2001), Author, Washington, DC
8. Aspin A.A. An examination and further development of a formula arising in the problem of comparing two mean values. Biometrika 35 (1947) 88-96
9. Bird K.D. Confidence intervals for effect sizes in analysis of variance. Educational and Psychological Measurement 62 (2002) 197-226
10. Bradley J.V. Nonrobustness in Z, t, and F tests at large sample sizes. Bulletin of the Psychonomic Society 16 (1980) 333-336
11. Brown M.B., and Forsythe A.B. The small sample behavior of some statistics which test the equality of several means. Technometrics 16 (1974) 129-132
12. Brunner E., Dette H., and Munk A. Box type approximations in nonparametric factorial designs. Journal of the American Statistical Association 92 (1997) 1494-1502
13. Brunner E., and Munzel U. The nonparametric Behrens-Fisher problem: Asymptotic theory and a small sample approximation. Biometrical Journal 42 (2000) 17-25
14. Cliff N. Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological Bulletin 114 (1993) 494-509
15. Cliff N. Answering ordinal questions with ordinal data using ordinal statistics. Multivariate Behavioral Research 31 (1996) 331-350
16. Cohen J. Some statistical issues in psychological research. In: Wolman B.B. (Ed). Handbook of clinical psychology (1965), Academic Press, New York 95-121
17. Conover W.J., and Iman R.L. Rank transformation as a bridge between parametric and nonparametric statistics. The American Statistician 35 (1981) 124-129
18. Cumming G., and Finch S. A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. Educational and Psychological Measurement 61 (2001) 532-574
19. Glass G.V., McGaw B., and Smith M.L. Meta-analysis in social research (1981), Sage, Newbury Park, CA
20. Hampel F.R., Ronchetti E.M., Rousseeuw P.J., and Stahel W.A. Robust statistics: The approach based on influence functions (1986), Wiley, New York
21. Harwell M.R., Rubinstein E.N., Hayes W.S., and Olds C.C. Summarizing Monte Carlo results in methodological research: The one- and two-factor fixed effects ANOVA cases. Journal of Educational Statistics 17 (1992) 315-339
22. Hays W.L. Statistics (1963), Holt, Rinehart and Winston, New York
23. Hedges L.V., and Olkin I. Statistical methods for meta-analysis (1985), Academic Press, Orlando, FL
24. Hochberg Y., and Tamhane A.C. Multiple comparison procedures (1987), Wiley, New York
25. Hogarty, K. Y., & Kromrey, J. D. (2001, April). We've been reporting some effect sizes: Can you guess what they mean? Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA.
26. Huber P.J. Robust statistics (1981), Wiley, New York
27. James G.S. The comparison of several groups of observations when the ratios of the population variances are unknown. Biometrika 38 (1951) 324-329
28. Johansen S. The Welch-James approximation to the distribution of the residual sum of squares in a weighted linear regression. Biometrika 67 (1980) 85-92
29. Kelly K. The effects of nonnormal distributions on confidence intervals around the standardized mean difference: Bootstrap and parametric confidence intervals. Educational and Psychological Measurement 65 (2005) 51-69
30. Keselman H.J. Testing treatment effects in repeated measures designs: An update for psychophysiological researchers. Psychophysiology 35 (1998) 470-478
31. Keselman H.J., Algina J., Wilcox R.R., and Kowalchuk R.K. Testing repeated measures hypotheses when covariance matrices are heterogeneous: Revisiting the robustness of the Welch-James test again. Educational and Psychological Measurement 60 (2000) 925-938
32. Keselman H.J., Carriere K.C., and Lix L.M. Testing repeated measures hypotheses when covariance matrices are heterogeneous. Journal of Educational Statistics 18 (1993) 305-319
33. Keselman H.J., Carriere K.C., and Lix L.M. Robust and powerful nonorthogonal analyses. Psychometrika 60 (1995) 395-418
34. Keselman H.J., Carriere K.C., and Lix L.M. Errata to "Robust and powerful nonorthogonal analyses.". Psychometrika 61 (1996) 191
35. Keselman H.J., Cribbie R.A., and Holland B. Pairwise multiple comparison test procedures: An update for clinical child and adolescent psychologists. Journal of Clinical Child and Adolescent Psychology 33 (2004) 623-645
36. Keselman H.J., Huberty C.J., Lix L.M., Olejnik S., Cribbie R., Donahue B., et al. Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analyses. Review of Educational Research 68 (1998) 350-386
37. Keselman H.J., Kowalchuk R.K., Algina J., Lix L.M., and Wilcox R.R. Testing treatment effects in repeated measures designs: Trimmed means and bootstrapping. British Journal of Mathematical and Statistical Psychology 53 (2000) 175-191
38. Keselman H.J., Kowalchuk R.K., and Lix L.M. Robust nonorthogonal analyses revisited: An update based on trimmed means. Psychometrika 63 (1998) 145-163
39. Keselman H.J., and Lix L.M. Improved repeated measures stepwise multiple comparison procedures. Journal of Educational and Behavioral Statistics 20 (1995) 83-99
40. Keselman H.J., and Lix L.M. Analyzing multivariate repeated measures designs when covariance matrices are heterogeneous. British Journal of Mathematical and Statistical Psychology 50 (1997) 319-338
41. Keselman H.J., and Rogan J.C. Repeated measures F tests and psychophysiological research: Controlling the number of false positives. Psychophysiology 17 (1980) 499-503
42. Kline R.B. Beyond significance testing: Reforming data analysis methods in behavioral research (2004), American Psychological Association, Washington, DC
43. Kohr R.L., and Games P.A. Robustness of the analysis of variance, the Welch procedure and a Box procedure to heterogeneous variances. Journal of Experimental Education 43 (1974) 61-69
44. Kraemer H.C., and Andrews G.A. A nonparametric technique for meta-analysis effect size calculation. Psychological Bulletin 91 (1982) 404-412
45. Kulinska E., and Staudte R.G. Interval estimates of weighted effect sizes in the one-way heteroscedastic ANOVA. British Journal of Mathematical and Statistical Psychology 59 (2006) 97-111
46. Lix L.M., and Keselman H.J. Approximate degrees of freedom tests: A unified perspective on testing for mean equality. Psychological Bulletin 117 (1995) 547-560
47. Lix L.M., and Keselman H.J. Interaction contrasts in repeated measures designs. British Journal of Mathematical and Statistical Psychology 49 (1996) 147-162
48. Lix L.M., and Keselman H.J. To trim or not to trim: Tests of mean equality under heteroscedasticity and nonnormality. Educational and Psychological Measurement 58 (1998) 409-429 853
49. Luh W., and Guo J. Approximate sample size formulas for the two-sample trimmed mean test with unequal variances. British Journal of Mathematical and Statistical Psychology 60 (2007) 137-146
50. Marazzi A., and Ruffieux C. The truncated mean of an asymmetric distribution. Computational Statistics & Data Analysis 32 (1999) 79-100
51. Maxwell S.E., and Delaney H.D. Designing experiments and analyzing data: A model comparison perspective. 2nd ed. (2004), Erlbaum, Mahwah, NJ
52. McGraw K.O., and Wong S.P. A common language effect size statistic. Psychological Bulletin 111 (1992) 361-365
53. Milligan G.W., Wong D.S., and Thompson P.A. Robustness properties of nonorthogonal analysis of variance. Psychological Bulletin 101 (1987) 464-470
54. Murphy K.R. Editorial. Journal of Applied Psychology 82 (1997) 3-5
55. Ramsey P.H. Exact Type 1 error rates for robustness of Student's t test with unequal variances. Journal of Educational Statistics 5 (1980) 337-349
56. SAS Institute. SAS/IML: User's guide (Version 8) (1999), Author, Cary, NC
57. Smithson M. Confidence intervals (2003), Sage, Thousand Oaks, CA
58. Staudte R.G., and Sheather S.J. Robust estimation and testing (1990), Wiley, New York
59. Steiger J.H. Beyond the F test: Effect size confidence intervals and tests of close fit in the analysis of variance and contrast analysis. Psychological Methods 9 (2004) 164-182
60. Steiger J.H., and Fouladi R.T. Noncentrality interval estimation and the evaluation of statistical models. In: Harlow L., Mulaik S., and Steiger J.H. (Eds). What if there were no significance tests? (1997), Erlbaum, Hillsdale, NJ 221-257
61. Thompson B. Guidelines for authors. Educational and Psychological Measurement 54 (1994) 837-847
62. Timm N.H. Applied multivariate analysis (2002), Springer, New York
63. Tukey J.W. A survey of sampling from contaminated normal distributions. In: Olkin I., et al. (Ed). Contributions to probability and statistics (1960), Stanford University Press, Stanford, CA 448-485
64. Vargha A., and Delaney H.D. A critique and improvement of the CL Common Language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics 25 (2000) 101-132
65. Welch B.L. The significance of the difference between two means when the population variances are unequal. Biometrika 29 (1938) 350-362
66. Welch B.L. The generalization of Students' problem when several different population variances are involved. Biometrika 34 (1947) 23-35
67. Welch B.L. On the comparison of several mean values: An alternative approach. Biometrika 38 (1951) 330-336
68. Westfall P.H., and Young S.S. Resampling-based multiple testing (1993), Wiley, New York
69. Wilcox R.R. ANOVA: A paradigm for low power and misleading measures of effect size?. Review of Educational Research 65 1 (1995) 51-77
70. Wilcox R.R. Introduction to robust estimation and hypothesis testing (1997), Academic Press, San Diego, CA
71. Wilcox R.R. Applying contemporary statistical techniques (2003), Academic Press, New York
72. Wilcox R.R. Introduction to robust estimation and hypothesis testing. 2nd ed. (2005), Elsevier, New York
73. Wilcox R.R., Charlin V.L., and Thompson K. New Monte Carlo results on the robustness of the ANOVA F, W, and F* statistics. Communications in Statistics. Simulation and Computation 15 (1986) 933-944
74. Wilcox R.R., and Keselman H.J. Modern robust data analysis methods: Measures of central tendency. Psychological Methods 8 (2003) 254-274
75. Wilcox R.R., Keselman H.J., and Kowalchuk R.K. Can tests for treatment group equality be improved?: The bootstrap and trimmed means conjecture. British Journal of Mathematical and Statistical Psychology 51 (1998) 123-134
76. Wilcox R.R., Keselman H.J., Muska J., and Cribbie R. Repeated measures ANOVA: Some new results on comparing trimmed means and means. British Journal of Mathematical and Statistical Psychology 53 (2000) 69-82
77. 2. British Journal of Mathematical and Statistical Psychology 52 (1999) 93-110
78. Wilkinson L., and the Task Force on Statistical Inference. Statistical methods in psychology journals. American Psychologist 54 (1999) 594-604
79. Yuen K.K. The two-sample trimmed t for unequal population variances. Biometrika 61 (1974) 165-170
80. Zimmerman D.W., and Zumbo B.D. Relative power of parametric and nonparametric statistical methods. In: Keren G., and Lewis C. (Eds). A handbook for data analysis in the behavioral sciences: Vol. 1. Methodological issues (1993), Erlbaum, Hillsdale, NJ 481-517