Making subjective judgments in quantitative studies: The importance of using effect sizes and confidence intervals

Human Resource Development Quarterly

Volumn 17, Issue 2, 2006, Pages 159-173

  Callahan, Jamie L ^a   Reio Jr , Thomas G ^b  

^a TEXAS A AND M UNIVERSITY   (United States)

^b UNIVERSITY OF LOUISVILLE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 33746092046     PISSN: 10448004     EISSN: 15321096     Source Type: Journal    
DOI: 10.1002/hrdq.1167     Document Type: Article

References (45)

1. American Psychological Association. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.
2. Anderson, D. R., Burnham, K. P, & Thompson, W. L. (2000). Null hypothesis testing: Problems, prevalence, and an alternative. Journal of Wildlife Management, 64, 912-923.
3. Arnemann, K. (2003, February 13-15). A review of the panoply of effect size choices. Paper presented at the Southwest Educational Research Association, San Antonio, TX.
4. Balluerka, N., Gómez, J., & Hidalgo, D. (2005). The controversy over null hypothesis significance testing revisited. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 1, 55-70.
5. Boring, E. G. (1919). Mathematical vs. scientific significance. Psychological Bulletin, 16, 335-338.
6. Capraro, M. M., & Capraro, R. M. (2003). Exploring the APA fifth edition Publication Manuals impact on the analytic preferences of journal editorial board members. Educational and Psychological Measurement, 63, 554-565.
7. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Mahwah, NJ: Erlbaum.
8. Cohen, J. (1994). The earth is round (p <.05). American Psychologist, 49, 997-1003.
9. Cohen, J., Cohen, P, West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences. Mahwah, NJ: Erlbaum.
10. Cortina, J. M. (2002). Big things have small beginnings: An assortment of "minor" methodological misunderstandings. Journal of Management, 28, 339-362.
11. Cortina, J. M., & Dunlap, W. P. (1997). On the logic and purpose of significance testing. Psychological Methods, 2 (2), 161-172.
12. Cumming, G., & Finch, S. (2001). A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. Educational and Psychological Measurement, 61, 530-572.
13. Gumming, G., & Finch, S. (2005). Inference by eye: Confidence intervals and how to read pictures of data. American Psychologist, 60, 170-180.
14. Fan, X. (2001). Statistical significance and effect size in education research: Two sides of a coin. Journal of Educational Research, 94, 275-282.
15. Fidler, F. (2002). The fifth edition of the APA Publication Manual: Why its statistics recommendations are so controversial. Educational and Psychological Measurement, 62, 749-770.
16. Fidler, F., Gumming, G., Burgman, M., & Thomason, N. (2004). Statistical reform in medicine, psychology and ecology. Journal of Socioeconomics, 33, 615-630.
17. Fidler, F., Thomason, N., Gumming, G., Finch, S., & Leeman, J. (2004). Editors can lead researchers to confidence intervals, but can't make them think. Psychological Science, 15, 119-126.
18. Fidler, F., Thomason, N., Gumming, G., Finch, S., & Leeman, J. (2005). Still much to learn about confidence intervals. Psychological Science, 16, 494-495.
19. Henson, R. K., & Smith, A. D. (2000). State of the art in statistical significance and effect size reporting: A review of the APA task force report and current trends. Journal of Research and Development in Education, 33, 285-296.
20. Huberty, C. J. (2002). A history of effect size indices. Educational and Psychological Measurement, 62, 227-240.
21. Huberty, C. J., &r Lowman, L. L. (2000). Group overlap as a basis for effect size. Educational and Psychological Measurement, 60, 543-563.
22. Huck, S. W. (2000). Reading Statistics and Research (3rd ed.). New York: Longman.
23. Judd, C. M., Smith, E. R., & Kidder, L. H. (1991). Research methods in social relations (6th ed.). Orlando, FL: Harcourt Brace.
24. Killeen, P. R. (2005). An alternative to null-hypothesis significance tests. Psychological Science, 16, 345-353.
25. Kirk, R. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.
26. Kirk, R. E. (2001). Promoting good statistical practices: Some suggestions. Educational and Psychological Measurement, 61, 213-218.
27. Krueger, J. (2001). Null hypothesis significance testing: On the survival of a flawed method. American Psychologist, 56 (1), 16-26.
28. Neyman, J. (1941). Fiducial argument and the theory of confidence intervals. Biometrika, 32, 128-150.
29. Nickerson, R. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5 (2), 241-301.
30. Nix, T. W, &r Barnette, J. J. (1998). The data analysis dilemma: Ban or abandon. A review of null hypothesis testing. Research in the Schools, 5, 3-14.
31. Olejnik, S., & Algina J. (2000). Measures of effect size for comparative studies: Applications, interpretations, and limitations. Contemporary Educational Psychology, 25, 241-286.
32. Rouder, J. N., & Morey, R. D. (2005). Relational and arelational confidence intervals: A comment on Fidler, Thomason, Cumming, Finch, and Leeman (2004). Psychological Science, 16, 77-79.
33. Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309-316.
34. Snyder, P., & Lawson, S. (1993). Evaluating results using corrected and uncorrected effect size estimates. Journal of Experimental Education, 61, 334-349.
35. Sohlberg, S., & Andersson, G. (2005). Extracting a maximum of useful information from statistical research data. Scandinavian Journal of Psychology, 46, 69-77.
36. Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher, 25 (2), 26-30.
37. Thompson, B. (1998). Statistical significance and effect size reporting: Portrait of a possible future. Research in the Schools, 5 (2), 33-38.
38. Thompson, B. (1999). Common methodology mistakes in educational research, revisited: Along with a primer on both effect sizes and the bootstrap. Invited address to the American Educational Research Association, Montreal. (ERIC Document Reproduction Service No. ED429110)
39. Thompson, B. (2002a). "Statistical," "practical," and "clinical": How many kinds of significance do counselors need to consider? Journal of Counseling and Development, 80, 64-71.
40. Thompson, B. (2002b). What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31, 24-31.
41. Thompson, B., Diamond, K. E., McWilliam, R., Snyder, P., & Snyder, S. W. (2005). Evaluating the quality of evidence from correlational research for evidence-based practice. Exceptional Children, 71, 181-194.
42. Vacha-Haase, T. (2001). Statistical significance should not be considered one of life's guarantees: Effect sizes are needed. Educational and Psychological Measurement, 61, 219-224.
43. Vaske, J. J., Gliner, J. A., & Morgan, G. A. (2002). Communicating judgments about practical significance: Effect size, confidence intervals and odds ratios. Human Dimensions of Wildlife, 7, 287-300.
44. Wilkinson, L., & APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604,
45. Yates, F. (1951). The influence of "Statistical methods for research workers" on the development of the science of statistics, journal of the American Statistical Association, 46, 19-34.