Statistical Significance and Replicability: Why the Former Does not Presage the Latter

Theory & Psychology

Volumn 8, Issue 3, 1998, Pages 291-311

  Sohn, David ^a  

^a UNIVERSITY OF NORTH CAROLINA AT CHARLOTTE   (United States)

Author keywords

null hypothesis testing; replicability and replication; significance testing; statistical significance

Indexed keywords

EID: 0032377959     PISSN: 09593543     EISSN: None     Source Type: Journal    
DOI: 10.1177/0959354398083001     Document Type: Article

References (43)

1. Bakan, D.(1966). The test of significance in psychological research. Psychological Bulletin, 66, 423-437.
2. Baril, G.L., & Cannon, T. (1995). What is the probability that null hypothesis testing is meaningless? American Psychologist, 50, 1098-1099.
3. Barlow, D.H., & Hersen, M. (1984). Single case experimental designs: Strategies for studying behavior change (2nd ed.). New York: Pergamon.
4. Bem, D.J., & Honorton, C. (1994). Does psi exist? Replicable evidence for an anomalous process of information transfer. Psychological Bulletin, 115, 4-18.
5. Bergin, A.E., & Strupp, H.H. (1970). New directions in psychotherapy research. Journal of Abnormal Psychology, 76, 13-26.
6. Carver, R.P.(1978). The case against statistical significance testing. Harvard Educational Review, 48, 378-399.
7. Carver, R.P.(1993). The case against statistical significance testing, revisited. Journal of Experimental Education, 61, 287-292.
8. Cohen, J.(1994). The earth is round (p <.05). American Psychologist, 49, 997-1003.
9. Elmes, D.G., Kantowitz, B.H., & Roediger, H. L., III (1995). Research methods in psychology (5th ed.). St Paul, MN: West.
10. Estes, W.K.(1991). Statistical models in behavioral research. Hillsdale, NJ: Erlbaum.
11. Evans, J.D.(1985). Invitation to psychological research. New York: Holt, Rinehart & Winston.
12. Falk, R.(1986). Misconceptions of statistical inference. Journal of Structural Learning, 9, 83-96.
13. Falk, R., & Greenbaum, C.W. (1995). Significance tests die hard: The amazing persistence of a probabilistic misconception. Theory & Psychology, 5, 75-98.
14. Fisher, R.A.(1925). Theory of statistical estimation. Proceedings of the Cambridge Philosophical Society, 22, 700-725.
15. Fisher, R.A.(1960). The design of experiments (7th ed.). New York: Hafner.
16. Gigerenzer, G.(1993). The Superego, the Ego, and the Id in statistical reasoning. In G. Keren & C. Lewis (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues (pp. 311-339). Hillsdale, NJ: Erlbaum.
17. Gigerenzer, G., & Murray, D.J. (1987). Cognition as intuitive statistics. Hillsdale, NJ: Erlbaum.
18. Goodman, S.N.(1992). A comment on replication, p-values and evidence. Statistics in Medicine, 11, 875-879.
19. Greenwald, A.G., Gonzalez, R., Harris, R.J., & Guthrie, D. (1996). Effect sizes and p values: What should be reported and what should be replicated? Psychophysiology, 33, 175-183.
20. Hagood, M.J.(1970). The notion of a hypothetical universe. In D. E. Morrison & R. E. Henkel (Eds.), The significance test controversy-A reader (pp. 65-78). Chicago, IL: Aldine.
21. Judd, C.M., McClelland, G.H., & Culhane, S.E. (1995). Data analysis: Continuing issues in the everyday analysis of psychological data. Annual Review of Psychology, 46, 433-465.
22. Levin, J.R.(1993). Statistical significance testing from three perspectives. Journal of Experimental Education, 61, 378-382.
23. Levy, P.(1967). Substantive significance of significant differences between two groups. Psychological Bulletin, 67, 37-40.
24. Lykken, D.T.(1968). Statistical significance in psychological research. Psychological Bulletin, 70, 151-159.
25. Lykken, D.T.(1991). What's wrong with psychology anyway? In D. Cicchetti & W. M. Grove (Eds.), Thinking clearly about psychology: Vol. 1. Matters of public interest (pp. 3-39). Minneapolis: University of Minnesota Press.
26. Melton, A.W.(1962). Editorial. Journal of Experimental Psychology, 64, 553-557.
27. Neyman, J., Iwaszkiewicz, K., & Kolosziejczyk, S. (1935). Statistical problems in agricultural experimentation. Supplement to Journal of the Royal Statistical Society, 2, 107-154.
28. Nunnally, J.C.(1975). Introduction to statistics for psychology and education. New York: McGraw-Hill.
29. Oakes, M.(1986). Statistical inference: A commentary for the social and behavioral sciences. New York: Wiley.
30. Phillips, L.D.(1973). Bayesian statistics for social scientists. London: Nelson.
31. Pollard, P., & Richardson, J. T. E. (1987). On the probability of making Type I errors. Psychological Bulletin, 102, 159-163.
32. Reaves, C.C.(1992). Quantitative research for the behavioral sciences. New York: Wiley.
33. Rorer, L.G.(1991). Some myths of science in psychology. In D. Cicchetti & W. M. Grove (Eds.), Thinking clearly about psychology: Vol. 1. Matters of public interest (pp. 61-87). Minneapolis: University of Minnesota Press.
34. Rosenthal, R.(1993). Cumulating evidence. In G. Keren & C. Lewis (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues (pp. 519-559). Hillsdale, NJ: Erlbaum.
35. Rozeboom, W.W.(1960). The fallacy of the null hypothesis significance test. Psychological Bulletin, 57, 416-428.
36. Shaughnessy, J.J., & Zechmeister, E.B. (1994). Research methods in psychology (3rd ed.). New York: McGraw-Hill.
37. Sidman, M.(1960). Tactics of scientific research. New York: Basic Books.
38. Sohn, D.(1993). Psychology of the scientist: LXVI. The idiot savants have taken over the psychology labs! Or why in science using the the rejection of the null hypothesis as the basis for affirming the research hypothesis is unwarranted. Psychological Reports, 73, 1167-1175.
39. Steering Committee of the Physicians' Health Study Research Group(1988). Preliminary report: Findings of the aspirin component of the ongoing physicians' health study. New England Journal of Medicine, 318, 262-264.
40. Thompson, B.(1993). The use of significance tests in research: Bootstrap and other alternatives. Journal of Experimental Education, 61, 361-377.
41. Thompson, B.(1994). The pivotal role of replication in psychological research: Empirically evaluating the replicability of sample results. Journal of Personality, 62, 157-176.
42. Tversky, A., & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76, 105-110.
43. Winch, R.F., & Campbell, D.T. (1969). Proof? No. Evidence? Yes. The significance of tests of significance. The American Sociologist, 4, 140-143.