The significance fallacy in inferential statistics Psychology

BMC Research Notes

Volumn 8, Issue 1, 2015, Pages

  Kühberger, Anton ^a   Fritz, Astrid ^b   Lermer, Eva ^c   Scherndl, Thomas ^a  

^a UNIVERSITY OF SALZBURG   (Austria)

^b Österreichisches Zentrum für Begabtenförderung und Begabungsforschung   (Austria)

^c UNIVERSITY OF MUNICH   (Germany)

Author keywords

Effect size; NHST; Practical significance; Sample size; Statistical significance

Indexed keywords

ADOLESCENT; ADULT; FEMALE; HUMAN; MALE; SAMPLE SIZE; STATISTICAL ANALYSIS; YOUNG ADULT;

ADOLESCENT; ADULT; DATA INTERPRETATION, STATISTICAL; FEMALE; HUMANS; MALE; SAMPLE SIZE; YOUNG ADULT;

EID: 84925842576     PISSN: None     EISSN: 17560500     Source Type: Journal    
DOI: 10.1186/s13104-015-1020-4     Document Type: Article

References (73)

1. Cumming G. The new statistics: why and how. Psychol Sci. 2014;25:7-29.
2. Dienes Z. Bayesian versus orthodox statistics: which side are you on? Perspect Psychol Sci. 2011;6:274-90.
3. Nickerson RS. Null hypothesis significance testing: a review of an old and continuing controversy. Psychol Methods. 2000;5:241-301.
4. Pashler H, Wagenmakers EJ. Editors' introduction to the special section on replicability in psychological science: a crisis of confidence? Perspect Psychol Sci. 2012;7:528-30.
5. Ives B. Effect size use in studies of learning disabilities. J Learn Disabil. 2003;36:490-504.
6. Cohen J. Things I have learned (so far). Am Psychol. 1990;45:1304-12.
7. Fan X. Statistical significance and effect size in education research: two sides of a coin. J Educ Res. 2001;94:275-83.
8. Greenwald AG, Gonzalez R, Guthrie DG, Harris RJ. Effect sizes and p values: what should be reported and what should be replicated? Psychophysiol. 1996;33:175-83.
9. Hedges LV. What are effect sizes and why do we need them? Child Dev Perspect. 2008;2:167-71.
10. Kirk RE. Effect magnitude: a different focus. J Stat Plan Inference. 2007;137:1634-46.
11. Thompson B. 'Statistical', 'practical', and 'clinical': How many kinds of significance do counselors need to consider? J Couns Dev. 2002;80:64-71.
12. Vacha-Haase T. Statistical significance should not be considered one of life's guarantees: effect sizes are needed. Educ Psychol Meas. 2001;61:219-24.
13. Castro Sotos AE, Vanhoof S, Van den Noortgate W, Onghena P. Students' misconceptions of statistical inference: a review of the empirical evidence from research on statistics education. Educ Res Rev. 2007;2:98-113.
14. Fidler F, Cumming G, Thomason N, Pannuzzo D, Smith J, Fyffe P, et al. Evaluating the effectiveness of editorial policy to improve statistical practice: the case of the Journal of Consulting and Clinical Psychology. J Consult Clin Psych. 2005;73:136-43.
15. Gliner JA, Leech NL, Morgan GA. Problems with null hypothesis significance testing (NHST): what do the textbooks say? J Exp Educ. 2002;71:83-92.
16. Kline RB. Beyond significance testing: reforming data analysis methods in behavioral research. Washington, DC: American Psychological Association; 2004.
17. Silva-Aycaguer LC, Suarez-Gil P, Fernandez-Somoano A. The null hypothesis significance test in health sciences research (1995-2006): statistical analysis and interpretation. BMC Med Res Methodol. 2010;10:No. 44.
18. Beyth-Marom R, Fidler F, Cumming G. Statistical cognition: towards evidence based practice in statistics and statistics education. Stat Educ Res J. 2008;7:20-39.
19. Hacking I. Logic of statistical iinference. Cambridge: Cambridge University Press; 1965.
20. Stigler SM. The history of statistics. The measurement of uncertainty before 1900. Cambridge, Mass: Belknap Press; 1986.
21. Fisher RA. The Design of experiments, 5th ed. 1951. Edinburgh: Oliver & Boyd; 1935.
22. Fisher RA. Statistical methods and scientific inference. Edinburgh: Oliver and Boyd; 1956.
23. Anderson DR, Burnham KP, Thompson WL. Null hypothesis testing: problems, prevalence, and alternatives. J Wildl Manag. 2000;64:912-23.
24. Bakan D. The test of significance in psychological research. Psychol Bull. 1966;66:423-37.
25. Balluerka N, Gomez J, Hidalgo D. The controversy over null hypothesis significance testing revisited. Methodology. Eur J Res Meth Behav Soc Sci. 2005;1:55-70.
26. Berkson J. Some difficulties of interpretation encountered in the application of the chi-square test. J Am Stat Assoc. 1938;33:526-36.
27. Carver RP. The case against statistical significance testing. Harv Educ Rev. 1978;48:378-99.
28. Jones A, Sommerlund N. A critical discussion of null hypothesis significance testing and statistical power analysis within psychological research. Nord Psychol. 2007;59:223-30.
29. Lakens D, Evers ERK. Sailing from the seas of chaos into the corridor of stability: practical recommendations to increase the informational value of studies. Perspect Psychol Sci. 2014;9:278-92.
30. Meehl PE. Theory-testing in psychology and physics: a methodological paradox. Philos Sci. 1967;34:103-15.
31. Meehl PE. Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. J Consult Clin Psychol. 1978;46:806-34.
32. Thompson B. If statistical significance tests are broken/misused, what practices should supplement or replace them? Theor Psychol. 1999;10:165-81.
33. Rosenthal R. Meta-analytic procedures for social research. 2nd ed. New York: Sage; 1991.
34. Kalinowski P, Fidler F. Interpreting significance: the differences between statistical significance, effect size, and practical importance. Newborn Infant Nurs Rev. 2010;10:50-4.
35. Wilkerson M, Olson MR. Misconceptions about sample size, statistical significance, and treatment effect. J Psychol. 1997;131:627-31.
36. Rosnow RL, Rosenthal R. Statistical procedures and the justification of knowledge in psychological science. Am Psychol. 1989;44:1276-84.
37. Steering Committee of the Physicians' Health Study Research Group. Final report on the aspirin component of the ongoing physicians' health study. N Engl J Med. 1989;321:129-35.
38. Parris KM, McCarthy MA. Identifying effects of toe clipping on anuran return rates: the importance of statistical power. Amphibia Repilia. 2001;22:275-89.
39. Macleod M. Why animal research needs to improve. Nature. 2011;477:511.
40. Button KS, Ioannidis JPA, Mokrysz C, Nosek BA, Flint J, Robinson ES, et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci. 2013;14:365-76.
41. Rosenthal R, Gaito J. The interpretation of levels of significance by psychological researchers. J Psychol. 1963;55:33-8.
42. Rosenthal R, Gaito J. Further evidence for the cliff effect in the interpretation of levels of significance. Psychol Rep. 1964;15:570.
43. Nelson N, Rosenthal R, Rosnow RL. Interpretation of significance levels and effect sizes by psychological researchers. Am Psychol. 1986;41:1299-301.
44. Poitevineau J, Lecoutre B. Interpretation of significance levels by psychological researchers: the.05 cliff effect may be overstated. Psychon Bull Rev. 2001;8:847-50.
45. Bradley MT, Brand A. A correction on the bradley and brand method of estimating effect sizes from published literature. Theor Psychol. 2014;24:860-2.
46. Bradley MT, Stoica G. Diagnosing estimate distortion due to significance testing in literature on detection of deception. Percept Mot Skills. 2004;98:827-39.
47. Bakker M, Wicherts JM. The (mis) reporting of statistical results in psychology journals. Behav Res. 2011;43:666-78.
48. Bakker M, van Dijk A, Wicherts JM. The rules of the game called psychological science. Perspect Psychol Sci. 2012;7:543-54.
49. Oakes M. Statistical inference: a commentary for the social and behavioral sciences. New York: Wiley; 1986.
50. Kahneman D, Tversky A. Subjective probability: a judgment of representativeness. Cogn Psychol. 1972;3:430-54.
51. IJzerman H, Semin G. The thermometer of social relations. Mapping social proximity on temperature. Psychol Sci. 2009;20:1214-20.
52. Koch S, Holland RW, Hengstler M, van Knippenberg A. Body locomotion as regulatory process. stepping backward enhances cognitive control. Psychol Sci. 2009;20:549-50.
53. Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. New York, NY: LEA; 1988.
54. Lynott D, Corker KS, Wortman J, Connell L, Donnellan BM, Lucas RE, et al. Replication of "Experiencing physical warmth promotes interpersonal warmth" by Williams and Bargh (2008). Soc Psychol. 2014;45:216-22.
55. Lenth RV. Some practical guidelines for effective sample-size determination. Am Stat. 2001;55:187-93.
56. Lenth RV. Statistical power calculations. J Anim Sci. 2007;85:E24-9.
57. Acklin MW, McDowell CJ, Orndoff S. Statistical power and the Rorschach: 1975-1991. J Pers Assess. 1992;59:366-79.
58. Bezeau S, Graves R. Statistical power and effect sizes of clinical neuropsychology research. J Clin Exp Neuropsychol. 2001;23:399-406.
59. Clark-Carter D. The account taken of statistical power in research published in the British Journal of Psychology. Br J Psychol. 1997;88:71-83.
60. Cohen J. The statistical power of abnormal-social psychological research: a review. J Abnorm Soc Psychol. 1962;65:145-53.
61. Kazantzis N. Power to detect homework effects in psychotherapy outcome research. J Consult Clin Psychol. 2000;68:166-70.
62. Rossi JS. Statistical power of psychological research: what have we gained in 20 years? J Consult Clin Psychol. 1990;58:646-56.
63. Sedlmeier P, Gigerenzer G. Do studies of statistical power have an effect on the power of studies? Psychol Bull. 1989;107:309-16.
64. Fritz A, Scherndl T, Kühberger A. A comprehensive review of reporting practices in psychological journals: are effect sizes really enough? Theor Psychol. 2013;23:98-122.
65. Hager W. Vorgehensweise in der deutschsprachigen psychologischen Forschung. Eine Analyse empirischer Arbeiten der Jahre 2001 und 2002. [Procedures in German empirical research - an analysis of some psychological journals of the years 2001 and 2002.]. Psychol Rundsch. 2005;56:191-200.
66. Henson RK, Hull DM, Williams CS. Methodology in our education research culture: toward a stronger collective quantitative proficiency. Educ Res. 2010;39:229-40.
67. Alhija FN, Levy A. Effect size reporting practices in published articles. Educ Psychol Meas. 2009;69:245-65.
68. Sun S, Pan W, Wang LL. A comprehensive review of effect size reporting and interpreting practices in academic journals in Education and Psychology. J Educ Psychol. 2010;102:989-1004.
69. APA (American Psychological Association). Publication manual of the American psychological association. 6th ed. Washington, DC: Author; 2010.
70. American Educational Research Association. Standards on reporting on empirical social science research in AERA publications. Educ Res. 2006;35:33-40.
71. Henson RK. Effect-size measures and meta-analytic thinking in counseling psychology research. Couns Psychol. 2006;34:601-29.
72. John LK, Loewenstein GM, Prelec D. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol Sci. 2012;23:524-32.
73. Kühberger A, Fritz A, Scherndl T. Publication bias in psychology: a diagnosis based on the correlation between effect size and sample size. PLoS ONE, 9(9), e105825. doi:10.1371/journal.pone.0105825.