Significance testing needs a taxonomy: Or how the fisher, Neyman–Pearson controversy resulted in the inferential tail wagging the measurement dog

Psychological Reports

Volumn 119, Issue 2, 2016, Pages 487-504

  Bradley, Michael T ^a   Brand, Andrew ^b  

^a UNIVERSITY OF NEW BRUNSWICK   (Canada)

^b BANGOR UNIVERSITY   (United Kingdom)

Author keywords

Correct use of statistical techniques; Error rates; Measurement; Measures and statistics; Probabilities; Psychometrics; Significance testing; Taxonomy

Indexed keywords

HUMAN; PSYCHOMETRY; STANDARDS; STATISTICAL ANALYSIS;

DATA INTERPRETATION, STATISTICAL; HUMANS; PSYCHOMETRICS;

EID: 84993968518     PISSN: 00332941     EISSN: 1558691X     Source Type: Journal    
DOI: 10.1177/0033294116662659     Document Type: Article

References (45)

1. American Psychological Association. (1999). Publication manual of the American Psychological Association (4th ed.). Washington, DC: Author.
2. American Psychological Association. (2002). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.
3. American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
4. Babbie, E. R. (2001). The practice of social research (9th ed.). Belmont, CA: Wadsworth.
5. Bakan, D. (1966). The test of significance in psychological research. Psychological Bulletin, 66, 423-437.
6. Bendtsen, L., Bigal, M. E., Cerbo, R., Diener, H. C., Holroyd, K., Lampl, C., Tfelt-Hansen, P. (2009). Guidelines for controlled trials of drugs in tension-type headache: Second edition. Cephalalgia, 30(1), 1-16.
7. Box, J. F. (1978). R. A. Fisher: The life of a scientist. New York: John Wiley.
8. Bradley, M. T., & Brand, A. (2013). Alpha values as a function of sample size, effect size, and power: Accuracy over inference. Psychological Reports: Measures & Statistics, 112(3), 1-10.
9. Bradley, M. T., & Brand, A. (2016). Accuracy when inferential statistics are used as measurement tools. BMC Research Notes, 9, 241. doi:10.1186/s13104-016-2045-z
10. Bradley, M. T., Brand, A., & MacNeill, A. L. (2012). Interpreting effect size estimates through graphic analysis of raw data distributions. In: P. Cox, P. Rodgers & B. Plimmer (Eds.), Diagrams (pp. 117-123). Berlin: Springer-Verlag.
11. Bradley, M. T., Brand, A., & MacNeill, A. L. (2013, June). Effect size reporting reveals the weakness Fisher believed inherent in the Neyman-Pearson approach to statistical analysis. Poster presented at the annual meeting of the Canadian Psychological Association. Quebec City, QC.
12. Bradley, M. T., & Gupta, R. D. (1997). Estimating the effect of the file drawer problem in meta-analysis. Perceptual & Motor Skills, 85, 719-722.
13. Bradley, M. T., & Stoica, G. (2004). Diagnosing estimate distortion due to significance testing in literature on the detection of deception. Perceptual and Motor Skills, 98(3), 827-839. doi:10.2466/Pms.98.3.827-839
14. Branch, M. (2014). Malignant side effects of null-hypothesis significance testing. Theory & Psychology, 24, 256-277.
15. Brand, A., Bradley, M. T., Best, L., & Stoica, G. (2008). Accuracy of effect size estimates from published psychological research. Perceptual & Motor Skills, 106, 645-649.
16. Brand, A., & Bradley, M. T. (2016). The precision of effect size estimation from published psychological research surveying confidence intervals. Psychological Reports, 118, 1154-1170. doi:10.1177/0033294115625265
17. Cohen, J. (1994). The earth is round (p<.05). American Psychologist, 49, 997-1003.
18. Cowles, M., & Davis, C. (1982). On the origins of the .05 level of statistical significance. American Psychologist, 37(5), 553-558.
19. David, H. A., & Edwards, A. W. F. (2001). Annotated readings in the history of statistics. New York: Springer.
20. Fisher, R. A. (1922). On the mathematical foundations of theoretical statistics. Philosophical Transactions of the Royal Society of London, 222, 309-368.
21. Fisher, R. A. (1926). The arrangement of field experiments. Journal of the Ministry of Agriculture, 33, 503-513.
22. Fisher, R. A. (1935). Statistical tests. Nature, 136, 474.
23. Fisher, R. A. (1942). The design of experiments (3rd ed.). London, England: Oliver & Boyd.
24. Fisher, R. A. (1955). Statistical methods and scientific induction. Journal of the Royal Statistical Society: Series B (Methodological), 17(1), 69-78.
25. Fisher, R. A. (1973). Statistical methods and scientific inference (3rd ed.). New York, NY: Hafner Press.
26. Gigerenzer, G. (1993). The superego, the ego, and the id in statistical reasoning. In: G. Keren & C. Lewis (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues (pp. 311-339). Hillsdale, NY: Lawrence Erlbaum Associates.
27. Halprin, P. F., & Stam, H. J. (2006). Inductive inference or inductive behavior: Fisher and Neyman-Person approaches to statistical testing in psychological research (1940- 1960). The American Journal of Psychology, 119, 625-653.
28. Hubbard, R., & Bayarri, M. J. (2003). Confusion over measures of evidence (p’s) versus errors (a) in classical statistical testing. The American Statistician, 57, 171-182.
29. Hurlbert, S. H., & Lombardi, C. M. (2009). Final collapse of the Neyman-Pearson decision theoretic framework and the rise of the neoFisherian. Annales Zoologic Fennici, 46, 311-349.
30. Keuzenkamp, H. A., & Magnus, J. R. (1995). On tests and significance in econometrics. Journa1 of Econometrics, 67, 5-24.
31. Kline, R. B. (2013). Beyond significance testing: Statistics reform in the behavioral sciences (2nd ed.). Washington, DC: American Psychological Association.
32. Lenhard, J. (2006). Models and statistical inference: The controversy between Fisher and Neyman-Pearson. British Journal for the Philosophy of Science, 57(1), 69-91. doi:10.1093/bjps/axi152
33. Mudge, J. F., Baker, L. F., Edge, C. B., & Houlahan, J. E. (2012). Setting an optimal a that minimizes errors in null hypothesis significance tests. PLoS ONE, 7, e32734.
34. Neyman, J., & Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 231(694-706), 289-337. doi:10.1098/rsta.1933.0009
35. Neyman, J. (1950). First course in probability and statistics. New York, NY: Henry Holt.
36. Perezgonzales, J. D. (2014). A reconceptualization of significance testing. Theory & Psychology, 24, 852-859. doi:10.1177/0959354314546157
37. Rosenthal, R. (1979). The “File Drawer Problem” and the tolerance for null results. Psychological Bulletin, 86(3), 638-641. doi:10.1037/0033-2909.86.3.638
38. Sharp, D., & Poets, S. (2015, June). Where are the boys? Statistical implications of the absence of men from psychology participant pools. Presented at the annual meeting of the Canadian Psychological Association, Ottawa, ON, Canada.
39. Sterling, T. D. (1959). Publication decision and the possible effects drawn from tests of significance - or Vice Versa. Journal of the American Statistical Association, 54, 30-34.
40. Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103(2684), 677-680. doi:10.1126/science
41. Stigler, S. M. (1986). The history of statistics: The measurement of uncertainty before 1900.
42. London, England: Belknap Press. Student (1908). The probable error of a mean. Biometrica, 6, 1-25.
43. Tukey, J. W. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley.
44. Wald, A. (1945). Sequential tests of statistical hypotheses. Annals of Mathematical Statistics, 16(2), 117-186. doi:10.1214/aoms/1177731118
45. Wasserstein, R. L., & Lazar, N. A. (2016). The ASA’s statement on p-values: Context, process, and purpose. The American Statistician, 70, 129-133. doi:10.1080/00031305.2016.1154108.