Blinding us to the obvious? The effect of statistical training on the evaluation of evidence

Management Science

Volumn 62, Issue 6, 2016, Pages 1707-1718

  McShane, Blakeley B ^a   Gal, David ^b  

^a NORTHWESTERN UNIVERSITY   (United States)

^b UNIVERSITY OF ILLINOIS AT CHICAGO   (United States)

Author keywords

Choice; Description; Evaluation of evidence; Inference; Judgment; Null hypothesis; p values; Significance testing; Sociology of science; Strength of evidence

Indexed keywords

MANAGEMENT; MANAGEMENT SCIENCE;

CHOICE; DESCRIPTION; EVALUATION OF EVIDENCE; INFERENCE; JUDGMENT; NULL HYPOTHESIS; P-VALUES; SIGNIFICANCE TESTING; SOCIOLOGY OF SCIENCE; STRENGTH OF EVIDENCE;

STATISTICAL TESTS;

EID: 84975110910     PISSN: 00251909     EISSN: 15265501     Source Type: Journal    
DOI: 10.1287/mnsc.2015.2212     Document Type: Article

References (68)

1. Anderson DR, Burnham KP, Thompson WL (2000) Null hypothesis testing: Problems, prevalence, and an alternative. J. Wildlife Management 64:912-923.
2. Asendorpf JB, Conner M, De Fruyt F, De Houwer J, Denissen JJA, Fiedler K, Fiedler S, et al. (2013) Recommendations for increasing replicability in psychology. Eur. J. Personality 27:108-119.
3. Bakan D (1966) The test of significance in psychological research. Psych. Bull. 66:423-437.
4. Berger JO, Berry DA (1988) Statistical analysis and the illusion of objectivity. Amer. Scientist 76:159-165.
5. Berger JO, Delampady M (1987) Testing precise hypotheses (with comments). Statist. Sci. 2:317-352.
6. Berger JO, Sellke T (1987) Testing a point null hypothesis: The irreconcilability of p values and evidence. J. Amer. Statist. Assoc. 82:112-122.
7. Brandt MJ, IJzerman H, Dijksterhuis A, Farach FJ, Geller J, Giner-Sorolla R, Grange JA, Perugini M, Spies JR, Van't Veer A (2014) The replication recipe: What makes for a convincing replication? J. Experiment. Soc. Psych. 50:217-224.
8. Brodeur A, Lé M, Sangnier M, Zylberberg Y (2012) Star wars: The empirics strike back. Technical report, Paris School of Economics, Paris.
9. Casella G, Berger RL (1987) Reconciling Bayesian and frequentist evidence in the one-sided testing problem (with comments). J. Amer. Statist. Assoc. 82:106-138.
10. Cochran WG (1976) Early development of techniques in comparative experimentation. Owen DB, ed. On the History of Statistics and Probability (Dekker, New York).
11. Cohen J (1994) The earth is round (p < 0.05). Amer. Psychologist 49:997-1003.
12. Cowles M, Davis C (1982) On the origins of the 0.05 level of significance. Amer. Psychologist 44:1276-1284.
13. Cramer H (1955) The Elements of Probability Theory (John Wiley & Sons, New York).
14. Cumming G (2012) Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis (Routledge, New York).
15. Cumming G (2014) The new statistics: Why and how. Psych. Sci. 25:7-29.
16. Fisher RA (1926) The arrangement of field experiments. J. Ministry Agriculture 33:503-513.
17. Fong GT, Krantz DH, Nisbett RE (1986) The effects of statistical training on thinking about everyday problems. Cognitive Psych. 18:253-292.
18. Francis G (2013) Replication, statistical consistency, and publication bias. J. Math. Psych. 57:153-169.
19. Freedman D, Pisani R, Purves R (2007) Statistics, 4th ed. (W. W. Norton & Company, New York).
20. Freeman PR (1993) The role of p-values in analysing trial results. Statist. Medicine 12:1443-1452.
21. Gelman A (2013) Interrogating p-values. J. Math. Psych. 57:188-189.
22. Gelman A (2015) The connection between varying treatment effects and the crisis of unreplicable research: A Bayesian perspective. J. Management 41:632-643.
23. Gelman A, Stern H (2006) The difference between "significant" and "not significant" is not itself statistically significant. Amer. Statistician 60:328-331.
24. Gelman A, Tuerlinckx F (2000) Type S error rates for classical and Bayesian single and multiple comparison procedures. Comput. Statist. 15:373-390.
25. Gelman A, Weakliem D (2009) Of beauty, sex and power. Amer. Scientist 97:310-316.
26. Gelman A, Carlin JB, Stern HS, Rubin DB (2003) Bayesian Data Analysis, 2nd ed. (Chapman & Hall/CRC, Boca Raton, FL).
27. Gigerenzer G (1987) The Probabilistic Revolution: Ideas in the Sciences, Vol. II (MIT Press, Cambridge, MA).
28. Gigerenzer G (2004) Mindless statistics. J. Socio-Econom. 33:587-606.
29. Gigerenzer G, Krauss S, Vitouch O (2004) The null ritual: What you always wanted to know about null hypothesis testing but were afraid to ask. Kaplan DW, ed. The Sage Handbook of Quantitative Methods for the Social Sciences (Sage, Thousand Oaks, CA), 391-406.
30. Gill J (1999) The insignificance of null hypothesis significance testing. Political Res. Quart. 52:647-674.
31. Greenland S, Poole C (2013a) Living with p-values: Resurrecting a Bayesian perspective on frequentist statistics. Epidemiology 24:62-68.
32. Greenland S, Poole C (2013b) Living with statistics in observational research. Epidemiology 24:73-78.
33. Haller H, Krauss S (2002) Misinterpretations of significance: A problem students share with their teachers? Methods Psych. Res. 7(1):1-20.
34. Hauer E (2004) The harm done by tests of significance. Accident Anal. Prevention 36:495-500.
35. Healy D (2006) The antidepressant tale: Figures signifying nothing? Adv. Psychiatric Treatment 12:320-328.
36. Hommes DW, Bura A, Mazzolai L, Buller H, ten Cate JW (1992) Subcutaneous heparin compared with continuous intravenous heparin administration in the initial treatment of deep vein thrombosis. Ann. Internal Medicine 116:279-284.
37. Hoover KD, Siegler MV (2008) Sound and fury: Mccloskey and significance testing in economics. J. Econom. Methodol. 15:1-37.
38. Hubbard R (2004) Alphabet soup: Blurring the distinctions between p's and α's in psychological research. Theory Psych. 14:295-327.
39. Hubbard R, Lindsay RM (2008) Why p values are not a useful measure of evidence in statistical significance testing. Theory Psych. 18:69-88.
40. Hunter JE (1997) Needed: A ban on the significance test. Psych. Sci. 8:3-7.
41. Ioannidis JPA (2005) Why most published research findings are false. PLoS Medicine 2:e124.
42. Kramer W, Gigerenzer G (2005) How to confuse with statistics or: The use and misuse of conditional probabilities. Statist. Sci. 20:223-230.
43. Krantz DH (1999) The null hypothesis testing controversy in psychology. J. Amer. Statist. Assoc. 94:1372-1381.
44. Ledgerwood A (2014) Introduction to the special section on moving toward a cumulative science: Maximizing what our research can tell us. Perspect. Psych. Sci. 9:610-611.
45. Loftus GR (1996) Psychology will be a much better science when we change the way we analyze our data. Current Directions Psych. 6:161-171.
46. McCloskey DR, Ziliak S (1996) The standard error of regression. J. Econom. Literature 34:97-114.
47. McShane BB, Böckenholt U (2014) You cannot step into the same river twice: When power analyses are optimistic. Perspect. Psych. Sci. 9:612-625.
48. McShane BB, Böckenholt U (2016) Planning sample sizes when effect sizes are uncertain: The power-calibrated effect size approach. Psych. Methods 21(1):47-60.
49. Meehl PE (1978) Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. J. Counsel. Clinical Psych. 46:806-834.
50. Messori A, Scrocarro G, Martini N (1993) Calculation errors in meta-analysis. Ann. Internal Medicine 118:77-78.
51. Morrison DE, Henkel RE (1970) The Significance Test Controversy (Aldine, Chicago).
52. Mukherjee S (2010) The Emperor of All Maladies: A Biography of Cancer (Scribner, New York).
53. Nickerson RS (2000) Null hypothesis statistical testing: A review of an old and continuing controversy. Psych. Methods 5:241-301.
54. Oakes M (1986) Statistical Inference: A Commentary for the Social and Behavioral Sciences (John Wiley & Sons, New York).
55. Pashler H, Wagenmakers E-J (2012) Editors' introduction to the special section on replicability in psychological science: A crisis of confidence? Perspect. Psych. Sci. 7:528-530.
56. Rosnow RL, Rosenthal R (1989) Statistical procedures and the justification of knowledge in psychological science. Amer. Psychologist 44:1276-1284.
57. Rothman KL, Lanes S, Robins J (1993) Causal inference. Epidemiology 4(6):555-556.
58. Rozenboom WW (1960) The fallacy of the null hypothesis significance test. Psych. Bull. 57:416-428.
59. Sawyer AG, Peter JP (1983) The significance of statistical significance tests in marketing research. J. Marketing Res. 20:122-133.
60. Schmidt FL (1996) Statistical significance testing and cumulative knowledge in psychology: Implications for the training of researchers. Psych. Methods 1:115-129.
61. Schwab A, Abrahamson E, Starbuck WH, Fidler F (2011) Researchers should make thoughtful assessments instead of null-hypothesis significance tests. Organ. Sci. 22:1105-1120.
62. Serlin RC, Lapsley DK (1993) Rational appraisal psychological research and the good enough principle. Keren G, Lewis C, eds. A Handbook for Data Analysis in the Behavioral Sciences: Methodological Issues (Lawrence Erlbaum Associates, Hillsdale, NJ), 199-228.
63. Spellman BA (2012) Introduction to the special section on research practices. Perspect. Psych. Sci. 7:655-656.
64. Spellman BA (2013) Introduction to the special section on advancing science. Perspect. Psych. Sci. 8:412-413.
65. Tukey JW (1991) The philosophy of multiple comparisons. Statist. Sci. 6:100-116.
66. Tversky A, Kahneman D (1974) Judgment under uncertainty: Heuristics and biases. Science 185:1124-1131.
67. Yong E (2012) Replication studies: Bad copy. Nature 485:298-300.
68. Yule GU, Kendall MG (1950) An Introduction to the Theory of Statistics, 14th ed. (Griffin, London).