Invited Commentary: The Need for Cognitive Science in Methodology

American Journal of Epidemiology

Volumn 186, Issue 6, 2017, Pages 639-645

  Greenland, Sander ^a,b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

behavioral economics; bias analysis; cognitive bias; motivated reasoning; nullism; overconfidence; sensitivity analysis; significance testing

Indexed keywords

COGNITION; ECONOMICS; EXPERIMENTAL STUDY; HYPOTHESIS TESTING; METHODOLOGY; STATISTICAL ANALYSIS;

ANALYTIC METHOD; BAYES THEOREM; CALCULATION; COGNITIVE BIAS; COMPULSION; DICHOTOMANIA; GENOMICS; HAZARD ASSESSMENT; HUMAN; MATHEMATICAL COMPUTING; MEDICAL PSYCHOLOGY; METHODOLOGY; NULL HYPOTHESIS; PROBABILITY; REPRODUCIBILITY; REVIEW; SENSITIVITY ANALYSIS; SOCIOLOGY; STATISTICAL BIAS; STATISTICAL MODEL; STATISTICAL REIFICATION; PSYCHOLOGY; SELECTION BIAS;

COGNITIVE SCIENCE; HUMANS; MODELS, STATISTICAL; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; SELECTION BIAS;

EID: 85029694410     PISSN: 00029262     EISSN: 14766256     Source Type: Journal    
DOI: 10.1093/aje/kwx259     Document Type: Review

References (67)

1. Tyson ND. The Sky Is Not the Limit: Adventures of an Urban Astrophysicist. 2nd ed. Amherst, NY: Prometheus Books; 2004.
2. Lash TL. The harm done to reproducibility by the culture of null hypothesis significance testing. Am J Epidemiol. 2017; 186(6): 627-635.
3. Lash TL. Heuristic thinking and inference from observational epidemiology. Epidemiology. 2007;18(1): 67-72.
4. Greenland S. Causal inference as a prediction problem: assumptions, identification, and evidence synthesis. In: Berzuini C, Dawid AP, Bernardinelli L, eds. Causality Inference: Statistical Perspectives and Applications. Chichester, United Kingdom: John Wiley&Sons Ltd.; 2012: 43-58.
5. Greenland S. Transparency and disclosure, neutrality and balance: shared values or just shared words J Epidemiol Community Health. 2012;66(11): 967-970.
6. Greenland S. For and against methodology: some perspectives on recent causal and statistical inference debates. Eur J Epidemiol. 2017;32(1): 3-20.
7. Greenland S, Senn SJ, Rothman KJ, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016;31(4): 337-350.
8. Neyman J. "Inductive behavior" as a basic concept of philosophy of science. Rev Int Stat Inst. 1957;25(1/3): 7-22.
9. Rothman KJ. A show of confidence. N Engl J Med. 1978; 299(24): 1362-1363.
10. Rothman KJ. Significance questing. Ann Intern Med. 1986; 105(3): 445-447.
11. Rothman KJ. Curbing type I and type II errors. Eur J Epidemiol. 2010;25(4): 223-224.
12. Amrhein V, Korner-Nievergelt F, Roth T. The earth is flat (p > 0. 05): significance thresholds and the crisis of unreplicable research. PeerJ. 2017;5: e3544.
13. McShane BB, Gal D. Blinding us to the obvious The effect of statistical training on the evaluation of evidence. Manag Sci. 2016;62(6): 1707-1718.
14. Schmidt M, Rothman KJ. Mistaken inference caused by reliance on and misinterpretation of a significance test. Int J Cardiol. 2014;177(3): 1089-1090.
15. Poole C. Beyond the confidence interval. Am J Public Health. 1987;77(2): 195-199.
16. Rothman KJ, Lanes S, Robins J. Causal inference (letter). Epidemiology. 1993;4(6): 555-556.
17. Rothman KJ, Greenland S, Lash TL. Precision and statistics in epidemiologic studies. In: Rothman KJ, Greenland S, Lash TL, eds. Modern Epidemiology. 3rd ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2008: 148-167.
18. Greenland S. A serious misinterpretation of a consistent inverse association of statin use with glioma across 3 casecontrol studies. Eur J Epidemiol. 2017;32(1): 87-88.
19. Trafimow D, Marks M. Editorial. Basic Appl Soc Psych. 2015; 37(1): 1-2.
20. Casella G, Berger RL. Comment. Stat Sci. 1987;2(3): 344-347.
21. Greenland S. Weaknesses of certain Bayesian methods for meta-analysis: the case of vitamin E and mortality. Clin Trials. 2009;6: 42-46.
22. Greenland S. Null misinterpretation in statistical testing and its impact on health risk assessment. Prev Med. 2011;53(4-5): 225-228.
23. Greenland S, Poole C. Living with p values: resurrecting a Bayesian perspective on frequentist statistics. Epidemiology. 2013;24(1): 62-68.
24. Greenland S, Poole C. Living with statistics in observational research. Epidemiology. 2013;24(1): 73-78.
25. Gelman A. P values and statistical practice. Epidemiology. 2013;24(1): 69-72.
26. Fiedler K, Kutzner F, Krueger JI. The long way from error control to validity proper: problems with a short-sighted false-positive debate. Perspect Psychol Sci. 2012;7(6): 661-669.
27. Neyman J. Frequentist probability and frequentist statistics. Synthese. 1977;36(1): 97-131.
28. Greenland S. The ASA guidelines and null bias in current teaching and practice. Am Statist. 2016;70(suppl 10). http: // amstat. tandfonline. com/doi/suppl/10. 1080/00031305. 2016. 1154108/suppl-file/utas-a-1154108-sm5079. pdf. Accessed July 21, 2017.
29. Greenland S. The need for critical appraisal of expert witnesses in epidemiology and statistics. Wake Forest Law Rev. 2004;39: 291-310.
30. Kelly KT. Simplicity, truth, and probability. In: Bandyopadhyay PS, Forster MR, eds. Philosophy of Statistics. 1st ed. (Handbook of the Philosophy of Science, vol. 7). Amsterdam, the Netherlands: Elsevier B. V.; 2011: 983-1026.
31. Popper KR. The Logic of Scientific Discovery. New York, NY: Basic Books; 1959.
32. Yates F. The influence of statistical methods for research workers on the development of the science of statistics. J Am Stat Assoc. 1951;46(253): 19-34.
33. Cox DR. Some problems connected with statistical inference. Ann Math Stat. 1958;29(2): 357-372.
34. Greenland S. Nonsignificance plus high power does not imply support for the null over the alternative. Ann Epidemiol. 2012; 22(5): 364-368.
35. Birnbaum A. A unified theory of estimation, I. Ann Math Stat. 1961;32: 112-135.
36. Lang JM, Rothman KJ, Cann CI. That confounded P-value. Epidemiology. 1998;9(1): 7-8.
37. Casella G, Berger RL. Reconciling Bayesian and frequentist evidence in the one-sided testing problem. J Am Stat Assoc. 1987;82(397): 106-111.
38. Poole C. Confidence intervals exclude nothing. Am J Public Health. 1987;77(4): 492-493.
39. Poole C. Low P-values or narrow confidence intervals: which are more durable Epidemiology. 2001;12(3): 291-294.
40. Gelman A. The problems with p-values are not just with pvalues: my comments on the recent ASA statement (with comments) [blog post]. http: //andrewgelman. com/2016/03/07/ 29212/. Accessed May 21, 2017.
41. Greenland S. Randomization, statistics, and causal inference. Epidemiology. 1990;1(6): 421-429.
42. Gelman A, Loken E. The statistical crisis in science: datadependent analysis-a "garden of forking paths"-explains why many statistically significant comparisons don't hold up. Am Sci. 2014;102(6): 460-465.
43. Kelly KT, Glymour C. Why probability does not capture the logic of scientific justification. In Hitchcock C, ed. Contemporary Debates in the Philosophy of Science. London, United Kingdom: Blackwell Ltd.; 2004.
44. Gigerenzer G, Marewski JN. Surrogate science: the idol of a universal method for scientific inference. J Manag. 2015;41(2): 421-440.
45. Sellke TM, Bayarri MJ, Berger JO. Calibration of p values for testing precise null hypotheses. Am Stat. 2001;55(1): 62-71.
46. Goodman SN. Introduction to Bayesian methods I: measuring the strength of evidence. Clin Trials. 2005;2(4): 282-290.
47. Wagenmakers EJ. A practical solution to the pervasive problem of p values. Psychon Bull Rev. 2007;14(5): 779-804.
48. Senn S. A comment on replication, p-values and evidence. S. N. Goodman, Statistics in Medicine 1992;11: 875-879 [letter].
49. Stat Med. 2002;21(16): 2437-2444.
50. Fraundorf P. Examples of surprisal. 2017. http: //www. umsl. edu/~fraundorfp/egsurpri. html. Accessed April 6, 2017.
51. Stanford K. Underdetermination of scientific theory. In: Zalta EN, ed. Stanford Encyclopedia of Philosophy Archive. Spring 2016 Edition. https: //plato. stanford. edu/archives/spr2016/ entries/scientific-underdetermination/. Published August 12, 2009. Revised September 16, 2013. Accessed April 6, 2017.
52. Greenland S. The sensitivity of a sensitivity analysis (invited paper). In: 1997 Proceedings of the Biometrics Section. Alexandria, VA: American Statistical Association; 1998: 19-21.
53. Broman K, Cetinkaya-Rundel M, Nussbaum A, et al. Recommendations to Funding Agencies for Supporting Reproducible Research. Washington, DC: American Statistical Association; 2017. http: //www. amstat. org/asa/files/ pdfs/pol-reproducibleresearchrecommendations. pdf. Accessed April 6, 2017.
54. Greenland S, Lash TL. Bias analysis. In: Rothman KJ, Greenland S, Lash TL, eds. Modern Epidemiology. 3rd ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2008: 345-380.
55. Kahan DM, Peters E, Dawson EC, et al. Motivated numeracy and enlightened self-government. Behav Public Pol. 2017; 1(1): 54-86.
56. Greenland S, Kheifets L. Leukemia attributable to residential magnetic fields: results from analyses allowing for study biases. Risk Anal. 2006;26(2): 471-482.
57. Rothman KJ. Policy recommendations in epidemiologic research papers. Epidemiology. 1993;4(2): 94-95.
58. Greenland S, Gago-Domiguez M, Castelao JE. The value of risk-factor ("black-box") epidemiology. Epidemiology. 2004; 15(5): 529-535.
59. Kahneman D, Slovic P, Tversky P, eds. Judgment Under Uncertainty: Heuristics and Biases. New York, NY: Cambridge University Press; 1982.
60. Gilovich T, Griffin D, Kahneman D. Heuristics and Biases: The Psychology of Intuitive Judgment. New York, NY: Cambridge University Press; 2002.
61. Baron J. Thinking and Deciding. 4th ed. New York, NY: Cambridge University Press; 2007.
62. Pohl RF, ed. Cognitive Illusions: Intriguing Phenomena in Thinking, Judgment and Memory. 2nd ed. New York, NY: Routledge, Taylor & Francis Group LLC; 2017.
63. Gilovich T. How We Know What Isn't So: The Fallibility of Human Reason in Everyday Life. New York, NY: Free Press; 1993.
64. Kahneman D. Thinking, Fast and Slow. New York, NY: Farrar, Straus and Giroux; 2011.
65. LessWrongWiki. Bias [blog post]. https: //wiki. lesswrong. com/ wiki/Bias. Modified November 19, 2013. Accessed April 25, 2017.
66. Wikipedia. Cognitive bias. https: //en. wikipedia. org/wiki/ Cognitive-bias. Modified July 3, 2017. Accessed April 24, 2017.
67. Wikipedia. List of cognitive biases. https: //en. wikipedia. org/ wiki/List-of-cognitive-biases. Modified July 17, 2017. Accessed April 24, 2017.