The Connection Between Varying Treatment Effects and the Crisis of Unreplicable Research: A Bayesian Perspective

Journal of Management

Volumn 41, Issue 2, 2015, Pages 632-643

  Gelman, Andrew ^a  

^a Och Spine at New York Presbyterian Hospitals   (United States)

Author keywords

hierarchical modeling; multilevel modeling; null hypothesis significance testing; p values; regression analysis; variation

Indexed keywords

EID: 84922021475     PISSN: 01492063     EISSN: 15571211     Source Type: Journal    
DOI: 10.1177/0149206314525208     Document Type: Editorial

References (45)

1. Anderson N.,Ones D. S.Rejoinder to Goldberg, Lee and Ashton: Explaining counterintuitive findings.European Journal of Personality. 2008;22:157-162
2. Bem D. J.Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect.Journal of Personality and Social Psychology. 2011;100:407-425
3. Berrington A.,Cox D. R.Interpretation of interaction: A review.Annals of Applied Statistics. 2007;1:371-385
4. Bloom H. S.,Raudenbush S. W.,Weiss M.Estimating variation in program impacts: Theory, practice, and applications. New York, NY: ; 2011:
5. Bryk A. S.,Raudenbush S. W.Heterogeneity of variance in experimental studies: A challenge to conventional interpretations.Psychological Bulletin. 1988;104:396-404
6. Dehejia R.Program evaluation as a decision problem.Journal of Econometrics. 2005;125:141-173
7. Durante K. M.,Arsena A. R.,Griskevicius V.The fluctuating female vote: Politics, religion, and the ovulatory cycle.Psychological Science. 2013;24:1007-1016
8. Edwards J. R.,Berry J. W.The presence of something or the absence of nothing: Increasing theoretical precision in management research.Organizational Research Methods. 2010;13:668-689
9. Emirbayer M.Manifesto for a relational sociology.American Journal of Sociology. 1997;103:281-317
10. Ferguson C. J.,Heene M.A vast graveyard of undead theories: Publication bias and psychological science’s aversion to the null.Perspectives on Psychological Science. 2012;7:555-561
11. Francis G.Replication, statistical consistency, and publication bias (with discussion).Journal of Mathematical Psychology. 2013;57 (5): 153-169
12. Freese J.Intergenerational Caregiving. Booth A., ed. Washington, D.C.: Urban Institute Press; 2007:145-178.
13. Fryer R. G.,Levitt S. D.,List J. A.Exploring the impact of financial incentives on stereotype threat: Evidence from a pilot study.American Economic Review. 2008;98:370-375
14. Ganley C. M.,Mingle L. A.,Ryan A. M.,Ryan K.,Vasilyeva M.,Perry M.An examination of stereotype threat effects on girls’ mathematics performance.Developmental Psychology. 2013;49:1886-1897
15. Gelman A.Applied Bayesian modeling and causal inference from an incomplete data perspective. Gelman A.Meng X. L., ed. New York, NY: Wiley; 2004:195-202.
16. Gelman A.Correction for “Should the Democrats Move to the Left on Economic Policy?”.Annals of Applied Statistics. 2013a 1248;7:
17. Gelman A.P values and statistical practice.Epidemiology. 2013b;24:69-72
18. Gelman A.,Carlin J. B.Design analysis, prospective or retrospective, using external information. New York, NY: Columbia University; 2013:
19. Gelman A.,Carlin J. B.,Stern H. S.,Vehtari A.,Dunson D. B.,Rubin D. B.Bayesian data analysis. (2013). Bayesian data analysis (3rd ed). London, UK: Chapman and Hall.London, UK: Chapman and Hall; 2013:
20. Gelman A.,Hill J.,Yajima M.Why we (usually) don’t have to worry about multiple comparisons.Journal of Research on Educational Effectiveness. 2012;5:189-211
21. Gelman A.,Huang Z.Estimating incumbency advantage and its variation, as an example of a before/after study (with discussion).Journal of the American Statistical Association. 2008;103:437-451
22. Gelman A.,Jakulin A.,Pittau M. G.,Su Y. S.A weakly informative default prior distribution for logistic and other regression models.Annals of Applied Statistics. 2008;2:1360-1383
23. Gelman A.,Loken E.The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. New York, NY: Columbia University; 2013:
24. Gelman A.,Tuerlinckx F.Type S error rates for classical and Bayesian single and multiple comparison procedures.Computational Statistics. 2000;15:373-390
25. Gelman A.,Weakliem D.Of beauty, sex, and power: Statistical challenges in estimating small effects.American Scientist. 2009;97:310-316
26. Gelman S.The essential child. Oxford, UK: Oxford University Press; 2005:
27. Gilovich T.,Vallone R.,Tversky A.The hot hand in basketball: On the misperception of random sequences.Cognitive Psychology. 1985;17:295-314
28. Goldberg L. R.,Lee K.,Ashton M. C.Comment on Anderson and Ones.European Journal of Personality. 2008a;22:151-156
29. GoldbergL. R.LeeK.AshtonM. C. (2008b). Response to Anderson and Ones (2008). Unpublished. Retrieved from http://people.ucalgary.ca/~kibeom/Anderson\%20Ones/AndersonOnes.htm
30. Greenland S.,Poole C.Living with p-values: Resurrecting a Bayesian perspective on frequentist statistics (with discussion).Epidemiology. 2013a;24:62-68
31. Greenland S.,Poole C.Living with statistics in observational research.Epidemiology. 2013b;24:73-78
32. Ioannidis J.Why most published research findings are false.PLOS Medicine. 2005;2:e124
33. Kanazawa S.Big and tall parents have more sons: Further generalizations of the Trivers-Willard hypothesis.Journal of Theoretical Biology. 2005;233:583-590
34. Kanazawa S.Beautiful parents have more daughters: A further implication of the generalized Trivers-Willard hypothesis.Journal of Theoretical Biology. 2007;244:133-140
35. Krantz D. H.The null hypothesis testing controversy in psychology.Journal of the American Statistical Association. 1999;44:1372-1381
36. Leung A. K.,Kim S.,Polman E.,Ong L. S.,Qui L.,Goncalo J. A.,Sanchez-Burks J.Embodied metaphors and creative “acts.”.Psychological Science. 2012;23:502-509
37. Mischel W.Personality and assessment. New York, NY: Wiley; 1968:
38. Pashler H.,Wagenmakers E. J.Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence?.Perspectives on Psychological Science. 2012;7:528-530
39. Raudenbush S.,Bryk A. S.A hierarchical model for studying school effects.Sociology of Education. 1986;59:1-17
40. Rubin D. B.The future of meta-analysis. Wachter K. W.Straf M. L., ed. New York, NY: Russell Sage Foundation; 1989:155-165.
41. Simmons J.,Nelson L.,Simonsohn U.False-positive psychology: Undisclosed flexibility in data collection and analysis allow presenting anything as significant.Psychological Science. 2011;22:1359-1366
42. Skrondal A.,Rabe-Hesketh S.Generalized latent variable modeling: Multilevel, longitudinal and structural equation models. London, UK: Chapman and Hall; 2004:
43. Tibshirani R.Regression shrinkage and selection via the lasso.Journal of the Royal Statistical Society B. 1996;58:267-288
44. Vul E.,Harris C.,Winkielman P.,Pashler H.Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition (with discussion).Perspectives on Psychological Science. 2009;4:274-324
45. YaariG. (2012, January12). “Hot hand” debate is warming up [Web log post]. Retrieved from Brainstorm Private Consulting Blog: http://blog.gostorm.net/2012/01/12/hot-hand-debate-is-warming-up/