Bayesian tests to quantify the result of a replication attempt

Journal of Experimental Psychology: General

Volumn 143, Issue 4, 2014, Pages 1457-1475

  Verhagen, Josine ^a   Wagenmakers, Eric Jan ^a  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

Bayes factor; Effect size; Prior distribution

Indexed keywords

BAYES THEOREM; HUMAN; METHODOLOGY; REPRODUCIBILITY;

BAYES THEOREM; HUMANS; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN;

EID: 84905094316     PISSN: 00963445     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0036731     Document Type: Article

References (81)

1. Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., . . . Wicherts, J. M. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108-119
2. Bayarri, M. J., Berger, J. O., Forte, A., & García-Donato, G. (2012). Criteria for Bayesian model choice with application to variable selection. Annals of Statistics, 40, 1550-1577
3. Bayarri, M. J., & García-Donato, G. (2007). Extending conventional priors for testing general hypotheses in linear models. Biometrika, 94, 135- 152
4. Bayarri, M. J., & Mayoral, A. M. (2002a). Bayesian analysis and design for comparison of effect sizes. Journal of Statistical Planning and Inference, 103, 225-243
5. Bayarri, M. J., & Mayoral, A. M. (2002b). Bayesian design of "successful" replications. American Statistician, 56, 207-214
6. Bem, D. J., Utts, J., & Johnson, W. O. (2011). Must psychologists change the way they analyze their data? Journal of Personality and Social Psychology, 101, 716-719
7. Berger, J. O. (2006). Bayes factors. In S. Kotz, N. Balakrishnan, C. Read, B. Vidakovic, & N. L. Johnson (Eds.), Encyclopedia of statistical sciences: Vol. 1 (2nd ed., pp. 378-386). Hoboken, NJ: Wiley.
8. Berger, J. O., & Pericchi, L. R. (1996). The intrinsic Bayes factor for model selection and prediction. Journal of the American Statistical Association, 91, 109-122
9. Bones, A. K. (2012). We knew the future all along: Scientific hypothesizing is much more accurate than other forms of precognition-a satire in one part. Perspectives on Psychological Science, 7, 307-309
10. Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., . . . van't Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217-224
11. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.
12. Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304-1312.
13. Cumming, G. (in press). The new statistics: Why and how. Psychological Science.
14. Cumming, G., & Finch, S. (2001). A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. Educational and Psychological Measurement, 61, 532-574
15. de Groot, A. D. (in press). The meaning of "significance" for different types of research (E.-J. Wagenmakers, D. Borsboom, J. Verhagen, R. Kievit, M. Bakker, A. Cramer, . . . H. L. J. van der Maas, Trans. & Annot.). Acta Psychologica. (Original work published 1956)
16. Dickey, J. M., & Lientz, B. P. (1970). The weighted likelihood ratio, sharp hypotheses about chances, the order of a Markov chain. Annals of Mathematical Statistics, 41, 214-226
17. Dienes, Z. (2011). Bayesian versus orthodox statistics: Which side are you on? Perspectives on Psychological Science, 6, 274-290
18. Dijksterhuis, A., Spears, R., Postmes, T., Stapel, D., Koomen, W., van Knippenberg, A., & Scheepers, D. (1998). Seeing one thing and doing another: Contrast effects in automatic behavior. Journal of Personality and Social Psychology, 75, 862-871
19. Dijksterhuis, A., & van Knippenberg, A. (1998). The relation between perception and behavior, or how to win a game of Trivial Pursuit. Journal of Personality and Social Psychology, 74, 865- 877
20. Edwards, W., Lindman, H., & Savage, L. J. (1963). Bayesian statistical inference for psychological research. Psychological Review, 70, 193- 242
21. Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. New York, NY: Chapman & Hall
22. Elliot, A. J., Kayser, D. N., Greitemeyer, T., Lichtenfeld, S., Gramzow, R. H., Maier, M. A., & Liu, H. (2010). Red, rank, and romance in women viewing men. Journal of Experimental Psychology: General, 139, 399-417
23. Francis, G. (in press-a). The frequency of excess success for articles in Psychological Science. Psychonomic Bulletin & Review.
24. Francis, G. (in press-b). Replication, statistical consistency, and publication bias. Journal of Mathematical Psychology.
25. Francis, G. (2013a). Publication bias in "Red, Rank, and Romance in Women Viewing Men," by Elliot et al. Journal of Experimental Psychology: General, 142, 292-296.
26. Francis, G. (2013b). Replication, statistical consistency, and publication bias. Journal of Mathematical Psychology, 57, 153-169
27. Gallistel, C. R. (2009). The importance of proving the null. Psychological Review, 116, 439-453
28. Gelman, A., & Stern, H. (2006). The difference between "significant" and "not significant" is not itself statistically significant. American Statistician, 60, 328-331
29. Jeffreys, H. (1961). Theory of probability (3rd ed.). Oxford, England: Oxford University Press.
30. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23, 524-532
31. Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773-795
32. Kruschke, J. K. (2010). Doing Bayesian data analysis: A tutorial introduction with R and BUGS. Burlington, MA: Academic Press.
33. LeBel, E. P., & Peters, K. R. (2011). Fearing the future of empirical psychology: Bem's (2011). Evidence of psi as a case study of deficiencies in modal research practice. Review of General Psychology, 15, 371-379
34. LeBoeuf, R. A., & Estes, Z. (2004). "Fortunately, I'm no Einstein": Comparison relevance as a determinant of behavioral assimilation and contrast. Social Cognition, 22, 607-636
35. Lecoutre, B. (1999). Two useful distributions for Bayesian predictive procedures under normal models. Journal of Statistical Planning and Inference, 79, 93-105
36. Lecoutre, B. (2007). Another look at confidence intervals for the noncentral T distribution. Journal of Modern Applied Statistical Methods, 6, 107-116.
37. Lee, M. D., & Wagenmakers, E.-J. (2013). Bayesian modeling for cognitive science: A practical course. New York, NY: Cambridge University Press.
38. Lempers, F. B. (1971). Posterior probabilities of alternative linear models. Rotterdam, the Netherlands: University Press Rotterdam.
39. Liang, F., Paulo, R., Molina, G., Clyde, M. A., & Berger, J. O. (2008). Mixtures of g priors for Bayesian variable selection. Journal of the American Statistical Association, 103, 410 - 423
40. Lindley, D. V. (1965). Introduction to probability and statistics from a Bayesian viewpoint. Part 2. Inference From a Bayesian Viewpoint. Cambridge, England: Cambridge University Press
41. Liu, C. C., & Aitkin, M. (2008). Bayes factors: Prior sensitivity and model generalizability. Journal of Mathematical Psychology, 52, 362-375
42. Milliken, B., Joordens, S., Merikle, P. M., & Seiffert, A. E. (1998). Selective attention: A reevaluation of the implications of negative priming. Psychological Review, 105, 203-229
43. Morey, R. D., & Rouder, J. N. (2011). Bayes factor approaches for testing interval null hypotheses. Psychological Methods, 16, 406-419
44. Morey, R. D., Rouder, J. N., Verhagen, J., & Wagenmakers, E.-J. (in press). Why hypothesis tests are essential for psychological science: A comment on Cumming. Psychological Science.
45. Morey, R. D., & Wagenmakers, E.-J. (2014). Simple relation between Bayesian order-restricted and point-null hypothesis tests. Manuscript submitted for publication.
46. Mussweiler, T. (2003). Comparison processes in social judgment: Mechanisms and consequences. Psychological Review, 110, 472-489
47. Neill, W. T., & Kahan, T. A. (1999). Response conflict reverses priming: A replication. Psychonomic Bulletin & Review, 6, 304 -308
48. Neuroskeptic. (2012). The nine circles of scientific hell. Perspectives on Psychological Science, 7, 643-644.
49. Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011). Erroneous analyses of interactions in neuroscience: A problem of significance. Nature Neuroscience, 14, 1105-1107
50. Nosek, B. A., & Bar-Anan, Y. (2012). Scientific utopia: I. Opening scientific communication. Psychological Inquiry, 23, 217-243
51. Nosek, B. A., & Lakens, D. (2013). Call for proposals: Special issue of Social Psychology on "replications of important results in social psychology." Social Psychology, 44, 59 - 60
52. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615-631.
53. O'Hagan, A., & Forster, J. (2004). Kendall's advanced theory of statistics: Vol. 2B. Bayesian inference (2nd ed.). London, England: Arnold.
54. Pashler, H., & Wagenmakers, E.-J. (2012). Editors' introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7, 528-530
55. Rosenthal, R. (1990). Replication in behavioral research. Journal of Social Behavior and Personality, 5, 1-30.
56. Rouanet, H. (1996). Bayesian methods for assessing importance of effects. Psychological Bulletin, 119, 149-158
57. Rouder, J. N. (in press). Optional stopping: No problem for Bayesians. Psychonomic Bulletin & Review.
58. Rouder, J. N., & Morey, R. D. (2011). A Bayes factor meta-analysis of Bem's ESP claim. Psychonomic Bulletin & Review, 18, 682-689
59. Rouder, J. N., & Morey, R. D. (2012). Default Bayes factors for model selection in regression. Multivariate Behavioral Research, 47, 877-903
60. Rouder, J. N., Morey, R. D., Speckman, P. L., & Province, J. M. (2012). Default Bayes factors for ANOVA designs. Journal of Mathematical Psychology, 56, 356-374
61. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225-237
62. Royall, R. M. (1997). Statistical evidence: A likelihood paradigm. London, England: Chapman & Hall.
63. Sellke, T., Bayarri, M. J., & Berger, J. O. (2001). Calibration of p values for testing precise null hypotheses. American Statistician, 55, 62-71
64. Shanks, D. R., Newell, B. R., Lee, E. H., Balakrishnan, D., Ekelund, L., Cenac, Z., . . . Moore, C. (2013). Priming intelligent behavior: An elusive phenomenon. PLoS ONE, 8, e56515
65. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366
66. Simonsohn, U. (2013). Evaluating replication results. Manuscript submitted for publication.
67. Ullrich, J., Krueger, J., Brod, A., & Groschupf, F. (in press). More is not less: Greater information quantity does not diminish liking. Journal of Personality and Social Psychology.
68. Vanpaemel, W. (2010). Prior sensitivity in theory testing: An apologia for the Bayes factor. Journal of Mathematical Psychology, 54, 491-498
69. Wagenmakers, E.-J., Lee, M. D., Lodewyckx, T., & Iverson, G. (2008). Bayesian versus frequentist inference. In H. Hoijtink, I. Klugkist, & P. A. Boelen (Eds.), Bayesian evaluation of informative hypotheses (pp. 181-207). New York, NY: Springer-Verlag
70. Wagenmakers, E.-J., Lodewyckx, T., Kuriyal, H., & Grasman, R. (2010). Bayesian hypothesis testing for psychologists: A tutorial on the Savage- Dickey method. Cognitive Psychology, 60, 158-189
71. Wagenmakers, E.-J., van der Maas, H. J. L., & Grasman, R. P. P. P. (2007). An EZ-diffusion model for response time and accuracy. Psychonomic Bulletin & Review, 14, 3-22
72. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: The case of psi. Journal of Personality and Social Psychology, 100, 426-432
73. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632- 638
74. Wetzels, R., Grasman, R. P. P. P., & Wagenmakers, E.-J. (2012). A default Bayesian hypothesis test for ANOVA designs. American Statistician, 66, 104-111
75. Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011). Statistical evidence in experimental psychology: An empirical comparison using 855 t tests. Perspectives on Psychological Science, 6, 291-298.
76. Wetzels, R., & Wagenmakers, E.-J. (2012). A default Bayesian hypothesis test for correlations and partial correlations. Psychonomic Bulletin & Review, 19, 1057-1064
77. Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61, 726 -728
78. Yong, E. (2012, October 10). Nobel laureate challenges psychologists to clean up their act: Social-priming research needs "daisy chain" of replication. Nature
79. Zellner, A. (1986). On assessing prior distributions and Bayesian regression analysis with g-prior distributions. In P. Goel & A. Zellner (Eds.), Bayesian inference and decision techniques: Essays in honor of Bruno de Finetti (pp. 233-243). Amsterdam, the Netherlands: North- Holland.
80. Zellner, A., & Siow, A. (1980). Posterior odds ratios for selected regression hypotheses. In J. M. Bernardo, M. H. deGroot, D. V. Lindley, & A. F.M. Smith (Eds.), Bayesian statistics (pp. 585-603). Valencia, Spain: University Press.
81. Zwaan, R. A., & Zeelenberg, R. (2013). Replication attempts of important results in the study of cognition. Frontiers in Psychology: Cognition. Retrieved from http://www.frontiersin.org/Cognition/researchtopics/ Replication_Attempts_of_Import/1461