Using Bayes to get the most out of non-significant results

Frontiers in Psychology

Volumn 5, Issue , 2014, Pages

  Dienes, Zoltan ^a  

^a UNIVERSITY OF SUSSEX   (United Kingdom)

Author keywords

Bayes factor; confidence interval; highest density region; null hypothesis; power; significance testing; statistical inference

Indexed keywords

EID: 85175261271     PISSN: None     EISSN: 16641078     Source Type: Journal    
DOI: 10.3389/fpsyg.2014.00781     Document Type: Article

References (115)

1. Allen, C. P. G., Sumner, P., and Chambers, C. D. (2014). The timing and neuroanatomy of conscious vision as revealed by TMS-induced blindsight. J. Cogn. Neurosci. 26, 1507–1518. doi: 10.1162/jocn_a_00557
2. American Psychological Association [APA] (2010). Publication Manual of the American Psychological Association, 5th Edn. Washington, DC: AP.
3. Armitage, P., McPherson, C. K., and Rowe, B. C. (1969). Repeated significance tests on accumulating data. J. R. Stat. Soc. Ser. A 132, 235–244. doi: 10.2307/2343787
4. Armstrong, A. M., and Dienes, Z. (2013). Subliminal understanding of negation: unconscious control by subliminal processing of word pairs. Conscious. Cogn. 22, 1022–1040. doi: 10.1016/j.concog.2013.06.010
5. Baguley, T. (2004). Understanding statistical power in the context of applied research. Appl. Ergon. 35, 73–80. doi: 10.1016/j.apergo.2004.01.002
6. Baguley, T. (2009). Standardized or simple effect size: what should be reported? Br. J. Psychol. 100, 603–617. doi: 10.1348/000712608X377117
7. Baguley, T. (2012). Serious Stats: A Guide to Advanced Statistics for the Behavioral Sciences. Basingstoke: Palgrave Macmillan.
8. Berger, J. O., and Delampady, M. (1987). Testing precise hypotheses. Stat. Sci. 2, 317–335. doi: 10.1214/ss/1177013238
9. Berger, J. O., and Wolpert, R. L. (1988). The Likelihood Principle, 2nd Edn. Beachwood, OH: Institute of Mathematical Statistic.
10. Berry, D. A. (1996). Statistics: A Bayesian Perspective. London: Duxbury Press.
11. Berry, S. M., Carlin, B. P., Lee, J. J., and Müller, P. (2011). Bayesian Adaptive Methods for Clinical Trials. London: Chapman and Hall.
12. Box, G. E. P., and Tiao, G. C. (1973). Bayesian Inference in Statistical Analysis (Addison-Wesley Series in Behavioral Science, Quantitative Methods). Harlow: Longman Higher Education.
13. Buhle, J. T., Stevens, B. L., Friedman, J. J., and Wager, T. D. (2012). Distraction and placebo: two separate routes to pain control. Psychol. Sci. 23, 246–253. doi: 10.1177/0956797611427919
14. Burnham, K. P., and Anderson, D. R. (2002). Model Selection and Multimodel Inference: A Practical Information–Theoretic Approach, 2nd Edn. New York: Springer.
15. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376. doi: 10.1038/nrn3475
16. Cohen, J. (1962). The statistical power of abnormal-social psychological research: a review. J. Abnorm. Soc. Psychol. 65, 145–153. doi: 10.1037/h0045186
17. Cohen, J. (1988). Statistical Power Analysis for the Behavioural Sciences, 2nd Edn. Hillsdale, NJ: Erlbaum.
18. Cohen, J. (1990). Things I have learnt so far. Am. Psychol. 45, 1304–1312. doi: 10.1037/0003-066X.45.12.1304
19. Cohen, J. (1994). The earth is round (p < 0.05). Am. Psychol. 49, 997–1003. doi: 10.1037/0003-066X.49.12.997
20. Cumming, G. (2011). Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis. Abingdon: Routledge.
21. Dienes, Z. (2008). Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference. Basingstoke: Palgrave Macmillan.
22. Dienes, Z. (2011). Bayesian versus orthodox statistics: which side are you on? Perspect. Psychol. Sci. 6, 274–290. doi: 10.1177/1745691611406920
23. Dienes, Z. (forthcoming). How Bayes factors change our science. J. Math. Psychol. (special issue on Bayes factors). Available at: http://www.lifesci.sussex.ac.uk/home/Zoltan_Dienes/Dienes%20How%20Bayes%20factors%20change%20our%20sci-ence.pdf
24. Dienes, Z. (in press). “How Bayesian statistics are needed to determine whether mental states are unconscious,” in Behavioural Methods in Consciousness Research, ed. M. Overgaard (Oxford: Oxford University Press). Available at: http://www.lifesci.sussex.ac.uk/home/Zoltan_Dienes/Dienes%20Bayes%20and%20the%20unconscious.pdf
25. Dienes, Z., Baddeley, R. J., and Jansari, A. (2012). Rapidly measuring the speed of unconscious learning: amnesics learn quickly and happy people slowly. PLoS ONE 7:e33400. doi: 10.1371/journal.pone.0033400
26. Dienes, Z., and Hutton, S. (2013). Understanding hypnosis metacognitively: rTMS applied to left DLPFC increases hypnotic suggestibility. Cortex 49, 386–392. doi:10.1016/j.cortex.2012.07.009
27. Edwards, A. W. F. (1992). Likelihood, Expanded Edn. London: John Hopkins University Press.
28. Elliot, A. J., Kayser, D. N., Greitemeyer, T., Lichtenfeld, S., Gramzow, R. H., Maier, M. A., et al. (2010). Red, rank, and romance in women viewing men. J. Exp. Psychol. Gen. 139, 399–417. doi: 10.1037/a0019689
29. Faul, F., Erdfelder, E., Buchner, A., and Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: tests for correlation and regression analyses. Behav. Res. Methods 41, 1149–1160. doi: 10.3758/BRM.41.4.1149
30. Fisher, R. A. (1935). The Design of Experiments. Tweeddale: Oliver and Boyd.
31. Freedman, L. S., and Spiegelhalter, D. J. (1983). The assessment of subjective opinion and its use in relation to stopping rules for clinical trials. Statistician 32, 153–160. doi: 10.2307/2987606
32. Fu, Q., Bin, G., Dienes, Z., Fu, X., and Gao, X. (2013a). Learning without consciously knowing: evidence from event-related potentials in sequence learning. Conscious. Cogn. 22, 22–34. doi: 10.1016/j.concog.2012.10.008
33. Fu, Q., Dienes, Z., Shang, J., and Fu, X. (2013b). Who learns more? Cultural differences in implicit sequence learning. PLoS ONE 8:e71625. doi: 10.1371/journal.pone.0071625
34. Gallistel, C. R. (2009). The importance of proving the null. Psychol. Rev. 116, 439–453. doi: 10.1037/a0015251
35. Glover, S., and Dixon, P. (2004). Likelihood ratios: a simple and flexible statistic for empirical psychologists. Psychon. Bull. Rev. 11, 791–806. doi: 10.3758/BF03196706
36. Goodman, S. N. (1999). Toward evidence-based medical statistics. 2: the Bayes factor. Ann. Intern. Med. 130, 1005–1013. doi: 10.7326/0003-4819-130-12-199906150-00019
37. Guo, X., Jiang, S., Wang, H., Zhu, L., Tang, J., Dienes, Z., et al. (2013a). Unconsciously learning task-irrelevant perceptual sequences. Conscious. Cogn. 22, 203–211. doi: 10.1016/j.concog.2012.12.001
38. Guo, X., Li, F., Yang, Z., and Dienes, Z. (2013b). Bidirectional transfer between metaphorical related domains in Implicit learning of formmeaning connections. PLoS ONE 8:e68100. doi: 10.1371/journal.pone.0068100
39. Greenwald, A. G. (1975). Consequences of prejudice against the Null Hypothesis. Psychol. Bull. 82, 1–20. doi: 10.1037/h0076157
40. Greenwald, A. G. (1993). “Consequences of prejudice against the null hypothesis,” in A Handbook for Data Analysis in the Behavioural Sciences: Methodological Issues, eds G. Keren, and C. Lewis (Lawrence Erlbaum: Hove), 419–449.
41. Hodges, J. L. Jr., and Lehmann, E. L. (1954). Testing the approximate validity of statistical hypotheses. J. R. Stat. Soc. Series B Stat Methodol. 16, 261–268.
42. Hoijtink, H., Klugkist, I., and Boelen, P. A. (eds). (2008). Bayesian Evaluation of Informative Hypotheses. New York: Springer. doi: 10.1007/978-0-387-09612-4
43. Howard, G. S., Maxwell, S. E., and Fleming, K. J. (2000). The proof of the pudding: an illustration of the relative strengths of null hypothesis, meta-analysis, and Bayesian analysis. Psychol. Methods 5, 315–332. doi: 10.1037/1082-989X.5.3.315
44. d Edn. Chicago: Open Court.
45. Jackman, S. (2009). Bayesian Analyses for the Social Sciences. Wiley: Eastbourne. doi: 10.1002/9780470686621
46. Jaynes, E. T. (2003). Probability Theory: The Logic of Science. Cambridge, England: Cambridge University Press. doi: 10.1017/CBO9780511790423
47. Jeffreys, H. (1939/1961). The Theory of Probability, 1st/3rd Edn. Oxford, England: Oxford University Press.
48. Kass, R. E., and Wasserman, L. (1996). The selection of prior distributions by formal rules. J. Am. Stat. Assoc. 91, 1343–1370. doi: 10.1080/01621459.1996.10477003
49. Kievit, R. A. (2011). Bayesians caught smuggling priors into Rotterdam Harbor. Perspect. Psychol. Sci. 6:313. doi: 10.1177/1745691611406928
50. Kirsch, I. (2010). The Emperor’s New Drugs: Exploding the Antidepressant Myth. New York: Basic Books.
51. Kirsch, I. (2011). Suggestibility and suggestive modulation of the Stroop effect. Conscious. Cogn. 20, 335–336. doi: 10.1016/j.concog.2010.04.004
52. Kiyokawa, S., Dienes, Z., Tanaka, D., Yamada, A., and Crowe, L. (2012). Cross cultural differences in unconscious knowledge. Cognition 124, 16–24. doi: 10.1016/j.cognition.2012.03.009
53. Kruschke, J. K. (2010a). Doing Bayesian Data Analysis: A Tutorial with R and BUGS. Amsterdam: Academic Press.
54. Kruschke, J. K. (2010b). Bayesian data analysis. Wiley Interdiscip. Rev. Cogn. Sci. 1, 658–676. doi: 10.1002/wcs.72
55. Kruschke, J. K. (2011). Bayesian assessment of null values via parameter estimation and model comparison. Perspect. Psychol. Sci. 6, 299–312. doi: 10.1177/1745691611406925
56. Kruschke, J. K. (2013a). Posterior predictive checks can and should be Bayesian: comment on Gelman and Shalizi, ‘philosophy and the practice of Bayesian statistics’. Br. J. Math. Stat. Psychol. 66, 45–56. doi: 10.1111/j.2044-8317.2012.02063.x
57. Kruschke, J. K. (2013b). Bayesian estimation supersedes the t test. J. Exp. Psychol. Gen. 142, 573–603. doi: 10.1037/a0029146
58. Kruschke, J. K. (2013c). How Much of a Bayesian Posterior Distribution Falls Inside a Region of Practical Equivalence (ROPE). Available at: http://doingbayesiandataanalysis.blogspot.co.uk/2013/08/how-much-of-bayesian-posterior.html [accessed August 9, 2013].
59. Lee, M. D., and Wagenmakers, E. J. (2005). Bayesian statistical inference in psychology: comment on trafimow (2003). Psychol. Rev. 112, 662–668. doi:10.1037/0033-295X.112.3.662
60. Lee, M. D., and Wagenmakers, E. J. (2014). Bayesian Cognitive Modeling: A Practical Course. Cambridge: Cambridge University Press.
61. Lewis, C. (1993). “Analyzing means from repeated measures data,” in A Handbook for Data Analysis for the Behavioural Sciences: Statistical Issues, eds G. Keren, and C. Lewis (Erlbaum: Hove), 73–94.
62. Liang, F., Paulo, R., Molina, G., Clyde, M. A., and Berger, J. O. (2008). Mixtures of g-priors for Bayesian variable selection. J. Am. Stat. Assoc. 103, 410–423. doi: 10.1198/016214507000001337
63. Lindley, D. V. (1957). A statistical paradox. Biometrika 44, 187–192. doi: 10.1093/biomet/44.1-2.187
64. Lindley, D. V. (1972). Bayesian Statistics: A Review. Philadelphia, PA: Society for Industrial and Applied Mathematics.
65. Lockhart, R. S. (1998). Introduction to Data Analysis for the Behavioural Sciences. New York: W. W. Freeman and Company.
66. Mayo, D. (1996). Error and the Growth of Experimental Knowledge. Chicago: University of Chicago Press. doi: 10.7208/chicago/9780226511993.001.0001
67. McGrayne, S. B. (2012). The Theory that Would Not Die: How Bayes’ Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy. Yale, MI: Yale University Press.
68. McLatchie, N. (2013). Feeling Impulsive, Thinking Prosocial: The Importance of Distinguishing Guilty Feelings From Guilty Thoughts. Unpublished Ph.D. thesis, University of Kent, Kent.
69. Mealor, A. D., and Dienes, Z. (2012). Conscious and unconscious thought in artificial grammar learning. Conscious. Cogn. 21, 865–874. doi: 10.1016/j.concog.2012.03.001
70. Mealor, A. D., and Dienes, Z. (2013a). The speed of metacognition: taking time to get to know one’s structural knowledge. Conscious. Cogn. 22, 123–136. doi: 10.1016/j.concog.2012.11.009
71. Mealor, A. D., and Dienes, Z. (2013b). Explicit feedback maintains implicit knowledge. Conscious. Cogn. 22, 822–832. doi: 10.1016/j.concog.2013.05.006
72. Morey, R. D., and Rouder J. N. (2011). Bayes factor approaches for testing interval null hypotheses. Psychol. Methods 16, 406–419. doi: 10.1037/a0024377
73. Newell, B. R., and Rakow, T. (2011). Revising beliefs about the merit of unconscious thought: evidence in favor of the null hypothesis. Soc. Cogn. 29, 711–726. doi: 10.1521/soco.2011.29.6.711
74. Nuijten, M. B., Wetzels, R., Matzke, D., Dolan, C. V., and Wagenmakers, E.-J. (in press). A default Bayesian hypothesis test for mediation. Behav. Res. Methods doi: 10.3758/s13428-014-0470-2 [Epub ahead of print].
75. Oakes, M. (1986). Statistical Inference: Commentary for the Social and Behavioural Sciences. Chichester: Wiley-Blackwell.
76. Parris, B. A., and Dienes, Z. (2013). Hypnotic suggestibility predicts the magnitude of the imaginative word blindness suggestion effect in a non-hypnotic context. Conscious. Cogn. 22, 868–874. doi: 10.1016/j.concog.2013.05.009
77. Pashler, H., and Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspect. Psychol. Sci. 7, 531–536. doi: 10.1177/1745691612463401
78. Raz, A., Fan, J., and Posner, M. I. (2005). Hypnotic suggestion reduces conflict in the human brain. Proc. Natl. Acad. Sci. U.S.A. 102, 9978–9983. doi: 10.1073/pnas.0503064102
79. Rogers, J. L., Howard, K. I., and Vessey, J. T. (1993). Using significance tests to evaluate equivalence between two experimental groups. Psychol. Bull. 113, 553–565. doi: 10.1037/0033-2909.113.3.553
80. Rosenthal, R. (1993). “Cumulating evidence,” in A Handbook for Data Analysis in the Behavioural Sciences: Methodological Issues, eds G. Keren and C. Lewis (Lawrence Erlbaum: Hove), 519–559.
81. Rosenthal, R., Rosnow, R. L., and Rubin, D. R. (2000). Contrasts and Effect Sizes in Behavioural Research: A Correlational Approach. Cambridge University Press Cambridge.
82. Rouder, J. N. (2014). Optional stopping: no problem for Bayesians. Psychon. Bull. Rev. 21, 301–308. doi: 10.3758/s13423-014-0595-4
83. Rouder, J. N., and Morey R. D. (2011). A Bayes-factor meta analysis of Bem’s ESP claim. Psychon. Bull. Rev. 18, 682–689. doi: 10.3758/s13423-011-0088-7
84. Rouder, J. N., Morey, R. D., Speckman, P. L., and Pratte, M. S. (2007). Detecting chance: a solution to the null sensitivity problem in subliminal priming. Psychon. Bull. Rev. 14, 597–605. doi: 10.3758/BF03196808
85. Rouder, J. N., Morey R. D., Speckman P. L., and Province J. M. (2012). Default Bayes factors for ANOVA designs. J. Math. Psychol. 56, 356–374. doi: 10.1016/j.jmp.2012.08.001
86. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., and Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychon. Bull. Rev. 16, 225–237. doi: 10.3758/PBR.16.2.225
87. Royall, R. M. (1997). Statistical Evidence: A Likelihood Paradigm. London: Chapman and Hall.
88. Sagarin B. J., Ambler J. K., and Lee E. M. (2014). An ethical approach to peeking at data. Perspect. Psychol. Sci. 9, 293–304. doi: 10.1177/1745691614528214
89. Sanborn, A. N., and Hills, T. T. (2014). The frequentist implications of optional stopping on Bayesian hypothesis tests. Psychon. Bull. Rev. 21, 283–300. doi: 10.3758/s13423-013-0518-9
90. Sedlmeier, P., and Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychol. Bull. 105, 309–316. doi: 10.1037/0033-2909.105.2.309
91. Semmens-Wheeler, R., Dienes, Z., and Duka, T. (2013). Alcohol increases hypnotic susceptibility. Conscious. Cogn. 22, 1082–1091. doi: 10.1016/j.concog.2013.07.001
92. Serlin, R., and Lapsley, D. (1985). Rationality in psychological research: the good-enough principle. Am. Psychol. 40, 73–83. doi: 10.1037/0003-066X.40.1.73
93. Shang, J., Fu, Q., Dienes, Z., Shao, C. and, Fu, X. (2013). Negative affect reduces performance in implicit sequence learning. PLoS ONE 8:e54693. doi: 10.1371/journal.pone.0054693
94. Smith, A. F. M., and Spiegelhalter, D. J. (1980). Bayes factors and choice criteria for linear models. J. R. Stat. Soc. Ser. B 42, 213–220.
95. Smithson, M. (2003). Confidence Intervals. London: Sage.
96. Song, M., Maniscalco, B., Koizumi, A., and Lau, H. (2013). “A new method for manipulating metacognitive awareness while keeping performance constant,” in Oral Presentation at the 17th Meeting of the Association for the Scientific Study of Consciousness, San Diego, 12–15 July, 2013.
97. Stone, J. V. (2013). Bayes’ Rule: A Tutorial Introduction to Bayesian Analysis. Sebtel Press.
98. Storm, L., Tressoldi, P. E., and Utts, J. (2013). Testing the Storm et al. (2010) metaanalysis using Bayesian and Frequentist approaches: reply to Rouder et al. (2013). Psychol. Bull. 139, 248–254. doi: 10.1037/a0029506
99. Strube, M. J. (2006). SNOOP: a program for demonstrating the consequences of premature and repeated null hypothesis testing. Behav. Res. Methods 38, 24–27. doi: 10.3758/BF03192746
100. Tan, L. F., Dienes, Z., Jansari, A., and Goh, S. Y. (2014). Effect of mindfulness meditation on brain–computer interface performance. Conscious. Cogn. 23, 12–21. doi: 10.1016/j.concog.2013.10.010
101. Tan, L. F., Jansari, A., Keng, S. L., and Goh, S. Y. (2009). “Effect of mental training on BCI performance.” in Human–Computer Interaction, Part II, HCI 2009. LNCS, Vol. 5611, ed. J. A. Jacko (Heidelberg: Springer), 632–635.
102. Terhune, D. B., Wudarczyk, O. A., Kochuparampil, P., and Kadosh, R. C. (2013). Enhanced dimension-specific visual working memory in graphemecolor synaesthesia. Cognition 129, 123–137. doi: 10.1016/j.cognition.2013.06.009
103. Vanpaemel, W. (2010). Prior sensitivity in theory testing: an apologia for the Bayes factor. J. Math. Psychol. 54, 491–498. doi: 10.1016/j.jmp.2010.07.003
104. Vanpaemel, W., and Lee, M. D. (2012). Using priors to formalize theory: optimal attention and the generalized context model. Psychon. Bull. Rev. 19, 1047–1056. doi: 10.3758/s13423-012-0300-4
105. Verhagen, A. J., and Wagenmakers, E.-J. (in press). Bayesian tests to quantify the result of a replication attempt. J. Exp. Psychol. Gen.
106. Wagenmakers, E. J. (2007). A practical solution to the pervasive problems of p values. Psychon. Bull. Rev. 14, 779–804. doi: 10.3758/BF03194105
107. Wagenmakers, E. J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., and Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspect. Psychol. Sci. 7, 632–638. doi: 10.1177/1745691612463078
108. Watson, J. C., Gordon, L. B., Stermac, L., Kalogerakos, F., and Steckley, P. (2003). Comparing the effectiveness of process–experiential with cognitive–behavioral psychotherapy in the treatment of depression. J. Consult. Clin. Psychol. 71, 773–781. doi: 10.1037/0022-006X.71.4.773
109. Wetzels, R., Grasman, R. P. P. P., and Wagenmakers, E. J. (2012). A default Bayesian hypothesis test for ANOVA designs. Am. Stat. 66, 104–111. doi: 10.1080/00031305.2012.695956
110. Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., and Wagenmakers, E. J. (2011). Statistical evidence in experimental psychology: an empirical comparison using 855 t tests. Perspect. Psychol. Sci. 6, 291–298. doi: 10.1177/1745691611406923
111. Wetzels, R., Raaijmakers, J. G. W., Jakab, E., and Wagenmakers, E.-J. (2009). How to quantify support for and against the null hypothesis: a flexible WinBUGS implementation of a default Bayesian t test. Psychon. Bull. Rev. 16, 752–760. doi: 10.3758/PBR.16.4.752
112. Wetzels, R., and Wagenmakers, E. J. (2012). A default Bayesian hypothesis test for correlations and partial correlations. Psychon. Bull. Rev. 19, 1057–1064. doi: 10.3758/s13423-012-0295-x
113. Wilcox, R. R. (2010). Fundamentals of Modern Statistical Methods: Substantially Improving Power and Accuracy, 2nd Edn. London: Springer. doi: 10.1007/978-1-4419-5525-8
114. Yu, E. C., and Sprenger, A. M., Thomas, R. P., and Dougherty, M. R. (2014). When decision heuristics and science collide. Psychon. Bull. Rev. 21, 268–282. doi: 10.3758/s13423-013-0495-z
115. Yuan, Y., and MacKinnon, D. P. (2009). Bayesian mediation analysis. Psychol. Methods 14, 301–322. doi: 10.1037/a0016972