Bayes factor design analysis: Planning for compelling evidence

Psychonomic Bulletin and Review

Volumn 25, Issue 1, 2018, Pages 128-142

  Schönbrodt, Felix D ^a   Wagenmakers, Eric Jan ^b  

^a UNIVERSITY OF MUNICH   (Germany)

^b UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

Bayes factor; Design analysis; Design planning; Power analysis; Sequential testing

Indexed keywords

BAYES THEOREM; HUMAN; INFORMATION PROCESSING; METHODOLOGY; PROBABILITY; SAMPLE SIZE;

BAYES THEOREM; DATA COLLECTION; HUMANS; PROBABILITY; RESEARCH DESIGN; SAMPLE SIZE;

EID: 85014101196     PISSN: 10699384     EISSN: 15315320     Source Type: Journal    
DOI: 10.3758/s13423-017-1230-y     Document Type: Article

References (73)

1. Armitage, P., McPherson, C.K., & Rowe, B.C. (1969). Repeated significance tests on accumulating data. Journal of the Royal Statistical Society. Series A (General), 132(2), 235–244.
2. Bacchetti, P., Wolf, L.E., Segal, M.R., & McCulloch, C.E. (2005). Ethics and sample size. American Journal of Epidemiology, 161(2), 105–110. doi:10.1093/aje/kwi014.
3. Bassler, D., Briel, M., Montori, V.M., Lane, M., Glasziou, P., Zhou, Q., & Guyatt, G.H. (2010). Stopping randomized trials early for benefit and estimation of treatment effects: Systematic review and meta-regression analysis. Journal of the American Medical Association, 303(12), 1180–1187.
4. Bayarri, M.J., Benjamin, D.J., Berger, J.O., & Sellke, T.M. (2016). Rejection odds and rejection ratios: A proposal for statistical practice in testing hypotheses. Journal of Mathematical Psychology, 72, 90–103. doi:10.1016/j.jmp.2015.12.007.
5. Berger, J.O. (1985). Statistical decision theory and Bayesian analysis, 2nd ed. Springer: New York.
6. Berger, J.O. (2006). Bayes factors. In Kotz, S., Balakrishnan, N., Read, C., Vidakovic, B., & Johnson, N.L. (Eds.), Encyclopedia of statistical sciences. 2nd ed., (Vol. 1, pp. 378–386). Hoboken, NJ: Wiley.
7. Berger, J.O., & Wolpert, R.L. (1988). The likelihood principle, 2nd ed. Hayward, CA: Institute of Mathematical Statistics.
8. Berger, J.O., Brown, L.D., & Wolpert, R.L. (1994). A unified conditional frequentist and Bayesian test for fixed and sequential simple hypothesis testing. The Annals of Statistics, 22(4), 1787–1807. doi:10.1214/aos/1176325757.
9. Berry, S.M., Bradley, P.C., & Connor, J. (2010). Bias and trials stopped early for benefit. JAMA, 304, 156–159. doi:10.1001/jama.2010.930.
10. Blume, J.D. (2002). Likelihood methods for measuring statistical evidence. Statistics in Medicine, 21(17), 2563–2599. doi:10.1002/sim.1216.
11. Blume, J.D. (2008). How often likelihood ratios are misleading in sequential trials. Communications in Statistics: Theory & Methods, 37(8), 1193–1206. doi:10.1080/03610920701713336.
12. Boehm, U., Hawkins, G.E., Brown, S., van Rijn, H., & Wagenmakers, E.-J. (2015). Of monkeys and men: Impatience in perceptual decision-making. Psychonomic Bulletin & Review, 23(3), 738–749. doi:10.3758/s13423-015-0958-5.
13. Bogacz, R., Brown, E., Moehlis, J., Holmes, P., & Cohen, J.D. (2006). The Physics of Optimal Decision Making: A Formal Analysis of Models of Performance in Two?alternative Forced Choice Tasks. Psychological Review, 113, 700–765.
14. Button, K.S., Ioannidis, J.P., Mokrysz, C., Nosek, B.A., Flint, J., Robinson, E.S., & Munafò, M.R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365–376.
15. Cavagnaro, D.R., Myung, J.I., Pitt, M.A., & Kujala, J.V. (2009). Adaptive design optimization: A mutual information-based approach to model discrimination in cognitive science. Neural Computation, 22(4), 887–905. doi:10.1162/neco.2009.02-09-959.
16. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. New Jersey, US: Lawrence Erlbaum Associates.
17. De Santis, F. (2004). Statistical evidence and sample size determination for Bayesian hypothesis testing. Journal of Statistical Planning and Inference, 124, 121–144.
18. Dienes, Z. (2008). Understanding psychology as a science: An introduction to scientific and statistical inference. New York: Palgrave Macmillan.
19. Dienes, Z. (2011). Bayesian versus orthodox statistics: Which side are you on?. Perspectives on Psycho-logical Science, 6(3), 274–290. doi:10.1177/1745691611406920.
20. Dienes, Z. (2014). Using Bayes to get the most out of nonsignificant results. Frontiers in Psychology: Quantitative Psychology and Measurement, 5, 781. doi:10.3389/fpsyg.2014.00781.
21. Dienes, Z. (2016). How Bayes factors change scientific practice. Journal of Mathematical Psychology, 72, 78–89. doi:10.1016/j.jmp.2015.10.003.
22. Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., & Johannesson, M. (2015). Using prediction markets to estimate the reproducibility of scientific research. Proceedings of the National Academy of Sciences, 112(50), 15343–15347. doi:10.1073/pnas.1516179112.
23. Eaton, M.L., Muirhead, R.J., & Soaita, A.I. (2013). On the limiting behavior of the probability of claiming superiority in a Bayesian context. Bayesian Analysis, 8(1), 221–232.
24. Edwards, W., Lindman, H., & Savage, L.J. (1963). Bayesian statistical inference for psychological research. Psychological Review, 70(3), 193–242. doi:10.1037/h0044139.
25. Emanuel, E.J., Wendler, D., & Grady, C. (2000). What makes clinical research ethical?. JAMA, 283(20), 2701–2711.
26. Fan, X., DeMets, D.L., & Lan, K.K.G. (2004). Conditional bias of point estimates following a group sequential test. Journal of Biopharmaceutical Statistics, 14(2), 505–530. doi:10.1081/BIP-120037195.
27. Forstmann, B.U., Ratcliff, R., & Wagenmakers, E.-J. (2016). Sequential Sampling Models in Cognitive Neuroscience: Advantages, Applications, and Extensions. Annual Review of Psychology, 67, 641–666.
28. Garthwaite, P.H., Kadane, J.B., & O’Hagan, A. (2005). Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100(470), 680–701. doi:10.1198/016214505000000105.
29. Gelman, A., & Carlin, J. (2014). Beyond power calculations: Assessing Type S (sign) and Type M (magnitude) errors. Perspectives on Psychological Science, 9(6), 641–651. doi:10.1177/1745691614551642.
30. Gelman, A., & Tuerlinckx, F. (2000). Type S error rates for classical and Bayesian single and multiple comparison procedures. Computational Statistics, 15(3), 373–390.
31. Good, I.J. (1979). Studies in the history of probability and statistics. XXXVII A. M. Turing?s statistical work in World War II. Biometrika, 66(2), 393–396. doi:10.1093/biomet/66.2.393.
32. Goodman, S.N. (2007). Stopping at nothing? Some dilemmas of data monitoring in clinical trials. Annals of Internal Medicine, 146(12), 882–887.
33. Halpern, S.D., Karlawish, J.H.T., & Berlin, J.A. (2002). The continuing unethical conduct of underpowered clinical trials. JAMA, 288(3), 358–362.
34. Hoijtink, H., Klugkist, I., & Boelen, P. (2008). Bayesian Evaluation of Informative Hypotheses. New York: Springer.
35. Ioannidis, J.P.A. (2005). Why most published research findings are false. PLoS Med, 2(8), e124. doi:10.1371/journal.pmed.0020124.
36. JASP Team (2016). JASP (Version 0.7.5.6)[Computer software].
37. Jeffreys, H. (1961). The theory of probability. Oxford University Press.
38. Kass, R.E., & Raftery, A.E. (1995). Bayes factors. Journal of the American Statistical Association, 90(430), 773–795.
39. Kruschke, J.K. (2010). Bayesian data analysis. Wiley Interdisciplinary Reviews: Cognitive Science, 1(5), 658–676. doi:10.1002/wcs.72.
40. Kruschke, J.K. (2014). Doing Bayesian data analysis: A tutorial with R, JAGS, and Stan, 2nd edn. Boston: Academic Press.
41. Lakens, D., & Evers, E.R.K. (2014). Sailing from the seas of chaos into the corridor of stability: Practical recommendations to increase the informational value of studies. Perspectives on Psychological Science, 9(3), 278–292. doi:10.1177/1745691614528520.
42. Lee, M.D., & Wagenmakers, E.-J. (2013). Bayesian cognitive modeling: A practical course. Cambridge University Press.
43. Lewis, S.M., & Raftery, A.E. (1997). Estimating Bayes Factors via posterior simulation with the Laplace Metropolis estimator. Journal of the American Statistical Association, 92, 648–655.
44. Lindley, D.V. (1956). On a Measure of the Information Provided by an Experiment. The Annals of Mathematical Statistics, 27, 986– 1005.
45. Lindley, D.V. (1997). The choice of sample size. Journal of the Royal Statistical Society. Series D (The Statistician), 46(2), 129– 138.
46. Ly, A., Verhagen, J., & Wagenmakers, E.-J. (2016). Harold Jeffreys’s default Bayes factor hypothesis tests: Explanation, extension, and application in psychology. Journal of Mathematical Psychology. Bayes Factors for Testing Hypotheses in Psychological Research: Practical Relevance and New Developments, 72, 19–32. doi:10.1016/j.jmp.2015.06.004.
47. Morey, R.D., & Rouder, J.N. (2015). BayesFactor: Computation of Bayes factors for common designs.
48. Morey, R.D., Romeijn, J.-W., & Rouder, J.N. (2016). The philosophy of Bayes factors and the quantification of statistical evidence. Journal of Mathematical Psychology. Bayes Factors for Testing Hypotheses in Psychological Research: Practical Relevance and New Developments, 72, 6–18. doi:10.1016/j.jmp.2015.11.001.
49. Morris, D.E., Oakley, J.E., & Crowe, J.A. (2014). A webbased tool for eliciting probability distributions from experts. Environmental Modelling & Software, 52, 1–4. doi:10.1016/j.envsoft.2013.10.010.
50. Muirhead, R.J., & Soaita, A.I. (2013). On an approach to Bayesian sample sizing in clinical trials. In Jones, G., & Shen, X. (Eds.), Advances in Modern Statistical Theory and Applications: A Festschrift in honor of Morris L. Eaton (pp. 126–137). Ohio: Institute of Mathematical Statistics: Beachwood.
51. Mulder, J., & Wagenmakers, E.-J. (2016). Editors? introduction to the special issue Bayes factors for testing hypotheses in psychological research: Practical relevance and new developments. Journal of Mathematical Psychology. Bayes Factors for Testing Hypotheses in Psychological Research: Practical Relevance and New Developments, 72, 1–5. doi:10.1016/j.jmp.2016.01.002.
52. O’Hagan, A., & Forster, J. (2004). Kendall’s Advanced Theory of Statistics Vol. 2B: Bayesian Inference (2nd ed.) London: Arnold.
53. O’Hagan, A., & Stevens, J.W. (2001). Bayesian assessment of sample size for clinical trials of cost effectiveness. Medical Decision Making: An International Journal of the Society for Medical Decision Making, 21(3), 219–230.
54. O’Hagan, A., Stevens, J.W., & Campbell, M.J. (2005). Assurance in clinical trial design. Pharmaceutical Statistics, 4(3), 187–201. doi:10.1002/pst.175.
55. Platt, J.R. (1964). Strong inference. Science, 146(3642), 347–353. doi:10.1126/science.146.3642.347.
56. Pocock, S.J., & Hughes, M.D. (1989). Practical problems in interim analyses, with particular regard to estimation. Controlled Clinical Trials, 10(4), 209–221.
57. R Core Team (2014). R: A language and environment for statistical computing. Vienna, Austria.
58. Rouder, J.N. (2014). Optional stopping: No problem for Bayesians. Psychonomic Bulletin & Review, 21(2), 301–308. doi:10.3758/s13423-014-0595-4.
59. Rouder, J.N., Morey, R.D., Speckman, P.L., & Province, J.M. (2012). Default Bayes factors for ANOVA designs. Journal of Mathematical Psychology, 56(5), 356–374. doi:10.1016/j.jmp.2012.08.001.
60. Rouder, J.N., Speckman, P.L., Sun, D., Morey, R.D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16(2), 225–237.
61. Royall, R.M. (2000). On the probability of observing misleading statistical evidence. Journal of the American Statistical Association, 95(451), 760–768. doi:10.2307/2669456.
62. Schönbrodt, F.D. (2016). BFDA: Bayes factor design analysis package for R. https://github.com/nicebread/BFDA.
63. Schönbrodt, F.D., Wagenmakers, E.-J., Zehetleitner, M., & Perugini, M. (2015). Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences. Psychological Methods. doi:10.1037/met0000061.
64. Spiegelhalter, D.J., Abrams, K.R., & Myles, J.P. (2004). Bayesian approaches to clinical trials and health-care evaluation. John Wiley & Sons.
65. Taroni, F., Bozza, S., Biedermann, A., Garbolino, P., & Aitken, C. (2010). Data analysis in forensic science: A Bayesian decision perspective. Chichester: JohnWiley & Sons.
66. van Erven, T., Grünwald, P., & de Rooij, S. (2012). Catching up faster by switching sooner: A predictive approach to adaptive estimation with an application to the AIC? BIC dilemma. Journal of the Royal Statistical Society B, 74, 361–417.
67. Wagenmakers, E.-J., Grünwald, P., & Steyvers, M. (2006). Accumulative prediction error and the selection of time series models. Journal of Mathematical Psychology, 50, 149–166.
68. Wagenmakers, E.-J., Morey, R.D., & Lee, M.D. (2016). Bayesian benefits for the pragmatic researcher. Current Directions in Psychological Science, 25(3), 169–176. doi:10.1177/0963721416643289.
69. Wagenmakers, E.-J., Verhagen, J., Ly, A., Bakker, M., Lee, M.D., Matzke, D., & Morey, R.D. (2014). A power fallacy. Behavior Research Methods 47(4):913–917. doi:10.3758/s13428-014-0517-4.
70. Wald, A. (1945). Sequential tests of statistical hypotheses. The Annals of Mathematical Statistics, 16(2), 117–186.
71. Wald, A., & Wolfowitz, J. (1948). Optimum character of the sequential probability ratio test. The Annals of Mathematical Statistics, 19(3), 326–339. doi:10.1214/aoms/1177730197.
72. Walley, R.J., Smith, C.L., Gale, J.D., & Woodward, P. (2015). Advantages of a wholly Bayesian approach to assessing efficacy in early drug development: a case study. Pharmaceutical Statistics, 14(3), 205–215. doi:10.1002/pst.1675.
73. Weiss, R. (1997). Bayesian sample size calculations for hypothesis testing. Journal of the Royal Statistical Society. Series D (The Statistician), 46(2), 185–191.