Bayesian hypothesis testing for psychologists: A tutorial on the Savage-Dickey method

Cognitive Psychology

Volumn 60, Issue 3, 2010, Pages 158-189

  Wagenmakers, Eric Jan ^a   Lodewyckx, Tom ^b   Kuriyal, Himanshu ^c   Grasman, Raoul ^a  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

^b UNIVERSITY OF LEUVEN   (Belgium)

^c INDIAN INSTITUTE OF TECHNOLOGY   (India)

Author keywords

Bayes factor; Hierarchical modeling; Model selection; Order restrictions; Random effects; Statistical evidence

Indexed keywords

ARTICLE; BAYES THEOREM; BEHAVIORAL RESEARCH; HUMAN; PSYCHOLOGY; STATISTICAL ANALYSIS; STATISTICAL MODEL; STATISTICS;

BAYES THEOREM; BEHAVIORAL RESEARCH; COGNITIVE SCIENCE; DATA INTERPRETATION, STATISTICAL; HUMANS; LIKELIHOOD FUNCTIONS; MODELS, STATISTICAL;

EID: 77649192250     PISSN: 00100285     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.cogpsych.2009.12.001     Document Type: Article

References (109)

1. Aitchison J., and Dunsmore I.R. Statistical prediction analysis (1975), Cambridge University Press, Cambridge
2. Anscombe F.J. Sequential medical trials. Journal of the American Statistical Association 58 (1963) 365-383
3. Bartlett M.S. A comment on D.V. Lindley's statistical paradox. Biometrika 44 (1957) 533-534
4. Batchelder W.H., and Riefer D.M. Theoretical and empirical review of multinomial process tree modeling. Psychonomic Bulletin & Review 6 (1999) 57-86
5. Berger J.O., and Berry D.A. The relevance of stopping rules in statistical inference. In: Gupta S.S., and Berger J.O. (Eds). Statistical decision theory and related topics Vol. 1 (1988), Springer Verlag, New York 29-72
6. Berger J.O., and Berry D.A. Statistical analysis and the illusion of objectivity. American Scientist 76 (1988) 159-165
7. Berger J.O., and Delampady M. Testing precise hypotheses. Statistical Science 2 (1987) 317-352
8. Berger J.O., and Mortera J. Default Bayes factors for nonnested hypothesis testing. Journal of the American Statistical Association 94 (1999) 542-554
9. Berger J.O., and Pericchi L.R. The intrinsic Bayes factor for model selection and prediction. Journal of the American Statistical Association 91 (1996) 109-122
10. Berger J.O., and Pericchi L.R. Objective Bayesian methods for model selection: Introduction and comparison (with discussion). In: Lahiri P. (Ed). Model selection Vol. 38 (2001), Institute of Mathematical Statistics Lecture Notes-Monograph Series, Beachwood, OH 135-207
11. Bernardo J.M., and Smith A.F.M. Bayesian theory (1994), Wiley, New York
12. Besag J. A candidate's formula: A curious result in Bayesian prediction. Biometrika 76 (1989) 183
13. Bowers J.S. Priming is not all bias: Commentary on Ratcliff and McKoon (1997). Psychological Review 106 (1999) 582-596
14. Bowers J.S., Vigliocco G., and Haan R. Orthographic, phonological, and articulatory contributions to masked letter and word priming. Journal of Experimental Psychology: Human Perception and Performance 24 (1998) 1705-1719
15. Brückner H., and Bearman P. After the promise: The STD consequences of adolescent virginity pledges. Journal of Adolescent Health 36 (2005) 271-278
16. Busemeyer J.R., and Stout J.C. A contribution of cognitive decision models to clinical assessment: Decomposing performance on the Bechara gambling task. Psychological Assessment 14 (2002) 253-262
17. Carlin B.P., and Chib S. Bayesian model choice via Markov chain Monte Carlo methods. Journal of the Royal Statistical Society, Series B 57 (1995) 473-484
18. Chen M.-H. Computing marginal likelihoods from a single MCMC output. Statistica Neerlandica 59 (2005) 16-29
19. Chib S. Marginal likelihood from the Gibbs output. Journal of the American Statistical Association 90 (1995) 1313-1321
20. Chib S., and Jeliazkov I. Marginal likelihood from the Metropolis-Hastings output. Journal of the American Statistical Association 96 (2001) 270-281
21. Christensen R. Testing fisher, Neyman, Pearson, and Bayes. The American Statistician 59 (2005) 121-126
22. Congdon P. Applied Bayesian modelling (2003), Wiley, Chichester, UK
23. Consonni G., and Veronese P. Compatibility of prior specifications across linear models. Statistical Science 23 (2008) 332-353
24. de Braganca Pereira C.A., and Stern J.M. Evidence and credibility: Full Bayesian significance test for precise hypotheses. Entropy 1 (1999) 99-110
25. Dennis S., and Humphreys M.S. A context noise model of episodic word recognition. Psychological Review 108 (2001) 452-477
26. Dennis S.J., Lee M.D., and Kinnell A. Bayesian analysis of recognition memory: The case of the list - length effect. Journal of Memory and Language 59 (2008) 361-376
27. In: Dey D.K., Ghosh S.K., and Mallick B.K. (Eds). Generalized linear models: A Bayesian perspective (2000), Taylor & Francis, Boca Raton, FL
28. Dickey J.M. The weighted likelihood ratio, linear hypotheses on normal location parameters. The Annals of Mathematical Statistics 42 (1971) 204-223
29. Dickey J.M., and Lientz B.P. The weighted likelihood ratio, sharp hypotheses about chances, the order of a Markov chain. The Annals of Mathematical Statistics 41 (1970) 214-226
30. Draper D. Assessment and propagation of model uncertainty. Journal of the Royal Statistical Society, Series B 57 (1995) 45-97
31. Edwards W., Lindman H., and Savage L.J. Bayesian statistical inference for psychological research. Psychological Review 70 (1963) 193-242
32. Forster K.I., Mohan K., and Hector J. The mechanics of masked priming. In: Kinoshita S., and Lupker S.J. (Eds). Masked priming: The state of the art (2003), Psychology Press, New York, NY 3-38
33. Gallistel C.R. The importance of proving the null. Psychological Review 116 (2009) 439-453
34. Gamerman D., and Lopes H.F. Markov chain Monte Carlo: Stochastic simulation for Bayesian inference (2006), Chapman & Hall, Boca Raton, FL
35. Gelfand A.E., Smith A.F.M., and Lee T.-M. Bayesian analysis of constrained parameter and truncated data problems using Gibbs sampling. Journal of the American Statistical Association 87 (1992) 523-532
36. Gelman A. Inference and monitoring convergence. In: Gilks W.R., Richardson S., and Spiegelhalter D.J. (Eds). Markov chain Monte Carlo in practice (1996), Chapman & Hall/CRC, Boca Raton, FL 131-143
37. Gelman A., and Hill J. Data analysis using regression and multilevel/hierarchical models (2007), Cambridge University Press, Cambridge
38. Gelman A., and Rubin D.B. Inference from iterative simulation using multiple sequences (with discussion). Statistical Science 7 (1992) 457-472
39. Geurts H.M., Verté S., Oosterlaan J., Roeyers H., and Sergeant J.A. How specific are executive functioning deficits in attention deficit hyperactivity disorder and autism?. Journal of Child Psychology and Psychiatry 45 (2004) 836-854
40. Gigerenzer G. The Superego, the Ego, and the Id in statistical reasoning. In: Lewis C., and Keren G. (Eds). A handbook for data analysis in the behavioral sciences: Methodological issues (1993), Erlbaum, Hillsdale, NJ 311-339
41. Gigerenzer G. Mindless statistics. The Journal of Socio-Economics 33 (2004) 587-606
42. Gigerenzer G., Krauss S., and Vitouch O. The null ritual: What you always wanted to know about significance testing but were afraid to ask. In: Kaplan D. (Ed). The Sage handbook of quantitative methodology for the social sciences (2004), Sage, Thousand Oaks, CA 391-408
43. In: Gilks W.R., Richardson S., and Spiegelhalter D.J. (Eds). Markov chain Monte Carlo in practice (1996), Chapman & Hall/CRC, Boca Raton, FL
44. Gilks W.R., Thomas A., and Spiegelhalter D.J. A language and program for complex Bayesian modelling. The Statistician 43 (1994) 169-177
45. Gill J. Bayesian methods: A social and behavioral sciences approach (2002), CRC Press, Boca Raton, FL
46. Gönen M., Johnson W.O., Lu Y., and Westfall P.H. The Bayesian two-sample t test. The American Statistician 59 (2005) 252-257
47. Good I.J. Weight of evidence: A brief survey. In: Bernardo J.M., DeGroot M.H., Lindley D.V., and Smith A.F.M. (Eds). Bayesian statistics 2 (1985), Elsevier, New York 249-269
48. Grant D.A., and Berg E.A. A behavioral analysis of degree of reinforcement and ease of shifting to new responses in a Weigl-type card-sorting problem. Journal of Experimental Psychology 38 (1948) 404-411
49. Green P.J. Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika 82 (1995) 711-732
50. Haller H., and Krauss S. Misinterpretations of significance: A problem students share with their teachers?. Methods of Psychological Research 7 (2002) 1-20
51. Heaton R.K., Chelune G.J., Talley J.L., Kay G.G., and Curtiss G. Wisconsin card sorting test manual: Revised and expanded (1993), Psychological Assessment Resources, Inc., Odessa, FL
52. Hoeting J.A., Madigan D., Raftery A.E., and Volinsky C.T. Bayesian model averaging: A tutorial. Statistical Science 14 (1999) 382-417
53. Hoijtink H., Klugkist I., and Boelen P. Bayesian evaluation of informative hypotheses that are of practical value for social scientists (2008), Springer, New York
54. Hsiao C.K. Approximate Bayes factors when a mode occurs on the boundary. Journal of the American Statistical Association 92 (1997) 656-663
55. Hubbard R., and Bayarri M.J. Confusion over measures of evidence (p's) versus errors (α's) in classical statistical testing. The American Statistician 57 (2003) 171-182
56. Jaynes E.T. Probability theory: The logic of science (2003), Cambridge University Press, Cambridge, UK
57. Jeffreys H. Theory of probability (1961), Oxford University Press, Oxford, UK
58. Kass R.E., and Raftery A.E. Bayes factors. Journal of the American Statistical Association 90 (1995) 773-795
59. Kass R.E., and Vaidyanathan S.K. Approximate Bayes factors and orthogonal parameters, with application to testing equality of two binomial proportions. Journal of the Royal Statistical Society, Series B 54 (1992) 129-144
60. Kass R.E., and Wasserman L. A reference Bayesian test for nested hypotheses and its relationship to the Schwarz criterion. Journal of the American Statistical Association 90 (1995) 928-934
61. Klugkist I., Laudy O., and Hoijtink H. Inequality constrained analysis of variance: A Bayesian approach. Psychological Methods 10 (2005) 477-493
62. Klugkist I., Laudy O., and Hoijtink H. Bayesian eggs and Bayesian omelettes: Reply to Stern (2005). Psychological Methods 10 (2005) 500-503
63. Kruschke J.K. ALCOVE: An exemplar-based connectionist model of category learning. Psychological Review 99 (1992) 22-44
64. Lauritzen S.L. Graphical models (1996), Clarendon, Oxford
65. Lee M.D. Three case studies in the Bayesian analysis of cognitive models. Psychonomic Bulletin & Review 15 (2008) 1-15
66. Lewis S.M., and Raftery A.E. Estimating Bayes factors via posterior simulation with the Laplace-Metropolis estimator. Journal of the American Statistical Association 92 (1997) 648-655
67. Liang F., Paulo R., Molina G., Clyde M.A., and Berger J.O. Mixtures of g priors for Bayesian variable selection. Journal of the American Statistical Association 103 (2008) 410-423
68. Lindley D.V. Bayesian statistics, a review (1972), SIAM, Philadelphia, PA
69. Lindley D.V. The philosophy of statistics. The Statistician 49 (2000) 293-337
70. Liu C.C., and Aitkin M. Bayes factors: Prior sensitivity and model generalizability. Journal of Mathematical Psychology 52 (2008) 362-375
71. Lodewyckx, T., Lee, M.D., & Wagenmakers, E.-J. (submitted for publication). A general computational method for estimating Bayes factors.
72. Lunn D., Spiegelhalter D., Thomas A., and Best N. The BUGS project: Evolution, critique and future directions. Statistics in Medicine 28 (2009) 3049-3067
73. Lunn D.J., Thomas A., Best N., and Spiegelhalter D. WinBUGS - a Bayesian modelling framework: Concepts, structure, and extensibility. Statistics and Computing 10 (2000) 325-337
74. MacKay D.J.C. Information theory, inference, and learning algorithms (2003), Cambridge University Press, Cambridge
75. Madigan D., and Raftery A.E. Model selection and accounting for model uncertainty in graphical models using Occam's window. Journal of the American Statistical Association 89 (1994) 1535-1546
76. Mulder, J., Klugkist, I., van de Schoot, R., Meeus, W.H.J., Selfhout, M., & Hoijtink, H. (in press). Bayesian model selection of informative hypotheses for repeated measurements. Journal of Mathematical Psychology.
77. Myung I.J. Tutorial on maximum likelihood estimation. Journal of Mathematical Psychology 47 (2003) 90-100
78. Myung I.J., Forster M.R., and Browne M.W. Model selection [Special issue]. Journal of Mathematical Psychology 44 1-2 (2000)
79. Myung I.J., and Pitt M.A. Applying Occam's razor in modeling cognition: A Bayesian approach. Psychonomic Bulletin & Review 4 (1997) 79-95
80. Nickerson R.S. Null hypothesis statistical testing: A review of an old and continuing controversy. Psychological Methods 5 (2000) 241-301
81. Ntzoufras I. Bayesian modeling using WinBUGS (2009), Wiley, Hoboken, NJ
82. O'Hagan A. Fractional Bayes factors for model comparison. Journal of the Royal Statistical Society B 57 (1995) 99-138
83. O'Hagan A., and Forster J. Kendall's advanced theory of statistics. Bayesian inference. 2nd ed. Vol. 2B (2004), Arnold, London
84. Ratcliff R., and McKoon G. A counter model for implicit priming in perceptual word identification. Psychological Review 104 (1997) 319-343
85. Rouder J.N., and Lu J. An introduction to Bayesian hierarchical models with an application in the theory of signal detection. Psychonomic Bulletin & Review 12 (2005) 573-604
86. Rouder J.N., Lu J., Morey R.D., Sun D., and Speckman P.L. A hierarchical process dissociation model. Journal of Experimental Psychology: General 137 (2008) 370-389
87. Rouder J.N., Lu J., Sun D., Speckman P., Morey R., and Naveh-Benjamin M. Signal detection models with random participant and item effects. Psychometrika 72 (2007) 621-642
88. Rouder J.N., Speckman P.L., Sun D., Morey R.D., and Iverson G. Bayesian t-tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review 16 (2009) 225-237
89. Royall R.M. Statistical evidence: A likelihood paradigm (1997), Chapman & Hall, London
90. Sellke T., Bayarri M.J., and Berger J.O. Calibration of p values for testing precise null hypotheses. The American Statistician 55 (2001) 62-71
91. Sheu C.-F., and O'Curry S.L. Simulation-based Bayesian inference using BUGS. Behavioral Research Methods, Instruments, & Computers 30 (1998) 232-237
92. Sisson S.A. Transdimensional Markov chains: A decade of progress and future perspectives. Journal of the American Statistical Association 100 (2005) 1077-1089
93. Smith A.F.M., and Spiegelhalter D.J. Bayes factors and choice criteria for linear models. Journal of the Royal Statistical Society B 42 (1980) 213-220
94. Spiegelhalter D.J. Bayesian graphical modelling: A case-study in monitoring health outcomes. Applied Statistics 47 (1998) 115-133
95. Stone C.J., Hansen M.H., Kooperberg C., and Truong Y.K. Polynomial splines and their tensor products in extended linear modeling (with discussion). The Annals of Statistics 25 (1997) 1371-1470
96. Vandekerckhove J., Tuerlinckx F., and Lee M.D. A Bayesian approach to diffusion process models of decision-making. In: Sloutsky V., Love B., and McRae K. (Eds). Proceedings of the 30th annual conference of the cognitive science society (2008), Cognitive Science Society 1429-1434
97. Verdinelli I., and Wasserman L. Computing Bayes factors using a generalization of the Savage-Dickey density ratio. Journal of the American Statistical Association 90 (1995) 614-618
98. Wagenmakers E.-J. A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review 14 (2007) 779-804
99. Wagenmakers E.-J. Methodological and empirical developments for the Ratcliff diffusion model of response times and accuracy. European Journal of Cognitive Psychology 21 (2009) 641-671
100. Wagenmakers E.-J., Lee M.D., Lodewyckx T., and Iverson G. Bayesian versus frequentist inference. In: Hoijtink H., Klugkist I., and Boelen P.A. (Eds). Bayesian evaluation of informative hypotheses (2008), Springer Verlag, New York 181-207
101. Wagenmakers E.-J., Ratcliff R., Gomez P., and Iverson G.J. Assessing model mimicry using the parametric bootstrap. Journal of Mathematical Psychology 48 (2004) 28-50
102. Wagenmakers E.-J., and Waldorp L. Model selection: Theoretical developments and applications [Special issue]. Journal of Mathematical Psychology 50 2 (2006)
103. Wagenmakers E.-J., Zeelenberg R., and Raaijmakers J.G.W. Testing the counter model for perceptual identification: Effects of repetition priming and word frequency. Psychonomic Bulletin & Review 7 (2003) 662-667
104. Weaver R. Parameters, predictions, and evidence in computational modeling: A statistical view informed by ACT-R. Cognitive Science 32 (2008) 1349-1375
105. Wetzels, R., Grasman, R.P.P.P., & Wagenmakers, E.-J. (submitted for publication). An encompassing prior generalization of the Savage-Dickey density ratio test.
106. Wetzels, R., Vandekerckhove, J., Tuerlinckx, F., & Wagenmakers, E.-J. (in press). Bayesian parameter estimation in the Expectancy Valence model of the Iowa gambling task. Journal of Mathematical Psychology.
107. Wetzels R., Raaijmakers J.G.W., Jakab E., and Wagenmakers E.-J. How to quantify support for and against the null hypothesis: A flexible WinBUGS implementation of a default Bayesian t-test. Psychonomic Bulletin & Review 16 (2009) 752-760
108. Wilkinson L., and The Task Force on Statistical Inference. Statistical methods in psychology journals: Guidelines and explanations. American Psychologist 54 (1999) 594-604
109. Zeelenberg R., Wagenmakers E.-J., and Raaijmakers J.G.W. Priming in implicit memory tasks: Prior study causes enhanced discriminability, not only bias. Journal of Experimental Psychology: General 131 (2002) 38-47