Recursive pathways to marginal likelihood estimation with prior-sensitivity analysis

Statistical Science

Volumn 29, Issue 3, 2014, Pages 397-419

  Cameron, Ewan ^a   Pettitt, Anthony ^a  

^a QUEENSLAND UNIVERSITY OF TECHNOLOGY   (Australia)

Author keywords

Bayes factor; Bayesian model selection; Importance sampling; Marginal likelihood; Metropolis coupled Markov chain Monte Carlo; Nested sampling; Normalizing constant; Path sampling; Reverse logistic regression; Thermodynamic integration

Indexed keywords

EID: 84907204566     PISSN: 08834237     EISSN: None     Source Type: Journal    
DOI: 10.1214/13-STS465     Document Type: Article

References (89)

1. AITKIN, M. (2001). Likelihood and Bayesian analysis of mixtures. Statist. Model. 1 287-304.
2. ANTONIAK, C. E. (1974). Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. Ann. Statist. 2 1152-1174.
3. ARIMA, S. and TARDELLA, L. (2012). Improved harmonic mean estimator for phylogenetic model evidence. J. Comput. Biol. 19 418-438.
4. BAELE, G., LEMEY, P., BEDFORD, T., RAMBAUT, A., SUCHARD, M. A. and ALEKSEYENKO, A. V. (2012). Improving the accuracy of demographic and molecular clock model comparison while accommodating phylogenetic uncertainty. Mol. Biol. Evol. 29 2157-2167.
5. BAILER-JONES, C. A. L. (2012). A Bayesian method for the analysis of deterministic and stochastic time series. Astron. Astrophys. 546 A89.
6. BILLINGSLEY, P. (1968). Convergence of Probability Measures. Wiley, New York.
7. BREWER, B. J. and STELLO, D. (2009). Gaussian process modelling of asteroseismic data. Mon. Not. R. Astron. Soc. 395 2226-2233.
8. CAIMO, A. and FRIEL, N. (2013). Bayesian model selection for exponential random graph models. Social Networks 35 11-24.
9. CALDERHEAD, B. and GIROLAMI, M. (2009). Estimating Bayes factors via thermodynamic integration and population MCMC. Comput. Statist. Data Anal. 53 4028-4045.
10. CAMERON, E. and PETTITT, A. N. (2013). On the evidence for cosmic variation of the fine structure constant (II): A semiparametric Bayesian model selection analysis of the quasar dataset. Preprint. Available at arXiv:1309.2737.
11. CAPPe´, O., GUILLIN, A., MARIN, J. M. and ROBERT, C. P. (2004). Population Monte Carlo. J. Comput. Graph. Statist. 13 907-929.
12. CHEN, M.-H. and SHAO, Q.-M. (1997). OnMonte Carlo methods for estimating ratios of normalizing constants. Ann. Statist. 25 1563-1594.
13. CHEN, M.-H., SHAO, Q.-M. and IBRAHIM, J. G. (2000). Monte Carlo Methods in Bayesian Computation. Springer, New York.
14. CHIB, S. (1995). Marginal likelihood from the Gibbs output. J. Amer. Statist. Assoc. 90 1313-1321.
15. CHOPIN, N. (2002). A sequential particle filter method for static models. Biometrika 89 539-551.
16. CHOPIN, N. and ROBERT, C. P. (2010). Properties of nested sampling. Biometrika 97 741-755.
17. CORNUET, J.-M., MARIN, J.-M., MIRA, A. and ROBERT, C. P. (2012). Adaptive multiple importance sampling. Scand. J. Stat. 39 798-812.
18. DAVIS, T. M. et al. (2007). Scrutinizing exotic cosmological models using ESSENCE supernova data combined with other cosmological probes. Astrophys. J. 666 716-725.
19. DEL MORAL, P., DOUCET, A. and JASRA, A. (2006). Sequential Monte Carlo samplers. J. R. Stat. Soc. Ser. B Stat. Methodol. 68 411-436.
20. DIEBOLT, J. and ROBERT, C. P. (1994). Estimation of finite mixture distributions through Bayesian sampling. J. Roy. Statist. Soc. Ser. B 56 363-375.
21. DOSS, H. (2012). Hyperparameter and model selection for nonparametric Bayes problems via Radon-Nikodym derivatives. Statist. Sinica 22 1-26.
22. DUDLEY, R. M. and PHILIPP, W. (1983). Invariance principles for sums of Banach space valued random elements and empirical processes. Z. Wahrsch. Verw. Gebiete 62 509-552.
23. ESCOBAR, M. D. and WEST, M. (1995). Bayesian density estimation and inference using mixtures. J. Amer. Statist. Assoc. 90 577-588.
24. EVANS, M., ROBERT, C. P., DAVISON, A. C., JIANG, W., TANNER, M. A., DOSS, H., QIN, J., FOKIANOS, K., MACEACHERN, S. N., PERUGGIA, M., GUHA, S., CHIB, S., RITOV, Y., ROBINS, J. M. and VARDI, Y. (2003). Discussion on the paper by Kong, McCullagh, Meng, Nicolas and Tan. J. Roy. Statist. Soc. B 65 604-618.
25. FAN, Y., RUI, W., CHEN, M.-H., KUO, L. and LEWIS, P. O. (2012). Choosing among partition models in Bayesian phylogenetics. Mol. Biol. Evol. 28 523-532.
26. FEROZ, F. and HOBSON, M. P. (2008). Multimodal nested sampling: An efficient and robust alternative to Markov Chain Monte Carlo methods for astronomical data analyses. Mon. Not. R. Astron. Soc. 384 449-463.
27. FEROZ, F., HOBSON, M. P., CAMERON, E. and PETTITT, A. N. (2013). Importance nested sampling and the multinest algorithm. Preprint. Available at arXiv:1306.2144.
28. FERRENBERG, A. M. and SWENDSEN, R. H. (1989). Optimized Monte Carlo data analysis. Phys. Rev. Lett. 63 1195-1198.
29. FRIEL, N. and PETTITT, A. N. (2008).Marginal likelihood estimation via power posteriors. J. R. Stat. Soc. Ser. B Stat. Methodol. 70 589-607.
30. FRIEL, N. and WYSE, J. (2012). Estimating the evidence-A review. Stat. Neerl. 66 288-308.
31. GELFAND, A. E. and DEY, D. K. (1994). Bayesian model choice: Asymptotics and exact calculations. J. Roy. Statist. Soc. Ser. B 56 501-514.
32. GELMAN, A. and MENG, X.-L. (1998). Simulating normalizing constants: From importance sampling to bridge sampling to path sampling. Statist. Sci. 13 163-185.
33. GELMAN, A., CARLIN, J. B., STERN, H. S. and RUBIN, D. B. (2004). Bayesian Data Analysis, 2nd ed. Chapman & Hall/CRC, Boca Raton, FL.
34. GEYER, C. J. (1992). Practical Markov chain Monte Carlo. Statist. Sci. 7 473-483.
35. GEYER, C. J. (1994). Estimating normalizing constants and reweighting mixtures in Markov chain Monte Carlo. Technical Report 568, School of Statistics, Univ.Minnesota,Minneapolis, MN.
36. GEYER, C. J. and THOMPSON, E. A. (1992). Constrained Monte Carlo maximum likelihood for dependent data. J. Roy. Statist. Soc. Ser. B 54 657-699.
37. GILL, R. D., VARDI, Y. andWELLNER, J. A. (1988). Large sample theory of empirical distributions in biased sampling models. Ann. Statist. 16 1069-1112.
38. GRu¨N, B. and LEISCH, F. (2010). BayesMix: An R package for Bayesian mixture modeling. Technical report.
39. GUNN, J. E. and GOTT, J. R. III (1972). On the infall of matter into clusters of galaxies and some effects on their evolution. Astrophys. J. 176 1-19.
40. HABECK, M. (2012). Evaluation of marginal likelihoods via the density of states. J. Mach. Learn. Res. 22 486-494.
41. HALMOS, P. R. (1950). Measure Theory. Van Nostrand, New York.
42. HALMOS, P. R. and SAVAGE, L. J. (1949). Application of the Radon-Nikodym theorem to the theory of sufficient statistics. Ann. Math. Statist. 20 225-241.
43. HESTERBERG, T. (1995). Weighted average importance sampling and defensive mixture distributions. Technometrics 37 185-194.
44. HOETING, J. A., MADIGAN, D., RAFTERY, A. E. and VOLINSKY, C. T. (1999). Bayesian model averaging: A tutorial. Statist. Sci. 14 382-417.
45. Ho¨RMANDER, L. (1983). The Analysis of Linear Partial Differential Operators. I: Distribution Theory and Fourier Analysis. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences] 256. Springer, Berlin.
46. JARA, A., HANSON, T. E., QUINTANA, F. A., Mu¨LLER, P. and ROSNER, G. L. (2011). DPpackage: Bayesian semi- and nonparametric modelling in R. J. Statist. Softw. 40 1-30.
47. JASRA, A., HOLMES, C. C. and STEPHENS, D. A. (2005). Markov chain Monte Carlo methods and the label switching problem in Bayesian mixture modeling. Statist. Sci. 20 50-67.
48. JAYNES, E. T. (2003). Probability Theory: The Logic of Science. Cambridge Univ. Press, Cambridge.
49. JEFFREYS, H. (1961). Theory of Probability, 3rd ed. Clarendon Press, Oxford.
50. JEFFREYS, W. H. and BERGER, J. O. (1991). Sharpening Ockham's razor on a Bayesian strop. Technical Report 91-44C, Dept. Statistics, Purdue Univ., West Lafayette, IN.
51. KASS, R. E. and RAFTERY, A. E. (1995). Bayes factors. J. Amer. Statist. Assoc. 90 773-795.
52. KILBINGER, M., WRAITH, D., ROBERT, C. P., BENABED, K., CAPPe´, O., CARDOSO, J.-F., FORT, G., PRUNET, S. and BOUCHET, F. R. (2010). Bayesian model comparison in cosmology with populationMonte Carlo. Mon. Not. R. Astron. Soc. 405 2381-2390.
53. KIRSHNER, R. P., OEMLER, A. JR., SCHECHTER, P. L. and SHECTMAN, S. A. (1981). A million cubic megaparsec void in Boo¨tes? Astrophys. J. 248 57-60.
54. KONG, A., LIU, J. S. and WONG, W. H. (1994). Sequential imputations and Bayesian missing data problems. J. Amer. Statist. Assoc. 89 278-288.
55. KONG, A., MCCULLAGH, P., MENG, X.-L., NICOLAE, D. and TAN, Z. (2003). A theory of statistical models for Monte Carlo integration. J. R. Stat. Soc. Ser. B Stat. Methodol. 65 585-618.
56. KUMAR, S., ROSENBERG, J. M., BOUZIDA, D., SWENDSEN, R. H. and KOLLMAN, P. A. (1992). The weighted histogram analysis method for free-energy calculations on biomolecules. I. The method. J. Comput. Chem. 13 1011-1021.
57. LARTILLOT, N. and PHILLIPE, H. (2006). Computing Bayes factors using thermodynamic integration. Syst. Biol. 55 195-207.
58. LEE, K., MARIN, J.-M., MENGERSEN, K. and ROBERT, C. P. (2008). Bayesian inference on mixtures of distributions. Preprint. Available at arXiv:0804.2413.
59. LEFEBVRE, G., STEELE, R. and VANDAL, A. C. (2010). A path sampling identity for computing the Kullback-Leibler and J divergences. Comput. Statist. Data Anal. 54 1719-1731.
60. LI, Y., NI, Z.-X. and LIN, J.-G. (2011). A stochastic simulation approach to model selection for stochastic volatility models. Comm. Statist. Simulation Comput. 40 1043-1056.
61. LIU, J. S. (2001). Monte Carlo Strategies in Scientific Computing. Springer, New York.
62. MARIN, J.-M., PUDLO, P. and SEDKI, M. (2012). Consistency of the adaptive multiple importance sampling. Preprint. Available at arXiv:1301.2548.
63. MARIN, J.-M. and ROBERT, C. P. (2010). On resolving the Savage-Dickey paradox. Electron. J. Stat. 4 643-654.
64. MENG, X.-L. andWONG, W. H. (1996). Simulating ratios of normalizing constants via a simple identity: A theoretical exploration. Statist. Sinica 6 831-860.
65. MILLER, J. W. and HARRISON, M. T. (2013). A simple example of Dirichlet process mixture inconsistency for the number of components. Preprint. Available at arXiv:1301.2708v1.
66. MUKHERJEE, P., PARKINSON, D. and LIDDLE, A. R. (2006). A nested sampling algorithm for cosmological model selection. Astrophys. J. 638 L51-L54.
67. NEAL, R. (1999). Erroneous results in "Marginal likelihood from the Gibbs output." Available at http://www.cs.toronto.edu/~radford/ftp/chib-letter.pdf
68. NEAL, R. M. (2000). Markov chain sampling methods for Dirichlet process mixture models. J. Comput. Graph. Statist. 9 249- 265.
69. NEAL, R. M. (2001). Annealed importance sampling. Stat. Comput. 11 125-139.
70. NEWTON, M. A. and RAFTERY, A. E. (1994). Approximate Bayesian inference with the weighted likelihood bootstrap. J. Roy. Statist. Soc. Ser. B 56 3-48.
71. ORTEGA, J. M. and RHEINBOLDT, W. C. (1967). Monotone iterations for nonlinear equations with application to Gauss-Seidel methods. SIAM J. Numer. Anal. 4 171-190.
72. PFANZAGL, J. (1979). Conditional distributions as derivatives. Ann. Probab. 7 1046-1050.
73. PHILLIPS, D. B. and SMITH, A. F. M. (1996). Bayesian model comparison via jump diffusions. In Markov Chain Monte Carlo in Practice 215-239. Chapman & Hall, London.
74. POSTMAN, M.,HUCHRA, J. P. andGELLER, M. J. (1986). Probes of large-scale structure in the Corona Borealis region. Astrophys. J. 92 1238-1247.
75. RAFTERY, A. E., NEWTON, M. A., SATAGOPAN, J. M. and KRIVITSKY, P. N. (2007). Estimating the integrated likelihood via posterior simulation using the harmonic mean identity. In Bayesian Statistics 8 371-416. Oxford Univ. Press, Oxford.
76. RICHARDSON, S. and GREEN, P. J. (1997). On Bayesian analysis of mixtures with an unknown number of components. J. Roy. Statist. Soc. Ser. B 59 731-792.
77. ROBERT, C. P. andWRAITH, D. (2009). Computational methods for Bayesian model choice. In Bayesian Inference and Maximum Entropy Methods in Science and Engineering: The 29th International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering. AIP Conference Proceedings 1193 251-262. American Institute of Physics, New York.
78. ROEDER, K. (1990). Density estimation with confidence sets exemplified by superclusters and voids in the galaxies. J. Amer. Statist. Assoc. 85 617-624.
79. ROEDER, K. and WASSERMAN, L. (1997). Practical Bayesian density estimation using mixtures of normals. J. Amer. Statist. Assoc. 92 894-902.
80. SCHECHTER, P. (1976). An analytic expression for the luminosity function of galaxies. Astrophys. J. 203 297-306.
81. SHIRTS, M. R. and CHODERA, J. D. (2008). Statistically optimal analysis of samples from multiple equilibrium states. J. Chem. Phys. 129 124105.
82. SKILLING, J. (2006). Nested sampling for general Bayesian computation. Bayesian Anal. 1 833-859 (electronic).
83. STEPHENS, M. (2000). Bayesian analysis of mixture models with an unknown number of components-An alternative to reversible jump methods. Ann. Statist. 28 40-74.
84. TAN, Z., GALLICCHIO, E., LAPELOSA, M. and LEVY, R. M. (2012). Theory of binless multi-state free energy estimation with applications to protein-ligand binding. J. Chem. Phys. 136 144102.
85. TIERNEY, L. (1994). Markov chains for exploring posterior distributions. Ann. Statist. 22 1701-1762.
86. VARDI, Y. (1985). Empirical distributions in selection bias models. Ann. Statist. 13 178-205.
87. WEINBERG, M. D. (2012). Computing the Bayes factor from a Markov chain Monte Carlo simulation of the posterior distribution. Bayesian Anal. 7 737-769.
88. WOLPERT, R. L. and SCHMIDLER, S. C. (2012). a-stable limit laws for harmonic mean estimators of marginal likelihoods. Statist. Sinica 22 1233-1251.
89. XIE, W., LEWIS, P., FAN, Y., KUO, L. and CHEN, M.-H. (2011). Improving marginal likelihood estimation for Bayesian phylogenetic model selection. Syst. Biol. 18 1001-1013.