Bayesian leave-one-out cross-validation approximations for Gaussian latent variable models

Journal of Machine Learning Research

Volumn 17, Issue , 2016, Pages 1-38

  Vehtari, Aki ^a   Mononen, Tommi ^a   Tolvanen, Ville ^a   Sivula, Tuomas ^a   Winther, Ole ^b  

^a AALTO UNIVERSITY   (Finland)

^b TECHNICAL UNIVERSITY OF DENMARK   (Denmark)

Author keywords

Expectation propagation; Gaussian latent variable model; Laplace approximation; Leave one out cross validation; Predictive performance

Indexed keywords

BAYESIAN NETWORKS; CASE BASED REASONING; GAUSSIAN DISTRIBUTION; INTELLIGENT SYSTEMS; LAPLACE TRANSFORMS; MARKOV PROCESSES; STATISTICAL METHODS;

EXPECTATION PROPAGATION; LAPLACE APPROXIMATION; LATENT VARIABLE MODELING; LEAVE-ONE-OUT CROSS VALIDATIONS; PREDICTIVE PERFORMANCE;

MONTE CARLO METHODS;

EID: 84988932233     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (57)

1. Mark Yuying An. Logconcavity versus logconvexity: a complete characterization. Journal of Economic Theory, 80(2):350-369, 1998.
2. José M. Bernardo and Adrian F. M. Smith. Bayesian Theory. John Wiley & Sons, 1994.
3. Edward Challis and David Barber. Gaussian Kullback-Leibler approximate inference. Journal of Machine Learning Research, 14:2239-2286, 2013.
4. Botond Cseke and Tom Heskes. Approximate marginals in latent Gaussian models. Journal of Machine Learning Research, 12:417-454, 2 2011. ISSN 1532-4435.
5. Ilenia Epifani, Steven N MacEachern, and Mario Peruggia. Case-deletion importance sampling estimators: Central limit theorems and related results. Electronic Journal of Statistics, 2:774-806, 2008.
6. Seymour Geisser and William F. Eddy. A predictive approach to model selection. Journal of the American Statistical Association, 74(365):153-160, March 1979.
7. Alan E. Gelfand. Model determination using sampling-based methods. In W. R. Gilks, S. Richardson, and D. J. Spiegelhalter, editors, Markov Chain Monte Carlo in Practice, pages 145-162. Chapman & Hall, 1996.
8. Alan E. Gelfand, D. K. Dey, and H. Chang. Model determination using predictive distributions with implementation via sampling-based methods (with discussion). In J. M. Bernardo, J. O. Berger, A. P. Dawid, and A. F. M. Smith, editors, Bayesian Statistics 4, pages 147-167. Oxford University Press, 1992.
9. Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. Bayesian Data Analysis. Chapman & Hall/CRC, third edition, 2013.
10. Andrew Gelman, Jessica Hwang, and Aki Vehtari. Understanding predictive information criteria for Bayesian models. Statistics and Computing, 24(6):997-1016, 2014.
11. Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of American Statistical Association, 102:359-379, 2007.
12. Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(2):243-268, 2007.
13. Leonhard Held, Birgit Schrödle, and Håvard Rue. Posterior and cross-validatory predictive checks: A comparison of MCMC and INLA. In Thomas Kneib and Gerhard Tutz, editors, Statistical Modelling and Regression Structures, pages 91-110. Springer, 2010.
14. Daniel Hernáandez-Lobato, José M. Hernández-Lobato, and Alberto Suárez. Expectation propagation for microarray data classification. Pattern Recognition Letters, 31(12):1618-1626, 2010.
15. José M. Hernández-Lobato, Tjeerd Dijkstra, and Tom Heskes. Regulator discovery from gene expression time series of malaria parasites: a hierarchical approach. In J.C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 649-656, 2008.
16. Edward L. Ionides. Truncated importance sampling. Journal of Computational and Graphical Statistics, 17(2):295-311, 2008.
17. Pasi Jylänki, Jarno Vanhatalo, and Aki Vehtari. Robust Gaussian process regression with a Student-t likelihood. Journal of Machine Learning Research, 12:3227-3257, 2011.
18. Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. The International Conference on Learning Representations (ICLR), 2014.
19. Augustine Kong, Jun S. Liu, and Wing Hung Wong. Sequential imputations and Bayesian missing data problems. Journal of the American Statistical Association, 89(425):278-288, 1994.
20. Siem Jan Koopman, Neil Shephard, and Drew Creal. Testing the assumptions behind importance sampling. Journal of Econometrics, 149(1):2-11, 2009.
21. Longhai Li, Shi Qiu, Bei Zhang, and Cindy X. Feng. Approximating cross-validatory predictive evaluation in Bayesian latent variables models with integrated IS and WAIC. arXiv preprint arXiv:1404.2918, 2014.
22. Finn Lindgren and Håvard Rue. Bayesian spatial modelling with R-INLA. Journal of Statistical Software, 63(19), 2015.
23. E. C. Marshall and D. J. Spiegelhalter. Approximate cross-validatory predictive checks in disease mapping models. Statistics in Medicine, 22(10):1649-1660, 2003.
24. Thiago G Martins, Daniel Simpson, Finn Lindgren, and Håvard Rue. Bayesian computing with INLA: new features. Computational Statistics & Data Analysis, 67:68-83, 2013.
25. Thomas Minka. A Family of Algorithms for Approximate Bayesian Inference. PhD thesis, Massachusetts Institute of Technology, 2001.
26. Iain Murray and Ryan Prescott Adams. Slice sampling covariance hyperparameters of latent Gaussian models. In J. Lafferty, C. K. I. Williams, R. Zemel, J. Shawe-Taylor, and A. Culotta, editors, Advances in Neural Information Processing Systems 23, pages 1732-1740, 2010.
27. Iain Murray, Ryan Prescott Adams, and David J.C. MacKay. Elliptical slice sampling. JMLR Workshop and Conference Proceedings: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 9:541-548, 2010.
28. Hannes Nickisch and Carl Edward Rasmussen. Approximations for binary Gaussian process classification. Journal of Machine Learning Research, 9:2035-2078, October 2008.
29. Manfred Opper and Ole Winther. Gaussian processes for classification: Mean-field algorithms. Neural Computation, 12(11):2655-2684, 2000.
30. Mario Peruggia. On the variability of case-deletion importance sampling weights in the Bayesian linear model. Journal of the American Statistical Association, 92(437):199-207, 1997.
31. Juho Piironen and Aki Vehtari. Comparison of Bayesian predictive methods for model selection. Statistics and Computing, 2016. Online first: DOI:10.1007/s11222-016-9649-y.
32. Joaquin Quiñonero-Candela and Carl Edward Rasmussen. A unifying view of sparse approximate Gaussian process regression. Journal of Machine Learning Research, 6(3):1939-1959, 2005.
33. Rajesh Ranganath, Sean Gerrish, and David M Blei. Black box variational inference. JMLR Workshop and Conference Proceedings: Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, 33:814-822, 2014.
34. Carl Edward Rasmussen and Hannes Nickisch. Gaussian processes for machine learning (GPML) toolbox. The Journal of Machine Learning Research, 11:3011-3015, 2010.
35. Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006.
36. Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. JMLR Workshop and Conference Proceedings: Proceedings of The 31st International Conference on Machine Learning, 32: 1278-1286, 2014.
37. Jaakko Riihimäki and Aki Vehtari. Laplace approximation for logistic Gaussian process density estimation and regression. Bayesian analysis, 9(2):425-448, 2014.
38. Jaakko Riihimäki, Pasi Jylänki, and Aki Vehtari. Nested expectation propagation for Gaussian process classification with a multinomial probit likelihood. Journal of Machine Learning Research, 14:75-109, 2013.
39. Håvard Rue, Sara Martino, and Nicolas Chopin. Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations. Journal of the Royal statistical Society B, 71(2):1-35, 2009.
40. Matthias Seeger. Bayesian inference and optimal design for the sparse linear model. Journal of Machine Learning Research, 9:759-813, 2008.
41. S. Sundararajan and S. S. Keerthi. Predictive approaches for choosing hyperparameters in Gaussian processes. Neural Computation, 13(5):1103-1118, May 2001.
42. Luke Tierney and Joseph B. Kadane. Accurate approximations for posterior moments and marginal densities. Journal of the American Statistical Association, 81(393):82-86, 1986.
43. Ville Tolvanen, Pasi Jylänki, and Aki Vehtari. Expectation propagation for nonstationary heteroscedastic Gaussian process regression. In 2014 IEEE International Workshop on Machine Learning for Signal Processing (MLSP), 2014. DOI:10.1109/MLSP.2014.6958906.
44. Marcel van Gerven, Botond Cseke, Robert Oostenveld, and Tom Heskes. Bayesian source localization with the multivariate Laplace prior. In Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems, pages 1901-1909, 2009.
45. Jarno Vanhatalo and Aki Vehtari. Modelling local and global phenomena with sparse Gaussian processes. In David A. McAllester and Petri Myllymäki, editors, Proceedings of the 24th Conference on Uncertainty in Artificial Intelligence, pages 571-578, 2008.
46. Jarno Vanhatalo and Aki Vehtari. Speeding up the binary Gaussian process classification. In Peter Grünwald and Peter Spirtes, editors, Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence, pages 1-9, 2010.
47. Jarno Vanhatalo, Pasi Jylänki, and Aki Vehtari. Gaussian process regression with Student-t likelihood. In Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems 22, pages 1910-1918, 2009.
48. Jarno Vanhatalo, Ville Pietiläinen, and Aki Vehtari. Approximate inference for disease mapping with sparse Gaussian processes. Statistics in Medicine, 29(15):1580-1607, 2010.
49. Jarno Vanhatalo, Jaakko Riihimäki, Jouni Hartikainen, Pasi Jylänki, Ville Tolvanen, and Aki Vehtari. GPstuff: Bayesian modeling with Gaussian processes. Journal of Machine Learning Research, 14:1175-1179, 2013.
50. Aki Vehtari. Bayesian Model Assessment and Selection Using Expected Utilities. PhD thesis, Helsinki University of Technology, 2001.
51. Aki Vehtari and Andrew Gelman. Pareto smoothed importance sampling. arXiv preprint arXiv:1507.02646, 2015.
52. Aki Vehtari and Jouko Lampinen. Bayesian model assessment and comparison using cross-validation predictive densities. Neural Computation, 14(10):2439-2468, 2002.
53. Aki Vehtari and Janne Ojanen. A survey of Bayesian predictive methods for model assessment, selection and comparison. Statistics Surveys, 6:142-228, 2012.
54. Aki Vehtari, Andrew Gelman, and Jonah Gabry. Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. arXiv preprint arXiv:1507.04544, 2016.
55. Sumio Watanabe. Equations of states in singular statistical estimation. Neural Networks, 23 (1):20-34, 2010a.
56. Sumio Watanabe. Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory. Journal of Machine Learning Research, 11:3571-3594, 2010b.
57. Jin Zhang and Michael A Stephens. A new and efficient estimation method for the generalized Pareto distribution. Technometrics, 51(3):316-325, 2009.