Mean field variational bayes for continuous sparse signal shrinkage: Pitfalls and remedies

Electronic Journal of Statistics

Volumn 8, Issue , 2014, Pages 1113-1151

  Neville, Sarah E ^a   Ormerod, John T ^b   Wand, M P ^c  

^a UNIVERSITY OF WOLLONGONG   (Australia)

^b UNIVERSITY OF SYDNEY   (Australia)

^c UNIVERSITY OF TECHNOLOGY SYDNEY   (Australia)

Author keywords

Approximate bayesian inference; Continued fraction; Generalized double pareto distribution; Horseshoe distribution; Lentz's algorithm; Normal Exponential Gamma distribution; Special function

Indexed keywords

EID: 84938886140     PISSN: 19357524     EISSN: None     Source Type: Journal    
DOI: 10.1214/14-EJS910     Document Type: Article

References (37)

1. Abramowitz, M. and Stegun, I.A. (Eds.) (1972). Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables. New York: Dover Publications.
2. Archambeau, C. and Bach, F. (2008). Sparse probabilistic projections. 21st Annual Conference on Neural Information Processing Systems, Vancouver, Canada, December 8–11.
3. Armagan, A. (2009). Variational bridge regression. Journal of Machine Learning Research, Workshop and Conference Proceedings, 5, 17–24.
4. Armagan, A., Dunson, D.B. and Clyde, M. (2011). Generalized beta mixtures of Gaussians. In Advances in Neural Information Processing Systems 24, J. Shawe-Taylor, R.S. Zamel, P. Bartlett, F. Pereira and K.Q. Weinberger (Eds.), 523–531.
5. Armagan, A., Dunson, D.B. and Lee, J. (2013). Generalized double Pareto shrinkage. Statistica Sinica, 23, 119–143. MR3076161
6. Attias, H. (1999). Inferring parameters and structure of latent variable models by variational Bayes. Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence, 21–30.
7. BISHOP, C.M. (2006). Pattern Recognition and Machine Learning. New York: Springer. MR2247587
8. Carbonetto, P. and Stephens, M. (2011). Scalable variational inference for Bayesian variable selection in regression, and its accuracy in genetic association studies. Bayesian Analysis, 6(4), 1–42. MR2896713
9. Carvalho, C.M., Polson, N.G. and Scott, J.G. (2010). The horseshoe estimator for sparse signals. Biometrika, 97, 465–480. MR2650751
10. Consonni, G. and Marin, J.M. (2007). Mean-field variational approximate Bayesian inference for latent variable models. Computational Statistics and Data Analysis, 52, 790–798. MR2418528
11. Cuyt, A., Petersen, V.B., Verdonk, B., Waadeland, H. and Jones, W.B. (2008). Handbook of Continued Fractions for Special Functions. New York: Springer. MR2410517
12. Flandin, G. and Penny, W.D. (2007). Bayesian fMRI data analysis with sparse spatial basis function priors. NeuroImage, 34, 1108–1125.
13. Galassi, M., Davies, J., Theiler, J., Gough, B., Jungman, G., Alken, P., Booth, M. and Rossi, F. (2009). GNU Scientific Library Reference Manual, 3rd Edition, Version 1.12, Bristol UK: Network Theory.
14. GRADSHTEYN, I.S. and Ryzhik, I.M. (1994). Tables of Integrals, Series, and Products, 5th Edition, San Diego, California: Academic Press. MR1243179
15. Griffin, J.E. and Brown, P.J. (2011). Bayesian hyper-lassos with non-convex penalization. Australian and New Zealand Journal of Statistics, 53, 423–442. MR2910027
16. Hankin, R.K.S. (2007). gsl 1.9. Wrapper for the Gnu Scientific Library. R package,http://cran.r-project.org.
17. Johnstone, I.M. and Silverman, B.W. (2004). Needles and straw in haystacks: empirical Bayes estimates of possibly sparse sequences. The Annals of Statistics, 32, 1594–1649. MR2089135
18. Johnstone, I.M. and Silverman, B.W. (2005). Bayes selection of wavelet thresholds. The Annals of Statistics, 33, 1700–1752. MR2166560
19. Lentz, W.J. (1976). Generating Bessel lunctions in Mie scattering calculations using continued fractions. Applied Optics, 3, 668–671.
20. Ligges, U., Thomas, A., Spiegelhalter, D., Best, N., Lunn, D., Rice, K. and Sturtz, S. (2011). BRugs 0.5: OpenBUGS and its R/S-PLUS interlace BRugs. R package,http://www.stats.ox.ac.uk/ pub/RWin/bin/windows/contrib/2.13.
21. Logsdon, B.A., Hoffman, G.E. and Mezey, J.G. (2010). A variational Bayes algorithm lor last and accurate multiple locus genome-wide association analysis. BMC Bioinformatics, 11:58, 1–13.
22. Lunn, D.J., Thomas, A., Best, N. and Spiegelhalter, D. (2000). WinBUGS – a Bayesian modelling framework: concepts, structure, and extensibility. Statistics and Computing, 10, 325–337.
23. McGrory, C.A. and Titterington, D.M. (2007). Variational approximations in Bayesian model selection lor finite mixture distributions. Computational Statistics and Data Analysis, 51, 5352–5367. MR2370876
24. NEVILLE, S.E. (2013). Elaborate Distribution Semiparametric Regression via Mean Field Variational Bayes. PhD Thesis, University of Wollon-gong.
25. Ormerod, J.T. and Wand, M.P. (2010). Explaining variational approximations. The American Statistician, 64, 140–153. MR2757005
26. Polson, N.G. and Scott, J.G. (2010). Shrink globally, act locally: sparse Bayesian regularization and prediction. In Bayesian Statistics 9, J.M. Bernardo, M.J. Bayarri, J.O. Berger, A.P. Dawid, D. Hecker-man, A.F.M. Smith and M. West (Eds.). Oxford: Oxford University Press.
27. Press, W., Teukolosky, S., Vetterling, W. and Flannery, B. (1992). Numerical Recipes: The Art of Scientific Computing, 2nd Edition. New York: Cambridge University Press. MR2371990
28. R Development Core Team (2014). R: A language and environment lor statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, http://www.R-project.org
29. Teschendorff, A.E., Wang, Y., Barbosa-Morais, N.L., Bren-TON, J.D. and Caldas C. (2005). A variational Bayesian mixture modelling framework lor cluster analysis of gene-expression data. Bioinformatics, 21, 3025–3033.
30. Tipping, M.E. and Lawrence, N.D. (2003). A variational approach to robust Bayesian interpolation. IEEE Workshop on Neural Networks for Signal Processing, 229–238.
31. Wainwright, M.J. and Jordan, M.I. (2008). Graphical models, exponential families, and variational inference. Foundation and Trends in Machine Learning, 1, 1–305.
32. Wand, M.P. and Ormerod, J.T. (2011). Penalized wavelets: embedding wavelets into semiparametric regression. Electronic Journal of Statistics, 5. 1654–1717. MR2870147
33. Wand, M.P. and Ormerod, J.T. (2012). Continued fraction enhancement of Bayesian computing. Stat., 1, 31–41.
34. Wand, M.P., Ormerod, J.T., Padoan, S.A. and Frühwirth, R. (2011). Mean field variational Bayes for elaborate distributions. Bayesian Analysis, 6, 847–900. MR2869967
35. Wand, M.P. and Ripley, B.D. (2010). KernSmooth 2.23. Functions for kernel smoothing corresponding to the book: Wand, M.P. and Jones, M.C. (1995) "Kernel Smoothing". R package,http://cran.r-project.org./
36. Whittaker, E T. and WATSON, G.N. (1990). A Course in Modern Analysis, 4th Edition, Cambridge UK: Cambridge University Press.
37. WUERTZ, D. and others. (2009). fAsianOptions 2100.76. Exponential Brow-nian motion and Asian option evaluation. R package, http://cran, r-project.org