Expectation propagation in linear regression models with spike-and-slab priors

Machine Learning

Volumn 99, Issue 3, 2015, Pages 437-487

  Hernández Lobato, José Miguel ^a   Hernández Lobato, Daniel ^b   Suárez, Alberto ^b  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^b UNIVERSIDAD AUTÓNOMA DE MADRID   (Spain)

Author keywords

Expectation propagation; Linear regression; Selective shrinkage; Spike and slab

Indexed keywords

ALGORITHMS; INFERENCE ENGINES; LAPLACE TRANSFORMS; MATHEMATICAL MODELS; SHRINKAGE;

APPROXIMATE INFERENCE; EXPECTATION PROPAGATION; GENETIC NETWORKS; LINEAR REGRESSION MODELS; PRIOR DISTRIBUTION; SHRINKAGE CAPACITIES; SPIKE-AND-SLAB; VARIATIONAL BAYES;

LINEAR REGRESSION;

EID: 84939976814     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-014-5475-7     Document Type: Article

References (58)

1. Alon, U. (2006). An introduction to systems biology. Boca Raton: CRC Press.
2. Attias, H. (1999). Inferring parameters and structure of latent variable models by variational Bayes. In K. B. Laskey & H. Prade (Eds.), UAI (pp. 21–30). Los Altos: Morgan Kaufmann.
3. Barabási, A. L., & Oltvai, Z. N. (2004). Network biology: Understanding the cell’s functional organization. Nature Reviews Genetics, 5(2), 101–113.
4. Bishop, C. M. (2006). Pattern recognition and machine learning. Berlin: Springer.
5. Blitzer, J., Dredze, M., & Pereira, F. (2007). Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th annual meeting of the ACL (pp. 440–447).
6. Brown, P., Fearn, T., & Vannucci, M. (2001). Bayesian wavelet regression on curves with application to a spectroscopic calibration problem. Journal of the American Statistical Association, 96, 398–408.
7. Calderhead, B., & Girolami, M. (2009). Estimating bayes factors via thermodynamic integration and population mcmc. Computational Statistics & Data Analysis, 53(12), 4028–4045.
8. Candès, E. (2006). Compressive sampling. Proceedings of the International Congress of Mathematicians, 3, 1433–1452.
9. Carbonetto, P., & Stephens, M. (2012). Scalable variational inference for bayesian variable selection in regression, and its accuracy in genetic association studies. Bayesian Analysis, 6(4), 1–42.
10. Carvalho, C. M., Polson, N. G., & Scott, J. G. (2009). Handling sparsity via the horseshoe. Journal of Machine Learning Research W&CP, 5, 73–80.
11. Cunningham, J. P., Hennig, P., & Lacoste-Julien, S. (2011). Gaussian probabilities and expectation propagation. arXiv:1111.6832v2.
12. Damien, P., Wakefield, J., & Walker, S. (1999). Gibbs sampling for Bayesian non-conjugate and hierarchical models by using auxiliary variables. Journal of the Royal Statistical Society Series B (Statistical Methodology), 61(2), 331–344.
13. Davis, J., & Goadrich, M. (2006). The relationship between precision-recall and ROC curves. In Proceedings of the 23rd international conference on machine learning (pp. 233–240). New York, NY: ACM.
14. Donoho, D. (2006). Compressed sensing. IEEE Transactions on Information Theory, 52(4), 1289–1306.
15. Faul, A. C., & Tippin, M. E. (2001). Analysis of sparse bayesian learning. In T. G. Dietterich, S. Becker, & Z. Ghahramani (Eds.), Advances in Neural Information Processing Systems, 14, 383–389.
16. Ferkinghoff-Borg, J. (2002). Monte Carlo methods in complex systems. PhD thesis, University of Cpenhagen.
17. Gardner, T. S., & Faith, J. J. (2005). Reverse-engineering transcription control networks. Physics of Life Reviews, 2(1), 65–88.
18. George, E. I., & McCulloch, R. E. (1997). Approaches for Bayesian variable selection. Statistica Sinica, 7(2), 339–373.
19. Geweke, J., et al. (1996). Variable selection and model comparison in regression. Bayesian statistics, 5, 609–620.
20. Gill, P. E., Golub, G. H., Murray, W., & Saunders, M. A. (1974). Methods for modifying matrix factorizations. Mathematics of Computation, 28(126), 505–535.
21. Hernández-Lobato, J. M., & Dijkstra, T. M. H. (2010). Hub gene selection methods for the reconstruction of transcription networks. In J. L. Balcázar, F. Bonchi, A. Gionis, & M. Sebag (Eds.), ECML-PKDD 2010. Lecture notes in artificial intelligence (Vol. 6321). Berlin: Springer.
22. Hernández-Lobato, J. M., & Hernández-Lobato, D. (2011). Convergent expectation propagation in linear models with spike-and-slab priors. arXiv:1112.2289.
23. Hernández-Lobato, J. M., Dijkstra, T., & Heskes, T. (2008). Regulator discovery from gene expression time series of malaria parasites: a hierachical approach. In J. Platt, D. Koller, Y. Singer, & S. Roweis (Eds.), Advances in neural information processing systems (Vol. 20, pp. 649–656). Cambridge, MA: MIT Press.
24. Hernández-Lobato, D., Hernández-Lobato, J. M., & Suárez, A. (2010). Expectation propagation for microarray data classification. Pattern Recognition Letters, 31(12), 1618–1626.
25. Ishwaran, H., & Rao, J. S. (2005). Spike and slab variable selection: Frequentist and Bayesian strategies. The Annals of Statistics, 33(2), 730–773.
26. Ji, S., Xue, Y., & Carin, L. (2008). Bayesian compressive sensing. IEEE Transactions on Signal Processing, 56(6), 2346–2356.
27. Johnstone, I. M., & Titterington, D. M. (2009). Statistical challenges of high-dimensional data. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 367(1906), 4237–4253.
28. Kuss, M., & Rasmussen, C. E. (2005). Assessing approximate inference for binary Gaussian process classification. The Journal of Machine Learning Research, 6, 1679–1704.
29. Lee, K. E., Sha, N., Dougherty, E. R., Vannucci, M., & Mallick, B. K. (2003). Gene selection: A Bayesian variable selection approach. Bioinformatics, 19(1), 90–97.
30. MacKay, D. J. C. (1992). Bayesian interpolation. Neural Computation, 4(3), 415–447.
31. MacKay, D. J. C. (2003). Information theory, inference, and learning algorithms. Cambridge: Cambridge University Press.
32. Manning, C. D., & Schütze, H. (2000). Foundations of statistical natural language processing. Cambridge: MIT Press.
33. Marbach, D., Schaffter, T., Mattiussi, C., & Floreano, D. (2009). Generating realistic in silico gene networks for performance assessment of reverse engineering methods. Journal of Computational Biology, 16(2), 229–239.
34. Minka, T. (2001). A family of algorithms for approximate Bayesian inference. PhD thesis, MIT.
35. Minka, T., & Lafferty, J. (2002). Expectation-propagation for the generative aspect model. In Proceedings of the 18th conference on uncertainty in artificial intelligence, (pp. 352–359).
36. Mitchell, T., & Beauchamp, J. (1988). Bayesian variable selection in linear regression. Journal of the American Statistical Association, 83(404), 1023–1032.
37. Nickisch, H., & Rasmussen, C. E. (2008). Approximations for binary Gaussian process classification. Journal of Machine Learning Research, 9, 2035–2078.
38. Opper, M., & Winther, O. (2005). Expectation consistent approximate inference. The Journal of Machine Learning Research, 6, 2177–2204.
39. Osborne, B. G., Fearn, T., Miller, A. R., & Douglas, S. (1984). Application of near infrared reflectance spectroscopy to the compositional analysis of biscuits and biscuit doughs. Journal of the Science of Food and Agriculture, 35(1), 99–105.
40. Osborne, B., Fearn, T., Hindle, P., & Hindle, P. (1993). Practical NIR spectroscopy with applications in food and beverage analysis. Longman food technology series. Canada: Wiley.
41. Rasmussen, C. E., & Williams, C. K. I. (2005). Gaussian processes for machine learning (adaptive computation and machine learning). Cambridge: MIT Press.
42. Rue, H., Martino, S., & Chopin, N. (2009). Approximate Bayesian inference for latent Gaussian models by using integrated nested laplace approximations. Journal of the Royal Statistical Society: Series B (statistical methodology), 71(2), 319–392.
43. Sandler, T., Talukdar, P. P., Ungar, L. H., & Blitzer, J. (2008). Regularized learning with networks of features. In D. Koller, D. Schuurmans, Y. Bengio, & L. Bottou (Eds.), Advances in neural information processing systems (vol. 21, pp. 1401–1408).
44. Scott, J. G. (2010). Parameter expansion in local-shrinkage models. arXiv:1010.5265.
45. Seeger, M. W. (2008). Bayesian inference and optimal design for the sparse linear model. The Journal of Machine Learning Research, 9, 759–813.
46. Seeger, M., Nickisch, H., & Schlkopf, B. (2010). Optimization of k-space trajectories for compressed sensing by Bayesian experimental design. Magnetic Resonance in Medicine, 63(1), 116–126.
47. Slonim, D. K. (2002). From patterns to pathways: Gene expression data analysis comes of age. Nature Genetics, 32, 502–508.
48. Steinke, F., Seeger, M., & Tsuda, K. (2007). Experimental design for efficient identification of gene regulatory networks using sparse Bayesian models. BMC Systems Biology, 1(1), 51.
49. Stolovitzky, G., Monroe, D., & Califano, A. (2007). Dialogue on reverse-engineering assessment and methods. Annals of the New York Academy of Sciences, 1115, 1–22.
50. Team, R. D. C. (2007). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. http://www.R-project.org, ISBN:3-900051-07-0
51. Thieffry, D., Huerta, A. M., Pérez-Rueda, E., & Collado-Vides, J. (1998). From specific gene regulation to genomic networks: A global analysis of transcriptional regulation in Escherichia coli. BioEssays, 20(5), 433–440.
52. Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society B, 58(1), 267–288.
53. Tipping, M. E., & Faul, A. (2003). Fast marginal likelihood maximisation for sparse Bayesian models. In C. M. Bishop & B. J. Frey (Eds.), Proceedings of the ninth international workshop on artificial intelligence and statistics.
54. Tipping, M. E. (2001). Sparse Bayesian learning and the relevance vector machine. The Journal of Machine Learning Research, 1, 211–244.
55. Titsias, M. K., & Lazaro-Gredilla, M. (2012). Spike and slab variational inference for multi-task and multiple kernel learning. In Advances in neural information processing systems (Vol. 24).
56. van Gerven, M., Cseke, B., Oostenveld, R., & Heskes, T. (2009). Bayesian source localization with the multivariate Laplace prior. In Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, & A. Culotta (Eds.), Advances in neural information processing systems (Vol. 22, pp. 1901–1909).
57. Wipf, D., Palmer, J., & Rao, B. (2004). Perspectives on sparse Bayesian learning. In S. Thrun, L. Saul, & B. Schölkopf (Eds.), Advances in neural information processing systems (Vol. 16). Cambridge, MA: MIT Press.
58. Zhu, H., & Rohwer, R. (1995). Bayesian invariant measurements of generalization. Neural Processing Letters, 2(6), 28–31.