Generalized spike-and-slab priors for Bayesian group feature selection using expectation propagation

Journal of Machine Learning Research

Volumn 14, Issue , 2013, Pages 1891-1945

  Hernández Lobato, Daniel ^a   Hernández Lobato, José Miguel ^b   Dupont, Pierre ^c  

^a UNIVERSIDAD AUTÓNOMA DE MADRID   (Spain)

^b UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^c UNIVERSITÉ CATHOLIQUE DE LOUVAIN   (Belgium)

Author keywords

Approximate inference; Expectation propagation; Generalized spike and slab priors; Group feature selection; Sequential experimental design; Signal reconstruction; Sparse linear model

Indexed keywords

APPROXIMATE INFERENCE; EXPECTATION PROPAGATION; GENERALIZED SPIKE-AND-SLAB PRIORS; LINEAR MODELING; SEQUENTIAL EXPERIMENTAL DESIGN;

BAYESIAN NETWORKS; INFERENCE ENGINES; SIGNAL RECONSTRUCTION;

ITERATIVE METHODS;

EID: 84884218772     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (64)

1. H. Attias. A variational Bayesian framework for graphical models. In In Advances in Neural Information Processing Systems, volume 12, pages 209-215. MIT Press, 2000.
2. F. R. Bach. Consistency of the group lasso and multiple kernel learning. Journal of Machine Learning Research, 9:1179-1225, 2008. ISSN 1532-4435.
3. J. A. Bazerque, G. Mateos, and G. B. Giannakis. Group-lasso on splines for spectrum cartography. Transactions on Signal Proccesing, 59(10):4648-4663, October 2011. ISSN 1053-587X.
4. C. M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer, August 2006. ISBN 0387310738.
5. J. Blitzer, M. Dredze, and F. Pereira. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 440-407, Prague, Czech Republic, 2007.
6. E. J. Candes and M. B. Wakin. An introduction to compressive sampling. IEEE Signal Processing Magazine, 25(2):21-30, march 2008. ISSN 1053-5888.
7. C. M. Carvalho, N. G. Polson, and J. G. Scott. Handling sparsity via the horseshoe. Journal of Machine Learning Research W&CP, 5:73-80, 2009.
8. K. Chaloner and I. Verdinelli. Bayesian experimental design: A review. Statistical Science, 10:273-304, 1995.
9. P. Damien, J. Wakefield, and S. Walker. Gibbs sampling for Bayesian non-conjugate and hierarchical models by using auxiliary variables. Journal of the Royal Statistical Society Series B, 61(2):331-344, 1999.
10. J. Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7:1-30, 2006. ISSN 1533-7928.
11. D. L. Donoho. Compressed sensing. IEEE Transactions on Information Theory, 52(4):1289-1306, 2006.
12. A. C. Faul and M. E. Tipping. Analysis of sparse Bayesian learning. In Thomas G. Dietterich, Suzanna Becker, and Zoubin Ghahramani, editors, Advances in Neural Information Processing Systems 14, pages 383-389. MIT Press, 2001.
13. V. Fedorov. Theory of Optimal Experiments. Academic Press, 1972.
14. E. I. George and R. E. McCulloch. Approaches for Bayesian variable selection. Statistica Sinica, 7(2):339-373, 1997.
15. M. Van Gerven, B. Cseke, R. Oostenveld, and T. Heskes. Bayesian source localization with the multivariate Laplace prior. In Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems 22, pages 1901-1909, 2009.
16. J. Geweke. Variable selection and model comparison in regression. Bayesian Statistics, 5:609-620, 1996.
17. P. E. Gill, G. H. Golub, W. Murray, and M. A. Saunders. Methods for modifying matrix factorizations. Mathematics of Computation, 28(126):505-535, 1974.
18. D. Hernández-Lobato. Prediction Based on Averages over Automatically Induced Learners: Ensemble Methods and Bayesian Techniques. PhD thesis, Computer Science Department, Universidad Autónoma de Madrid, 2009. Online available at: http://arantxa.ii.uam.es/dhernan/docs/Thesis color links.pdf.
19. D. Hernández-Lobato, J. M. Hernández-Lobato, T. Helleputte, and P. Dupont. Expectation propagation for Bayesian multi-task feature selection. In José L. Balcázar, Francesco Bonchi, Aristides Gionis, and Michèle Sebag, editors, Proceedings of the European Conference on Machine Learning, volume 6321, pages 522-537. Springer, 2010.
20. J. M. Hernández-Lobato. Balancing Flexibility and Robustness in Machine Learning: Semiparametric Methods and Sparse Linear Models. PhD thesis, Computer Science Department, Universidad Autónoma de Madrid, 2010.
21. J. M. Hernández-Lobato and T. Dijkstra. Hub gene selection methods for the reconstruction of transcription networks. In José L. Balcázar, Francesco Bonchi, Aristides Gionis, and Michèle Sebag, editors, Machine Learning and Knowledge Discovery in Databases, European Conference, Proceedings, Part I, volume 6321, pages 506-521, 2010.
22. J. M. Hernández-Lobato, T. Dijkstra, and T. Heskes. Regulator discovery from gene expression time series of malaria parasites: A hierarchical approach. In J. C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 649-656. The MIT Press, 2008.
23. J. M. Hernández-Lobato, D. Hernández-Lobato, and A. Suárez. Network-based sparse Bayesian classification. Pattern Recognition, 44:886-900, 2011.
24. J. Huang and T. Zhang. The benefit of group sparsity. The Annals of Statistics, 38:1978-2004, 2010.
25. T. S. Jaakkola. Tutorial on variational approximation methods. In M. Opper and D. Saad, editors, Advances in Mean Field Methods: Theory and Practice, pages 129-159. MIT Press, 2001.
26. S. Ji and L. Carin. Bayesian compressive sensing and projection optimization. In Proceedings of the 24th International Conference on Machine Learning, pages 377-384. ACM, 2007. ISBN 978-1-59593-793-3.
27. S. Ji, Y. Xue, and L. Carin. Bayesian compressive sensing. IEEE Transactions on Signal Processing, 56(6):2346-2356, june 2008. ISSN 1053-587X.
28. S. Ji, D. Dunson, and L. Carin. Multitask compressive sensing. IEEE Transactions on Signal Processing, 57(1):92-106, January 2009. ISSN 1053-587X.
29. I. M. Johnstone and D. M. Titterington. Statistical challenges of high-dimensional data. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 367(1906):4237, 2009.
30. H. J. Kappen and Vicenç Gómez. The variational garrote. Machine Learning, pages 1-17, 2013. Submitted.
31. S. Kim and E. P. Xing. Feature selection via block-regularized regression. In David A. McAllester and Petri Myllymäki, editors, Proceedings of the 24th Conference in Uncertainty in Artificial Intelligence, pages 325-332. AUAI Press, 2008.
32. Y. Kim, J. Kim, and Y. Kim. Blockwise sparse regression. Statistica Sinica, 16(2):375, 2006. ISSN 1017-0405.
33. N. Lawrence, M. W. Seeger, and R. Herbrich. Fast sparse Gaussian process methods: The informative vector machine. In S. Becker, S. Thrun, and K. Obermayer, editors, Advances in Neural Information Processing Systems 15, pages 609-616. MIT Press, Cambridge, MA, 2003.
34. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.
35. K. E. Lee, N. Sha, E. R. Dougherty, M. Vannucci, and B. K. Mallick. Gene selection: A Bayesian variable selection approach. Bioinformatics, 19(1):90-97, 2003.
36. Y. Li, C. Campbell, and M. Tipping. Bayesian automatic relevance determination algorithms for classifying gene expression data. Bioinformatics, 18(10):1332-1339, 2002.
37. D. J. C. MacKay. Information-based objective functions for active data selection. Neural Computation, 4:590-604, 1991.
38. D. J. C. MacKay. Information Theory, Inference & Learning Algorithms. Cambridge University Press, Cambridge, UK., 2003. ISBN 0521642981.
39. D. M. Malioutov, M. Çetin, and A. S. Willsky. A sparse signal reconstruction perspective for source localization with sensor arrays. IEEE Transactions on Signal Processing, 53 (8-2):3010-3022, 2005.
40. L. Meier, S. Van De Geer, and P. Bühlmann. The group lasso for logistic regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 70(1):53-71, 2008. ISSN 1467-9868.
41. T. Minka. A Family of Algorithms for Approximate Bayesian Inference. PhD thesis, Massachusetts Institute of Technology, 2001.
42. T. Minka and J. Lafferty. Expectation-propagation for the generative aspect model. In Adnan Darwiche and Nir Friedman, editors, Proceedings of the 18th Conference on Uncertainty in Artificial Intelligence, pages 352-359. Morgan Kaufmann, 2002.
43. T. J. Mitchell and J. J. Beauchamp. Bayesian variable selection in linear regression. Journal of the American Statistical Association, 83(404):1023-1032, 1988.
44. R. M. Neal. Probabilistic inference using Markov chain Monte Carlo methods. Technical Report CRG-TR-93-1, Dept. of Computer Science, University of Toronto, 1993.
45. H. Nickisch and C. E. Rasmussen. Approximations for binary Gaussian process classification. Journal of Machine Learning Research, 9:2035-2078, 10 2008.
46. B. Pang, L. Lee, and S. Vaithyanathan. Thumbs up? Sentiment classification using machine learning techniques. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 79-86, 2002.
47. A. T. Puig, A. Wiesel, G. F., and A. O. Hero. Multidimensional shrinkage-thresholding operator and group lasso penalties. IEEE Signal Processing Letters, 18(6):363-366, 2011.
48. R Development Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2011. URL http://www.R-project.org/. ISBN 3-900051-07-0.
49. S. Raman, T. J. Fuchs, P. J. Wild, E. Dahl, and V. Roth. The Bayesian group-lasso for analyzing contingency tables. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML'09, pages 881-888, New York, NY, USA, 2009. ACM. ISBN 978-1-60558-516-1.
50. V. Roth and B. Fischer. The group-lasso for generalized linear models: uniqueness of solutions and efficient algorithms. In A. McCallum and S. Roweis, editors, Proceedings of the 25th International Conference on Machine Learning, pages 848-855, 2008.
51. T. Sandler, J. Blitzer, P. P. Talukdar, and L. H. Ungar. Regularized learning with networks of features. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems 21, pages 1401-1408. 2009.
52. F. Scheipl, L. Fahrmeir, and T. Kneib. Spike-and-slab priors for function selection in structured additive regression models. Journal of the American Statistical Association, 107:1518-1532, 2012.
53. J. G. Scott. Parameter expansion in local-shrinkage models. ArXiv E-prints, 2010. arXiv:1010.5265v1.
54. M. W. Seeger. Bayesian inference and optimal design for the sparse linear model. Journal of Machine Learning Research, 9:759-813, 2008.
55. M. W. Seeger, H. Nickisch, R. Pohmann, and B. Schölkopf. Optimization of k-space trajectories for compressed sensing by Bayesian experimental design. Magnetic Resonance in Medicine, 63(1):116-126, 2009.
56. R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), 58(1):267-288, 1996.
57. M. E. Tipping. Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research, 1:211-244, 2001.
58. M. E. Tipping and A. Faul. Fast marginal likelihood maximisation for sparse Bayesian models. In Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics, pages 3-6, 2003.
59. 1, ∞ regularization. In Goesele et al., editor, 32nd Anual Symposium of the German Association for Pattern Recognition, volume 6376, pages 252-261. Springer, 2010.
60. M. Wakin, M. Duarte, S. Sarvotham, D. Baron, and R. Baraniuk. Recovery of jointly sparse signals from few random projections. In Y. Weiss, B. Schölkopf, and J. Platt, editors, Advances in Neural Information Processing Systems 18, pages 1433-1440. MIT Press, Cambridge, MA, 2006.
61. M. West. Bayesian factor regression models in the "large p, small n" paradigm. In Bayesian Statistics 7, pages 723-732. Oxford University Press, 2003.
62. T. J. Yen and Y. M. Yen. Grouped variable selection via nested spike and slab priors. ArXiv E-prints, 2011. arXiv:1106.5837v1.
63. M. Yuan and Y. Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1):49-67, 2006. ISSN 1467-9868.
64. H. Zhu and R. Rohwer. Bayesian invariant measurements of generalization. Neural Processing Letters, 2:28-31, 1995. ISSN 1370-4621.