Expectation propagation for neural networks with sparsity-promoting priors

Journal of Machine Learning Research

Volumn 15, Issue , 2014, Pages 1849-1901

  Jylänki, Pasi ^a,c   Nummenmaa, Aapo ^b   Vehtari, Aki ^c  

^a AALTO UNIVERSITY   (Finland)

^b MASSACHUSETTS GENERAL HOSPITAL   (United States)

^c RADBOUD UNIVERSITY   (Netherlands)

Author keywords

Automatic relevance determination; Expectation propagation; Linear model; Multilayer perceptron; Neural network; Sparse prior

Indexed keywords

ALGORITHMS; MATHEMATICAL MODELS; MODEL STRUCTURES; NEURAL NETWORKS;

AUTOMATIC RELEVANCE DETERMINATION; EXPECTATION PROPAGATION; LINEAR MODELING; MULTI LAYER PERCEPTRON; SPARSE PRIOR;

COMPLEX NETWORKS;

EID: 84902841459     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (44)

1. Kevin Bache and Moshe Lichman. UCI machine learning repository, 2013. URL http://archive.ics.uci.edu/ml.
2. Bart Bakker, Tom Heskes, Jan Neijt, and Bert Kappen. Improving Cox survival analysis with a neural-Bayesian approach. Statistics in Medicine, 23:2989-3012, 2004. (Pubitemid 39200136)
3. David Barber and Christopher M. Bishop. Ensemble learning for multi-layer networks. In M.I. Jordan, M.J. Kearns, and S.A. Solla, editors, Advances in Neural Information Processing Systems 10, pages 395-401. MIT Press, 1998.
4. Botond Cseke and Tom Heskes. Approximate marginals in latent Gaussian models. Journal of Machine Learning Research, 12:417-454, 2011.
5. Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 11:625-660, 2010.
6. Joao F. G. de Freitas. Bayesian Methods for Neural Networks. PhD thesis, University of Cambridge, 1999.
7. Marcel van Gerven, Botond Cseke, Robert Oostenveld, and Tom Heskes. Bayesian source localization with the multivariate Laplace prior. In Y. Bengio, D. Schuurmans, J. Lafferty, C.K.I. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems 22, pages 1901-1909. Curran Associates, Inc., 2009.
8. Marcel van Gerven, Botond Cseke, Floris de Lange, and Tom Heskes. Efficient Bayesian multivariate fMRI analysis using a sparsifying spatio-temporal prior. Neuro Image, 50: 150-161, 2010.
9. Alex Graves. Practical variational inference for neural networks. In J. Shawe-Taylor, R.S. Zemel, P.L. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 24, pages 2348-2356. Curran Associates, Inc., 2011.
10. Daniel Hernández-Lobato, José M. Hernández-Lobato, and A. Suárez. Expectation propagation for microarray data classification. Pattern Recognition Letters, 31(12):1618-1626, 2010.
11. Daniel Hernández-Lobato, José M. Hernández-Lobato, and Pierre Dupont. Generalized spike-and-slab priors for Bayesian group feature selection using expectation propagation. Journal of Machine Learning Research, 14:1891-1945, 2013.
12. José M. Hernández-Lobato, Tjeerd Dijkstra, and Tom Heskes. Regulator discovery from gene expression time series of malaria parasites: a hierarchical approach. In J.C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 649-656. Curran Associates, Inc., 2008.
13. Tom Heskes and Onno Zoeter. Expectation propagation for approximate inference in dynamic Bayesian networks. In Proceedings of the Eighteenth Conference Annual Conference on Uncertainty in Artificial Intelligence (UAI-02), pages 216-223. Morgan Kaufmann, 2002.
14. Tom Heskes, Bart Bakker, and Bert Kappen. Approximate algorithms for neural-Bayesian approaches. Theoretical Computer Science, 287:219-238, 2002. (Pubitemid 35019167)
15. Geoffrey E. Hinton and Drew van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the Sixth Annual Conference on Computational Learning Theory, COLT '93, pages 5-13. ACM, 1993.
16. Antti Honkela and Harri Valpola. Unsupervised variational Bayesian learning of nonlinear models. In L.K. Saul, Y. Weiss, and L. Bottou, editors, Advances in Neural Information Processing Systems 17, pages 593-600. MIT Press, 2005.
17. Pasi Jylänki, Jarno Vanhatalo, and Aki Vehtari. Gaussian process regression with a Student-t likelihood. Journal of Machine Learning Research, 12:3227-3257, 2011.
18. Jouko Lampinen and Aki Vehtari. Bayesian approach for neural networks - review and case studies. Neural Networks, 14(3):7-24, 2001.
19. David J. C. Mackay. Probable networks and plausible predictions - a review of practical Bayesian methods for supervised neural networks. Network: Computation in Neural Systems, 6(3):469-505, 1995.
20. Thomas Minka. A Family of Algorithms for Approximate Bayesian Inference. PhD thesis, Massachusetts Institute of Technology, 2001a.
21. Thomas Minka. Expectation propagation for approximate Bayesian inference. In Proceedings of the Seventeenth Conference Annual Conference on Uncertainty in Artificial Intelligence (UAI-01), pages 362-369. Morgan Kaufmann, 2001b.
22. Thomas Minka. Power EP. Technical report, Microsoft Research, Cambridge, 2004.
23. Thomas Minka. Divergence measures and message passing. Technical report, Microsoft Research, Cambridge, 2005.
24. Thomas Minka and John Lafferty. Expectation-propagation for the generative aspect model. In Proceedings of the Eighteenth Conference Annual Conference on Uncertainty in Artificial Intelligence (UAI-02), pages 352-359. Morgan Kaufmann, 2002.
25. Radford M. Neal. Bayesian Learning for Neural Networks. Springer, 1996.
26. Hannes Nickisch and Carl E. Rasmussen. Approximations for binary Gaussian process classification. Journal of Machine Learning Research, 9:2035-2078, Oct. 2008.
27. Manfred Opper and Ole Winther. Mean field approach to Bayes learning in feed-forward neural networks. Physical Review Letters, 76:1964-1967, Mar. 1996. (Pubitemid 126638911)
28. Manfred Opper and Ole Winther. Expectation consistent approximate inference. Journal of Machine Learning Research, 6:2177-2204, December 2005. (Pubitemid 41832629)
29. Gintaras V. Puskorius and Lee A. Feldkamp. Decoupled extended Kalman filter training of feedforward layered networks. In Proceedings of the International Joint Conference on Neural Networks, pages 307-312. Seattle, 1991.
30. Yuan (Alan) Qi, Thomas P. Minka, Rosalind W. Picard, and Zoubin Ghahramani. Predictive automatic relevance determination by expectation propagation. In Proceedings of Twenty-first International Conference on Machine Learning, pages 671-678, 2004. (Pubitemid 40290867)
31. Carl E. Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006.
32. Fabiano Ribeiro and Manfred Opper. Expectation propagation with factorizing distributions: A Gaussian approximation and performance results for simple models. Neural Computation, 23(4):1047-1069, 2011.
33. Jaakko Riihimäki, Pasi Jylänki, and Aki Vehtari. Nested expectation propagation for Gaussian process classification with a multinomial probit likelihood. Journal of Machine Learning Research, 14:75-109, 2013.
34. Anton Schwaighofer and Volker Tresp. Transductive and inductive methods for approximate Gaussian process regression. In S. Thrun S. Becker and K. Obermayer, editors, Advances in Neural Information Processing Systems 15, pages 953-960. MIT Press, 2003.
35. Matthias Seeger. Bayesian inference and optimal design for the sparse linear model. Journal of Machine Learning Research, 9:759-813, 2008.
36. Matthias Seeger and Hannes Nickisch. Fast convergent algorithms for expectation propagation approximate Bayesian inference. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 652-660. JMLR W & CP, Vol. 15, 2011.
37. Michael E. Tipping. Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research, 1:211-244, Dec. 2001. (Pubitemid 33687203)
38. Jarno Vanhatalo, Jaakko Riihimäki, Jouni Hartikainen, Pasi Jylänki, Ville Tolvanen, and Aki Vehtari. GPstuff: Bayesian modeling with Gaussian processes. Journal of Machine Learning Research, 14:1175-1179, 2013.
39. Eric A. Wan and Rudolph van der Merwe. The unscented Kalman filter for nonlinear estimation. In Proceedings of IEEE Symposium on Adaptive Systems for Signal Processing, Communications, and Control (AS-SPCC), pages 153-158, 2000.
40. Christopher K. I. Williams. Computation with infinite neural networks. Neural Computation, 10(5):1203-1216, 1998. (Pubitemid 128463669)
41. Peter M. Williams. Bayesian regularisation and pruning using a Laplace prior. Neural Computation, 7(1):117-143, 1995.
42. Ole Winther. Computing with finite and infinite networks. In T.K. Leen, T.G. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems 13, pages 336-342. MIT Press, 2001.
43. David Wipf and Srikantan Nagarajan. A new view of automatic relevance determination. In J.C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 1625-1632. Curran Associates, Inc., 2008.
44. David Wipf, Bhaskar D. Rao, and Srikantan Nagarajan. Latent variable Bayesian models for promoting sparsity. IEEE Transactions on Information Theory, 57(9):6236-6255, Sept. 2011.