Practical variational inference for neural networks

Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011, NIPS 2011

Volumn , Issue , 2011, Pages

  Graves, Alex ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

BAYESIAN NETWORKS; INFERENCE ENGINES; RECURRENT NEURAL NETWORKS; STOCHASTIC SYSTEMS;

BAYESIAN INFERENCE; LOSS FUNCTIONS; MINIMUM DESCRIPTION LENGTH; NETWORK WEIGHTS; NEURAL-NETWORKS; SIMPLE NETWORKS; SIMPLE++; STOCHASTIC VARIATIONAL METHODS; VARIATIONAL INFERENCE; VARIATIONAL METHODS;

NETWORK ARCHITECTURE;

EID: 85162557101     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (26)

1. D. Barber and C. M. Bishop. Ensemble learning in Bayesian neural networks., pages 215-237. Springer-Verlag, Berlin, 1998.
2. D. Barber and B. Schottky. Radial basis functions: A bayesian treatment. In NIPS, 1997.
3. G. E. Dahl, M. Ranzato, A. rahman Mohamed, and G. Hinton. Phone recognition with the meancovariance restricted boltzmann machine. In J. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems 23, pages 469-477. 2010.
4. DARPA-ISTO. The DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus (TIMIT), speech disc cd1-1.1 edition, 1990.
5. B. J. Frey. Graphical models for machine learning and digital communication. MIT Press, Cambridge, MA, USA, 1998.
6. K. fu Lee and H. wuen Hon. Speaker-independent phone recognition using hidden markov models. IEEE Transactions on Acoustics, Speech, and Signal Processing, 1989.
7. C. L. Giles and C.W. Omlin. Pruning recurrent neural networks for improved generalization performance. IEEE Transactions on Neural Networks, 5:848-851, 1994.
8. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the International Conference on Machine Learning, ICML 2006, Pittsburgh, USA, 2006.
9. A. Graves and J. Schmidhuber. Offline handwriting recognition with multidimensional recurrent neural networks. In NIPS, pages 545-552, 2008.
10. G. E. Hinton and D. van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In COLT, pages 5-13, 1993.
11. S. Hochreiter and J. Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735-1780, 1997.
12. A. Honkela and H. Valpola. Variational learning and bits-back coding: An information-theoretic view to bayesian learning. IEEE Transactions on Neural Networks, 15:800-810, 2004.
13. K.-C. Jim, C. Giles, and B. Horne. An analysis of noise in recurrent neural networks: convergence and generalization. Neural Networks, IEEE Transactions on, 7(6):1424-1438, nov 1996.
14. N. D. Lawrence. Variational Inference in Probabilistic Models. PhD thesis, University of Cambridge, 2000.
15. Y. Le Cun, J. Denker, and S. Solla. Optimal brain damage. In D. S. Touretzky, editor, Advances in Neural Information Processing Systems, volume 2, pages 598-605. Morgan Kaufmann, San Mateo, CA, 1990.
16. D. J. C. MacKay. Probable networks and plausible predictions - a review of practical bayesian methods for supervised neural networks. Neural Computation, 1995.
17. S. J. Nowlan and G. E. Hinton. Simplifying neural networks by soft weight sharing. Neural Computation, 4:173-193, 1992.
18. M. Opper and C. Archambeau. The variational gaussian approximation revisited. Neural Computation, 21(3):786-792, 2009.
19. D. Plaut, S. Nowlan, and G. E. Hinton. Experiments on learning by back propagation. Technical Report CMU-CS-86-126, Department of Computer Science, Carnegie Mellon University, Pittsburgh, PA, 1986.
20. M. Riedmiller and T. Braun. A direst adaptive method for faster backpropagation learning: The rprop algorithm. In International Symposium on Neural Networks, 1993.
21. J. Rissanen. Modeling by shortest data description. Automatica, 14(5):465-471, 1978.
22. D. E. Rumelhart, G. E. Hinton, and R. J.Williams. Learning representations by back-propagating errors, pages 696-699. MIT Press, Cambridge, MA, USA, 1988.
23. C. E. Shannon. A mathematical theory of communication. Bell system technical journal, 27, 1948.
24. P. Smolensky. Information processing in dynamical systems: foundations of harmony theory, pages 194-281. MIT Press, Cambridge, MA, USA, 1986.
25. C. S. Wallace. Classification by minimum-message-length inference. In Proceedings of the international conference on Advances in computing and information, ICCI'90, pages 72-81, New York, NY, USA, 1990. Springer-Verlag New York, Inc.
26. I. H. Witten, R. M. Neal, and J. G. Cleary. Arithmetic coding for data compression. Commun. ACM, 30:520-540, June 1987.